Monday, April 21, 2008

INTERVIEW - Sidewinder Linden

By Tauri Tigerpaw.
Exclusive to SurfWatch.

SurfWatch managed to track down one of the busiest men in Second Life this year: Sidewinder Linden, the program manager for the Havok 4 project. Since Second Life surfing has been so drastically affected by the latest Havok server code, we wanted to ask the man himself what was going on, how current Havok4-related surfing issues might be addressed and what's the plan for the future.

SurfWatch: How did the issue with Second Life's surfing waves come to your attention?

Sidewinder: One thing I hope you'll see as a consistent theme throughout our work is that we made a point of reaching out in many ways to many communities: hundreds of hours (maybe more - I didn't count) of in-world time, day and night, working with people. In all of that, no one mentioned surfing, and I honestly didn't think to ask about it.

My first encounter with Heather Goodliffe's waves was in trying to figure out why certain regions were having performance problems. After some 'look for high impact scripts'-work with some regions, we figured out that those waves were a particularly heavy impact, and noticed that if they were removed, the simulator stats were much better. Even after meeting with several residents to talk about this, none mentioned that they had any function past visual (which is amazing to me at this point). So it was post-Havok4 update that we understood that these are built for some 'purpose' other than to look nice.

SurfWatch: Now that the wave functionality is understood, is Linden Labs working with developers like Heather Goodliffe towards a fix?

Sidewinder: The current status is that we looked at the waves when we started getting performance complaints on some regions. We did not evaluate them in detail, so we have not done any in depth analysis yet (yet - that doesn't mean that we won't), and had encouraged the owners of them to talk to the builder, since I had guessed that the builder might have already done an update - many have. We do have a variety of high priority items on the deck at the moment.

The next step will be to talk to Heather about some strategies for creating a wave set that has lower impact, and at the same time discuss whether there are any efficiency optimisations we can do to improve things from the simulator perspective. Although Havok4 does many things faster than Havok1, there are some physics operations that are less efficient in the Havok4 physics engine, and we have discovered that some inworld content trips on those areas. In some cases, we have found optimisations that will help with speed, and in others the scripts or approach need to be tweaked a bit to avoid the issues, and we do not know just yet what the mix of those two solution strategies will be for this case.

At the moment, the team has been engaged in a few issues. For example, one is a problem with targetomega, which has caused some rotating objects inworld to spontaneously stop for what was mysterious reasons. That absorbed a lot of team time over the last week, along with a few high priority issues having to do with internals. I've talked with Heather, and explained that due to that, and internal coordination on this update release, we are still talking about sequence of addressing the rest of the open issues. There are a couple of sailing issues, and this one - that we now understand has wider impact than we thought - and other issues besides.

I don't have a specific fix date for the waves, because as you can see, we don't at the moment really know what the problem is. As far as we can tell, the float heights for Havok1 were not fully correct, so in our first Havok4 builds we 'fixed that bug' only to find out that many folks relied on the incorrect behavior. There are a number of other issues related to flotation and boats that are in this category, and we are doing some research to figure out a best path on this.

SurfWatch: So, any insight on possible solutions?

Sidewinder: The two obvious paths are:

1) to re-implement the bugs that used to be in Havok1 (in the cases that that is possible) so that Havok4 emulates the broken behavior, and thus 'old things work the way they did'. This strategy is nicer to existing content, but has the side effect of 'breaking the future' - in other words, we'll never get new products that can be better for using a physics infrastructure that works better and more consistently. That's always a hard call... in many cases we've found workarounds to support older content without compromising the future, but it is not always possible.

2) to not re-implement bad behaviors from Havok1, and accept the fact that there will be older content that either needs to be updated or will not work the same in Havok4. There are some cases where path 2 has been the 'righter choice'. One example is with cars. Many, many builders wanted us to increase the maximum motor speed on cars to 256m/S, whereas now it's 64 m/S. We did that, as a collective decision with the builders, knowing that older cars that used 'tricks' to go faster than the motor speed would run excessively fast on Havok4. We faced a similar issue in the internals of boat and 'floatie' behavior. We thought we'd identified some bad behavior in Havok1 in that area that was exploited to make things 'work' with the older simulator, and we are trying to figure out whether we had to break some old boat behavior to fix the future. In this case, we just figured out a way to avoid that, and are preserving the old boat behavior. These calls can go either way depending on the details.

SurfWatch: Surfers currently have a number of post-Havok4 issues with board performance, for example the older surf boards floating under water level. Some surfers are reporting boards riding in different positions on the waves and board maneuvers are handling differently. Can you comment?

Sidewinder: Different handling is actually expected. This is true of other vehicles as well. The jump from Havok1 to Havok4 involved some major changes in how the Havok physics engine calculates physics and its behavior around collisions, friction and damping are different.

I am guessing (hypothetically) that the board float height is the same issue that we see with some boats, and that they may need script tweaks to work well. But, since we are just in the process of figuring out which way to go with these issues, I would not want people to rush off and assume one way or the other. The real answer is: it's in discussion as we speak. We have office hours twice a week and I strongly encourage any builders of boards to join the process because this way they can actually discuss issues - which for some things is much more effective at figuring out issues than just posting comments.

SurfWatch: What about wave performance, specifically?

Sidewinder: On the wave front, I suspect that the solution may be a combination of script or design adjustments and some optimizations (if we can find any - none really stood out as obvious when we looked at the waves the first time).

So I think part of this is the reality that, as with operating system upgrades, some things can be made transparent and 'just the same', and others will change and need updates for people who want 'very detailed behaviors' to match. In some cases they can match, and in others it may be slightly different no matter what we all do. We are working through the list, and are trying to make things work well for everyone, within the constraint that we don't want to cripple the future to support the past - and that's always a hard line to walk with these kinds of upgrades.

SurfWatch: Is there anything else you would like to pass along, Sidewinder?

Sidewinder: I'd like to thank everyone for being involved, and if there are specific behaviors, in specific cases that are problematic, those will be very helpful to know about. We'll keep working the issues, and want involvement from people who are seeing problems. It's most useful for us to hear from the builders and scripters, as they have a deep understanding of how their products work internally. We have held informal "vehicle workshops" so that builders can demonstrate behaviors that are not correct, and maybe it would make sense to do this again with surfing, once we get the wave mechanics figured out.

1 comment:

Sidewinder Linden said...

As a quick followup to this article, we have since identified one possible cause of the wave performance overhead, and have a test build that includes a possible resolution. This is in the test queue, and is not likely to be deployed this week, but hopefully not long thereafter.

The reason for being hesitant about claiming a fix, is that until we see this in operation on the main Second Life system we won't know if we have really taken care of the one and only root cause, or whether this is a multi-part problem.

We'll continue to look at this and work with Heather (which we've already been doing) to resolve remaining issues.

Best regards,

Sidewinder Linden