Spirit Sol 118

This was not our best drive. The rover traversed about 37m (not that long ago, I keep reminding myself, this would have been a fantastic sol's drive), then the autonav failed out, deciding that there was no safe path from its present position. Oddly, according to the rover's on-board navigation maps, the terrain underneath the rover itself is flagged as too dangerous to drive into, which should never happen.

What appears to have happened is this. Going on autonav, the rover drove into a kind of ditch. (This was safe -- the ditch was wide enough, and its walls shallow enough, to enter.) When it was trying to climb out, its pitch changed significantly -- it's going up the ditch wall -- and this changed its evaluation of the immediately surrounding terrain. It decided that the path in front of it was a huge cliff (an illusion caused by its inability to see the side of the ditch it was climbing, only the terrain beyond) and behind it was a huge wall (another illusion -- this was really the floor and rear wall of the ditch). So it stopped where it was. The vehicle is safe and healthy, just a little confused about how ditches work.

For nextersol, John's plan is to floor it. He's planning a 100m drive -- which will be yet another record for us, if we make it. Art and I both tell him, tongue in cheek, that that's a mistake. We should set a 91m record, then a 92m record, and so on, not jump all the way to 100m! Art starts semi-seriously pushing for this, comparing it to the old Soviet weightlifters. They'd be able to break the record by, say, five pounds, but they'd do it by setting five new records, one pound apart. The more he pushes for the idea, the more I think he starts to like it. But John stands firm: "No, I want us to set a big record, and make them all quail in terror ...." Of course, I'm all for this, and in the end I actually extend the drive another 4m or so to ensure we actually make it up onto the rim. Total planned distance: about 106m.

We want to drive as much as possible nextersol, but the rover needs to nap in the middle of the day for power reasons. So we end up splitting the drive sequence into two parts (blind pre-siesta and autonav post-siesta), something we haven't been allowed to do since the Humphrey Bump Incident. I've been pushing to enable this, and it looks like I've finally won. The Curse of the Split Drive has been lifted!

A piece of delayed good news: we nailed the MTES of Potrillo. A week ago, we did that drive where we had to drive 12m before trying to hit a bullseye with the MTES, a hard problem because of the unknown degree of slip and the MTES's narrow field of view. Today's MTES PUL, Laura, cheerfully plunks down a black-and-white NAVCAM image with the MTES field of view shaded in purple. The rock we were aiming for, Potrillo, is in bounds -- it's at the bottom right-hand edge, but it's in there. Our perfect record is unbroken.

During a lull, Larry Soderblom distracts the entire room by showing a gorgeous 3-D model of the Columbia Hills on the room's projection screen. The model was compiled from orbital images taken by MGS. Larry shows us where we're heading, and breaks the news that we're not likely to climb the hills when we get there. "We would be crazy to climb those hills," he says. "We'd get a great view, but there's nothing up there when you get there." However, there's a lot of good stuff near the hills' base, including some cool "steps" about 500m away from our current destination. Best of all, we see formations that might be bedrock -- just the stuff we need to tell Gusev's "water story" -- assuming, as everyone does, that it has such a story to tell.

We finish up and everyone else leaves; I'm still working on my uplink report. And I see a problem with the drive.

At the end of every drive, we perform a "wheel wiggle" -- the rover turns its four steerable wheels all the way in, then straightens them out again. The theory behind this is that it will help settle the rover in its current position, mitigating the possible effect of settling into the soil, falling off a rock, etc. It's important, especially for IDD work, that the rover move as little as possible after sending us its final images. Even a few millimeters of unexpected settling can be a bad thing.

Ordinarily, the rover won't perform any driving maneuvers, not even a wheel wiggle, when it has a goal error (meaning, usually, that it didn't drive all the way to the target we specified; this is fairly common, as we often deliberately set goals we know the rover won't reach). We want the rover to wiggle its wheels even when it didn't reach its goal, though, because the science team might just want to use the IDD wherever we ended up. So, while wiggling the wheels, we "mask" the goal error. We send a command telling the rover to ignore the goal error, then we have it wiggle the wheels, then we send another command saying to pay attention to the goal error again ("unmasking" it).

Here's what we didn't think of. Suppose the first drive sequence gets deactivated while the rover is wiggling its wheels. Then it won't get to the command that says to pay attention to the goal error. That means that when the afternoon drive sequence executes, the rover will ignore any goal errors that may arise. This would not be good.

Now, it's unlikely that this will even happen. We expect that we allowed adequate time for the morning drive, meaning that it won't be running when we go to sleep -- it will have already finished. And the wheel wiggle takes less than a minute, so it's a somewhat narrow window of vulnerability. But we can't be certain, and the fact that the wiggle occurs at the end makes the odds a little worse for us.

Ah, hell. The Curse of the Split Drive is back.

I call Art -- at home, oy vey -- to alert him to the problem. He thinks of something I hadn't: is the mask volatile (that is, forgotten when the rover shuts down) or non-volatile (remembered even across reboots)? If the state is volatile, it doesn't matter; the mask will be forgotten when the rover takes its nap, and when it awakes, everything will be fine. I don't know, so I call Mark Maimone at home, poor guy. Mark greps[1] through the flight software's source code -- and calls me back with bad news. The mask is remembered across reboots. We have a problem to solve.

Solving the problem in the sequence is easy, taking me just a couple of minutes. I remove the commands that mask and unmask the goal error; if the wheel wiggle doesn't happen, we just won't worry about it. The rover's going to drive again in the afternoon, anyway; the odds that we were going to care about this particular wiggle were already slim.[2] The harder problem is: how are we going to get these commands to the rover? Everyone on the Spirit uplink team has already gone home, including the people who know how to convert the commands I've written into the binary form we uplink.

Ah, but the Opportunity uplink team is still here -- this is a late day for them. So I run upstairs with the intention of -- on Art's behalf -- begging their mission manager, Rick Welch, for the use of their Sequence Integration Engineer, Dan Moyers. But when I arrive, they're having their sequence walkthrough. I don't want to interrupt this, so I sit in a corner and try to stay quiet. Internally, I keep hearing Inigo Montoya talking to Miracle Max: "Sir, we are in a terrible hurry ...." But I'm mindful of Miracle Max's reply, and I keep my mouth shut.

The moment they're finished, I ask Rick for the favor. They're running late today, but it turns out this is a good time -- Dan is sitting idle, waiting for some other sequences to be delivered. So Dan and Rick and Bill Bensler wrap up the sequences, sign the paperwork, and run it downstairs for me. The MER-B team saves the day.

I have one more thing to do before I go. I go back downstairs, log in, and write a big, big thank-you to the MER-B team in my uplink report.

Courtesy NASA/JPL-Caltech. The final front HAZCAM from the previous drive. Doesn't look scary to me, but it looked scary to Spirit.


[1] A geek term meaning "searches."

[2] It seems a better solution would be to unconditionally unmask the goal error at the top of the second sequence. I don't know why I didn't think of that then. Time pressure, maybe.


Anonymous said...

Nextersol! Awesome word!

Also like your second footnote. I'm about to graduate BS from UT's aerospace program and feel that real engineering/operations perspective is too often absent from our curriculum. The availability of stories like this is going to be a tremendous boon to future generations of planetary explorers.

Anonymous said...

Oh yes, I meant to go and mention that the "now we know" bits being in footnotes is structurally wonderful. It reinforces that this is an actual problem that doesn't just get magically solved in the next sentence, there's time and sweat involved and maybe a few non-optimal transient solutions before we work out the one that's good and sticks.

Thanks for it, rover driver.

Anon E. Mouse said...

If you rush a miracle, you get rotten miracles

ermanno said...

All my troubles seem so firewall.."

Mhh.. it doesn't sound good. ;-)