Spirit Sol 205

We didn't make it work. I come in a little early and sit down at my workstation with Richard Kornfeld, who's now working on MSL and wants to gather some lessons from MER to improve MSL's design. Just as we get started, Chris comes over and asks, "Did you hear what happened?"

"Did you hear what happened?" is bad. People don't ask that when what happened is that things went perfectly, they ask it when what happened is that things went wrong. And things went wrong.

What went wrong is pretty embarrassing. The rover tracks its attitude using something called the Inertial Measurement Unit, or IMU. Because the IMU draws a lot of power, we want to turn it off when the rover is going to be sitting still (so that its attitude won't be changing) for more than a minute or so, and on when the rover is moving. Since the rover sits still and thinks for long periods when doing visodom -- as in yestersol's drive -- visodom sequences turn the IMU off and then back on several times.

Before turning the IMU off, you have to wait a few seconds to ensure that the flight software isn't still trying to read data from it. And after turning it on, you have to wait a few seconds for it to warm up.

What happened yestersol was that we screwed up this trivial procedure the very first time we turned the IMU off. Rather than telling the sequence to wait a few seconds and then turn the IMU off, we told it to turn the IMU off and then (uselessly) wait a few seconds. We did this right after a drive command, so the flight software tried to talk to the IMU while it was off, which caused the rover to decide that something was wrong and abort the rest of the drive. The result is that after all that work yestersol, the rover drove a total of 25cm.

I can think of something they could improve on MSL.[1]

To make this especially embarrassing (and worrisome), there were lots of places we should have caught this. Chris and I never should have made the mistake in the first place, but the uplink team also reviewed the sequence twice -- as usual -- before uplinking it, and nobody caught it during either review. Which means we could miss something else, something that could really damage the rover.

Even worse is that I foresaw this scenario the first time I learned there was this procedure for turning the IMU on and off. I knew we'd screw this up eventually -- like anything else that depends on humans to get it right -- and that I should automate the check. Well, better late than never. While Chris works on thisol's drive, I automate the check for this mistake. One less thing to go wrong in the future.

Ordinarily, a recovery sequence in a case like this would be simple enough -- cut out the commands that executed successfully, so that the rover picks up where it left off. But the rover shows signs of having slipped significantly even over the 25cm it tried to drive, which is probably because we're in a sandy hollow. So Chris plans a whole new drive, one that gets us out of the sand trap sooner. This is probably the right thing to do, but it takes time, and it makes us late once again.

Meanwhile, Art is trying to prevent an international incident. In a couple of sols, we have a coordinated experiment with ESA's MEX spacecraft. In order to do our part of the experiment, we need to be oriented at 260 degrees. But that isn't a scientifically useful heading for the experiments we want to perform on the outcrop, and if we make it to the outcrop on this drive, we won't want to move again just to support the experiment. We might be able to get in a single sol of IDDing the outcrop, but then we'll just have to stow and turn to perform the experiment. For some reason, our LTP folks didn't have the MEX experiment included in their plan, which is why we're scrambling now. Not only is the timing awkward, there's also a significant energy hit involved -- the experiment happens about 3 AM Mars time, so the rover has to be awake and well heated -- and our energy is low and not getting better.

They call Ray about it (Ray's on the freeway, on his way home). Art would rather bag the MEX experiment altogether to simplify things for us, but, as he tells Ray, "It's above my badge level -- this has been negotiated by governments on both sides of the Atlantic. You want to complain to [Project Manager] Jim Erickson about it?" Art knows this isn't going to do any good, since he's already complained to Jim himself, and Jim said MEX wins. But if he can get Ray on board, he can at least go back into the fight with more ammo. I don't expect Ray to say yes, but he does: "We'll be at Clovis" -- the outcrop -- "only a limited time; we don't want to blow two or three sols on this MEX thing," Ray says. "MEX will still be there, you know?" Art sympathizes, but notes that Erickson has already used his veto power -- "But we'll retry," he shrugs. In the end, Erickson does the right thing, the only thing he can do -- he plays the bad guy, and says we have to honor our international agreement to support MEX, and that's that.

Though that still leaves room for good-natured dissension in the ranks. If the drive errors out in the right way, we could still make it to Clovis but be at the wrong heading to support MEX, and it would look like an "unfortunate" accident. Chris jokingly volunteers to make this happen ....

Naturally, we do no such thing. And anyway, damn it, Jim, we're engineers, not diplomats. So we go back to doing what we do best.

We're running behind the (unrealistic) schedule, and Saina -- who always wants to just get the heck out of here -- is antsy. After the walkthrough, she asks how soon I can deliver. I look at my watch. It's 18:40 now ... "Well, this is pushing it," I say, "but maybe I can deliver by 19:00?"

"That would be great," Saina says. "If you deliver by 19:00, we will have a party."
Well, with incentive like that ... I get to work, going by the book, and when I deliver I look at my watch again. 18:58. I tell Saina, "You owe us a party."[2]

---

Footnotes:

[1] I don't actually know whether MSL works this way or not, but they eventually improved this on MER: a later flight software upgrade eliminated the need to wait before turning the IMU off, though you still have to wait after turning it on.

[2] But she never came through. Or, more likely, she did come through, and I just wasn't invited. ;-)