Spirit Sol 36

I get up early so I can come in for a traversability meeting at 1800. That meeting doesn't happen, which doesn't bother me because I manage to catch part of Ray Arvidson's lunchtime science briefing. He's talking about the little blue spheres found at Meridiani -- "Marsberries," he calls them. I'm ravenous because all I've eaten for the last two days is a little Chinese food and half a bad sandwich, so I'm almost deliriously happy to find that there's pizza in the room for some reason. I eat it while I listen to Ray talk.

When that's over, the traversability meeting happens after all. We haven't gotten yesterday's telemetry yet, so we have to assume that all went well and that we're starting from last night's planned destination. Our next stop will be to swing by the lander on our way to Bonneville. We'll drive to maybe 1.5m from the lander, image the astrobot experiment, and then drive maybe 15-25 additional meters (10-15m direct and 10-15 with autonav) if the Odyssey pass brings us good enough data. Adler says that the science team now understands that the rover has a terminal disease. They've been looking at science they can do 20m from the lander; Adler recommends they look at 25-30m targets, and reminds them they'll have to just plan from wherever we end up, especially until we get confidence in our autonav behavior. 30 days of driving lie ahead, and the scientists will take what they can get along the way.

As I leave that meeting I get sucked into a discussion of an APXS anomaly. They got only about half the data they were supposed to get from the other night, and nobody knows why, but since I wrote the sequences that started them up, it might be my fault. We start looking into the problem but can't find anything wrong -- my sequences appear to have done what they were supposed to do.

It's while I'm working on this that the Odyssey pass data comes back. Spirit didn't drive. Didn't fucking move. Not a millimeter. "Excuse me," I tell the others, and I go look at the new problem.

My first thought, of course, is: it's my fault. The first possibility that occurs to me is that I was careless with the first command in the sequence, a backward move. If I told the rover to avoid hazards, it would have failed, because it doesn't know what the adjacent terrain looks like. I look at this possibility right away, but I did the right thing. So that's not the problem. And anyway, these sequences ran in the testbed. I don't think there's anything wrong with them.

But Spirit didn't move, and I don't find anything that might explain that. I go back to the SMSA, where a number of hypotheses are being explored at once. They saw an unexpected contact switch trip during the IDD stow before the drive, but the stow succeeded and Spirit reported being ready to drive, so we don't see how that could be the problem. Or maybe it was because the HGA wasn't stowed -- but no, that turns out not to be the problem either.

The stress level in the room is rising. They're calling people at home, grepping through warning records. I look over at one of the TV monitors, which is tuned to CNN Headline News. They're doing a story about a seven-week-old baby born with two heads, who died during surgery to remove the extra head. This poor infant's family has been suffering the whole time we've been on Mars, and now they're in for worse. This gives me much-needed perspective on our little problem.

Which is cleared up soon afterwards. Way back on sol 18, when Spirit started behaving erratically, it was running a MTES checkout sequence. One of the commands in this sequence told it not to drive until the sequence finished. But the sequence never did finish. As it turns out, this was caught two or three times by different people already, but in the frenzy of the anomaly investigation, it was overlooked. The PMA and HGA were disabled by the same sequence and were later re-enabled, but driving wasn't. When they discover this, I'm sitting in front of a monitor to which someone has taped a fortune-cookie fortune: "You have the ability to overcome obstacles on the way to success." From the moment we discovered the problem, the whole process has taken only about 30 minutes.

The good news is that the vehicle is healthy. We might be able to just re-run the drive sequence tomorrow, and add a second drive later in the day. We talk about this with Mark Adler, who agrees to make it the tentative plan.

At the downlink assessment meeting, Larry Haskins gives a presentation whose title is almost longer than the presentation itself. "A Possible Partial Explanation for the State of the Martian Surface Materials as We Understand Them" is as far as I get in copying it down before he flips the slide. "Larry Haskins Explains the Dirt" would have been a much better title. It's got punch.

Anyway, the presentation goes something like this. There's now a global dust coating on Mars. In the past, there was more distribution of larger amounts of material by wind and water. This would have included sulfur and chlorine emitted by volcanoes, which were also more active then. The sodium and chlorine would have reacted to produce H2SO4 and HCl, which in turn would have helped to nucleate H2O, producing acid rain. The acid rain percolated down through the Martian soil, leaching salts out of the regolith (and dunes, dust, impact ejecta, etc.). The thickness and cohesiveness of the resulting cemented layer varies by the depth to which water could penetrate (in the range of centimeters to meters), and the rainwater did not evaporate but reacted with rock minerals. This helped convert the Fe2 in the rocks to the Fe3+ now prevalent on Mars. The wind covered the hardened regolith with dust in varying depths, but this cover may obscure evidence of water features. Once again, we'll need to dig -- or, maybe, get to Bonneville, where a meteor did the digging for us -- to find out.

Carr reinforces this last point, also adding that on Earth we have more help in this regard than on Mars. On Earth, plate tectonics also helps bring lower layers to the surface, but tectonic activity is so much slower on Mars that we depend on the craters.

Steve Gorevan is up next, reporting on the state of the RAT grind. The RAT hole is the first-ever planned hole dug on another planet, which -- along with Spirit's return to health, generally -- made us a top story in the news again today. The hole is a modest one, 2.7mm deep, but a first is a first. The notches (or "divots" -- there are a lot of golf analogies on this project, for some reason) which were thought to reflect where material was chipped away are really (he says) existing depressions in the rock that were simply uncovered as we excavated; if we'd dug another millimeter, the surface would have been completely flat. As it is, we got 93% of the surface exposed.

(At this point a news flash comes in, which frequently happens in these meetings. Even though the drive didn't happen yestersol, the imaging did. Much of this was wasted, but important parts weren't -- in particular, the PANCAM of the RAT hole succeeded, producing a stereo image of the hole and part of the surrounding area. So far we only have thumbnails, but we'll get the whole thing eventually.)

Tom Myrick, another RAT team member, follows Steve to report more details. He starts by trying to express how thrilled he is -- how thrilled the whole team is -- at this success, after the years of work they've all invested in it. They were the last instrument to be used, so they've had to wait longer than everyone else.

Tom reports that the RAT timed out before reaching the commanded depth (5mm). This is because it's their first grind, so they used conservative parameters. The RAT grind could have stopped for other reasons, but didn't -- this is important. In particular, the drill wasn't stopped by the penetration-rate timeout, which triggers if the RAT isn't making fast enough progress (indicating that the material is too strong for it, which might damage it). Consequently, the rock isn't as hard as some had initially feared.

How hard is Adirondack? Tom's looking into that, and it will take some time to determine. He gives a range: harder than limestone, not as hard as Palisades basalt.

In support of his presentation, Tom also shows a graph that plots the RAT's current draw vs. its Z-position. You expect it to draw more current the farther it descends, as it progresses past dusty surface layers and gets into the rock proper. This is exactly what happened; the actual numbers plot nicely against the prediction. They can do maybe 20 more such RATtings -- of rocks similar in hardness and composition to Adirondack -- over the course of the mission. That, at least, is how long the grinding bits should last; other factors might limit the RAT's lifetime further. This doesn't sound like much, but if we explore 20 more rocks, we'll be here forever. Not that I'm complaining.[1]

My day is shaping up to be simple, since it's mostly a replay of yestersol. So I get some soup and check out the new images. Opportunity seems to be going for more whimsical names than Spirit. Badger. Pancake. Krispies. Tapioca. Chani (a character from Dune, the protagonist's girlfriend). Tarmac.

In order to plan the lander-passing drive, I need to know exactly where the lander is. Strangely, this is not easy to determine, because the rover resets its position knowledge when we tell it to, which we did immediately after egress -- it no longer knows where the lander is. I ask the mobility/IDD team to tell me where the lander now is with respect to the rover. But the guy on shift needs to run RSVP to do that (and other things), and it's not working for him. I don't want to get sucked into this, but he's helpless, so I solve his problem for him. Now that he's back in business, I escape before I can get sucked into something else.

I should get to work and finish now, hours early, but I don't. Instead I look into what's known about the APXS problem -- why didn't they get their data? The problem, it turns out, was a timing issue. The rover woke up in the middle of the night to take the MB off of Adirondack and replace it with the APXS, which is the sequence I built. The master sequence needs to allow enough time for the tool change to happen, so they asked us for a time estimate. In turn, we asked the people whose sequences were run from ours, and their estimates, it turns out, were low. We added margin anyway -- but not quite enough. We missed it by literally four seconds. Four seconds short, and the APXS guys lost five hours of data. They got the other five hours anyway, which was OK, but could have been a lot better. Damn it!

By now it's time for the SOWG meeting. I have very little to do there today, especially because they decide to cut the second drive. They want to unstow the IDD again, touch Adirondack, and restow the IDD before the first drive -- an idea I already tried to discourage at the downlink assessment meeting. I hate to say no to science, but if they do the IDD stuff, it introduces a chance for something else to go wrong, which will cause us to be stuck at Adirondack one more sol. In the end that idea loses, which is bad, but also good.

Andy Mishkin leans over and asks if I'm ready to plan a 10m drive. The second drive -- the one they ended up cutting -- would have been that long or longer. If we exceed 10m, he says, we'll beat Sojourner's record. Well, it won't happen today, but I'm driving the rover again tomorrow. Maybe I'll set an interplanetary distance record.

After the SOWG, the night is straightforward. I pick up yesterday's drive sequence, make a few simple modifications, and am finished in no time. As a result, I leave more or less on time (forgetting about the midnightly science briefing, stupidly, but I saw the lunchtime briefing for once, so it evens out). Despite starting later than I should, I finish with plenty of time to spare -- not neglecting to include the commands that will clear the error that kept us from driving today.

I volunteer to stay late if needed, as I've been doing for the last few sols, but the TUL, Kevin Talley, says not to worry about it. "Go home and see if you can find your family," he says. I tell him I'm going to go home and watch my wife sleep. Which I do.


[1] Spirit's now up to 15 grindings, plus 95 brushings, with her RAT. And no, I'm not complaining. :-)


Anonymous said...

I'm vaguely surprised RSVP didn't catch the don't_drive flag (however it exists) problem. Or that there wasn't a big light at the driving stations showing that state.

That's hindsight, of course. Hopefully this is one of those anecdotes that percolates around, though, since it's a simple problem (not being fully aware of the current rover state, I suppose) with a few apparently simple solutions.

Scott Maxwell said...

@Anonymous You're absolutely right, and I appreciate that you see this as one of those things that's a lot more obvious in hindsight. When (long before landing) we defined the file RSVP uses to capture the rover's state, we were very concerned about *physical* state -- how the rover was tilted, how the suspension was articulated, etc. We did not then think about the rover's *logical* state. For similar reasons, while there is such a red flag at the downlink stations, we just weren't used to looking for it yet. (It's a small red flag, not a big red flag, I'm afraid. In fairness, there are lots and lots and lots of telemetry channels, so they can't all be big.)

Since this experience, we have significantly improved both our software and our procedures. The Tactical Downlink Lead reports the state of the appropriate flags, and I think it's in the Mission Manager's checklist to ensure they're cleared when we walk through our sequences; they're also highlighted somewhat better in the automated portions of the downlink reports. In addition, my automated flight rule checker looks up the state of the drive-related and IDD-related flags and spits out an error if you're trying to drive or use the IDD without clearing the appropriate state.

It took time to put all this in place, and (though I'm embarrassed to admit it) we had to stumble over this and related problems a few more times before we got our act entirely together. But we learned from each mistake, got a little better each time, and now I can't imagine a problem like this slipping by us.

A good lesson for future missions!

Anonymous said...

I am really enjoying your blog. I watched the moon landing at my grandparents on a B&W TV in the hills of eastern KY. But I would recommend you tone down the off color language a little.

Still a good blog.

Scott Maxwell said...

@Anonymous I'm glad to hear you're enjoying reading this, and I appreciate the politeness and positive nature of your request. However, since I wrote the material five years ago and have already decided not to edit my notes before posting them, there's little I can do to accede except to strike a different tone in newly written material (such as the footnotes and my comment replies), which I shall cheerfully do.

I will, however, caution you to just skip sol 47. Not kidding.

Anonymous said...

I don't agree. This real life. It's a wonderful "behind the scenes" blog and the colorness of the words are part of the story (though I don't always understand it). We are grown up people and even if there were young readers, they would learn those colorful words in their schools, anyway.

Mark Adler said...

"Terminal disease". I remember using that term back when even the most optimistic of us thought that Spirit wouldn't last the calendar year. That terminal disease sure is taking a heck of a long time to run its course!