Spirit Sol 471

It's a commonplace among engineers that you do your best work in the shower. Your mind and body are relaxed, and you see solutions that you just don't see under other circumstances.

That's not to say you always like the answers you get. While I'm getting ready in the morning, I realize we completely screwed up the huge MI mosaic of Keystone. It's the largest MI mosaic of a single object ever; it took about three hours for the IDD to do the work; and I single-handedly managed to get nearly all of the images out of focus. I've got my eyes closed, I'm putting on the shampoo, and suddenly I visualize the relationship between the rock face and the motion of the IDD as it moves the MI across the surface, and it becomes blindingly clear that the angle is completely wrong for the majority of the surface.

See, the MI has a very shallow field of view. You can think of it as being like a small square sheet of paper, held about 4.6cm (call it two inches) in front of the lens. The "piece of paper" is maybe 3cm on a side and 3mm thick -- maybe instead of a single sheet of paper, you should think of it as a pad of just a few Post-It notes. Anything not within that volume will be either invisible or unfocused. If you look at a surface at too oblique an angle, most of the "notepad" will be either above or below that surface. A small amount of the image will be in focus, but most won't.

We held the paper at the wrong angle.

Some small fraction of the MIs will be in focus, but most won't. We basically just blew two or three sols, four or five hours of IDD actuator life, and hundreds of megabits of downlink. And it's all my fault.

I get to JPL a little early, anxious to know how bad the situation is. Nobody says anything in particular to me, which is usually a bad sign. I fire up mphoto (the image browser application I wrote during solar conjunction), bring up the MI thumbnails, and click on one of them. It happens to be one of the in-focus images, and looks pretty good. Then I click on another one. It's in focus, too. I click on a few more. They're all in focus.

Um. Evidently, I was mistaken. Happily, it would seem that I do better IDD work in the sequencing room than in the shower after all.

And I realize I'm now the holder of another record (not that any of us holds these individually, of course; it's always a team effort). Not only am I the single-sol distance-record holder on both sides of Mars, I've now also done the most MIs of a single object ever.

And they're even in focus!

Sadly, the MI sky flats don't work out again -- there's no time to do them in the weekend plan. When they officially sink the idea, I hit the mike and say, "Screw you guys, I'm doing it anyway!" Which provokes a laugh, and Ken says, "Go, Scott!" But he's philosophical about the lost chance: "I waited 400 sols, I can wait a little longer."

I did make one mistake in the MI sequence. We were performing the MI mosaic in six columns of four stacks apiece, covering the broad rectangular face of Keystone. Now, normally, we take an image of the IDD each time we place an instrument, taking higher-quality images when we place the MI to give us data points for calibrating the HAZCAMs. (When the MI is down, the MB is up, right where the cameras can see it, and its shiny contact ring is easy to pick out in the images.) But we didn't want to spend time or downlink volume taking those images for every stack, since there were so many. Instead, we want to image the IDD only once per column.

So I came up with a clever way to do that. When we're driving, we sometimes use the OK-to-IDD flag as a simple marker -- for instance, if we get to a certain point in the drive, we drop the OK-to-IDD flag. Then later on, the sequence checks if it's OK to IDD; if not, we know we got that far in the sequence and can do something special for that case. So I figured, why not turn that around? In the backbone sequence, we'll drop the OK-to-drive flag as we start each column, then make four calls to the helper sequence (each call to the helper sequence performs one stack and moves to set up for the next stack). The helper sequence will check whether it's OK to drive and, if not, will take the IDD image and then re-raise the flag so that the next call to the helper sequence won't take the image. That way, we'll take the image for only the first stack in each quartet.

Like so many solutions, this one is simple, elegant, and wrong. What I stupidly forgot was that when the IDD is unstowed, we're never OK to drive. So instead of imaging the IDD placement on every fourth call to the helper, we imaged it on every call -- sixteen IDD placement images, not four. And we reused that code when we completed the mosaic, so we're going to repeat the mistake. Oops.

So we got a whole bunch of images of the IDD moving across the surface. High-quality images. Hey ... on a whim, I make a simple animation and project it on the shared screen in the sequencing room. It's awesome -- not only do you see the IDD doing its thing, you even see the rover's own shadow marching away as the sun passes overhead. Steve sees the movie and loves it. "Maybe we should do more things like this," he remarks.

"Yeah, if only all our mistakes turned out this well," I reply.

Brenda Franklin also pipes up. "When life gives you lemons, make lemonade. This is lemonade. This is lemonade with whiskey!"

"Hey, you know what you should do?" Steve says. "You should send this to Guy Webster!" (That's JPL's main press dude for MER.) "When we release the MI mosaic, we can release this movie to go along with it."

Which is a great idea, and displays Steve's flair for outreach. So I put together a version of the movie for release -- actually, two versions, one from each eye -- and email the press people about it. We'll see what happens.[1]

I finish up the sequencing and hand over to Chris, then hang out to work on some other stuff. A little later, Steve comes on the teleconference line again. This time he's raving about the MIs. You can practically hear him jumping up and down with excitement. "Those MIs just look great!" he exclaims. "This is gonna be one of the premier data sets of the mission. I just talked to Ken, and they're going crazy in Flagstaff!" (This is not some strange expression; Flagstaff -- the University of Arizona -- is where Ken and his team are.)

I couldn't be happier, especially since this is the exact diametrical polar opposite of what I expected to hear when I came in today. I tell Steve I'll pass his message along to Chris and Ashitey, and I do.

Steve's got one more thing to mention. He was unable to cover the SOWG meeting this morning because he was teaching a class -- "Astrophysics 102 -- 300 students. At the start of every class, I give them a five-minute update on what's going on on Mars. It's the only part they pay attention for."

[Next post: sol 476, May 5.]

---

Courtesy NASA/JPL-Caltech. An accidental animation of the first sol's work on Keystone: best mistake ever. Follow the link for a full-sized version.


Courtesy NASA/JPL-Caltech. The mosaic of MI images on Keystone. Again, follow the link to get a full-sized version.

---

[1] As always, Steve's instincts were spot on. They loved it. They released the animation at the same time as the Keystone press release, and it's shown up more places than I can count since then. Best mistake ever.

Spirit Sol 470

We're in that weird phase where we're planning in the blind. The sequences we wrote yestersol haven't even been uplinked to the rover yet, much less executed, so clearly we haven't gotten the downlink yet. And today's are going to look similar to yesterday's, so we could blow two whole days if we did something wrong yesterday. So I ask Chris -- RP-2 today -- to look them over. In theory, if he finds a problem, we could fix it and uplink the result instead of what we planned yesterday. Not that our mission manager would let us get away with that. But Chris doesn't find anything anyway, so it's moot. At least I'll be able to sleep a little better.

In theory.

Yesterday we did a 4x4 subset of the total 4x6 mosaic. Today we pick up where we left off, doing the remaining 2x6 subset of the mosaic. Then we brush off a spot with the RAT and place the APXS there. Overall, not such a bad day -- yesterday was harder.

The hard part is to fit in the MI sky flats. This is where we take microscopic images of the sky. Yes, you read that right. It's a calibration thing. The problem is that there are restrictions on how we take the sky flats, and it's awkward to satisfy them all at once. First, the MI must be aimed at least 90 degrees away from the sun. Second, the sky-flat images must be taken late in the day, after about 14:00. Third, the MI needs an unobstructed view of the sky, so no part of the rover body or the terrain can be in the way. Fourth, it must be aimed about 40 degrees above the horizon.

The first and second constraints combine to force the MI to aim to the rover's right. So now the problem is to find a solution where the MI is aimed to the right, 40 degrees above the horizon, without any terrain or rover body in view. There doesn't seem to be any way to do this. In one wrist configuration, the rover deck blocks our view when the MI is close to the WEB and the IDD self-collides when the MI is pushed farther from the WEB. In the other wrist configuration, we can choose between having the rover deck in the way or running into a hardstop -- with the arm otherwise in the needed pose, the 40-degree elevation is in a "blind spot" where the turret can't aim the MI.

This means we can't do the sky flats, which makes me feel very bad. The MI PUL, Ken, is such a nice guy, and it's been literally 400 sols since he's had this calibration done. Then at last they make room for it in the plan, and I can't find a solution for him. It's not my fault that the solution's not there to be found, but I still feel bad about it.

But otherwise, the sol goes well. We're practically drunk on power -- 937 Watt-hours thisol, more than we had when we landed. Thanks, dust devils! And the sequencing proceeds without any major hiccups.

At the end of the day, Steve fills us in on the science team's current thinking about where we want to go from here. First, images taken from the other side -- that is, the current side -- of Larry's Lookout (a landmark partway up Husband Hill, which we passed on the way up) have apparently made the scientists even more interested in it, so we're going to head back downhill to it. Downhill is the wrong direction, I want to say (damn, I want to see the view from that summit!) -- but I'm just the chauffeur, so I wisely keep my mouth shut. The right direction is wherever the science is, and if that's downhill from here, then so be it.

On the way back down to Larry's Lookout, we'll check out another rock near this one, then this area called Jib Sheet. We probably won't get quite close enough to Larry's Lookout to touch it, since there's a steep pile of soft sand in the way, but we'll take highly detailed pictures from as close as we can. (Like Burns's Cliff, Steve points out.) Then we turn back around and start slogging up the hill again.

"Sorry about that," Steve says. "This is the best exposed geology we'll see this side of Husband Hill, so we gotta do it."

That's okay, it really is. The big disappointment from today comes from an unexpected direction, when Chris finds a way we could have done the MI sky flats after all. It's a configuration I thought I'd tried, but obviously not. So now I feel twice as bad about not finding a way to do it, because there was a way to be found. I just didn't find it.

(And writing this, it occurs to me there was always a way anyhow. The IDD's elbow joint is fully double-jointed; we could simply have swung the turret up so that it was above the elbow rather than below it. Then aiming the MI for the sky flats would have been trivial -- the rover deck is never in the way in that pose. Oh, I am dumb sometimes, I truly am.)

I swallow my pride and send Ken and Steve email about this. We couldn't do the sky flats today, but we'll still be here over the weekend. Maybe we'll squeeze it in tomorrow.

Spirit Sol 469

It's been nearly two weeks since I was last on shift. The drive we planned last time I was on shift went splendidly, putting us just where we wanted. Once there, as we started climbing up the long way around, the scientists started finding more stuff they just had to take a look at. So Spirit spent a while investigating some crumbly stuff called Methuselah, and now we've just driven up to a rock called Keystone, which they're practically salivating over. (And I'm not the only one who thinks so: "Steve's like a kid in a candy store," Dan Moyers remarks to me during the SOWG meeting.)

Thisol we're starting to IDD Keystone. Just to make sure I don't get bored, this is going to be the biggest MI mosaic of a single object ever, a total of 6x4x5+stereo = 144 images total. Our approach is to sequence as many of them today as the rover will have time to execute, and finish the rest tomorrow.

Before we can even really get started, we have to figure out what we want to image. This is troublesome, because we first have to know the angle of the MI with respect to the roughly rectangular rock surface -- that affects how the image footprints are oriented, which affects how you lay out the mosaic -- and none of our tools does that very well. So we basically just eyeball it, and it turns out it's reasonably close to the same orientation as the overall rectangle of rock itself, which simplifies things.

For the next step in the process, Ashitey and I (mostly Ashitey) pick a candidate center for the mosaic and use an image processing program to sketch out what the mosaic will cover with that as its center. Drawing the rectangle is easy, but figuring out where it lies is hard. No tools show you what the MI image covers (RSVP shows you a predicted MI image, but it's hard to figure out exactly where this lies in the original images), so you have to do a lot of eyeballing and detective work, and fit the results back into the original image. Oy. We email the resulting sketch to Steve, who prints it out, plays around with a ruler, and asks us to shift the whole thing 4.5cm to the left. So we do.

I think I have some ideas for improvements for future missions.

Anyway, with that done, the rest of the work isn't so hard. There are a lot of images in the mosaic, but they fall into groups, a total of 24 stacks of six images (including the stereo image) apiece. I further break down the mosaic into six columns of four stacks each, and then there's only a little overhead work to set up each column. It takes time, but I've had harder sequences to build. In the end, it turns out the rover won't have time to do more than four of the columns thisol, but that's still two-thirds of the total; we'll fill in the rest nextersol.

The only serious issue that arises is a question of using the IMU during the IDD sequence. This is something we do when there's a chance the vehicle's overall attitude would change during the sequence, which is something that could potentially endanger the IDD: if the vehicle tips forward when the arm's extended, suddenly it's that much closer to the ground, and could run into something we thought it would be clear of.

We're supposed to need to turn the IMU on when IDDing only when the rover's preloading the RAT or when we're at a significant pitch -- or anyway, that's what I thought the rules were. Ashitey thinks we need to do it when we're tilted, not just when we're pitched. (That is, he thinks we should do it even when we have zero pitch but lots of roll.) Thisol's pitch is only about 6.5 degrees but the overall tilt is a much higher 14.5 degrees, so it's going to matter which of us is right. John Wright is around, so I bring him in as a tie-breaker, and he votes with Ashitey, darn it. ("That was the wrong answer," I tell him.[1])

Why not just turn on the IMU anyway? Spirit's got power to burn, so it's not that. The problem is that there's ten seconds of overhead in every IDD move when the IMU is on, and what with all the crazy stuff we're doing thisol, the ten seconds per move add up to a whopping 38 minutes of overhead! And we don't have that to spare.

We could make back the 38 minutes by cutting another column, but John comes up with a better idea than that. It's a compromise: turn on the IMU during the unstow and first placement, and thereafter only when we actually contact the surface. Those are the times when there's the greatest risk (not that there's any real risk, but we're paid to be paranoid and we earn every penny) that the attitude would change. And that compromise adds only about six or eight minutes to the run-time, not 38. So that's what we do.

It's another fairly long day, but still less than ten hours. And if these images look as cool as I hope they will, it will have been well worth it.

---

Courtesy NASA/JPL-Caltech. Poised to examine Keystone.

---

Footnotes:

[1] Turns out I was right: pitch matters, roll doesn't. Fortunately, with the maturity and distance of five years, I have the class not to say nyaah nyaah.

Spirit Sol 455

The good news is that we're not dug in. The bad news is that we're not uphill.

Instead, we experienced alarmingly high levels of side-slip, gravity pulling us downhill as we tried to drive across the slope. The backoff worked, but then, only a couple of meters or so into the uphill segment, the sequence detected that we came too close to an obstacle and shut us down.

Ashitey tells me something I don't want to hear. "We can't make it," he says. "We should go back downhill and try to go around."

I hate it, but he's right. Straight uphill doesn't work. Crossing at a 45-degree angle doesn't work. Shallower angles -- either east or west -- take us into impassable slopes on one side, or rocks and an uncrossable berm on the other. The only way out is down.

Ashitey sets up a meeting among all the concerned parties. We gather in the SOWG room and, in short, agree that Ashitey's right. We'll have to go down.

Twelve meters or so downhill, there's what looks like a graded path around the side. My big concern is that we can't see all the way along the grade, so for all we know, we'll follow it to a certain point and then find that that way is blocked as well. But there's no way up from here, so it's the right next thing to try.

It's a complex drive. Ashitey and I are dumb enough to make it even harder on ourselves by adding a cross-slope leg after the downhill leg. But at least if it works, we'll have made real progress. If you can call a retreat progress. I guess we'll have made real regress, or something like that.

We also manage to pack in a couple of visits. The first is by science
fiction writer Bruce Sterling, whom Frank invited to come see us drive
a Mars rover. This is so wild I can't describe it.

Sterling's not what I expected. I don't know what I expected, but he's not it. He's short, dumpy, long-haired, laconic -- and creative, that part's no surprise. He takes one look at the six-wheeled rover with solar-panel wings and calls it a bug, which is a comparison that somehow never occurred to me, and I'm practically an avocational entomologist.

Our other visitor -- or group of visitors -- is brought around by Eric Baumgartner. They're a quartet of science fair winners from his kid's school, all around eleven or twelve years old. Plus two teachers. Ashitey and I do a good job with them, I think. He makes sure they look through the 3-D glasses, which appeals to everyone but especially kids. I don't think I'm generally very good with kids, but I remember being their age and being excited about science, and I'd have died for a trip like this. They're just like that -- a little intimidated at first, but grooving on it. I manage to get them to laugh and loosen up a little, and then they're just a bunch of smart kids having a blast. I love my job.

The long day goes on. We work apace. Jim Erickson shows up, with his usual grim look, and subtly chides us for adding the cross-slope leg. He's not the type to second-guess our decisions, but he's got a point to make and he makes it. We're putting in a lot of late nights, which exhausts the team. We've got to keep it simple. The extended mission is a marathon, not a sprint.

At this point, it's too late to gain anything by cutting the cross-slope leg. Since it's a sunk cost, we decide to leave it in for now, but we'll take it out if it ends up sucking up any more of our time. (It doesn't: the cross-slope leg makes it into the final sequence.)

We reach that point in the day where we're just waiting for the TAP/SIE to put everything in its final form. Today we put the time to good use. I've been worried about the fact that nobody really pays attention in the walkthroughs any more, which is partly because the complexity of the sequences, and the amount of stuff you have to know about the rover in order to follow along, is really overwhelming. So I break out a few pieces of the sequence and start going over them in detail -- an impromptu rover-driving course. When I see that people's brains are getting full, I declare success and move on to other things. It's only a little progress, but it's progress.

You can tell it's late, by the way. Jeff Favretto, following up on Erickson's earlier comments, says something about the value of getting out by 8:00 so we can spend time with our wives. (It's now nearly 10:00, and we'll be there another hour. We started twelve hours ago.) "I'm single, but if I don't leave by 8, I'll never get a girlfriend," Dan Moyers chimes in. And Favretto quips: "Dan, you'll have to leave a lot earlier than 8 if you want to get a girlfriend."

Ouch!

On a lighter note, Steve Squyres's book (Roving Mars) is coming out soon, and he's asked people on the project to help him make sure that the name of everyone involved in the project is listed in an appendix. This includes everyone from the project managers down to off-site contractors who sewed the parachutes. He had a good start already, but for the past week or so he's been beefing up the list, adding names as he's sent them.

So far he's got four thousand.

[Next post: sol 469, April 28.]

Spirit Sol 454

We've recovered from the weekend's fault, and the downlink shows we're in pretty much the state we expected: somewhat dug in. A small potato hovers by the left front wheel, and another possible potato -- the scientists bet strongly it's just a dirt clod, not a rock -- sits in the left rear wheel. So it'll be tricky just to get away.

Fortunately, we have a little more time than expected. On the large scale, that is: "There's nothing magic about sol 500 any more," Steve declares at the SOWG meeting. This had previously been the date by which we had to make it to the summit, before the sun angle started to favor south-facing slopes. The resulting lack of energy on our current north-facing slope would mean we'd face serious consequences. In the benign scenario, we'd have less and less time to drive each day, making the top farther and farther away. In the less benign scenario, we'd freeze to death.

But the dust devil that cleaned the solar panels bought us more time than I'd thought: even with the reduced energy level, we can last well past sol 500 now. The pace is now to be driven by the Martian terrain and scientific discoveries, not a mad scramble for the top.

Which is a good thing, because we haven't exactly been making a lot of progress lately. The next thing we're going to try is cross-slope driving. The last drive failed because it was fighting directly uphill -- the same mistake we made around Wopmay, back on the other side of the planet. This time we're going to try to drive at about a 45-degree angle to the slope instead. Before we can even do that, we have to carefully back away downhill, to extricate ourselves from our dug-in position. If you thought the going had been slow recently, just think what it would be like if we picked up another potato ....

Just a few meters away, the slope breaks, and should be easier going. Once the rover gets that far, the drive aims a little more uphill, to avoid a gap in the terrain mesh (representing an area we can't quite see from our current position, and therefore can't evaluate).

This is the first drive I've written for Spirit in quite some time. I have this fantasy of coming downstairs and magically putting everything back into shape, coming up with some brilliant scheme that puts the rover on the top of this hill in no time. Awing the crowd. But no such thing will happen: the guys who've been doing this are plenty smart already, and if there were any such clever trick, they'd have thought of it. Getting to the top is just going to be a long, tough slog, and it's really possible that we just won't make it.

I have a talk coming up in a few days, down in Anaheim at the USENIX 2005 conference. I'll be talking about how to drive a Mars rover. I spend some time thinking about how to put the audience in the right mindset. This is what I come up with:

You're 450 sols into a 90-sol mission. Threats surround you; every path is blocked. You're chasing the sun to the top of a hill and if you don't beat it you'll die. Welcome to Mars.

A little too dramatic, I think; I decide not to use it. But it sounds cool.

Spirit Sol 453

On my way in I notice we're on the front page. Our funding was extended for 18 more months, making the front page of the Pasadena Star-News. I remember when I'd come in and we were front-page news every day. I buy a copy for nostalgia's sake.

I'm in an hour early, to get a jump on the drive sequence, but Jim Erickson needs a minute of my time. "Did you hear about the problems over the weekend?" he asks. Oh, dear.

Both rovers had bad weekends. Opportunity's drive faulted out before it even drove anywhere. Last week the suspension limits tripped during a drive, and tripped again during the auto-recovery -- an off-nominal circumstance, but one we plan for. For the weekend, they sequenced a drive where they backed off the ripple they tripped on, then continued, ignoring the suspension limits during the backoff. But they didn't explicitly disable the limits, they just assumed the limits were i the "ignored" state for the first part of the drive. However, it turns out that the flag that tells the rover to check the suspension limits (not ignore them) was accidentally saved to EEPROM as the default weeks ago, so Opportunity actually applied the most recently set suspension limits, and since they'd tripped before, and Opportunity hadn't moved, it went right ahead and tripped them again. This was a problem that was just waiting to bite us. And then, possibly because of that error, they mis-timed the solar-array stomp, which meant the rover didn't nap when it was supposed to, and thus is power-poor.

Spirit had problems, too. They didn't properly clean up after last Thursday's fatal. (Which fatal, you will recall, was my damn fault.) The fatal caused Spirit to decide it didn't know where its HGA was pointed, so it didn't listen for the uplink from Earth. As a result, none of our cleanup sequences made it up over the weekend. Which means we can't drive today; we still need to do our imaging and update the rover's position. Lucky for us, Beth Dewell noticed the problem and already got Spirit back into decent shape over the weekend, which is a good thing -- else we'd be down for another day.

So, for those of you keeping score at home: last time I was on Spirit, I jammed a potato in the wheel, which blew about ten or twenty sols. This time I fataled it and blew Thursday through today (Monday) at least. The Opportunity problem is not my fault, but I couldn't feel worse if it were. I am a bad rover driver. I'm glum. I want to kill myself.

But if we're not driving today, I have time to answer Jim's questions, at least. He ticks them off. "These problems we've been having with the vehicles lately: are they problems due to unfamiliarity with the new version of the flight software? Are the RPs overworked? Are they complacent?"

I answer these as best I can. I know some of the problems were due to unfamiliarity with the new flight software; it's a partial explanation. I don't think the RPs are generally overworked. (I personally have a lot more work than I have time for, but this afternoon I'm going to talk to someone about coming in to the RSVP development team to pick some of it up, which will mitigate the problem.)

As for complacency ... well, I want to say no, but the honest answer is probably yes. We're not as paranoid about these vehicles as we used to be. Some of that's understandable and even proper. Some of it, maybe not.

Jim nods thoughtfully and goes away. As if on cue, Frank enters, fresh from the Land of Opportunity. He tells me he wouldn't have caught the problem that led to a fatal, either, but: "I was gonna give you a ribbing about it anyway, but then we had our problem over the weekend, and now I can't give you a hard time."

"The universe smiles on Scott," Al Herrera remarks.

If he's right, it means the problem on Opportunity was my fault. You know when I said I couldn't feel worse? I might have been wrong.

Upstairs, while we're waiting for the SOWG meeting to start, we try to look on the bright side of this extended anomaly. "The batteries will be topped off," Chris points out.

"And now the bar is lowered," I add. "If we can just manage to drive without fataling, we'll be so happy." It's black humor, but it's humor.

Squyres is -- what's the equivalent of "visibly unhappy" for a disembodied voice, "audibly unhappy?" Well, he's unhappy. We've got basically nothing to show for last week, with the ODY problem followed by the fatal, and today will be basically spent on recovery. He pokes at this to see if we can drive anyway, but we really can't do it without more data. He manages to talk Callas into allowing a MTES checkout, so I suppose that's something, from his point of view.

But we're 120 sols into the climb and only halfway up the hill, and we're 50 sols away from our deadline. We're power-rich right now, but in about 50 more sols, the sun's wandering position in the sky starts to favor south-facing slopes over north-facing slopes such as the one we're on. Our friendly, solar-panel-cleaning dust devils might have extended the deadline, but we'll eventually have to cross to the far slope or die. Steve wants to go, go, go, and we've got to just sit here. At least for today.

"We're gonna have a good week," Steve says emphatically at the close of the meeting. "Power of positive thinking."

Spirit Sol 450

I have to go give an RSVP demo, but first I have to know how Spirit did on my first attempt to drive it for months. I borrow a Web browser window from Colette Wilklow and quickly bring up the automatically generated Web page that summarizes the results.

The first line says more than eight meters -- not too shabby, for this terrain. Then I notice the next line, the usually more accurate line -- the one that tells you not just how far the wheels spun but how far the rover actually moved. About 87cm. That's all.

Fuck!

Just then my cell phone rings.

"Hello?"

It's Mark Maimone. "Well, I ruined my record," he says. "I fataled a spacecraft."

"Fataled" doesn't mean the rover is dead, but it is a bad thing. It means the rover rebooted. And it's our fault.

After I get back from my demo, I get the story. What happened here, it transpires, is that we screwed up something really stupid. It has to do with a feature in the new release of the flight software. The original flight software let you write sequences that made decisions based on a limited set of parameters -- so that the rover could decide to do one thing or another based on its distance to a defined target, whether the previous command had succeeded or failed, and about a dozen more. The new flight software added a bunch more of these parameters, one of them being the rover's current heading. (So now we can write a sequence that says, for instance, "if you're not already heading more or less east, turn to the east.")

Now, there was only one way to add these new parameters in a way that was backward-compatible with the old command dictionary (a requirement imposed from above), and that was sort of a kludge. The trick was to take one of the old parameters -- one we picked because we never actually used it in flight -- and use a command to redefine that one so that it's an alias for one of the new ones, and then tell the rover to test that. The "if" command itself resets the alias to an undefined state afterward. And when you test this parameter when it's in an undefined state, the flight software fatals.

Now, we did everything right. In the main sequence, that is. What we forgot is that we also had a background sequence running, one that checks periodically to see if the rover is too close to an obstacle, so that it can abort driving. And this sequence is basically a long string of "if you're too close to obstacle #1, stop driving; if you're too close to obstacle #2, stop driving ...." One of these commands just happened to undefine the special parameter set in the main sequence, triggering the fatal.

Ashitey and I didn't realize that any "if" reset the special parameter -- we thought only the ones that used that parameter reset it. Mark Maimone knew better, but he was awfully tired and stressed out yesterday. I fear he's overloaded.

John Callas declares by fiat that the team is too tired to work this weekend. We're just going to clean up after our boneheaded error and do a minimal weekend plan. No driving, no IDD. It's a hint that we should go home and rest up. But do I? Nope: too much other work to do.[1]

I'm not surprised the team is tired. This whole hill-climbing campaign with Spirit looks like it could become frustrating. For the rover, it's a long climb. We've been at it three months, and we're only halfway up. The hell of it is, two decently strong men could pick the damn rover up and carry it to the summit in half an hour.

When Andy's new schedule comes out, it shows I'm not downstairs just temporarily. Looks like I'm back on Spirit at least through the end of May, which probably means for good. I'm surprised that I'm disappointed by this. I'm almost ashamed to admit it, but there was a lot to be said for working on Opportunity. The team's more laid back. It's easier. You drive really far. It's a cheap kind of glory but I'll take it.

But in the end, I'm not too disappointed. This rover and I have been through a lot together already, and this just means there's more adventure ahead.

I can't wait to get started.

[Next post: sol 453, April 12.]

---

Footnotes:

[1] Easy for me to say now, but that was a bad decision: incompatible with rover safety. Shoulda gone home and slept. Shame on me.

Spirit Sol 449

ODY is back online; our broadband connection to Mars has been restored. I still don't know what the heck was wrong with it, but I guess I'll find out.

So we're driving. Our plan is to continue the drive that faulted out this past weekend; we've been sitting still since then because of the relay problem. I'm glad we're going to do some driving; else I'd come in next week and have to be RP-1, without any way to get properly up to speed. Sure, I've read through the drive sequences they've been building, but it's never the same as when the pressure's on.

My brain still hasn't caught up with this side of the world. I go over to MIPL to ask them to build us a new terrain mesh with the extra NAVCAM images that just came down, and unconsciously I go over to talk to the Opportunity support guy. Who's as puzzled as I am until I realize I'm supposed to be talking to the Spirit support guy. Sheesh.

But it's fun sequencing this rover again. I'd almost forgotten what challenging drives were like. Opportunity is just a slow car in a parking lot full of speed bumps. Driving Spirit up a rocky hill is delicate, exacting work. Obstacles to the left of you, obstacles to the right of you -- and a slope that's only too happy to help you slide into them.

One of the innovative techniques the Spirits have developed is a parallel obstacle-check sequence. The way we used to drive through hazardous zones was to first construct a list of hazards visible in the imagery. Then we'd write a sequence that would drive a little way, check if we were too close to the obstacles, drive a little farther, check again, and so on. At each stage, you'd tend to check only for obstacles you thought you'd be likely to be near at that point, because it was just a nightmare to try to check for all of them all of the time. The resulting drive sequences had a lot of repetition and were hard to modify: if you wanted to change your driving path, you'd have to revisit a lot of the places where you stopped and checked for hazards to make sure you were checking for the right ones.

So the Spirits worked out a way to separate those concerns. They now build one sequence that checks whether the rover is near any of the known hazards; this is padded out with wait commands so that it takes one minute to run. Then they have a second sequence that calls the first one ten times, and a third sequence that calls the ten-minute version 24 times (so it runs for 240 minutes -- or four hours, longer than their drives). Finally, this third sequence is activated in parallel with the drive. So the drive can just contain the needed commands to take the rover along the desired path, and the parallel obstacle-check sequence will stop it if it gets too close to any of the hazards. The parallel sequence checks all of the hazards, and checks them every minute, and doesn't interfere with your development of the drive sequences; in all three ways, it's better than the old approach.[1]

There's a lot more to come up to speed on as well -- a lot of fiddling with parameters that we don't need to do on Opportunity, but which are needed on Spirit, mainly to avoid catching another potato. Finally, as gravy, thisol we're doing two things for the first time on Mars: at the end of the drive, we're using autonav with visual odometry (so the rover will autonomously avoid obstacles as well as keeping careful track of its location), and we're using a new flight software feature that lets us make run-time decisions based on the vehicle's heading. If all goes well, we'll come in tomorrow to find we've climbed Husband Hill another 15 meters or so.

There's another difference between the two rovers: Opportunity's drive distances have more digits.

---

[1] A later flight software update added a notion of "keepout zones," which replaced this elaborate tree of obstacle-checking sequences with just a few commands, one command per obstacle. Incomparably better.

Spirit Sol 448

Drive? IDD? Neither: with ODY still out of commission, if we drive today, we might not get enough data down to do anything else until the weekend. So we're taking a gamble, doing a little remote sensing and basically sitting still. If ODY comes back tonight as expected, we'll be able to do normal work tomorrow.

For today: sit and stare at the ceiling.

Spirit Sol 447

So I'm back on Spirit. I'm a couple of minutes late, but it doesn't matter: as I'm walking in, they're saying what I expected them to say. Until we've recovered from the ODY thing, no driving, no IDD. Maybe Saturday.

But at least I'm not the only one who blows drives around here. Like Opportunity, Spirit had a two-sol drive over the weekend. The scraps of data we get back today show that the first sol's drive faulted out (early, too), so the second one didn't execute at all.

This shouldn't make me feel better. But it does, damn it, it does.

Opportunity Sol 426 (Spirit Sol 446)

Today's my last day on Opportunity for a while, and it's off to an inauspicious start. Our main relay orbiter, Mars Odyssey (ODY), went into safe mode over the weekend. As a result, we didn't get any of our downlink. Not a sausage.

When I look at it the right way, I see our "no news" as good news: I checked in on the rover over the weekend and was alarmed that there had been no updates. I wasn't sure why that would happen, but of course the explanation that immediately occurred to me was that I had killed it. So it's not that, and that's good.

Of course, maybe ODY went into safe mode and I killed Opportunity. Heck, maybe whatever I did to kill Opportunity took ODY with it. ODY learned that Opportunity was gone and then ODY died from grief, maybe. But what are the odds?

Well, if we assume Opportunity is healthy, maybe we can even drive it. Our uplink isn't affected, so we could send up another all-autonav drive sequence. I have it all worked out in my head, and start to pitch the idea. But I'm brought up short by the fact that there's no room in flash: the drive will generate so much data that it will push out other stuff we might not want to discard, and that's a risk we can't take. Plus, nobody seems to know for sure, but several people suspect there's a rule about that: you have to confirm that the rover is healthy before you uplink mobility sequences. (We have someone do that for weekend drives, for instance -- they don't re-plan in the middle, but they can pull the plug on continued driving.) So that idea's out.

So we can't drive, and without images, we can't IDD, either. So you'd think I'd have the day off. But it doesn't work out like that. Instead, I decide to use this unexpected holiday to try to get other work done. I figure if I can get some stuff done for Phoenix today, I won't feel so bad about ignoring them on days when I want to focus on the rovers.

Meanwhile, the ops team has reconfigured the rover to try to send a little data directly down to Earth. This takes lots of power and doesn't return a lot of data, but it'll be better than what we have now. The data brings good news and bad news.

The good news is, we have data, so Opportunity really is alive. (Maybe ODY is grieving for the pope instead.) The bad news is, the drive faulted out. Analysis shows that the first drive got closer to Voyager Crater than we'd planned -- a lot closer. Rather than being 10m or 15m away, we're right on the lip, with 8 degrees of tilt.

But that drive worked. Up to this point, the rover was happy and safe. (Not quite where we thought it would be, but happy and safe all the same.) But then the next sol's drive, the all-autonav continuation drive, kicked in. In the course of this drive, autonav climbed a little higher on the crater lip, hitting the max tilt limit of 12 degrees, which stopped the drive. It drove only about two meters.

Part of this is just due to a dumb misconfiguration. The autonav software has its own notion of what tilts to stay away from, and our default setting for this limit is 15 degrees. Thus, autonav doesn't know to avoid the 12-degree limit that stops driving. (Both limits are configurable; our mistake was to lower the max-tilt limit and not tell autonav.) If we'd lowered the autonav tilt limit as well, autonav would have known it should stay lower on the crater lip, and the drive might well have worked.

The other problem is that the first drive got much closer to the crater than we thought it would. This would be alarming even if the second drive had succeeded, and somebody (not me, thankfully) gets the responsibility for finding out why it happened.

So now we're trying to debug and recover from a drive failure, and we have no data. On a normal sol, this wouldn't be a problem -- we'd just clear the error, aim downhill, and step on the gas. But with no data, there's not a lot we can do. I work with Emily and Charles to re-prioritize the images already taken, so we'll get what we most need a little sooner. But the big problem is that we've gone from a fire hose to a drinking straw. We'll just have to wait.

As it happens, given the ODY situation, we don't want to drive until Wednesday or so anyway, at the earliest. So there's not any particular pressure. Just for the sake of it, I design a drive that would let us return to a clear zone and proceed to our next destination even if we get no further data, but we'll never send it. I just do it for the exercise.

I also realize one other piece of bad news. It's an unusual distinction, but for some reason, I happen to have crossed more kilometer boundaries with these rovers than anyone else. I don't know if I've done more total driving than anyone else (though I wouldn't doubt it), but I just happen to be on shift when the km number clicks over. Well, when the autonav drive faulted out, Opportunity was less than five meters from reaching the 5km mark. Someone else (Frank, I think) will get the glory this time. I mention this to the team, and Geoff Lake cocks an eyebrow at me. "Has anyone ever told you that you're really focused?" he asks. Only twice a day ....

Meanwhile, it occurs to me that we should exploit our fabulous position. We're poised on the lip of this crater; might as well take some pictures! But Charles Budney just shrugs. "Voyager is going to look like Viking," he says. "We'll take a low-res NAVCAM-360 and move on." That's a crime. This would make a spectacular full-color PANCAM panorama, but try as I might, I can't light a fire under the science team to do it, so I give up. But I'll tell you, it's a sad day when the engineers want science more than the scientists do.

Oh, who am I kidding? Science, hell. I just want pretty pictures.

Opportunity Sol 423 (Spirit Sol 443)

The previous drive went splendidly, just splendidly. We're right where we wanted to be, a few meters off Viking Crater, with a fabulous view into the hole.

Thisol we're planning two drives, to execute over the weekend. On 423, the first drive takes us from Viking to its twin, Voyager. The rover will take some pictures and then, on sol 424, we'll start an autonav-only drive from Voyager toward Erebus.

Erebus is another 800m away or so. That's a distance that used to take months to cover, but at the rate we're now progressing, it'll take only about a week. The sol-424 drive should eat about 100m of that all by itself, and leave us set up Monday to do a drive-by of a small flat crater we can discern in the orbital imagery.

Since we got the downlink late yesterday afternoon, I spent a few hours working ahead last night, so that when I come in, I'm almost done. Of course, "almost done" in rover-driver terms means "I just need to spend a few more hours obsessing over it." Which is exactly what I do, of course.

Evidently I have a reputation for it. At the CAM, Jeff Favretto asks Cooper -- thisol's RP-2 -- if he wants to go over the sequence in detail. "Or, no," Jeff corrects himself, "you like to just skim through them, right?"

"Yeah," Cooper jokes, "-- unlike Scott, who goes over it and over it until you beg him to stop."

I guess it's not a bad reputation, as these things go. There are worse things you could be known for.

[Next post: sol 446 (Opportunity sol 426), April 5.]

---

Courtesy NASA/JPL-Caltech. Viking Crater, right where it's supposed to be.