2009-08-28

Spirit Sol 232

The longer we stay at a rock target, the easier exploring it becomes. In part this is because we can reuse successful commands from previous sols, leveraging our previous problem solving.

It's thanks in part to this effect that Ashitey is already done with the sequence by the time I come in. And it looks like it's going to be a short day. Kevin Talley, our TUL today, wants to get out of here to play with a new flight simulator he's hoping UPS will bring him this afternoon.

Such an easy sol. For the first time in a long time, we actually run ahead of schedule and finish an hour early.

So now we know: want to stay on schedule? Buy Kevin a new flight simulator. If only we'd known that back on sol 1.[1]

[Next post: sol 237, September 2.]




Footnotes:

[1] Kevin left for a while to work on Phoenix, and is now back on MER, training as a rover driver. I'd forgotten this particular insight until now, but will have to keep it in mind for some time when I'm on shift with him and need to hurry the day along.

2009-08-27

Spirit Sol 231

This morning is one of those mornings that makes me appreciate all the mornings that aren't like this morning.

But there's good news when I get to work. Yestersol's sequence worked out fine, despite the wacky joint moves we used. Yestersol's focus was on brushing Ebenezer with the RAT; thisol we're following that up by grinding. Leaving an eternal mark on another world -- it's a strange kind of immortality, but I have an inexhaustible appetite for it.

After the grind, we APXS the RAT hole. As a measure of how cold it's getting on Mars, we can APXS at mid-day -- since the APXS works best in the cold, we used to have to wait until late at night to start it. So the warmest part of our day is now about as cold as the coldest part of our day used to be. Poor Spirit. Hang in there, baby, summer's coming. Just a few more months ....

Of course, the colder temperatures, and reduced sun exposure, mean we have less energy to play with. So the RAT's tail gets amputated -- what starts as a three-hour grind (insert Gilligan's Island reference here) is cut to two and a half hours, then to two. Like any of the scientists, the RAT guys' faces fall a little more each time they're cut. I slide over to one of them, Phil Chu, and commiserate. "That's the same thing that used to happen to us all the time on drive sols -- it starts out as a four-hour drive and you end up with fifteen minutes." Phil laughs. "Ah, it's not a problem," he says. And he's probably right, at that -- Ebenezer's a soft rock; even with only two hours to run, the RAT will take a good chunk out of it.

I'll never have kids, but I'll make an indelible hole in a rock on Mars. Take that, posterity.

2009-08-26

Spirit Sol 230

"Let me tell you, this target sucks," Chris exclaims. Truly, this target is in a very odd spot. The flight software will happily place the APXS and MB on it, but it generates errors when you try to place the RAT or MI. Which is supposed to be impossible -- if you can get there with one tool, you can get there with all of them.[1] Yet we've found an exception to the rule. Still stranger, the software will happily place the MI all around the target (which we did the other sol) -- just not on the target itself. I'd look into that if I had the time.

Still, our explorations here continue to go well. The scientists have gained enough confidence in the instrument placements here that they've cut down the MIs from 5-stacks to 3-stacks. That is, instead of taking a series of five images at each position, we're taking just three. The more images you take, the more likely it is one will be in focus; thus, shorter stacks indicate more confidence (or less importance, or tighter downlink). We've gone as high as eleven. Insert Spinal Tap joke here.

There are limits to their confidence, though. Since the APXS doors recently didn't open fully when we tried to open them on a rock target, they've asked us to open the APXS on the CCT instead. The CCT, or Compositional Calibration Target, is a spot on the rover's own body, just above the space the arm stows into. As its name implies, it's used to help calibrate the MB and APXS -- we know what it's made of, so taking readings of it helps us interpret readings of unknown stuff. And since it's a hard and accessible surface, we can also open the APXS doors on it.

But doing so requires a more-than-usual amount of clearance under the rover. The HAZCAMs can't see that area, so Chris asks me to look back at imagery from previous sols to see if we have room. He tells me about some weird trick Frank uses for this, which basically amounts to an abuse of the IDD workspace display[2], but it doesn't seem to work for me and I quickly lose patience. Instead, I do what I usually do in this situation -- bring up the terrain mesh built from the old imagery, and slide the rover around until the simulated camera view looks like the real one. This isn't precise, but it's easy and quick -- and in a case like this one, where it turns out there's nothing of consequence under the rover that could possibly be a problem, it turns out to be good enough.

One problem solved.

Chris is solving another one. There are basically two ways to move the arm. One way is called a joint move: tell one or more of its five joints what position you want it in. The other way is called a Cartesian move: give it the three-dimensional (i.e., Cartesian) coordinates of a point in space and tell it to put the current tool there. A very useful variant on this second way can also be used to move the arm in so-called "tool frame," a coordinate system centered on the current tool. Z is "down" in this frame, so you can place the tool 10cm above the target, then tell the arm to move 10cm "down" to place the tool on the target. (Actually, we normally tell it to move 11cm "down" in this case, to account for uncertainty in the terrain mesh and the arm's own behavior, but never mind that.)

We mostly use Cartesian moves, and RSVP has more facilities for using Cartesian moves than joint moves, because the original plan was to use Cartesian moves all the time. But internally, the IDD flight software converts Cartesian moves into joint moves, and there are some cases it can't convert -- cases like the one that's biting us thisol. We routinely avoid this problem by using a joint move to get near the desired position, then a Cartesian move to get to exactly the right position. (We'd just use joint moves all the time, except that RSVP doesn't make that easy to do; also, Cartesian moves result in easier-to-read sequences.) Chris tries that trick, but it doesn't work, so he just gives up and does it the hard way, computing the needed joint angles and plugging them into RSVP.

Let me tell you, this target sucks.




Footnotes:

[1] Not true, and I'm not sure now why I ever thought that.

[2] A technique now called "Hartman Localization," a clever use of a couple of RSVP features. You start with an old terrain mesh covering an area the rover has since driven into; you know the rover is now somewhere in that mesh, but you don't know exactly where. Then you load in a second mesh, a current one, which is attached to the simulated rover. Then you can slide the simulated rover around until the two meshes match up, and that tells you where the rover actually is in terms of the old mesh. Having done that, you can examine, for instance, what's near the rover that you can't see from the current position (because, say, the rover's own structure now blocks its view).

2009-08-25

Spirit Sol 229

Amazingly, it worked. I don't know if we're better than I think we are or if we just dodged a bullet this time, but it worked. There's no word yet on whether the MIs were in focus -- we've just got the thumbnails, not the full images -- but there's no IDD fault.

Phew.

So we worked around the terrain mesh problem yestersol, but the problem is still there. Indeed, if anything, we've taken a step backward: Chris is going to have to redo his analysis of it, because his previous analysis was performed using a version of RSVP that incorrectly transformed site to rover coordinates[1]. So not only do we still have the problem, we know even less about it than we thought we did.

I've come in early to check on yestersol's success and also to meet with a couple of reporters from Aviation Week. Aviation Week is going to run a story on MER, and they want to include a wealth of 3-D images illustrating what we've found and how we run the mission. In particular, they're interested in RSVP.

Frank's the artist of the group, and as such he's the best person for them to meet with. But he's leaving on vacation and hasn't packed yet, so he asks me to take care of them. They're just going to need to understand what RSVP is, and what its capabilities are, he tells me. I can handle that.

Only it turns out they've already met with people and they know all that already. What they need is cool screen shots illustrating how we've used RSVP, particularly in the second half of the mission. Preferably artsy stuff.

I can't handle that.

Not that I don't try. What we settle on, after half an hour of fumbling around, is taking a series of screen shots that communicate a sense of what they're after, without actually being what they're after. We email these to Frank, who will make versions of them that actually look cool. When he gets back from vacation. I just hope I haven't wasted their time and put them off the whole idea.

Sigh. Some days I feel useless.

As they're leaving, they catch sight of what Chris is working on. He's looking up at the rover from in front and below, as it reaches out its arm to explore the world before it. "That's perfect!" one of the reporters exclaims. "Can we get a shot of that? Wow, the longer I hang around here, the more cool stuff I see." He should try working here.

Thisol, of course, we continue prodding Ebenezer. I miss out on a lot when I'm not working as RP-1 -- stuff like, why are we at this particular rock instead of another? In this case, I happen to overhear Steve Squyres explaining it to someone: Ebenezer is the unaltered rock they're comparing Clovis to. But Steve's way of putting it is more entertaining: "Ebenezer is whatever Clovis was before whatever happened to it happened to it."

Thisol's exploration turns out to be fairly simple, at least as compared to yestersol, and they don't change their minds about what they want at the last minute, which always helps. This leaves me with time to follow up with Jim Erickson -- who, in case I haven't mentioned this, is now the project manager (Richard Cook moved on). Yesterday he asked me if I had a minute, and I told him no -- I really didn't. "When you have the time," he said. I have the time today, so when I spot him I ask what he wanted.

"You need to come to my office," he says. We head out to the elevator, and he turns around and says, "It's a good thing." He punches the "up" button. While we wait for the elevator, he muses, "Nobody ever got fired 'when you have the time.'"

So we go up to his office and close the door and sit down. I'm getting an award -- it's a JPL thing called a SPOT award, which is basically a pat on the back plus a small amount of money (minus taxes). This one is for "outstanding performance as a Rover Planner ... and in the continuous improvement of the sequence generation and validation software ...."

It was initiated by Arthur Amador ("and I heartily endorsed it and sent it on," Jim adds). Andy Mishkin and Brian Cooper were the ones who hired me onto the project in the first place, and Arthur was the one who (albeit under pressure from Brian) selected me as a rover driver. He was taking a chance on me when he did it, so it's been particularly important to me that I not let him down.

Jim makes me a photocopy of the award form. "This must be one of the best parts of your job," I say to kill time as he navigates the photocopier's needlessly complex menus. "Yeah," he says. "As project manager, you're always having to tell people, 'Cut your budget, do it faster.' You rarely get to say 'Good job.'" He hands me the photocopy and the check. "Good job," he says.

The award's not much, in itself. But then, it doesn't need to be, especially when it comes at the right time.

Some days I feel useless. But not today.




Footnotes:

[1] It took me a while to wrap my head around the plethora of coordinate frames on MER. Some things as seemingly simple as locating a point in the world -- just asking a question like "where am I?" -- turn out to be more subtle and complex than I'd previously considered.

So. As we all remember from high-school geometry, you can designate any point with three numbers -- say, one representing how far north/south the point is, one representing how far east/west the point is, and one representing how far up/down the point is. But those numbers have to be measured relative to some starting point, and for different purposes you want to use a different starting point.

For most purposes, on MER, we designate points in the world -- for example, drive targets, or science targets of interest -- relative to a "site frame," which we periodically reset. Resetting it makes all your old targets useless because it restarts the numbering at zero in a new place, but this is a more useful thing to do than you might think: when you've driven far enough (usually tens or hundreds of meters) from the last place you reset the site frame, all the numbers for designating a new point become annoyingly large, and the points you might have designated before have been left behind anyway.

For some purposes, though, we use another coordinate system called "rover frame," which is centered on a certain point on the rover body. As the rover moves around in the world, it sort of carries that "rover frame" with it. So, for example, in rover frame, the numbers (1, 0, 0) might designate the point exactly one meter in front of the rover, wherever the rover happens to be.

Coordinate frames are also oriented differently: site frame is oriented so that it aims north, while rover frame is oriented so that it pokes out the front of the rover, no matter how the rover is aimed or tilted.

Now, you sometimes need to refer to the same point using two different coordinate frames -- for example, to satisfy two different pieces of software that have different needs -- and transforming between them takes a bit of matrix math. Here, the problem was that the points Chris was looking at were picked out in site frame (which is the usual way to do it) but what mattered in this case was how the IDD flight software referred to them, which was in rover frame. Chris was using RSVP to transform one set of numbers into the other, but at the time there was a bug in how RSVP did this, which spoiled his work.

Driving a rover on Mars can be complicated.

2009-08-24

Spirit Sol 228

I'm the only rover driver who hasn't worked on both rovers. Brian and Jeff did most of Spirit's first drive before moving to Opportunity. Eric did the initial IDD checkout sequences, which Frank edited slightly, before moving to Opportunity; and Eric came back for a sol or two. Chris has spent several sols on Opportunity, and John is racking up a few as well. Ashitey, too. Bob never worked on Opportunity as far as I can recall, but he's off the project now.

Which leaves me. But not for long: I have a shift scheduled on Opportunity tomorrow. Technically, I'm a floater, which means in practice that I'm working on both, but it counts.

The only bad news is, I don't know anything about what they're up to or when they expect me to come in. So I go upstairs to the Opportunity room to find out.

"When do you need me tomorrow?" I ask Emily.

"Tomorrow's a restricted sol[1]," she replies. "We don't need a rover planner."

Well, that's just ducky. I guess I won't be working on Opportunity tomorrow, after all. And on the current schedule, I don't get another chance at it for a while.

And Opportunity has cool stuff going on right now, too, as Chris Salvo shows me. They found this weird mudlike stuff -- it looked like rock, but driving over it nearly destroyed much of it -- which they're going to drive back to and examine in detail with the IDD. Nobody knows what this is. One thing is certain, though: I won't be helping them find out.

Ah, well, Spirit's my first love, anyway. While I was away, they finished up at the previous location (where the rover slipped almost double the predicted amount during my last drive -- turns out it's a good thing I shortened it) and then performed a short drive to a new rock called "Ebenezer." The targets on Ebenezer are named for characters from "A Christmas Carol" -- TinyTim, Cratchit, Scrooge, Marley.

We expect to stay at Ebenezer for a while. The scientists like this rock and our power situation is excellent, so we might stay here through solar conjunction[2]. In the meantime, we'll be performing the usual extended exploration campaign -- APXS, MI, MB, RAT-brush, APXS, MI, MB, RAT-grind, APXS, MI, MB, and then maybe do it all over with different targets.

But not all in one sol, of course. Today we're just getting started, unstowing and beginning to poke at the surface. John has the work largely done by the time I get in, so it should be a cakewalk.

But, reminiscent of Ebenezer Scrooge himself, we're haunted by the Ghost of Sols Past. Yestersol they did a touch-and-go, and we discovered in thisol's downlink that the instrument placements very nearly didn't work. The APXS placement, in fact, wouldn't have opened the APXS doors (a requirement for good data) had John not happened to sequence an unusual preload move at the end of the instrument placement.

This looks like the same problem we've seen several times before, where the surface isn't where the images and terrain mesh seem to say it is. John thinks it's nothing to worry about, a combination of the fact that yestersol's images were taken in shadow (meaning they had low contrast), were heavily compressed, and were of a uniform surface. This combination of factors makes it hard for the stereo correlation to find the same features in the left-eye and right-eye images, and therefore it has a harder time figuring out how far away the terrain is. Thisol we're working with less heavily compressed images of a less uniform surface, so we shouldn't have the problem again.

But Julie's worried about it. She wants the rover to trust its sense of touch more than its sense of sight, by telling it to reach out and touch the surface so it knows where it is. "What would it take to do a MB touch before each instrument placement?" she asks.

John shrugs. "We can do that," he says. "We'll have to redo all the sequencing to use relative moves instead."

"How long will that take?"

He shrugs again. "A couple of hours," he says.

This is designed to make Julie give up on the idea. We're already an hour behind schedule, and nobody likes to slip even later. But she doesn't take the bait. "Okay," she says, "go ahead."

John looks annoyed, but it's not his problem anyway -- it's mine. I pick up the sequence and get to work.

As predicted, the changes take a couple of hours to implement. Then it gets worse. At the walkthrough, they ask for even more changes. By the time all is said and done, we're almost four hours late.

And I'm not really sure we gained anything. I've made a lot of changes under time pressure, which is always risky. If we'd been smart, we'd have weighed that risk against the risk of just leaving the original version of the sequence alone. Instead, we weighed risk against time, which was the wrong tradeoff. By now, we ought to be smarter about this stuff.

As I'm leaving, I can't help but think of that bit from The Princess Bride:


Valerie Max: Think it'll work?
Miracle Max: It would take a miracle!


Footnotes:

[1] Because of the difference in the two planets' day lengths, the Martian day appears to rotate through our planning day -- Martian noon is 40 minutes later each Earth day, more or less. When the phasing reaches a point where the sol's downlink (whose timing depends on the Martian-day time seen by a particular rover) will arrive so late in the Earth day that we'll no longer have time to plan the next sol's activities for the rovers, we plan differently: we plan two sols' worth of activities for that rover, every other day.

Planning ahead this way restricts what we can safely do with the rover, though -- we can't plan anything for the second sol that might be unsafe if the first sol goes awry, for instance, since we have to plan them both at the same time -- so these are known as "restricted sols."

Incidentally, because the two rovers are at almost exactly opposite longitudes on Mars, their Mars times are about 12 hours apart, so at most one rover is ever in restricted sols at any particular time.

[2] When the sun will be between us and Mars, making it impossible for us to reliably communicate with the rovers for about two weeks.

2009-08-19

Spirit Sol 223

So I'm RP-1, bright and shiny at 8 AM. And Art's here, so it's almost like the old days. "How ya doing, Art?" I ask, and he groans, "So good ... if it were any better, I don't know if I'd survive it." So I'm not the only non-morning person around here.

"Are we still driving today?"

"Today we're driving" -- he pauses dramatically -- "maybe one wheel rotation. Hey, don't knock it -- they could take that away from us."

Since I'm here at the right time, as opposed to the time the schedule said, I'm actually present for the SOWG meeting. That's been a long time -- so long, I have to ask Art if they're still having them in the same location. They are, so I head upstairs for that.

This SOWG is easy, since our sol is so easy. They're finished RATting this spot right in front of us and want to RAT another spot near the right front wheel. So all we're doing is turning 30 degrees clockwise and backing up 25cm. There's a little concern that we'll drive over the location we were just RATting before we have a chance to image it, but I model it in RSVP and demonstrate to their satisfaction that we won't be coming anywhere near it.

Of course, nothing is ever that easy, and today's no exception. Since we're on a steep slope, we expect to slip during the drive. The question is, how much? Fortunately, I'm getting better at delegating, so I just ask Rich Petras -- Mobility/IDD support thisol -- to figure it out for me. Between the turn and the drive, we expect to slip about 10cm downhill. RSVP shows that the RAT can still hit a target 10cm further upslope, meaning that the new RAT spot will be reachable even if we slip. I end up cutting the drive slightly, though, from 25cm to 20cm, just to split the difference.

We're holding a press conference today. On the agenda: what Opportunity's been up to, why the scientists care about Spirit again, and news about Opportunity's RAT. Art turns on the TV and starts flipping to the NASA channel, but gets stuck on the Olympics. I'm busy polishing the drive, but the grunts of sympathetic pain behind me keep me amused.

The press conference itself was intelligently done. Opportunity has a problem with its RAT -- it looks like there's a chunk of rock caught in the grinding wheel, which keeps the wheel from rotating. This is a known vulnerability in the design; they can issue low-level motor commands to run the wheel backward, and the obstructing rock should pop right back out.

But if they led the press conference with that, the headlines would read, "MARS MISSION DOOMED: NASA Embarrassed by Failed Science Instrument." And it's all the press would ask about at the end, because they'd get their heads filled up with questions about the RAT and stop paying attention. So they don't do that. They leave the RAT hiccup to the end, and lead with the positive stuff instead. The positive stuff includes Steve Squyres saying Spirit has found the best bedrock yet found at either site. It's exciting because we have altered rock adjacent to unaltered rock -- this gives you a chance to figure out what changed the rock. They also have an astonishing three-frame animation of water-ice clouds moving across the pink Martian sky -- just faint wisps of stuff, but unmistakable. And they have a gorgeous color PANCAM mosaic from Spirit -- the pictures we've been taking the last few sols so I could have an unexpected holiday.

Then they bring up Opportunity's RAT, and sure enough, the press appears to put the matter in its proper perspective. Check that out.

Even though we've been at the hill a long time, we haven't ascended very far -- we're just 9m above "sea level" now. At one point, we were planning to take a path through the hills that would have taken us only about 35m above "sea level," but Larry Soderblom says the plan now is to go all the way to the top of the hill -- 80m or 90m up. Those pictures are going to rock.

Speaking of exciting pictures in our future, Dan Maas is working on another MER video. This one is going to be an IMAX movie, and it's actually been in the works for months. Dan wants the rover to articulate realistically as it drives along -- and who should have software that realistically articulates the rover as it drives along, but yours truly. That, of course, is one of the things RSVP was built for. RSVP can also produce an XML file describing the rover's motion in detail, which is exactly the kind of thing Dan would need. So Justin Wick, an SAP developer who's working with Dan now, asks me to show him how to use RSVP. Which I do, and it's possible -- unlikely, and not a big deal anyway, but possible -- that I'll end up with some kind of credit at the end of the film. "Additional technical assistance provided by ...", or something like that. Heck, maybe they'll make me an associate producer!

Well, let's not get carried away. "Associate Producer" might be an impressive title, but I'll take "Mars Rover Driver" any day.

[Next post: sol 228, August 24.]





Courtesy NASA/JPL-Caltech. A beautiful color PANCAM image showing the RAT work we've been doing on Clovis while sitting here. The full-sized version is even more impressive.

2009-08-17

Spirit Sol 222

I've been scheduled for the last few sols, but there's been no rover motion. We're taking advantage of our good power situation and our excellent location to take a bunch of images and charge the battery. So I keep coming in and being told, "Nothing interesting today; you can go home." And going home.

I came in today mainly to drop off my copy of the Linux Journal that has Frank's and my RSVP article in it. We're duly congratulated.

I'm on as RP-1 tomorrow, and I think we'll actually be doing stuff. But it looks like tomorrow will be a simple sol. All we're doing is a very short drive, not much more than a turn in place, and Ashitey already has a preliminary version ready. After I go over this with him, there's not much more to do but go home. So I do that.





Our issue of the Linux Journal. Not too surprisingly, we made the cover.

2009-08-11

Spirit Sol 216

I'm RP-1 again. I come in at 8:30, expecting the SOWG meeting at 9:00. But 9:00 rolls around, and no SOWG meeting. In fact, from listening to people talking, it sounds like they had it already.

Turns out the schedule was erroneous. The schedule said the SOWG meeting would be at 9:00, but in fact it was at 8:00, and I'm apparently the only one who read the schedule. Everyone else goes to the SOWG meeting every day, so they know when it's happening, and don't look at the schedule any more.

So I missed it completely. A great way to start your day.

Well, the other news is good. They got usable data from the APXS the other sol after all, Arthur tells me -- the doors were partly open, though they didn't latch. We're going to hold off on uplinking thisol's sequences until we get confirmation that the doors closed, but we're all pretty confident they will.

Thisol we're grinding this rock -- always a fun thing to do. And Ashitey's my RP-2. Thisol goes a lot better than the last time he and I worked together (two sols back), and I like to think it's not only because I'm the guy at the keyboard this time. (It was true in first grade, and it's true today: I need to learn to play better with others.)

The only blight on the day is that we have to cut the RAT grind. The day is constrained on one end by the fact that we can't start IDD work until after noon (it's getting too cold on Mars as Gusev approaches winter) and on the other end by a comm pass. In between, there's just not enough time to do everything the scientists have asked for. We end up cutting some stuff out, and Ashitey and I find clever ways to squeeze down the timing of the other stuff, and in the end they have to cut only 30 minutes out of the grind. It's still a shame, but since 30 minutes translates to only about half a millimeter of depth, the RAT guys don't look all that upset.

And as I'm always saying, if science is happy, I'm happy. And I'm happy.

[Next post: sol 222, August 17.]

2009-08-10

Spirit Sol 215

This is my first RP-1 sol since I-don't-know-when, and moreover I'm doing it solo -- the scheduling snafu left us no RP-2, though Chris is here as a "floater."

I'm so happy.

It's still shaping up to be a simple sol, a MB-to-APXS tool change. Since we're in this weird out-of-phase part of the daily schedule, they haven't even uplinked yesterday's sequences yet, which means I have time to fix something if it's wrong. But I check them out, and to my surprise I can't find anything wrong with them. There's stuff I'd change in retrospect, but nothing really wrong. So I leave well enough alone.

Anyway, I've got thisol to worry about. It might be simple, but I don't like to procrastinate, so I go ahead and whip out the sequence.

Then we get the sol-213 downlink. Just as no plan ever survives first contact with the enemy, so thisol's plan will not survive first contact with the downlink. The joint angles from sol 213 were very close to the predicted values -- that is, the IDD went just about exactly where we asked it to go -- but the APXS contact switches report no contact and the APXS dust door reports that it's still closed.

Which is mildly bad. The APXS dust doors have to open in order for the scientists to get good data from the instrument. If they're not open, we'll have to redo this observation. We can't redo it on this exact spot, since we're going to RAT it on 214 -- unless we try to modify the sequence before uplinking it a few hours from now -- but that will be OK, since we'll get effectively identical results by just APXSing other parts of the outcrop. We can do that without even driving. And anyway, the doors might be open even though the rover says they're not -- on Opportunity, the switches are flaky, and maybe Spirit's switches are starting to get flaky, too. If so, the APXS observation will be just fine; we'll know when we get the results back later tonight. So it's not the end of the world.

But it does leave us with a puzzle: why did the IDD stop moving? Normally, the IDD will stop moving for one of three reasons: it reaches the commanded position, one of the instrument switches reports contact, or the joints draw more current than allowed (suggesting an unexpected contact). It came close to the commanded position, but didn't quite make it there, so it didn't stop for that reason. The switches report no contact, and the joint currents were well under the limit.

So it shouldn't have stopped. But it did.

In a sense, it doesn't matter. The IDD ended up in the right spot -- more or less -- didn't it? But it's a mystery, and engineers can no more ignore a mystery than they can stop breathing. So Rich Petras and I start trying to figure it out, and we end up sucking in Jessica Brooks and Jeff Favretto and Jeng Yen and some other people. Maybe the APXS contact switches were disabled? No, they're enabled, and anyway we'd see evidence of the switches tripping even if they were disabled, and we don't. Maybe the switches tripped long enough to stop the IDD but then "untripped"? No, we'd still see the temporary trip in the downlink, and we don't. Maybe we don't have all the data from the pass? No, then the data product would be marked as partial, and it isn't.

Oh, hell. We give up and call Bob Bonitz. For once, he doesn't know either, but he does move us forward a step. It looks like we did some calculations incorrectly, and the IDD might think it did reach the commanded position after all.

That's good enough for me. Well, no it isn't, but I have to get back to other work, so it will have to do for now. The other work I have to get back to is related to the fact that the APXS doors aren't known to be open. To avoid contaminating the APXS, we're not supposed to RAT with the APXS doors open. The sol-214 IDD sequence closes the APXS doors before doing the RAT brush, but sometimes the doors open only partially, and then the door-close move doesn't work (you have to open the doors fully before you can close them). So we might be about to RAT with the APXS open. Two hours to uplink ....

The science team decides to take the risk of the doors being open during the RAT brush, so we won't need to change 214 after all. But if the doors are open during the 216 RAT grind, that would be bad. So thisol -- 215 -- we'll have to open the doors fully, so that they'll close when we tell them to. This involves a maneuver that opens the doors on the CCT, a calibration target mounted on the rover's underside.

"Is that going to be a problem?" asks a worried-looking John Grant.

"No," I say cheerfully.

John relaxes instantly. "I love it when he says that," he says.

Changing the sequence takes only minutes. Then I realize that since we're now bringing the IDD up to the rover body, we can take a picture and see whether the doors are open. I throw that in too, for good measure.

And I'm pretty much done for the sol. I start both RP workstations rendering movies for the uplink report and kick back a while. Frank swings by, trailing a huge tour group consisting mainly of Japanese graphic artists and animators -- all SIGGRAPH contacts -- and I tell them a little about what's going on. They're all very polite and thank me profusely, and I kick myself for not remembering how to say "You're welcome" in Japanese. ("Dou-itashi-mashite," dumbass.)

Anticlimactically, I don't get to see the day quite through to the end. Just when the CAM starts, I have to go upstairs for a Change Control Board meeting -- management approval to deliver a slightly updated version of RSVP. Since I can't be in both places at once, Chris helpfully comes downstairs to handle the CAM.

Darn, and I really wanted a challenge.

2009-08-09

Spirit Sol 214

This is new: Ashitey's RP-1 thisol.

This is also new: there's no RP-1 scheduled for nextersol. But that's a scheduling snafu, one I fix by volunteering to do it myself. (Oh, twist my arm.)

I'll need to be a little late, but Arthur says that won't be a problem. "Tomorrow will just be a simple tool change," he reassures me -- little realizing, I think, that he's disappointing me instead. Hey, if I'm going to be RP-1 again for the first time in what seems like months, I want a damn challenge. But I'll take what I can get.

Thisol is the kind of complicated sol I'm lusting for. We're RATting a spot that we'll then investigate with the MB. But there's another issue: the sol-212 MI images we took appear to be out of focus, so we want to redo them thisol. Now, stay with me. The lowest -- that is, closest-to-rock -- MI images appeared to be most nearly in focus. And Ashitey's plan for thisol is to redo the stacks[1] using a technique that will start farther away from the rock. Which will make their focus worse, not better. Which is a complete waste of time, energy, and downlink data volume. I can't seem to communicate this concept to Ashitey to save my life, so it goes up like that.[2] Oy.

But Ashitey and I have worse problems. Thisol's IDD work requires that we move the IDD away from its current position a few times. But because sol 213's sequences haven't executed yet, we don't know exactly where the IDD contacted, so we have to pull some tricks to command the IDD back to its current position during thisol's business. The IDD automatically remembers a single location, so the only way to do this is to use a special command that tells the IDD to put a different tool on the last saved location. And that command doesn't seem to be working, at least not in the simulation. Neither Ashitey nor I can understand why not. It doesn't make sense: we're putting the IDD in the configuration it would have after the change-tool command would execute, then issuing the change-tool command, and the simulation says the command fails. We can't figure out why we would get an error at all, and the message itself isn't very useful.

This is where certain personality differences between me and Ashitey start to be a problem. We like each other a lot, but, well, we're very different. Most relevant, Ashitey's cool with being late, and I'm just not. Ashitey is the epitome of Type B, and I am Type A. No, make that Type A+. Type A+++. You get the picture.

We're not ready for the activity plan approval meeting. "Got an animation for me?" Saina asks. "Just a few minutes," Ashitey reassures her, but the minutes pass and we've made no progress. Saina and the scientists are clearly growing more and more concerned as Ashitey keeps putting them off, "just a few minutes" at a time. An hour passes this way, and I am starting to go insane. We're slipping later and later, and I'm not in control of the situation: Ashitey's sitting in front of the keyboard, not I, and he naturally wants to argue about every idea I have for fixing the situation (not that his -- or, for that matter, my -- ideas are working anyway).

I heroically summon every ounce of self-control I possess, and smile grandly at Ashitey. By itself, that act calms me down a lot.

But not for long. (Long enough to sign a small poster for Arthur C. Clarke, but not much longer.) Another hour goes by, and we're as stuck as we were to start with. At this point I decide to believe that my problem is that I'm not in front of the keyboard, so under the pretense of checking something, I log into the other RP workstation and start working out my ideas directly, rather than arguing over them with Ashitey one at a time. They're still not panning out, but at least I can explore them faster; and I'm driving, which makes me feel better for no good reason.

What we need is a fresh pair of eyes, and luckily we get one. John was working on Opportunity today, and when he moseys downstairs to say howdy I rope him into helping out. He and I look at it a while, and eventually we realize that we're helping the IDD too much. Remember that we're moving the IDD away and then back to where it started, then using the tool-change command to fix up its position more precisely. When we moved the IDD back in, we put it in approximately the configuration it would put itself in after the tool change, thinking that would simplify things for it, but the software can't handle that; it wants to compute its own path to the post-tool-change position, and gets upset if we do the work for it. When we make things "harder" by using the pre-change configuration instead (helped along by another trick with the amusing name of "the guarded move to nowhere"), it works like a charm.

We're three and a half hours late, I've probably destroyed my relationship with Ashitey, the new MIs are going to be in worse focus than the old ones, and hell if I know whether this hacked-up sequence is even going to work.

Did I say I like a challenge?

Can I just have a simple tool change?

Please?




Footnotes:

[1] When you look at something under a microscope on Earth, you look through the eyepiece and twiddle the focus knob until the subject is focused. We can't do that on the rovers because light-time delays would make the knob-twiddling take forever. Instead, our sequence takes a whole "stack" of images -- some above, one at, and some below where we think the right focus distance is. We're pretty good at nailing the best-focus position, but when we miss it, the images taken above and below that position are our record of what happened when we twiddled the focus knob; one of the images is almost always well-focused.

[2] A serious mistake. Yes, we were busy and stressed and running late. There was still no excuse for uplinking a sequence that I really didn't think would correctly implement the science observation. Since Ashitey and I clearly weren't communicating well here, a smarter move would have been to politely involve else in the conversation, someone who might have been better at explaining things to Ashitey (or to me, if I actually was in the wrong).

2009-08-08

Spirit Sol 213

But of course I didn't go. Too much moving to do.

"You should have gone," says Chris. He's nursing a hangover, and he's not the only one. Sounds like it was a wild time. Well, having fun isn't really my thing anyway.

This is, of course, all related to the sad fact that the science teams are, for the most part, leaving us. At least they're going out with a bang: Spirit's findings at Gusev are the focus of a special issue of Science magazine, one of the two preeminent science magazines in the world.

So that's cool.

JPL has access to Science online, so Art's flipping through the images. One of the scientists asks what he's doing, and Art says, "Reliving past glory."

Heck with that, I think; we've got plenty of future glory to achieve. Thisol's little chunk of glory will be to finish up yestersol's MB integration on the outcrop, then calibrate the MB and APXS before starting an APXS integration on the rock. It's not terribly complicated, modulo some confusion about how many times we're supposed to calibrate the APXS and whether we can calibrate the two instruments in parallel, and the day goes relatively smoothly.

So smoothly, indeed, that I have some time to entertain a visitor brought in by Justin Wick, one of the SAP developers. The visitor's name is Rupert Scammell; he's an Aussie who ported SAP to Irix -- he's not a JPLer, so this was just for the fun of it, as far as I can tell -- and who is, obviously, a huge fan of the mission. Justin wants me to show Rupert RSVP, so I do, and I talk to him about being a rover driver, and he gives the impression of being suitably impressed.

It's a supremely geeky experience, of course, and at some point I start to feel self-conscious about this. Then I remember that if I weren't a geek, I wouldn't be here. And I shrug it off. It's cool.

2009-08-07

Spirit Sol 212

The switchback drive worked -- we've reached the outcrop, and the scientists are awfully excited about it. John and I get to be the first to IDD it.

This still amazes me. This rock has been there, right in this spot, for four billion years, almost since Mars was born. We're the first people ever to even see it, much less explore it.

We're starting out simply enough. Indeed, thisol is so simple that when I walk in, John tells me, "The sequence is done -- you can go home."

Of course, life on Mars is never quite that simple. Now that we've made it up onto the outcrop, we're oriented a little more toward the sun than we have been in recent sols, so we have a lot more energy to play with. So much energy, indeed, that the batteries will be fully charged and shunting -- "spilling over," thus wasting energy, unless we find a way to use up more. So the science team asks if I can add in another stack of MI images near the existing stack, and I'm only too happy to oblige. (It's always nice to have something moderately creative to do.)

The location they chose for the first MI stack is called "Cochiti," an allusion to the Cochiti pueblo in New Mexico. I Google for a map of the area, find a nearby pueblo named "Jemez," and name the second stack's position that. I know it's not official, but it's damn cool to name things on another planet.

The visiting scientists are leaving us. It's their last weekend here, and some of the younger ones are throwing a wild party at their apartments to say goodbye.

I'm not planning to go, but then one of the scientists, Nicole Spanovich, asks me if I'm going.

"Nah, I'm way too busy moving into my new house."

She looks almost hurt. "You should go," she admonishes me.

"Okay."

I'm so easy.





Courtesy NASA/JPL-Caltech. Persistence pays off. At last, we're ready to start unlocking the secrets of this ancient stone.

2009-08-02

Spirit Sol 207

The news starts off good, at least. No errors on the drive, and we turned as expected for comm.

But where the hell are we? It looks like we have a little bit of outcrop under our wheels, and we'll just need another drive to back up a little. But first we need to know where we are.

What happens next is something we call "localization." You look at pictures from the previous sol -- or earlier, if you have them -- and try to figure out where you are relative to that picture, by comparing it to the new picture. Generally, this is a game of hunting rocks, trying to find the same set of rocks in each image to use as landmarks. "Let's see, if this guy here in this image is this guy over here in that image, then there would have to be a pointy rock over there ... but there isn't ... so maybe that guy there is the same as that guy over there ...." It helps if you know roughly where you should be looking, which we do. This is laborious, but can be fun in its way.

This time it's not working out. There's a clump of rocks right in front of us that should be obvious from the previous images, but hell if we can find them. We stare at the images for at least fifteen minutes, with plenty of false starts but no luck.

More images come in, and one of the scientists takes a look at one and says, "You know, I don't think we moved."

Ah-hah. The reason we weren't able to localize the rover was that we were trying to find a solution near where we were supposed to end up. Instead we look at the area near where we started -- and sure enough, there's the rock clump. We might have gained a whole meter.

As more images and engineering data arrive, they dispel the mystery. Looks like the rover experienced about 66% slip through the first part of the drive (already more than we'd planned), then the slope became so steep that it just couldn't climb any further. The more it tried to climb, the more it slipped. In one case, visodom showed the rover experiencing 125% slip -- the rover tried to step 60cm upslope, and ended up 15cm downslope.

In the process -- and perhaps exacerbating the problem -- the left rear (goalward) wheel lost contact with the surface. When this happens, the middle wheel continues to drive goalward, causing the bogie pair to tilt the rear wheel further from the surface. The slope was already too steep, but having only five wheels to climb with sure didn't help.

"Ray, I'm sorry," says Squyres, "but we're going to have to take the rover away from you."

Not that it's Ray's fault, of course. I had the foresight to warn them yestersol that something like this might happen, but I don't feel better to have been right.

Squyres and Ray don't want us to drive nextersol. Instead they ask us to take some time to come up with a plan. John and I talk about it a while, and work something out. We clearly can't fight our way directly uphill. Instead, we can drive switchback -- find a series of gentler slopes leading uphill, mostly transverse to the goal. After a few minutes of searching, we're able to find a path.

We'd have time to implement and deliver that drive thisol, but by then the planning has already proceeded with the assumption that we won't be driving, and it's too late to change it. And anyway, there's a possible problem with the PMA, which means we can't use it thisol, which means we can't do visodom thisol, and we definitely don't want to do this drive without visodom. So instead we just get a jump on the next sol, building the sequences now; we'll execute them a sol later.

We show the switchback plan to Ray, and he nods approval. He doesn't mind that it might take us two or three sols to get back uphill to Clovis. "It's going to be worth it," he says. "This outcrop is gonna tell all." Squyres emphatically agrees: "We drove three kilometers to get to this outcrop."

[Next post: sol 212, August 7.]

2009-08-01

Spirit Sol 206

Oh, damn it. I'm there when the downlink starts, and one of the first things we see is an indication of another goal error. So at least the rover thinks we didn't make it.

Scott Doudrick is sitting to my right, monitoring the incoming data stream. "Bus voltage is 26.9," he says. "It didn't go very far."

But then we get the rover's reported coordinates, and they're only about a meter away from the target. Maybe we're not in such bad shape after all. The rover apparently entered a limit cycle, which means it was unable to find the goal and danced around it. Eventually, it was stopped by the time-of-day limit.

What the hell happened?

The mid-drive and post-drive images clear up the matter. Just before it took the last step -- what should have been the final 2m of driving -- the rover was perfectly positioned, with the outcrop dead ahead. (Well, dead behind -- we were driving backward. But right where it should have been, anyway) The next image shows the rover two or three meters downslope of the target, with a lot of back-and-forth tracks visible in part of the image.

It's a bug. That final step of the drive told the rover to go to a Cartesian position 2m behind it, but the rover was unable to find that position because steep slopes confuse it. So it roverdanced, trying to find the right spot. This wouldn't have taken it so far from the target by itself, but the rover kept slipping downhill as it went -- most of the resulting position error is from slip. (Or at least this is my explanation. John has a different one. Pointing to a nearby copy of the _Weekly World News_ that shows a family of Martian Eskimos, he deadpans, "I think the Eskimos did it.")

In retrospect, we could have just told it to back up 2m, rather than the more complex command we used. But if the rover hadn't started from the right spot, the more complex command might have worked better. It was a gamble, and we lost.

So now we're about 3m downslope of our target, and have to spend another sol climbing uphill.

Which makes it twice as much of a shame that the drive there is turning into a huge pain, but at least Squyres -- who's as eager as anyone to get there -- has a good attitude about it. "We're all of nine meters above sea level," he notes, "but in the annals of interplanetary robotic mountaineering, this is an unprecedented achievement."

So we'll keep trying.





Courtesy NASA/JPL-Caltech. The Clovis outcrop, perfectly positioned smack behind us -- right before Spirit becomes confused and roverdances downhill.