2009-06-30

Spirit Sol 175

The conversation going on behind me piques my curiosity. "I hear visodom-related words," I say to Chris.

"Yeah," Chris replies, "we're gonna do it on our next drive on Spirit."

I cock an eyebrow at him. "For real, or are we just teasing Mark again?"

He laughs. "For real," he says. Mark will be pleased. I hope.

I continue checking my email. The press office sent me email asking me to do an interview with one of the reporters who's here covering Cassini -- he also wants to write an article about the challenges of driving the rovers on slopes. I'm willing, but since I haven't done this yet, I'm not the best person for them to talk to.

But the Opportunity drivers have all kinds of experience on slopes, stemming from the months they spent in Eagle Crater. So I head upstairs to see if Frank's interested in it.

As it turns out, he's already been offered a chance at this one, and declined for lack of time. We decide to throw it to Brian and Jeff instead. We actually end up talking mostly about houses -- he and his wife, Leah, spent a year and a half searching for their house. And that was their second round of looking; they had given up for a couple of years, after becoming discouraged by an unsuccessful earlier round of house-hunting.

We eventually get distracted from that semi-depressing discussion by rover power curves. Art's got a graph plotting the available solar power against time, for different assumptions about atmospheric dust settling. Even under the pessimistic assumptions, we still have enough energy each sol (200 Watt-hours) to do useful work at sol 720 -- that's two years.

Unless something else breaks first, of course. Spirit will become significantly less mobile when (if) the right front wheel seizes up, but at least we'll still be able to drive it. Other than that, there aren't any looming problems.

Oddly, Opportunity seems to be developing similar trouble with one of its wheels. There's a joking suggestion that this is simply sympathetic pain, like husbands whose backs and feet start to hurt when their wives are pregnant. It might turn out to be nothing, or so we all hope.

Today I earn my keep as an RP-2. One of thisol's goals is to get some microscopic images of the side of the rock we're currently exploring. Chris chooses a target and sequences the MI stack, then hands off to me and leaves. Later I notice that, because of the angle at which we're having the IDD approach the rock's side, the APXS gets awfully close to the soil. We'd probably be OK, but to be on the safe side, I choose a shallower angle, one that still lets us see the rock face they're interested in without endangering the IDD.

I miss being RP-1.

But I guess a side benefit of being an RP-2 is that I'm here when the NASA Code T folks stop by. These are the NASA HQ people working on implementing the President's return-to-Moon/go-to-Mars thing. They're interested in our tools and processes and stuff, so as a way of demonstrating our current capabilities, I show them an animation of thisol's IDD sequence. When they ask if we have any ideas for them about the manned Mars mission, I don't even have to think about it: "Send me!"

2009-06-25

Spirit Sol 170

We're back in the phase where we start sequencing before we have the downlink -- that is, we're planning the next sol before we've seen the results of this one. We're still in that state when I arrive, and the problem is exacerbated due to some kind of downlink delay.

We might have something interesting to look at when the data comes in. Based on some soil analysis they've been doing, Larry Soderblom says that we might have actually pushed hard enough on the rock to push it into the surface. (Apparently, this started as a joke some scientists were playing on Ray, and then they realized it might actually be true.)

It's been a long time since I've been around for the downlink. I'd almost forgotten that it's fun and exciting. There are usually several people looking at different data sources, each of which provides different hints about what's going on, and part of the fun is competing to be the first person who can authoritatively say what happened.

This time's a classic case. If all went well, we should see signs that we successfully RATted, and the APXS should be planted in the RAT hole. "The active tool is the APXS," Rich Petras points out. "No red alarms. We're good." But this doesn't necessarily mean that the RAT succeeded -- most common kinds of IDD failures automatically preclude further IDD motion, but not the RAT; even if the RAT fails to drill a hole, the IDD will usually proceed with its work.

Leo Bister, watching the EVR stream (a running list of status messages from the spacecraft), asks us about a message that indicates a failed RAT placement. "Are we in trouble?" Maybe, maybe not. Bob Bonitz, who's hanging around for the fun of it, says that we get those messages sometimes even on success. They're generated as the RAT tries to find the surface it's going to drill into, and can just mean that it hasn't found anything quite yet, not that it's given up.

We go through a couple more cycles of hope and dejection before the full, complex picture emerges. Bob, reading carefully through the full list of RAT EVRs, works out that the RAT did make contact and start to grind, but not for long -- maybe for only 20 minutes or so, out of a planned two hours. Then the RAT lost contact and gave up.

The reason for this becomes clear when we get the images back. Larry was right: we pushed the rock into the surface. Because the RAT was placed slightly off center, the rock actually rotated: the left side of the rock got pushed in, and the right side came up. (We attacked, and the rock turned tenkan![1]) The right-hand side brought some crusty soil up with it, like mud clinging to a shoe. There's no visible damage from the RAT -- we don't seem to have so much as scratched it.

The rock has foiled us again, or at least fought us to a draw. "'Pot of Gold?' Chris remarks sardonically. "More like 'Pot of Crap.'" He later has to pick a target on the rock for further IDD work, and he names it "Fool's Gold."

Scott Doudrick is similarly sour. He asks if we can use the IDD to flip Pot of Gold the bird as we drive by -- "the L.A. driving experience," he says. "For the true L.A. driving experience, we'd have to shoot at it," I remark.

Later, it turns out that we hurt the rock more than we thought. Larry Soderblom goes off and takes a more careful look at pre- and post-RAT images, and discovers that there is some visible change. Some of the irregularly shaped "fingers" broke off, and the RAT's outer ring truncated a couple of high points on the rock, leaving behind shiny clean surfaces, when we pushed the RAT down hard. (We always do this when we're about to grind, so that the RAT will stay firmly connected to the rock surface. Curiously, the sequence of images we took during the RATting shows that it wasn't this initial push that forced the rock into the surface; that happened later, maybe as a result of the grind-induced vibration.)

Squyres reports that we've done some other interesting damage as well. They've just discovered that the rocks we recently drove across are shiny (in the PANCAM's blue filter) where the cleats hit them -- so shiny, they saturate the image.

"What does that mean?" Doudrick asks him.

Squyres grins, deliriously happy. "I have no idea," he says.

Steve's quiet for a minute, staring at the image and shaking his head slowly. "Every day, it's something new and bizarre." He pauses again. When he speaks again, he voices a thought similar to one I've often had myself. "I feel like Mars is giving us our final exam," he says, "and we're not doing very well."

"It's been a long semester," shrugs Leo.

Despite our having had some limited success thisol -- for once -- the team continues to be jokingly superstitious about the nefarious power of Pot of Gold. Julie's trying to print some stuff out and go home, but she keeps being thwarted in the effort -- her (Windows) laptop is glitchy, and then when she turns on the printer (which wasn't supposed to be turned off), there are days' worth of other people's print jobs queued up, which all has to print out before she can print her stuff.

"It's this damned rock!" says Leo. Nobody disagrees.

[Next post: sol 175, June 30.]





Courtesy NASA/JPL-Caltech. The RAT shoved Pot of Gold into the soil. Take that, you damned rock!




Footnotes:

[1] An aikido joke. You are not expected to understand this. :-) But, if you're curious: tenkan.

2009-06-24

Spirit Sol 169

Driving Spirit is about to get harder. Up until now, we've been spoiled: we've been driving a vehicle that does exactly what you tell it, in the exact kind of terrain it was designed for. That's about to change. Not only is the right front wheel in danger of seizing up, which will make Spirit much less maneuverable and much less predictable, we're also about to start driving uphill, which means we'll be slipping more. No more 100m drives for us: we'll be inching our way up along "fire roads," shallow-sloped, relatively rocky paths that generally take the long way uphill. (Thanks to slip, more direct but steeper routes would actually require us to spin our wheels more times to get to the same location.) Increasing the difficulty, we'll also be trying to find a route that keeps the solar panels aimed toward the sun.

Fortunately, the Opportunity team's experience in Eagle and Endurance craters gives us some useful data for driving on slopes. Because of their situation, Randy Lindemann has already done a whole bunch of slip testing, producing graphs that show how the rover slips on various slopes, from zero up to 20 degrees. His results match up well against Opportunity's actual performance on Mars, making them the baseline for our expectations. So we Spirit rover drivers (sans John) sit in a meeting, talking over the results with Randy, trying to figure out how we're going to drive this rover under the new conditions.

I've decided to look at this as a challenge -- it's like we have a new vehicle and have to start all over learning about it. Pity it's not as good as the old one, but what are you gonna do?

Or they could take my suggestion, which I make half-jokingly: why not swap the Spirit and Opportunity rover-driving teams? There's a move afoot to pull Opportunity back out of its crater, which means they'd go back to driving on flat terrain. Why not let us deal with the flat-terrain driving we're used to, and let them deal with slipping on slopes, as they're used to?

I don't think it'll happen, though.

As it happens, Randy is also doing some slip testing this afternoon in the Mars Yard. I still have about an hour before I'm on shift, so I go check it out. They've set up a ten-foot-tall ramp, held up by a forklift and supports, which they can (laboriously) adjust to various angles. On this ramp they've placed a model of the rocky lip Opportunity is perched in front of, and they're testing whether the rover can drive over the lip and get back out. I have to leave before I see whether they succeeded (although the first test looks promising). But I do get to see a NOVA crew hanging around -- apparently they're doing a follow-up show on us. I'm looking forward to seeing it. It's fascinating to see events in person and then see how they're portrayed on TV; it sort of calibrates you. You also get to see how people you know react to the cameras -- who shies away, and who seeks them out.

When I get back to the SMSA, Jeng tells me the drive executed successfully. PotOfGold is RATtable at last!

But before we start drilling holes in it, it's time for a science lecture. Happily, the science team has agreed to resume these. The first one -- today's -- is given by Squyres, who uses the time to bring everyone up to speed on what's happened since the lectures were suspended (around the end of the nominal mission). He does it off the cuff ("No slides, no PowerPoint, no headsets, all analog"), and of course does a terrific job.[1]

He starts by reviewing the history of our explorations in Gusev -- Spirit's stuff. The whole point of landing at Gusev was to try to find sediments laid down by water, which (we think) flowed in through what is now the Ma'adim Valley. So we went hunting for sedimentary rocks, but found only basalt. All the rocks were identical -- interesting rocks, but not sedimentary ones.

"Mars," he says, "had faked us out. The sediments were there, but they were buried by lava." So we went to Bonneville, which should have punched a nice hole through the basaltic stuff into the underlying sedimentary rock, but all we saw when we got there was more basalt. Such a disappointment!

The only remaining prospect for anything fundamentally different was the Columbia Hills, 2.5km away. This was much farther than the rovers were supposed to be able to travel, but what else could we do? So we laid out a plan to get there by sol 160, and made great time.

There was one surprise along the way. We stopped long enough to dig a couple of trenches, in order to ground-truth the results from Odyssey's gamma-ray experiment (which Steve is also a member of). The gamma-ray experiment sees 20-30cm below the ground, and the trenches see maybe 10cm down. As it turns out, this was just the right depth to see something that can't be seen from space: highly elevated levels of sulfur and correlated elevated levels of magnesium, peaking about 5cm down. The reason they care: the only plausible explanation for this combination is a reaction between sulfuric acid and the basalt. The likely model is that Mars's groundwater was dilute sulfuric acid, which reacted with the basalt rocks in Gusev's floor over a long period. This isn't proof of large amounts of standing water -- it might have just been just a kind of watery sulfuric acid fog, reacting with the basalts over three billion years.

This finding helps ties the Gusev and Meridiani sites together. Small amounts of water on this side of the planet, large amounts on the other side, producing different results in the two locations. (At Meridiani, the evaporating water left behind large amounts of many different kinds of salts.)

So now we're at the hills. "It would have really sucked to get all this way, only to find the same old stuff," he says. Happily, that's not what happened. Upon our arrival at the base of the hills, we immediately found something morphologically different -- PotOfGold (née EndOfTheRainbow), the rock that's been such a pain to investigate. As Steve describes it, PotOfGold looks like a potato with toothpicks stuck in it, and jellybeans stuck on the ends of the toothpicks.

This shows differential erosion, a rock made of different materials of varying hardness being weathered over time. But what it's made of, we're not sure -- except that we're sure it includes hematite, revealed by the MB. That's our first sign of hematite -- more proof of water -- at Gusev. Oddly, the MTES doesn't show hematite in this rock, but there might be an explanation for that. Hematite comes in different forms, and some small-grained forms of hematite can be visible to the MB but not the MTES. So maybe that's what we're dealing with.

The rock will be a challenge to RAT, but the results will be worth it. ("It's so far outside the design specs of the RAT, it's not funny," Steve says. "But what the hell, it's sol 168, so let's go for it.") For one thing, they'll tell us whether it's rich in hematite all the way down, or only on the surface. The rock also shows signs of elements commonly deposited by water (though not olivine). The conclusion, Steve says, is that the rock started as basalt and has seen water somewhere along the way.

After RATting this rock, we'll find out what the hills are made of. It'll be the first Martian mountaineering expedition.

Then Steve starts relating the progress on Opportunity. At Eagle Crater, Opportunity was looking only at the very top layer of the geological record of Mars -- "like a book with all but the last chapter torn out."

They suspect there's a layer of soft, loose basaltic sandstone below the top layer. There are lots of impact craters, but no ejecta to speak of. But there is a ton of basaltic sand. That suggests that the asteroid impacts simply pulverize soft rock. (This is in contrast to Gusev, where the asteroid impacts throw out big chunky rocks like Humphrey.)

But they don't see this layer at Eagle Crater, because it's not deep enough. So they went to the much larger and deeper Endurance Crater in search of the basaltic layer. In contrast to Bonneville, they found what they were looking for. They wanted to jump right in and start measuring, but first we had to set up an extensive test program to evaluate the safety of entering the crater. So while that was going on, they had Opportunity start running a lap around the crater, taking pictures as it went. Once they worked out how to do it, they dipped their toes into the crater, using the RAT, MB, and APXS to produce a stratigraphic survey (the first such survey of another planet).

So far, the survey is showing lots of sulfates, not much basalt. They've also demonstrated that the evaporites go down for meters, which implies that there was lots of water here.

At this point I'm interrupted to help out with a minor problem they're having on Opportunity. It turns out to be a non-issue, and by the time I get back, the lecture is pretty much over. Ah, well. It's time for work anyway.

Thisol's emphasis is on the IDD, specifically the RAT. We're going to RAT PotOfGold or know the reason why. PotOfGold is a small rock in an awkward spot, maybe just barely large enough to fit the RAT on. But we're going for it; we're at least trying everything we can do. Besides -- as I point out at the CAM -- with all the trouble we've gone through to get to this stupid rock, seeing that RAT hole tomorrow is going to be so satisfying.




Footnotes:

[1] I recently took a memorable trip to the Scott Joplin House in St. Louis -- his former residence, now a museum to the famous ragtime composer. It struck me that there were people alive at the time who could sit in a room with Joplin and listen to the master play music for fun. Listening to Squyres give an impromptu science lecture is exactly like that, just exactly like that, and don't think for an instant that I don't know it.

2009-06-23

Spirit Sol 168

Larry's prediction was not as good as Ray's. We overshot the target by about 35cm, and will need to back up thisol. Yestersol I'd thought of shortening the drive by about 20cm, so in a way I'm glad we overshot it by more than that: we'd have missed the target even if I'd made that change. This overshoot will delay our RATting yet another sol, and is consistent with the running joke that we're on an ancient Martian burial ground and are therefore cursed.

So thisol we're IDDing the soil and then doing a very short drive back, so that we'll be able to RAT nextersol.

Or that's the plan until fairly late in the game. I'm catching up on my email and stuff when I realize that an animated discussion is shaping up behind me. The first thing I hear is Larry Soderblom saying, "We spent ten sols getting to this rock only to find out we don't have power to RAT. This could be the straw that -- I mean, I can just hear Ray screaming." He shakes his head. "OK, folks, we have to do some surgery. We're gonna cut two of the four NAVCAM octants and all the IDD work." (Which Chris just spent his whole day on.) "The highest priority is to get to the rock, get it in the right place, and be power-positive so we can RAT. Sorry about that -- but you still got your practice."

So the team gets to work, pruning back the sol. The RPs are left with only a dead-simple sequence, a 27cm backup drive -- with about 10% slip, that should put us 30cm back. That would leave us right in the middle of a zone where we can get a full-strength preload with the RAT, which is exactly what we want. I have only minor changes to make to Chris's version of the sequence, adding a final correcting turn at the end of the drive and one or two other small things, before I'm done.

Emily apologizes for the cutbacks to our sequences -- "Sorry to spoil your fun," she says.

"Ah, no problem," I reply, "it's nice to have a day to catch up on my email." Which it is: my email needed some serious weeding. So that's good, at least.

2009-06-22

Spirit Sol 167

One of the many things that's been great about being a rover driver is: no meetings.

Actually, we have meetings all day long -- the downlink assessment meeting, the SOWG meeting, the activity plan approval meeting, the command approval meeting -- but unlike most meetings, these are useful, necessary, productive, and focused. Also, they're exactly as long as they need to be -- no shorter, but no longer. And we skip the ones we don't need on any given sol (this is rare, but it's happened). If all meetings were like that, I wouldn't mind them at all.

Today I get invited to three meetings, all to be scheduled this week. First, Rick Welch wants me to attend a meeting to discuss the rovers' slip in the current terrain; this is preparatory work for climbing the hills. Second, Jake Matijevic wants to have what is apparently a different meeting on the same subject. (Maybe they're the same meeting, but it doesn't sound like it.) And Justin Maki wants to have a meeting about the terrain-mesh problem, where we've twice discovered the ground wasn't where the telemetry said it was.

It was nice while it lasted.

Ah well, this is all stuff that has to be done. And because there's usually a rover planner on shift at any given time when these meetings might be held, it's tough to get more than three of the four Spirit rover planners together for any given meeting. Maybe I'll be the lucky one. (Hmm, let's see -- drive the rover, or go to a meeting ... drive the rover, or go to a meeting ....)

The sol-163 sequence -- the last one I was here for -- seems to have gone perfectly. (Ray was right.) Since then, they drove away from the rock, and then drove back to it, trying to get a better angle for RATting. Unfortunately, we're in new terrain -- soft stuff under the wheels, and at a sharp angle -- and we're slipping like crazy. So they've spent three sols driving away and trying to drive back, and we're still not done; we've got yet another short drive in the plan for thisol, and we expect to have to complete the job nextersol. (Thisol we're climbing up the last part of a ridge, and the target is on top. We have slip data that show what happens as we try to climb at a given angle, but we don't have data to show what happens as we climb over an edge, as we're doing in this drive. It's anyone's guess.) Annoyingly, it turns out that there was a RAT solution in our original orientation after all -- we didn't have to drive away from the rock in the first place.

Since we're having so much trouble with this drive, and we expect similar trouble driving up and down the hills, John decides we should experiment with visual odometry. Visodom has displayed a few bugs on Opportunity recently, so we don't want to use it in actual rover-driving mode, but we could at least run it in a data collection mode, to try to get a sense of how it will work for us. I fetch Mark Maimone, and he shows me and John what we need to do to use visodom in this sequence. We put in all the commands we need, then find out there's not nearly enough time in the plan, so we have to rip it all back out. We do manage to sequence some NAVCAM images during the drive; Mark can process these on the ground and see what the rover would have done with them, so it's not a total loss. But this is at least the third time we've hauled Mark in as an expert in order to sequence visodom for Spirit, only to recant. Mark takes it in good humor, though. "This is becoming Xeno's visodom testing," I say, and he grins ruefully.

Thisol is a touch-and-go: we're unstowing the IDD, taking measurements of a soil patch dubbed "Jaws," then driving on. We're commanding the rover to drive about 2.2m, and expect it to cover about 1m due to slip. We'll have to see what it actually does, and I make sure to warn the team that we may need yet another sol to get into position to use the RAT on PotOfGold. Larry Soderblom quips, "So tomorrow, it's either RATs, or 'Rats!'"

But later he (unknowingly) echoes Ray's prediction from a few sols back. "I have this feeling it's going to work," he says.

2009-06-18

Spirit Sol 163

"This rock," Steve Squyres says, "is kicking our butts."

He's talking about yet another problem we've had in our efforts to IDD EndOfTheRainbow (or PotOfGold, or whatever the hell they call it). Yestersol's IDD sequence faulted out in the middle of the MI mosaic, and for an unusual reason: the rock wasn't where our telemetry said it was. It was about 3cm higher, well out of the usual range of uncertainty (which is about 1cm). The 3cm difference was enough that pulling the arm back after contact with the side of the rock would have caused the IDD turret to run up against its hardstop; as a result, the flight software decided we were dumb and stopped moving for the sol. (As an unfortunate result, the MI was left pointing up all night with its dust cover open, something we try hard to avoid. If we make that mistake too often, settling atmospheric dust will spoil the lens.)

We're not sure what caused the inaccurate terrain mesh that led to this problem, but we've seen this once before, on Mazatzal. Now that we have two instances of the problem, we might have some luck tracking it down. We're going to take a NAVCAM image to get another perspective on the problem. "We drove three kilometers to get to this rock," Adler dryly observes, "and we're foiled by a matter of three centimeters."

So we'll fix the problem and continue nextersol. At least the MI images we did take, before we were so rudely interrupted, came out beautifully focused. The MI PULs have already made a stereo anaglyph from the images we've gotten so far, and it looks beautiful.

Or so Chris tells me. Apparently, they had the image up on the big projection monitors in the science area all morning, but when I go to take a look, it's gone and the room is empty. I log into my workstation and start a search for recent images, but I become busy with other things before it completes. So I'll have to take his word for it.

Speaking of cool MI images ... Chris and I get to talking about the wheel problem, and we realize that we can probably take pictures of the wheel with the MI. It's kinda crazy, but it just might work. I run into Joe Melko, who seems to be in charge of the investigation, and broach the idea. He turns it down because of a problem we didn't think of: the arm has lots of angles and corners that can snag on the cables and things under the rover. So that's out. But it does prompt him to suggest imaging the wheels with high-quality HAZCAM images, which is something we could do in the next couple of sols. We can take two images -- one with the troublesome wheel straight, one toed in -- and it won't cost us much downlink volume, because we can subframe the images, downlinking only the part that shows the wheel. Joe doesn't think this will be useful because the problem is almost certainly not something physically jammed in the wheels, but: "At the press conference, they'll ask if we have images, and if not, why not." He shrugs. "So we can just avoid the whole issue and take them."

I work out the details with Joe and Julie, then head back to work with Chris on tomorrow's recovery plan. Since we can't move the rock and don't want to move the rover, our recovery strategy has to work around the rock's awkward location. Chris settles on retracting the arm part of the originally planned distance, then rotating it slightly and retracting it the rest of the way in a different direction, clearing the rock while avoiding the hardstop. We make these changes to all of the moves on the side of the rock, cut the stuff that succeeded yestersol before the fault, and that's about it.

I show this at the CAM, walking the participants through it, and everyone seems satisfied. "Let's cross our fingers and upload the load," says Julie.

Ray has a more optimistic view. Walking out, he nods and smiles and says in his quiet voice, "I feel it's gonna work this time." Naturally, we all hope he's right.

I'm shutting down everything and preparing to leave, when I come across the results from the search I started hours ago. I'd completely forgotten about it. It was successful, turning up not one but two new MI stereo images. I grab a handy pair of 3-D glasses and immerse myself in the view for a few minutes. It's fantastic -- I don't know what the scientists see when they look at it, but the micro-scale shape of the rock reminds me of Hokusai's famous woodcut painting "The Breaking Wave Off Kanagawa" (a.k.a. "The Great Wave"). Water, again.

[Next post: sol 167, June 22.]





Courtesy NASA/JPL-Caltech. One of the microscopic images of "End of the Rainbow."





Courtesy Wikipedia. Hokusai's "The Great Wave off Kanagawa" ("Kanagawa Oki Nami Ura").


2009-06-17

Spirit Sol 162

At the end of yestersol's sequence was a simple turn-in-place intended to reposition ourselves so that the rock we've been IDDing (dubbed "End of the Rainbow") would be right in front of us, giving us better access for continued IDD work. This maneuver executed perfectly, so thisol is all about poking at the top and side of this small rock.

Chris has only just gotten started on the sequencing when I arrive, meaning they're running a little late. So he asks me to help out by doing a couple of the instrument placements. This doesn't take much time to whip out, and it's nice to be doing the relatively creative part of the job again.

It turns out that there's a reason they're so interested in this rock. The preliminary MB results show it contains hematite. And hematite means water, baby. Water, at last! We couldn't let Opportunity have all the water, I guess.

Despite, or because of, our late start, we have spare time: the meeting schedule gets delayed. (It ends up being delayed two hours.) Since this leaves me a little time to kill, I check out the most recent downlinked images, including a NAVCAM looking up the spur we've just reached. Jake Matijevic, looking over my shoulder, asks, "What's the slope up those hills?"

I don't know that, and I should. So I work it out quickly. "About ... um, that's in radians ... about 10 degrees from here to the top of the hill."

"Not bad -- that's well within the vehicle limits."

"Yeah. Of course, that's just the overall slope -- the surface isn't uniform, so the instantaneous slope might be worse." I think about it for a second. "But that's probably not a problem. We can still climb. It'll be just like avoiding obstacles, only avoiding slopes."

"That may be bad if we have to drag a dead wheel, though," Jake says.

"Ah, it's nothing. We'll just drive backward uphill. What could be simpler?"

But Jake turns serious. "The tests they did in the ISIL weren't encouraging. Dragging a wheel causes positional error and digs a trench behind us." That was the worst case -- with the wheel totally dead -- which probably won't be our situation. We'll probably be able to feed the wheel enough power that it will turn some. But he's right, it will limit us.

"Last I heard anything on this was about a week ago," I say. "Any new thoughts since then on what's causing the problem?"

"Well, if it were the motors, we'd see inconsistent power delivery, and we're not seeing that. Joe Melko thinks it's lubricant congealing in the gearbox due to the cold."

"So is this something we can work around? Run heaters, park with that wheel in the sun? Maybe after the winter solstice, when Mars warms up again, it will just magically get better?" I'd better face it, I think to myself. I'm in denial.

But Jake is right there with me. "Maybe," he chuckles. "Mobility system, heal thyself!" He turns back to his keyboard. "Well," he adds, "stranger things have happened."

Chris takes some time to finish up the IDD sequence, but when he's done, it's in very good shape. I have to fix up the comments and add a safety-deactivate sequence, but not much else changes. At the CAM, Emily asks for science's opinion on the result. "It's gonna be cool," Ray says.

George Chen is going to be doing the flight director webcast thisol, and he has a question for Ray. "Ray, when are we going to announce this rock has hematite? Is it OK to reveal that?"

"Yeah, go ahead," Ray says. "It's definitely hematite. We discovered it on the recovery sol, so we only have the short -- four-hour -- Moessbauer results. That's why we're redoing the observation thisol with a thirteen-hour integration, to get better results. But it's definitely hematite."

Ray grows animated -- by Ray's laconic standards, at least -- describing the importance of this find, and the current thinking about the composition of the hills we labored so hard to reach. He compares them to Mazatzal, the coelocanth-like rock that so interested the science team back around sol 77. "Our model is that the Columbia Hills are like a giant Mazatzal," he says. "Mazatzal gone wild!"





Courtesy NASA/JPL-Caltech. Spirit centers herself for more work on "End of the Rainbow" (smack dab between the wheels).

2009-06-16

Spirit Sol 161

I walk into an empty room. Usually, the place is buzzing with activity -- or, if not buzzing, at least murmuring quietly. Today it's empty.

Dan Moyers clears up the mystery when he arrives a few minutes later. Once again, there's been a late schedule change. This time, at least, they moved the meetings later, not earlier. So they're all upstairs, and I'm extra-early, as it turns out.

Which is probably a good thing, because there's another mystery. Thisol's IDD sequence errored out, and nobody knows why. That is, they know the command that caused the error -- they just don't know how it got to the spacecraft, because it doesn't seem to be in the IDD sequences the rover planners wrote yestersol.

Which is very, very disturbing.

While I'm waiting for everyone else to show up, I start investigating, and I soon have a working theory. It looks like they delivered the wrong version of the sequence, one that included so-called overdrive correction commands that aren't meant to be sent to the spacecraft.

The overdrive correction commands are needed because RSVP doesn't know about a fundamental law of the universe -- that two objects can't occupy the same space at the same time. Because there's always a little bit of uncertainty in the system as to the exact location of the terrain with respect to the rover, we deliberately tell the IDD to move farther than we think it needs to when placing an instrument on rocks or soil. The rover's contact sensors detect when the instrument actually touches the terrain, and the arm stops. It's a little like feeling your way around in a dark room -- you gently extend your hand toward where you think the wall is, and stop moving when you touch it.

Only RSVP's simulation doesn't stop when the arm touches the terrain. If you tell it to put the arm half a meter inside a rock, it cheerfully goes that far in. This mismatch between the behavior of the real and simulated arms leads to other problems when we're sequencing follow-up motions -- just recently, the real arm shut down because it detected an impending self-collision, a self-collision that didn't show up in the simulation. As a result, we add commands to fix up the simulation -- for every command where the simulation puts the arm inside a rock or under the ground, we add a command that pulls it back out to a more plausible position. With these added commands, the simulation behaves much more like the real thing. When it's doing what we want, we remove the overdrive correction commands and send the rest to the real rover.

Only ... that doesn't seem to have happened this time. Deepening the mystery, when we look at the files that were delivered yesterday, the overdrive correction commands aren't there. But when we look at what was sent up to the spacecraft, two of them were included.

Eventually, a bunch of us -- John, Sharon, Marc Pack, I, and others -- manage to solve the mystery together. They had to change one of the IDD sequences yesterday, after an initial version had already been delivered.[1] That initial delivery had the wrong sequences, but the second time, one of the sequences was updated and one wasn't. For complicated reasons, the reviewed versions of the sequences weren't the ones that we sent up -- we reviewed the right ones, without the overdrive correction. So one of the sequences sent to the rover still had overdrive correction commands in it, but those commands weren't in the version everyone was looking at.

That last part is the really scary part. We have a system in place that ordinarily guarantees that the versions of the sequences that get sent up to the spacecraft are the same as the versions we review. That system can break down, though, when a sequence is updated late in the game, as happened yesterday. Fortunately, Sharon and Marc already have most of a script that can compare the reviewed versions of the sequences to the uplinked versions. They immediately set to work arranging for the script to be run as part of the uplink process every sol, which should help catch this problem in the future.

It makes me think, though. All that happened as a result of this mistake is that we lost most of a sol -- the spacecraft is fine. But one of these days, we might be telling a similar story with a much worse ending. If we're lucky, we're not telling that story to a Congressional committee.

At least this anomaly made today's sequencing easy. John copies yesterday's sequences, cuts the parts that completed already, prepends a command to clear the error flag, and goes to lunch. There's a little more to it than that, though (isn't there always?). In one way, we're lucky that the error killed the sequence, because one of the moves in yesterday's sequence would have left the MB hovering a couple of centimeters off the rock surface, which would have made its data puzzling at best, and perhaps entirely useless. I fix it.

I also write up the detailed story of the anomaly and send it out to the rover drivers, mission managers, and a few others. Bob Bonitz, who's still on the rover-driver email list, comes downstairs to laugh about the matter. Since he's not on the project any more, he can freely poke fun at our mistakes. "That was like a description of a Keystone Kops movie," he chuckles.[2]

He also tells me a friend of his saw the KCET "Life & Times" segment that featured me and called him to ask, "Who's this Scott Maxwell guy, and what is he doing driving the rover?" Apparently, Bob was also interviewed by an NPR reporter who was here at the same time as the KCET guys, spoiling Bob's plan of never talking to the press throughout the mission. "But I still kind of met my goal, because they mispronounced my name," he says. "They called me 'Bob Buntz.'"

Even though it amuses Bob, I hope we don't have another problem tomorrow. And so does Emily, who looks like she's had a long day. "Will you be here later if there are any problems?" she asks George Chen.

"Yeah," he says. "Why?"

"Have you ever been around Steve Squyres when we lose a sol?"

"Not a pleasant experience?" he asks.

Emily purses her lips. "I just don't want to be around."[3]




[1] This doesn't mean it had been uplinked to the spacecraft, just that it had reached a certain advanced point in our process -- a point at which we'd normally consider the sequence ready to go up.

[2] Fair enough -- it was. But it's interesting to note that mistakes in a complex process always look like that -- they have to, because the mistakes that get through to the end are by definition the ones that just happen to involve screw-up after screw-up. What you don't see are all the errors that get caught because of those multiple gates in the process that normally stop them. (In a sense, this sort of complex system is constantly in a state of partial failure.) As a result, if you look only at the mistakes, the people involved look stupid and/or incompetent, but that's an inaccurate view. Application of this idea to, say, high-profile government security failures is painful but sobering.

A while back, fellow rover driver Paolo Bellutta pointed out that we're missing something by focusing our attention almost exclusively on mistakes we don't catch. We should also pay more attention to mistakes we do catch, he said, and try to formalize the way we catch them, because otherwise we'll someday let one of those slip through. I thought this was a brilliant insight on his part and have tried my best to apply it.

[3] I don't know what was behind this comment -- and maybe I don't want to know -- but I can attest that I've never seen Steve be anything less than scrupulously professional. Maybe she'd just had a long day.

2009-06-11

Spirit Sol 156

Yestersol's drive tilted out. I think that's a first for us. The rovers have tilt sensors to tell them when they're on dangerous slopes -- sort of like a glorified pinball machine. Yesterday the rover climbed onto a slope that was just a little bit too steep, and, well, game over.

We knew yestersol that there was some risk of this. So when we were working on yestersol's drive sequence, I wrote a script that turned RSVP's simulation results into a graphical plot of tilt versus time. The graph showed a maximum tilt of about 17 degrees as the simulated rover climbed a ridge toward the end of the blind drive -- which was less than the current on-board limit of 20 degrees, so it should have been fine. Unfortunately, as it turns out, the simulation was just a little bit wrong, causing us to exceed the tilt limit. Well, these things happen.

Recovering is simple enough. We're going to increase the tilt limit, which was set lower than it should have been anyway, to finish climbing the ridge. Then we'll drive another 30m or so down the back of the ridge and across a little swale, and we're at the base of the Columbia Hills at last. (Or so it seems from here. But the slope at the base of the hills is gentle enough that there's no clear dividing line. Some say we're already there, but -- as in a nightmare -- the closer we get, the farther away it seems to be. It's become something of a running joke.)

Of course, it's never that simple. Continuing our effort to figure out what's starting to go wrong with the right front wheel, Joe Melko has devised a sort of exercise program for the rover. We're supposed to drive the rover for varying distances at varying speeds -- two meters at full speed, followed by 25cm very slowly, etc. -- with one series of these motions at the beginning (when the actuators are relatively cold) and a similar but different series at the end (when they'll be warmer).

The first problem with that is, none of us knows how to tell the rover to drive at anything less than full speed. There are no rover drivers upstairs -- they're not needed on Opportunity thisol -- and the best possible person to ask, Jeff Biesiadecki, is at home today. I poke around in the command dictionary and find what I think is the right command for this, but it's not very clearly documented. Then I ask around a little, and everybody I talk to ends up saying, "But you should call Jeff Biesiadecki to be sure." I hate doing that -- but they're right. So I call him.

At least I had pinned down the right command, and I had the right idea about how to use it. The basic idea is simple enough: there are six drive motors, one for each of the wheels, and you just use this command six times to set the maximum turn rate for each. There's a little more to it than that, and Jeff researches the matter and emails me the details later, but we already know enough to get started.

Which is a good thing, because these extra commands, along with associated data prioritization commands we also need to uplink so Joe can get the results quickly, complicate the drive considerably. Not to mention that we actually have to ensure that the rover is safe while it's performing the exercises Joe thought up for it, and that it ends up reaching thisol's destination. It turns into what I'm pretty sure is our most complex drive sequence ever. I quickly write a couple of new RoSE macros to help, but it's going to be a lot of work anyway.

Naturally, to make things worse, they had a traversability meeting this morning that effectively pushed back the start of the day. While Chris is working hard on the drive sequence, Julie asks him, "Will you be ready for the walkthrough in ten to fifteen minutes?" Chris turns around and says, "How much do you want me to laugh at that right now?" Which I interpret as a no.

Chris finishes his part and hands the rough-cut drive sequence off to me. I start fixing it up and making sure the details are right while he straps on the LCD shutter goggles and revisits his high-level decisions. Every once in a while he comes over and says, "You know, I don't think I like that waypoint after all," and I have a bunch of things to change. Under other circumstances, this is the sort of behavior that makes me want to strangle people. But he's not doing this to be an asshole or because he was sloppy, he's doing it because we're solving an unusually hard problem on an unusually tight schedule. And, as always, the rover's safety is paramount. If we need to do extra work, then fine, we need to do extra work. The rover matters more than our time.

With all this going on, it falls to Justin Maki, as usual, to ask: "Did you guys know that today is the one-year anniversary?" Holy smoke, I can't believe I didn't notice! Spirit launched June 10, 2003 -- one year ago today. And appropriately enough, this is the day we'll reach the Columbia Hills. Naturally, Justin's observation triggers reminiscences (from those who have time for it).

"Seems like a lot longer and a lot shorter time than that," Julie observes.

"We flew those damn things here --" Scott Doudrick says.

"That took a long time."

"-- the sun blew up on us --"[1]

"We lived two years in that six months," Julie says. She starts to say something else -- and is interrupted by the exigencies of the moment. Spirit needs her attention, again.

As Chris changes the waypoints, Jeff emails me other things we need to change in the sequence. The first is a potential buffer-size problem. The flight software allocates only a 300KB memory buffer for gathering motor data. The way it works is, at the start of each low-level drive command, the rover starts writing data into the buffer; it flushes the buffer (moving the data to another area so that it can be downlinked later) when the command completes. But since some of the steps in Joe's exercise program are taken at a very slow speed, some of the commands will take several minutes to execute -- much longer than normal. Also, we're gathering data at a higher-than-usual rate, to give Joe the maximum information possible. But this combination of long commands and high data rates will produce more than 300KB of data, causing the buffer to overflow. I don't think the software will crash, but we'll lose some of the data we're trying to gather. So we have to go back and break up the arc commands into multiple shorter arcs, short enough that each one will come in under the limit.

Then Jeff realizes there's another problem. Joe wants to run the motors at a much lower speed than anybody has ever tried before, and Jeff doesn't know for sure that that will work. When we're in a situation like that, we're supposed to try it in the testbed first.

Ah, wonderful.

Well, maybe we don't need to do that, and maybe we do. It's not my call to make; and fortunately, we have a meeting coming up that will include the right people to make this decision. It also occurs to me that Mark Maimone has probably run these rovers in the testbed more than anyone, and even if he's never run them so slowly, he may have a valuable opinion. So I go find him -- he's eating lunch while hacking away at some code in the SMSA -- and ask him to join us.

"You didn't think you were going to do anything useful today, did you?" I ask him.

"I guess not," he sighs.

The options as I see them, I tell him, are three: (1) try this in the testbed before sending the sequence up today, (2) punt and do it tomorrow, giving us more time in the testbed, or (3) ignore Jeff and try this for the first time on the flight vehicles. (I am no fan of option 3; I include it mainly to make Mark laugh.) Waiting wouldn't normally be a problem ... except we know we can fit the exercise into today's drive, but once we start climbing the hills, our opportunities to try it may be curtailed.

Mark looks at the Mars clock. "How long do we have until the sequence goes up?"

"About nine hours."

"So theoretically, if someone went to the testbed for an hour or two ...." he says. He knows he'll be the chosen someone, and he clearly isn't in the mood to sacrifice a productive day to this. But he's a good guy, and he's willing.

"Yeah, theoretically," I say. "But we're going to have a meeting to see whether we even want to do that, so don't head over there just yet."

At the meeting, we need to show the somewhat tricky drive in 3-D. Which means we need to scare up a bunch of 3-D glasses. I steal some from the science room, and snag my own five-dollar glasses as well (determined, as I am, to wring some value from them). As I hand a pair to Julie, I tease her, "Remember, men don't make passes at girls who wear 3-D glasses."

"My boyfriend will make me wear them all the time, then," she replies.

While people are still filtering in for the meeting, Mark Adler mentions that he's bringing a Senator around on Saturday. I think he says it's Senator Stevens, the head of the Senate Appropriations Committee and, consequently, one of the most powerful Senators. It would be fun to be there, but I'll be out house-hunting, so I'll have to give it a miss. Besides, after the President and Vice President, mere Senators are yawnsville.

The meeting starts. We've got three mission managers ("which guarantees we'll have one eye open among us," Andy quips), among others -- more than enough firepower to decide whether to run Joe's experiment tomorrow (though that matter has to wait until the goal of the meeting, planning our near-term drive strategy, is met). The decision is to split the difference: do as much of Joe's experiment as we can, but don't drive the wheels any slower than Jeff feels comfortable with. Joe had us driving the wheels as little as 1/34th of normal speed; Jeff feels comfortable going down only as far as 1/3rd. So I delve into the sequence once again, raising the minimum speed into Jeff's comfort zone.

One other matter that comes up in the meeting is the name of an upcoming destination -- Tacopa. As soon as I hear this, I start singing, "At Tacopa, Copacabana." Mark Maimone chimes in: "Her name was MOLA ...." Now that's something: a joke you can get only if you know old Barry Manilow songs and if you're enough of a space geek to know about the Mars Orbiter Laser Altimeter.

I think Mark is glad he came to the meeting after all. Just so he could make that pun.

As I'm finishing up for the day, I happen to look at the rover in its current state as determined from telemetry, frozen climbing the hill. I've hidden the terrain and am just looking at the rover itself from a front three-quarter view, when I'm struck by something. Tilted back to an unusual degree and with the front wheels extended in front of it, Spirit looks uncommonly like a rearing horse.

Steady on, girl, I think. And take care of that right front hoof.

[Next post: sol 161, June 16.]




Footnotes:

[1] While the rovers were on their months-long flight to Mars, they experienced a burst of unusually high solar radiation -- caused by sunspots, if I recall correctly. There was nothing we could do about it, and the rovers survived unharmed, but it was a scary incident.

2009-06-10

Spirit Sol 155

Lately the meeting schedule has been running early, so I've been coming in early. Today they're holding the meetings on time, which means I'm a lot earlier than I need to be.

This gives me time to help out Larry Soderblom and a couple of the other scientists, who want to know how to use the fancy 3-D shuttered LCD goggles to view 3-D images in SAP.

"You can't," I tell them. "They only work with RSVP. Want me to show you?"

They do, so I do. I set them up to use a (ahem) preview copy of the next release, which has extra features that make this a little easier. We load up some NAVCAM images they want to see, and I show them how to switch the system into true 3-D mode. One of them takes careful notes so that he can tell everyone else how to do it, then repeats the whole thing from scratch while I watch. He gets it exactly right the first time. What a good teacher I am!

The start of thisol's drive has an odd feature, a California U-turn. (Chris likes the term so much, we name the sequence after it.) This is a short drive forward, a 180-degree turn-in-place, a short drive backward, then another 180-degree turn so we're facing forward again. After that, we drive on normally.

This odd little dance is at mechanical engineer Chris Voorhees's request. It's to help us gather some engineering data on the right front wheel, which has started drawing more current recently. We're not sure why it's doing this, but it's not good news. At the rate it's going, it will start to be a problem by the end of August, so we're starting to think about what to do about it now. I talk with Voorhees about how we'll cope with it. Driving backward is one solution, since it works better to drag a dead wheel along than to try to push it ahead of us. He also says we can feed the wheel as much power as possible, so that the mechanical system offers as little resistance as possible to its turning. This makes the wheel a little less dead ("only mostly dead"), if you see what I mean, which might help.

"I realized the other day that I'm predestined to have a failure review," he sighs. With the rover at less than twice its designed age, most components of the rover have experienced less than twice their design limits -- the cameras have taken fewer than twice the expected number of pictures, the IDD has been used less than twice as much as planned, the mechanical system as a whole has experienced fewer than twice the expected thermal cycles, and so on. But the mobility subsystem, Voorhees's baby, has already been used for something like five times its design limit. We were only supposed to drive the thing 600m, and we're already over 3000m. No wonder the wheels are falling off.

In the spirit (ha) of always-look-on-the-bright-side-of-life, I point out that he'll have a lot to be proud of in that failure review. We're passing five times the design limit and still going strong. He agrees, but he doesn't look any less worried.

Since I got picked to be a rover driver, I've been more careful than usual when driving in real life. This is because I don't want to hear any lame jokes about being able to drive on Mars but not on Earth, and suchlike. I've avoided any problems, but Chris Leger didn't -- he just got a speeding ticket. John Wright got one, too, not long ago.

I drive home very, very carefully.

2009-06-09

Spirit Sol 154

Yesterday, I conservatively estimated we'd cover 60m. We did 70. So we're 100m from the hills now, and we should roll across another 70m of that thisol. The way ahead looks reasonably clear, so we have high hopes for a good long drive.

There must be a lot of politics going on among the scientists that I hear nothing about. As I'm working, I overhear one of the upper-level science folks say to another, "Poor Mike Carr has been persona non grata since he accused the Gusev crowd of wandering aimlessly, kicking the dirt." If this is true (as it likely is, considering the source), it's a reference to (some of) the Gusev science team's behavior in the month or two after landing, when they had to stop at every rock they saw, even though it was pretty clear they were all just dusty basalts. If he kicked their ass a little to stop them kicking the dirt and move on to bigger things already, Mike was right. I hope they're not really low enough to be holding it against him.

The day's relatively uneventful, a typical drive. Chris finishes his bit and hands over to me just in time for Larry Soderblom to start talking about our plan once we get to the hills. He's describing the geology of the area, laying out the high-level plan -- and I can't listen, because I gotta work. Foo. I guess I'll just plan to pick it up as we go along.

My happiest news of the day is this. Leo Bister, our mission manager for the sol, mentions at the end of the CAM that we need wakeup song suggestions for when we reach the hills, now only a day or two away. Way back around sol 2, Mark Adler asked for suggestions for rover wakeup songs, and one of the suggestions I sent him was a Red Hot Chili Peppers cover of the Stevie Wonder song "Higher Ground." It never got played. Later, Adler asked for another round of suggestions, and I included "Higher Ground" again -- and it never got played.

Now, a hundred and fifty sols later, we've at last reached our once-distant target, the Columbia Hills -- what better time to play a song whose lyrics include "Gonna keep on tryin' / Till I reach the higher ground"? I nonchalantly suggest it, and Leo likes the idea. (It probably helps that one of the scientists knows the exact cover I'm speaking of, and adds his recommendation.) I email Leo the MP-3, and he puts it on the schedule for that day.

Third time's the charm!

2009-06-08

Spirit Sol 153

We're just 170m from the hills now -- and I'm on shift for the next four sols, so we might just reach them on my watch. Of course, lately we've been regularly making 90 or 100m in a single sol, so that might not seem to be a problem. But now that we're closer to the hills, the terrain is getting rougher again, so we'll make less progress per sol, making this something of a race.

We've reached Plymouth Rock, a rock about 50cm tall and 1m wide, which has been our aim point for the last several weeks. One of our PANCAM images happened to capture it perfectly. Fortunately, it's off to our right; today's drive will have to go only about 1m out of its way to skirt this monster rock.

I'm early, but I try to make myself useful as soon as I arrive. Eavesdropping on the intense activity planning discussion in the corner, I gather they're trying desperately to save a few minutes in the plan, and are thinking about cutting the end-of-drive stutter step.

"What if we drop the stutter step and unconditionally dump the HAFIQ?" I suggest. This gives us a set of low-resolution pictures leading up to the final position (or so we hope; autonav may aim the rover in the wrong direction for our purposes, making this something of a gamble) without the overhead normally associated with our end-drive imaging. This saves about five minutes, which turns out to be enough to be useful. (Indeed, better than merely useful, it turns out to be enough to save a science observation. Jeff Norris says, only half joking, "Some grad student writing his thesis six years from now will be very happy we took this PANCAM image.")

As has been usual of late, we're running well ahead of schedule -- nearly an hour ahead. But scientists are going to be participating remotely -- that is, calling in -- starting soon, and that doesn't work if the meetings start early. So John and I hand over early, and for the first time in some weeks, I handle the activity plan approval meeting when it happens.

This drive sequence is r0101 -- our 101st drive sequence. (Or thereabouts -- for some now-forgotten reason we didn't have a r0000 or r0001, but we have some drive sequences with other wacky numbers.) "Annoyingly," John says, "Opportunity has more drive sequences than we do, even though they've done less driving." This is because they split the drive sequences more often than we do, inflating their number. As of today, they're on r0102.

And incidentally, their r0102 has a scary description: "position_for_entry." This is, of course, their entry into Endurance Crater, which they've officially decided to go for. Cooper told me the other day he was hoping they wouldn't do this -- there's a lot of interesting stuff outside the crater they could go look at instead. One of the ideas floating around was to drive Opportunity to its own heatshield and RAT it. This would be the first time we'd ever done that kind of investigation into what happens to a spacecraft landing on another planet, and could teach us lessons we could use to make future landers lighter and more reliable. But if Opportunity gets stuck in Endurance Crater, that option is foreclosed.

With all the obstacles in our way, we're not sure how far we'll be from the hills following thisol's drive. We know we'll go about 40m closer, but after that it's up to autonav's ability to take us through this more difficult terrain. When one of the scientists asks, Julie says, "We'll be 100m from the hills if autonav does its job."

"Autonav's number one job is to keep the rover safe," I point out. "Moving you closer to the target is job number two."[1]

So however far we get thisol, we'll get another shot at the hills tomorrow. If autonav does its job, and we do ours.





Courtesy NASA/JPL-Caltech. Plymouth Rock, an important guidepost on our way to the Columbia Hills.




Footnotes:

[1] Kuh-rist, I can really be a pedantic asshole sometimes. Sorry about that, Julie.

2009-06-03

Spirit Sol 149

The big poster outside my office is drooping. The duct tape that holds it up (can you tell we're engineers around here?) has come unstuck at one corner, and it's folded over itself. I don't want it to get damaged, so I stick it back up.

"What were we thinking when we dumped the HAFIQ?" I ask Chris as I walk into the sequencing MSA.

"What do you mean?"

The Hazard Avoidance Fault Image Queue is a buffer that contains the last few images the rover has taken as it proceeds on autonav. The idea is that if the rover encounters certain kinds of errors while driving, it automatically dumps the queue -- sending those images back to Earth -- so that we can see what's wrong. We've also gotten in the practice of adding commands to manually dump the HAFIQ at the end of a drive in other cases, because that gives us more data about the rover's surroundings when we need it most. Yestersol, we dumped the HAFIQ unconditionally.

But that was a dumb thing to do, because there was no autonav yestersol -- it was all blind driving. So there's nothing useful in the HAFIQ. All we'll get is some useless overhead data that comes down with the empty queue.

I point this out to Chris. "Oh," he says. Then he shrugs. "Well, it was just two megabits."

Which is actually a fair point. Early in the mission, I got in the habit of working to eliminate every unnecessary bit of downlink. We were packing every sol so full of science observations that anything we could save on the engineering side was significant -- it meant we could work in another science observation, and that's what we're here for, after all. Nowadays, with our less efficient planning process, there's a lot more slop. Also, we're now normally constrained by energy, not by data volume, and since we doubled our top data rate, we often have more available downlink data volume than we used to. So it doesn't matter so much any more.

I'm still going to cut it if this happens again. For old times' sake, if nothing else.

Yestersol's drive was successful. We ended up in a spot where we can deploy the IDD and poke at stuff -- including the magnets mounted at the base of the PANCAM mast, which is always fun because the rover's arm has to swing up over its own body. Some of the IDD observations are cut for time or energy reasons, but much remains.

As usual, I start my shift by independently reviewing the state of the rover while Chris charges ahead with the sequencing. While we're doing this, we hear a Labwide announcement: "Please join NASA administrator Sean O'Keefe for a discussion of exploration." "Sorry," Chris says, "we're too busy exploring."

Bob Bonitz, passing by, stops in to say hi. "So, do you miss MER?" Chris asks.

"Yes and no," Bob shrugs. "It's definitely easier doing one job than two." Apparently, they're moving him out of the building -- Phoenix development is being moved into a trailer. You should have stayed on MER, Bob. Then again, when Cassini reaches Saturn (less than a month from now) and crowds us out of the news entirely, maybe we'll be moved into a trailer, too.

Bob also congratulates us on passing the 3km mark. Justin Maki says: "You know, we passed the 3km mark and nobody noticed. I brought it up in a meeting, and everybody was like, eh." But even if familiarity has bred contempt, it's still a hell of an achievement: we've now gone five times our required distance, and we're still zooming along. "Remember all the meetings where they were saying the 600m requirement was too much and we should take it out?" Justin remarks. He adds, though, that we're inadvertently making life harder for MSL. "MSL is like, before MER landed, we had all these requirements, and now MER has done half of them."

Aw, poor MSL. I can guarantee them one thing: we're not stopping any time soon, certainly not to make their jobs easier. They'll just have to deal.

Chris finishes with his part of the work, and we walk through the sequence together as part of the handover. Emily asks him for the estimated duration of one of the sequences, and he tells her.

"Does that include the 10-second-per-move overhead that RSVP doesn't model?" I ask him. This is overhead imposed by the IMU, which needs to go into the mode where it tracks the vehicle's attitude every time we move the IDD, and entering that mode takes a few seconds.

"No," he says, "remember, that's not needed any more because we're not using the IMU while moving the IDD now."

"Oh, right. Hey, if you wait long enough, the bug turns into a feature."

Emily forgets to have Chris sign the activity plan approval form before he leaves, so she asks me to sign it instead.

"I always wondered what would happen if I didn't sign one of these," I say.

"Nothing," she answers. "I forgot to do one yesterday."

Ah, we're a well-oiled machine.

Sounds like Opportunity's going into Endurance Crater. Squyres lets this slip as he's telling Mark Adler that Spirit might climb the hills -- "but keep it in the family," he says; it's not official yet, but the handwriting is on the (crater) wall.

Adler shakes his head as Steve leaves. "Those guys have been having meeting after meeting with high-level management about going into that crater," he says. "High-level" means, at least, Ed Weiler, the guy who's basically the NASA-side man in charge of JPL. Since entering the crater is a potentially mission-limiting decision -- once they go in, they may never get back out -- it had to get his approval. And apparently he's decided the scientists can have their way.

Steve pops back in to add, "We should figure out where we're going to winter" -- that is, where we want the rovers to be during the winter solstice, when solar energy will be at its lowest. Mark has clearly been thinking about this already. "We want to aim the solar panels north to maximize sun exposure," he says. "That's the great thing about hills -- you can find a way to aim pretty much wherever you want." If Opportunity's in the crater, it can pull the same trick -- climb partway up the south wall of the crater, so its solar panels are aimed north. And wait.

Art Thompson is working on Opportunity this month, but he stops by to keep in touch. This is Emily's last shift as TUL for a few days -- Julie Townsend is in tomorrow -- and Art asks if Emily has depleted Spirit's batteries. (This is part of an ongoing good-natured rivalry of sorts, where each team tries to leave the other in a difficult position.) "She's gonna have to drive it into the gas station?" Art asks.

"She's gonna have to push it into the gas station!" Emily says.

"Good girl!" Art laughs. "I trained you --" he's obviously about to say "well," but thinks of a better word and corrects himself on the fly. He finishes instead with, "maliciously."

The poster corner has come unstuck again. I put it back up, smooth the tape down carefully, and go home.

[Next post: sol 153, June 8.]

2009-06-02

Spirit Sol 148

As expected, today's another drive sol, as Chris tells me when bringing me up to speed. "But not another drive-like-hell sol. We're just going 50-70m to the top of a ridge." There, we're going to take a nice color PANCAM of the hills, while we're still far enough away to be able to take them all in. And we're setting up to do IDD work on the next sol. To improve our chances of being able to use the IDD, we're going to blow off the autonav; this gives us a lot more control over where the rover ends up, and therefore whether it will be safe to deploy the arm.

Yestersol's drive went about 50m or so, leaving us around 20m short of the 3km mark. So we'll break the 3km mark with this drive.

While Chris plugs away at the drive, I start doing my own analysis of the terrain. In the far corner of the room, a couple of people are watching our press conference, where they're discussing the current thinking on driving Opportunity into Endurance Crater. This bugs me -- not because they're watching it, but because I can't. I really want to follow it, so I keep stealing glances, but eventually I give up and concentrate on my rover.

When Chris has finished and we're ready to hand over, I don't see much I would change. I tell him I'm going to slightly change the tolerance on one of the waypoints, just so I'll get my stink on the sequence.

"Sorry," he says. "Next time I'll try to leave you a less complete sequence. Or deliberately put some bugs in it."

So I tell him about Professor DeJong's theorem prover. This was for an AI class I took at the University of Illinois at Urbana-Champaign. DeJong gave us a series of projects that centered on this LISP-based theorem prover, but the first project was to study its code to find and remove some bugs he'd put in it on purpose.

"That's evil," Chris says.

"Yeah," I reply, "but it was a great way to learn about the thing; you really had to understand it to get the bugs out. And it was a good idea for him, because any bugs you found in it, he could claim were deliberate -- whether they were or not."

One of the activities on the plate is to PANCAM a rock named "Chisel." Mark Adler asks thisol's SOWG chair, John Grant, what's the deal with that rock.

"Well, we're seeing these oddly shaped bi-toned rocks," John says, "and Chisel is one of those."

We've made some bi-toned rocks ourselves -- drilling into them with the RAT tends to leave a dark hole, while the rest of the rock is still covered with lighter-colored dust. "Maybe someone was here before us with a RAT," Adler says, comically cocking an eyebrow.

"Except with Chisel, the dark bits are raised, as if someone had been here with a chisel," John replies. He cocks an eyebrow back. "Hm? Hm?"

Our new uplink process has been running well through most of the extended mission, but today there's a little glitch. RSVP overestimates the time required for blind driving, and since we only have blind driving today, that's causing a problem -- the software models believe that no post-drive science is going to be done. This isn't a huge problem, since the spacecraft will do the right thing, but we'd like the science team to be able to see a more realistic model at the Command Approval Meeting.

It's at moments like this that one asks oneself, "What would Scotty do?" The command durations are recorded in a file, so all I have to do is work out how long each command will actually take and edit the file to contain accurate values, so that the SIE can run it back through the modeling process.

"How long will that take?" asks Emily, the TUL.

"I don't know, we've never done it before," I tell her dramatically. "Let me invent a fix for it -- say, ten minutes?"

If this were really Star Trek, she'd have said, "You've got five!" (Of course, if this were Star Trek, I'd have known she was going to say that, so I'd have told her it would take twenty so she'd say I have ten, which is what I need. Star Trek is complicated.) But she gives me the whole ten minutes, and I get it done.

After that, I don't have much more work to do. I have a brief chat with Chris Lewicki, who says they're estimating the rovers' motors are at maybe 1/3 to 1/2 of their maximum possible lifetime (8-10 million revs out of a lifetime of 20-25 million). By that estimate, they'll live at most two or three times as long as they have to this point.

I also have an interview in the afternoon, this one with Disney. I think about declining, because it's Disney, or, as I like to call them, "pure evil." But I relent, because the interview is going to promote this mission and (more important) space exploration generally. I think it's safe to say that if you wanted to get kids fired up about space exploration, you could involve someone worse than Disney. And that's a lot more important to me than the fact that a company I dislike will also benefit. So I hold my nose and go.

When I show up, I discover that it's a camera interview -- something to do with a new ride at EPCOT (named "Mission: Space," I think). The story of the ride is that you're an astronaut on a training mission -- to Mars. Which is where we come in.

Apparently, Deborah Bass worked with them on developing this ride. When I come in, she's being interviewed along with an Imagineer who worked on the ride (whose name, I learn later, is Sue). She's saying things like, "Disney was a great partner and the Imagineers wanted to make this ride as realistic as possible," or something like that. The interviewer keeps making hand gestures, apparently urging her to add more enthusiasm or something, but I'm sitting behind him and can't see very well.

Also off camera, off to my left, is a Disney Suit. Her name, I learn later, is Marilyn. Her suit is yellow. She's watching the proceedings like a hawk, making sure it's Disney enough, and making sure the interviewees do enough "branding" (e.g., you say "Disney" or "JPL" or "Imagineering," not just "we"). Periodically they'll pause the interview and have an exchange like this:


Interviewer [to Suit]: You happy?

Sue [embarrassed and nervous]: I know, I forgot to say "Imagineering" again, didn't I?

Suit: You know, I think you need to say a little more about the passion around us.


Then they'll restart the interview -- "show" would be a better word -- and Sue will say, "JPL's passion was an inspiration to our Imagineering team ...."

I feel sorry for Sue.

Deborah could do this for a living; she's smooth and articulate, stays on message, looks comfortable. (Later, when she's done and they're setting up for my turn, I tell her I'm just going to do my Deborah Bass impression.) They finish the segment with Deborah and Sue together, then spend a little time interviewing Deborah only.

While they're setting up for the Deborah-only interview, Sue and Marilyn are waiting around for someone to escort them out -- apparently, whatever I say doesn't need on-site Disney Corporation approval, so Sue and the Suit can leave. (I'll be edited out later, no doubt.) I chat with them a little. Apparently, they've only just found out about the Mickey head we made on Humphrey back on sol 55, and they're aching to get a picture of it. I have a feeling this will be prominently displayed somewhere as part of the ride ....

Then it's my turn. The interviewer (Steve) gives me a list of the questions they're going to ask, so I can look it over while they hook me up with a mike and get the sound and lighting set up. Steve reminds me to answer in complete sentences and at least capture the sense of the question in the answer, because the viewer won't hear the question in the final product. So when he says, "What is the most exciting moment of the mission?" you're supposed to reply, "The most exciting moment of the mission was when ...." (Then I promptly screw this up on the first question, but I get it right after that.)

Knowing what questions they're going to ask helps tremendously. Even having a few seconds per question to get the answers in my mind enables me to structure my responses a lot better. Some parts of it don't go well, even so. For example, one question is about what it's like during EDL -- the "six minutes of terror." Here's the truth: I was nervous and anxious, but controllably so. I knew I'd done everything I could do (which amounted to, approximately, nothing plus staying out of the way). But this is not what Steve wants me to say. I'm not sure at first what he does want me to say, but that's not it.

"Could you say something like, 'Every second feels like an hour'?" he finally prods.

"I don't know if I could say that truthfully," I reply.

"Well, don't say anything if it's not true."

"Let me see," I muse, trying to think back to the moment. "What was it like, what was it like ...." Then I realize what it was like, and he's not going to like the answer. I smile brightly yet sardonically for the camera. "It's like when you're waiting to get a test result back from the doctor, and he finally says, 'OK, Mr. Smith, you don't have a disease after all.'"

Lisa Townsend (the JPL Media Relations person here monitoring the interview) and I laugh. Steve laughs politely but seems nonplussed.

"But you won't want to use that," I say.

"We're not going to use that," he nods, and we move on.

Steve has an interesting interview technique. He frames the question as a negative, as in: "What was it like developing and testing these rovers? I'm sure it was really easy, right?" Obviously this is meant to prompt you to respond more energetically: "Developing and testing the rovers, especially on such a tight schedule, was extraordinarily challenging ...."

This is all part of a more general aim: ensuring that we look upbeat and energetic. Think of how Disney wants people to look on this video -- that Disney-happy-happy that borders on mania; that's what he's after. So another thing he does is a kind of weird ongoing pantomime. As you're answering the question, he'll sort of body-English your answer, making hand gestures and facial grimaces to urge you to do it bigger and brighter.

Unluckily for Steve, that's over my limit. I did earlier interviews where I felt compelled to display more enthusiasm than I had, and I felt dirty about it afterward. If there's a difference between that and lying, I don't see it. So I've already decided I'm not faking it for anybody any more, period. I'm plenty excited and enthusiastic about what we're doing, and if it doesn't come across naturally when I'm speaking, then too bad.

Luckily for Steve, some of his questions do get the kind of reaction he wants -- and they get it genuinely. The best of these is when he asks about the most memorable moment on the mission so far, or something like that. "The most exciting moment for me was when we drove from a rock called Adirondack, where we'd been sitting for a couple of weeks, to another rock called White Boat," I say. "It was the first time I'd driven the rover solo, and it was a big deal to me. Later, I went home and tried to get some sleep -- and couldn't. Because a hundred million miles away, on the surface of another planet, there was a robot doing what I told it to." The whole time I'm saying this, Steve is doing a kind of "yeah, yeah!" reaction, as if he'd just hit the home run in the World Series.

So I guess he liked that one. And a few others elicit a similar response, so I guess he's getting what he's after. "You guys are great," he beams afterward. "You're going to get me another job." (He's a contractor, not a Disney employee.)

Despite Steve's odd interview techniques, the guy is affable enough. I'm reflexively suspicious of flattery, so I discount the nice things he says about my interview performance. But he encourages the crew to suggest interview questions, and when they do, he asks them. Which is a nice way to treat people. And he and the cameraman both seem honestly interested and excited about being here. I spend some time with them afterward, explaining the THEMIS images and guiding them through what I know of the Opportunity panorama mounted on the wall behind us. Steve gets Lisa to take his picture with me, twice -- once with us holding up a MER poster, and once with us pointing to the blueberries visible in the Opportunity pan.

"Something to impress my parents," he explains.

2009-06-01

Spirit Sol 147

"Have a homemade cupcake," John says.

I hesitate. "Made by you, or by Helen?"[1]

"Helen."

"Ah, so they're edible." I take one and start eating it. Spice, with vanilla frosting. "Hey, not bad."

"We have a firm rule," he says. "When the smoke detector goes off, dinner is ready."

Thisol is a garden-variety drive. We have to maneuver a little to avoid a largish rock right in front of us, then it's a reasonably straight shot toward the hills.

We're tantalizingly close to breaking the 3km mark. At the CAM, Mark Adler asks me whether we're going to break it with this drive. I do the math. "Best case, we'll cover about 69m thisol," I tell him. "So with the 85cm stutter step at the end, that takes us to ...."

"Two thousand nine hundred ninety-nine point six meters," he finishes. 40cm short -- just 16 inches.

"Think we can scare up another two minutes for driving?"

Not today, we can't. And it probably wouldn't matter anyway; the numbers we're using are optimistic. If autonav finds a single obstacle to avoid -- a likely enough event, in this terrain -- we'll fall even shorter of the mark. But we'll be set up to cross the line nextersol. And I'll be here tomorrow, so whether it happens now or then is all the same to me.

For most of the long drive from Bonneville to the hills, we've been able to meander a little, and that helped a lot when avoiding larger obstacles such as impassable ridges. It didn't matter if our drive direction was off by five degrees this way, or ten degrees that way, as long as we were trending in the right direction. Now that we're getting so close to the hills, though, we're going to lose some of that flexibility. Indeed, the scientists are working to choose a specific target point for us at the base of the hills.

But they haven't chosen it yet. Andy, writing his mission manager report, asks Mark Adler about this. Mark shrugs. "John Grant says that if you ask ten scientists where we're going, you'll get ten answers. Actually, that's not too bad; you usually get eleven or twelve."




Footnotes:

[1] His fiancée.