I checked my email at 8:30 last night -- still no information on today's schedule. All I know is the time we expect downlink to start (6:45 AM), so I add the usual amount of time to that and estimate that I should be there at 9:15 or 9:30 AM.
Two hours after I last checked my email -- at 10:30 last night, fer Chrissakes -- they sent out today's schedule. And the schedule says they're doing an unusually compressed sol, so I should be there at something like 8:00 AM. Not 9:30. 8:00.
Thanks so much.
So I'm late. By the time I arrive, they're way past the Activity Plan Approval Meeting -- they've already finished the master/submaster walkthrough. I just have time to glance over the sequences before delivering them. I also show Chris how to do the delivery, just in case this happens again. (Though I don't think it will: mustering all the meager subtlety at my command, I point out the problem to the responsible parties. Subtle or not (answer: not), the message gets through.)
While I'm trying to figure out what the hell happened, Ashitey brings some guests by. I'm a little distracted, but I try my best to switch modes and give them the usual this-is-how-we-drive-the-rover spiel.
On the bright side, I still have my health.
[Next post: sol 147, June 1.]
2009-05-28
2009-05-27
Spirit Sol 142
Of course, they've corrected the anomaly. Once again, it was not my fault. Surprisingly, it was another race condition, this one in the imaging subsystem: a command to take an image was dispatched at almost the same instant as a command to stop imaging. This caused the spacecraft to try to write to the camera interface hardware when the interface hardware was turned off, which triggers a self-protective reboot. The window of vulnerability was all of an eighth of a second wide, which helps explain why we hadn't seen this before -- it's another one of those one-in-a-million (well, maybe one-in-a-few-hundred) things.
In his Mission Manager Report, summarizing the anomaly and the resulting investigation, Mark Adler wrote:
Then, in the "Concerns" section of his report, Adler wrote:
Well, the rover drivers on shift (John and Ashitey) had a few easy days. Due to the anomaly, they were off for at least one of those days, and then spent yesterday and the day before (sols 140 and 141) just re-running sequences that had already been built for sols 137 and 138 -- doing the stuff that didn't happen because of the anomaly. Today we're all caught up from the anomaly and are driving away -- a simple drive, if a short one.
Art, as usual, is spending his time kidding around. With a mix of rue and pride, he describes the Pathfinder-era Thompson Loop -- the combination of a sequencing error (in a sequence Art wrote) and flight software behavior that seemed it might cause us to lose the Sojourner rover. Since both the last Spirit anomaly and this one happened to occur on sols Art planned -- and, in each case, his last day on-shift before going off-shift (just like today) -- he tells Mark Adler to expect the TWA (Thompson Weekly Anomaly) to occur tomorrow. "Keep that in mind," Adler says, "someday HQ is going to ask us how do we kill these things." "Put Thompson on it," several people say at once.
In his Mission Manager Report, summarizing the anomaly and the resulting investigation, Mark Adler wrote:
"This was the second 'very low probability' fatal in less than
a week. After verifying the theory, the flight software team
then went out to buy lottery tickets."
Then, in the "Concerns" section of his report, Adler wrote:
"Concerns: I'm not sure that the flight software guys are
clear on the 'very low probability' concept."
Well, the rover drivers on shift (John and Ashitey) had a few easy days. Due to the anomaly, they were off for at least one of those days, and then spent yesterday and the day before (sols 140 and 141) just re-running sequences that had already been built for sols 137 and 138 -- doing the stuff that didn't happen because of the anomaly. Today we're all caught up from the anomaly and are driving away -- a simple drive, if a short one.
Art, as usual, is spending his time kidding around. With a mix of rue and pride, he describes the Pathfinder-era Thompson Loop -- the combination of a sequencing error (in a sequence Art wrote) and flight software behavior that seemed it might cause us to lose the Sojourner rover. Since both the last Spirit anomaly and this one happened to occur on sols Art planned -- and, in each case, his last day on-shift before going off-shift (just like today) -- he tells Mark Adler to expect the TWA (Thompson Weekly Anomaly) to occur tomorrow. "Keep that in mind," Adler says, "someday HQ is going to ask us how do we kill these things." "Put Thompson on it," several people say at once.
2009-05-23
Spirit Sol 138
We're planning sol 138, even though sol 137 hasn't started yet. More precisely, it's sol 137 on Mars, but it's too early for Spirit to operate -- she's still sleeping. Jeng has generated initial conditions files that show what the rover's state should be after sol 137, and we'll base the sequences on those. In a little while, Spirit will awake, and they'll uplink the sol 137 sequences. If anything goes wrong, we'll have to adapt on the fly.
Confused yet? You're in good company.
Jeng's back from Montreal, and he tells me my presentation nearly got canceled. Apparently, someone didn't get the word that John was going to present it for me -- they just saw that I wasn't there, and they canceled it. Fortunately, Jeng found out about this and straightened it out.
"How was the conference?" I ask.
"Eh, okay. It was mostly about satellite operations and stuff," he says. "I liked Montreal, but technically speaking, you didn't miss much. The next one is in Rome, though."
Rome? They do SpaceOps every two years; given two years, I'm sure I can come up with something to present, even if it's just to update the presentations they just finished giving in Montreal. "I wouldn't mind going to Rome in two years. Let's see, if I start on the paperwork now ...." Then again, maybe I won't.[1]
Thisol's plan is to finish poking at the trench, then stow the IDD and back up 85cm to MTES it. Chris is working on the IDD stuff and asks me if I'd mind sequencing the drive.
"What, the 85cm backup?" This is one command, plus some boilerplate. I grimace with mock concern as he laughs. "Well, I'll see if I can handle it."
It does turn out to be a little more interesting than that. Our downlink hasn't been very good the last few sols -- when we're working in a trench, we've got to face the trench; we're not free to turn to the best comm direction. But we're finishing up at the trench, leaving us free to turn at the end of thisol. Julie asks if we'd mind building a second simple drive sequence to do the turn, with the idea that they'll run this turn sequence after the backup and the MTES. That's trivial to do, but it seems like needless complexity to me. We can just do the turn immediately after the backup, all in the same sequence, and then do the MTES -- as long as the rover's own body won't obscure the trench when we turn. I simulate it quickly in RSVP, and it looks to me like we're clear. Laura Mehall, thisol's MTES PUL, agrees with me, so we go with that.
So the big drive sequence turned out to be two commands plus boilerplate. Hey, I just doubled my productivity. W00t.
Meanwhile, Julie's stressing because we don't have a Sequence Integration Engineer in yet. The SIE needs to be here to build the sol's master and submaster sequences for the nominal case, so they usually sit in on the Activity Plan Approval Meeting to see the sol's structure. We'll also need an SIE to help build the sequences we'll use to diagnose problems if our downlink shows there's something wrong with the rover. (Not likely, of course, but there's a little extra concern about that today because we didn't get yesterday's afternoon Odyssey pass. Which might be because Odyssey was low on the horizon from the rover's perspective, and we just got too much terrain interference -- or might be because we have a problem and don't know it yet.) She calls the SIE on duty (Marc Pack) but gets no answer, then calls Arthur Amador to alert him to the potential problem.
When she hangs up with Arthur, she talks it over with Scott Doudrick. "We don't really need him until we don't get the comm beep," she says.
"Yeah, but I don't want to get caught with my pants down," Scott says.
"Good point ... well, Opportunity's team should be here; maybe we can steal their SIE in an emergency."
The Activity Plan Approval meeting starts right on schedule -- and moments later, Marc walks in nonchalantly. So much for that problem.
Chris leaves, and I start doing my usual RP-2 thing, polishing the IDD sequences and whatnot. I must be having fun, because time flies -- next thing I know, we're starting the master/submaster walkthrough. This is where we walk through the penultimate versions of the master/submaster sequences, as well as the rover planner sequences, making it Marc's and my show. Marc starts leading the team through the day's master sequence, and it's maybe thirty seconds later when Scott Doudrick comes in with a deeply worried expression.
"We didn't get the comm beep," he says. They uplinked the sol 137 sequences anyway (sometimes failing to receive the comm beep just means there's a minor problem at the DSN, and we'll be able to salvage the sol), but the odds are, this means tomorrow's off. They'll probably have to spend the sol trying to debug the problem and getting the rover back up to snuff. Assuming it's even alive, which we can't prove at the moment. So Julie calls a halt to the meeting.
"Damn," says Marc, "just as I was really starting to build up some steam."
The news goes from bad to worse. Sol 137 was not received on the rover (we would have gotten acknowledgment of this) and so its sequences did not execute. At best, this means we'll delay everything one sol. None of the expected comm sequences appears to have executed on the rover, either. The working assumption is that a sequencing error of some kind caused all of the sequences to be killed, including the ones that set up the comm sequences for the sol. The next thing they try is "pinging" the rover at every valid comm rate, to figure out whether it's listening at all. With a 20-minute light-time delay, this means it's 40 minutes before we can know anything new.
I'm not the only one who's become more relaxed about these problems. In fact, looking around, I see that people are tense, but the responses are healthy: there's more laughter than panic. Jason Fox "confesses" that it's all his fault: he jinxed the sol by planning to go to the beach this afternoon.
A little while later, Scott Doudrick comes back in, looking a little better now. "Carrier in lock," he says, "we got the first beep."
"That's the fault rate?" Julie asks.
"That's the fault rate," he nods. "We're in some fault state -- but we're alive."
Good old Spirit. Drama queen.
"All right," Julie says briskly, all business. "We have a fault state of some sort, so stop planning. Science, you may go home: we're not planning any science on sol 138."
She tells me I can go home, too. "Thanks, but I've got other work to do, so I'm going to stick around a while and do it. Besides, I can't leave without finding out what the problem is."
"Yeah, I know," she grins. "It's like walking out of a mystery movie."
"Besides," I continue, "some part of me imagines I'll actually be able to help. Like, 'Oh, we could drive our way out of this fault, if only we had a rover driver around.'"
I'm not the only one with that irrational response. Everyone has it; it's natural. When a problem like this occurs, people start accreting, like worried family outside an operating room. Mostly, we stand around with worried expressions, continuously rehashing what we know and trying to work out what the problem might be.
Andy does it, too. Julie brings him up to speed.
"It could be another one of those one-time things," Andy says. He doesn't sound hopeful, and I don't blame him: we just had "one of those one-time things" the other day, the race condition that interrupted our backward drive. Strictly speaking, that doesn't have any bearing on the odds that we're seeing one today, but it doesn't feel right.
Just then I'm about to send out some email, and I notice the quote the system has randomly chosen for my email signature: "If [space travel] were [risk-free], it wouldn't be glorious." That's William Gibson, writing on the Columbia disaster. I'm not sure whether, if I were superstitious, I'd see that as a bad omen or a good one.
The SMSA is the place to be when these things are happening, so I give up on trying to get work done and go hang out there. They've already got a working theory: the spacecraft fataled for some reason during the sol. When this happens, the spacecraft considers the state of the HGA encoders unknown and switches to a low data rate -- which is what we see. We should get downlink in 13 minutes with EVRs saying what happened to the vehicle.
Mark Maimone walks through, trailing a couple of visitors -- his parents, I think. He sees right away that something's wrong. "Were we driving?" he asks me, sotto voce. "No," I tell him, "it's not your fault." He looks relieved and moves on.
The time for the expected downlink comes and goes. This is not encouraging. We'll learn nothing more today. Not quite what I was expecting at the end of this particular mystery movie: a big placard reading, "To be continued ...."
[Next post: sol 142, May 27.]
Footnotes:
[1] Yeah. Didn't. Sigh.
Confused yet? You're in good company.
Jeng's back from Montreal, and he tells me my presentation nearly got canceled. Apparently, someone didn't get the word that John was going to present it for me -- they just saw that I wasn't there, and they canceled it. Fortunately, Jeng found out about this and straightened it out.
"How was the conference?" I ask.
"Eh, okay. It was mostly about satellite operations and stuff," he says. "I liked Montreal, but technically speaking, you didn't miss much. The next one is in Rome, though."
Rome? They do SpaceOps every two years; given two years, I'm sure I can come up with something to present, even if it's just to update the presentations they just finished giving in Montreal. "I wouldn't mind going to Rome in two years. Let's see, if I start on the paperwork now ...." Then again, maybe I won't.[1]
Thisol's plan is to finish poking at the trench, then stow the IDD and back up 85cm to MTES it. Chris is working on the IDD stuff and asks me if I'd mind sequencing the drive.
"What, the 85cm backup?" This is one command, plus some boilerplate. I grimace with mock concern as he laughs. "Well, I'll see if I can handle it."
It does turn out to be a little more interesting than that. Our downlink hasn't been very good the last few sols -- when we're working in a trench, we've got to face the trench; we're not free to turn to the best comm direction. But we're finishing up at the trench, leaving us free to turn at the end of thisol. Julie asks if we'd mind building a second simple drive sequence to do the turn, with the idea that they'll run this turn sequence after the backup and the MTES. That's trivial to do, but it seems like needless complexity to me. We can just do the turn immediately after the backup, all in the same sequence, and then do the MTES -- as long as the rover's own body won't obscure the trench when we turn. I simulate it quickly in RSVP, and it looks to me like we're clear. Laura Mehall, thisol's MTES PUL, agrees with me, so we go with that.
So the big drive sequence turned out to be two commands plus boilerplate. Hey, I just doubled my productivity. W00t.
Meanwhile, Julie's stressing because we don't have a Sequence Integration Engineer in yet. The SIE needs to be here to build the sol's master and submaster sequences for the nominal case, so they usually sit in on the Activity Plan Approval Meeting to see the sol's structure. We'll also need an SIE to help build the sequences we'll use to diagnose problems if our downlink shows there's something wrong with the rover. (Not likely, of course, but there's a little extra concern about that today because we didn't get yesterday's afternoon Odyssey pass. Which might be because Odyssey was low on the horizon from the rover's perspective, and we just got too much terrain interference -- or might be because we have a problem and don't know it yet.) She calls the SIE on duty (Marc Pack) but gets no answer, then calls Arthur Amador to alert him to the potential problem.
When she hangs up with Arthur, she talks it over with Scott Doudrick. "We don't really need him until we don't get the comm beep," she says.
"Yeah, but I don't want to get caught with my pants down," Scott says.
"Good point ... well, Opportunity's team should be here; maybe we can steal their SIE in an emergency."
The Activity Plan Approval meeting starts right on schedule -- and moments later, Marc walks in nonchalantly. So much for that problem.
Chris leaves, and I start doing my usual RP-2 thing, polishing the IDD sequences and whatnot. I must be having fun, because time flies -- next thing I know, we're starting the master/submaster walkthrough. This is where we walk through the penultimate versions of the master/submaster sequences, as well as the rover planner sequences, making it Marc's and my show. Marc starts leading the team through the day's master sequence, and it's maybe thirty seconds later when Scott Doudrick comes in with a deeply worried expression.
"We didn't get the comm beep," he says. They uplinked the sol 137 sequences anyway (sometimes failing to receive the comm beep just means there's a minor problem at the DSN, and we'll be able to salvage the sol), but the odds are, this means tomorrow's off. They'll probably have to spend the sol trying to debug the problem and getting the rover back up to snuff. Assuming it's even alive, which we can't prove at the moment. So Julie calls a halt to the meeting.
"Damn," says Marc, "just as I was really starting to build up some steam."
The news goes from bad to worse. Sol 137 was not received on the rover (we would have gotten acknowledgment of this) and so its sequences did not execute. At best, this means we'll delay everything one sol. None of the expected comm sequences appears to have executed on the rover, either. The working assumption is that a sequencing error of some kind caused all of the sequences to be killed, including the ones that set up the comm sequences for the sol. The next thing they try is "pinging" the rover at every valid comm rate, to figure out whether it's listening at all. With a 20-minute light-time delay, this means it's 40 minutes before we can know anything new.
I'm not the only one who's become more relaxed about these problems. In fact, looking around, I see that people are tense, but the responses are healthy: there's more laughter than panic. Jason Fox "confesses" that it's all his fault: he jinxed the sol by planning to go to the beach this afternoon.
A little while later, Scott Doudrick comes back in, looking a little better now. "Carrier in lock," he says, "we got the first beep."
"That's the fault rate?" Julie asks.
"That's the fault rate," he nods. "We're in some fault state -- but we're alive."
Good old Spirit. Drama queen.
"All right," Julie says briskly, all business. "We have a fault state of some sort, so stop planning. Science, you may go home: we're not planning any science on sol 138."
She tells me I can go home, too. "Thanks, but I've got other work to do, so I'm going to stick around a while and do it. Besides, I can't leave without finding out what the problem is."
"Yeah, I know," she grins. "It's like walking out of a mystery movie."
"Besides," I continue, "some part of me imagines I'll actually be able to help. Like, 'Oh, we could drive our way out of this fault, if only we had a rover driver around.'"
I'm not the only one with that irrational response. Everyone has it; it's natural. When a problem like this occurs, people start accreting, like worried family outside an operating room. Mostly, we stand around with worried expressions, continuously rehashing what we know and trying to work out what the problem might be.
Andy does it, too. Julie brings him up to speed.
"It could be another one of those one-time things," Andy says. He doesn't sound hopeful, and I don't blame him: we just had "one of those one-time things" the other day, the race condition that interrupted our backward drive. Strictly speaking, that doesn't have any bearing on the odds that we're seeing one today, but it doesn't feel right.
Just then I'm about to send out some email, and I notice the quote the system has randomly chosen for my email signature: "If [space travel] were [risk-free], it wouldn't be glorious." That's William Gibson, writing on the Columbia disaster. I'm not sure whether, if I were superstitious, I'd see that as a bad omen or a good one.
The SMSA is the place to be when these things are happening, so I give up on trying to get work done and go hang out there. They've already got a working theory: the spacecraft fataled for some reason during the sol. When this happens, the spacecraft considers the state of the HGA encoders unknown and switches to a low data rate -- which is what we see. We should get downlink in 13 minutes with EVRs saying what happened to the vehicle.
Mark Maimone walks through, trailing a couple of visitors -- his parents, I think. He sees right away that something's wrong. "Were we driving?" he asks me, sotto voce. "No," I tell him, "it's not your fault." He looks relieved and moves on.
The time for the expected downlink comes and goes. This is not encouraging. We'll learn nothing more today. Not quite what I was expecting at the end of this particular mystery movie: a big placard reading, "To be continued ...."
[Next post: sol 142, May 27.]
Footnotes:
[1] Yeah. Didn't. Sigh.
2009-05-22
Spirit Sol 137
This not-living-on-Mars-time thing is weird. RP-1s used to show up at 15:00 rover-time, RP-2s at 22:00 rover-time. I'm here as RP-2 today (Ashitey is RP-1 -- filling in for Chris, who has a visitor), and it's only 09:16 rover-time. It works out because the rover hasn't moved since yestersol (that's what restricted sols are for), and we used the time to get imagery we can use for planning thisol. But it feels so unnatural.
Ashitey has done about a third of the sequencing by the time I arrive, and he works on the second third while I tackle the, uh, third third.
Ashitey also conducts an intense series of cell phone conversations with Eric Baumgartner. Since winter is approaching, bringing shorter days and colder temperatures, our poor solar-powered rovers have to start making do with less energy. To help with this, Eric and others have worked out that we don't have to have the IMU on while using the IDD. The IMU -- Inertial Measurement Unit -- keeps track of changes in the rover's attitude (that is, its roll, pitch, and yaw). For example, if the rover is heading straight down a gentle hill, the IMU might report that we were pitched nose-down at five degrees. Well, there was concern that the rover's attitude might change when it extended the IDD -- doing this changes the vehicle's center of mass slightly, and could cause us to pitch forward a little, for instance. So to keep track of these changes, we've had a rule that we must turn on the IMU when doing any IDD motion. But the IMU is, relatively speaking, a power hog, and measurements have shown that the original concern was overstated. So the rules have changed: under most circumstances, we can leave the IMU off and save some power. So Ashitey was going to leave the IMU off during thisol's IDD sequences.
That, at least, was the plan. To set up for that, we already set some parameters on the rover that tell it not to worry that the IMU is off when the IDD is moving, and we got no flak about setting those parameters because the sequence that set them had already been run on Opportunity. But it turns out today that Opportunity hasn't actually used the IDD since setting the parameters this way -- so, in other words, the effect of the new settings hasn't been tested on the flight vehicles. They've sort of been tested on the testbed vehicles -- only one of them ever had an IMU, and that IMU was removed some time ago. But it's not quite the same thing, and after some discussion, Art wimps out. Because of our current weird timing, we'd lose two or more sols if something goes wrong with this. "We don't want to be the guinea pig on this one," he says.
"Well, how about leaving the IMU off for the overnight sequence only?" Ashitey proposes. It's a reasonable suggestion: this way, the sol-137 daytime IDD sequences would still succeed; only the overnight sequence could possibly fail.
Art muses, frowns, shakes his head. "No, we'd still lose that observation, which we'd have to redo," he points out. "Plus, we'd lose sol 138. It's not worth it."
The IMU commands go back in. Looks like Opportunity will have to be the risk-taker this time.
Meanwhile, I'm having some fun writing an IDD sequence for the first time since -- well, I can't remember. It's been long enough that I've almost forgotten how to do it, or so it seems at first. Turns out, though, that it's like riding a bike: you have to keep pedaling or you'll fall off and hurt yourself. Or something like that. I've kind of missed it.
The IDD work we're doing is all to explore a trench we dug the other sol. One of the targets the scientists want us to poke at is a spot called "Mill Basin," located on the trench's floor. When we show the animation at the Activity Plan Approval Meeting, Art doesn't like the instrument placement on that target. Not our fault, it just puts the instruments a little too close to the trench wall for his taste.
So we work with the scientists to choose another target that's a little closer to the middle of the trench floor, and when we place the instruments there, we have a new problem. When we switch from the MB to the APXS, we normally place the APXS 4mm closer to the soil than the last position of the MB. We have the MB touch the soil, but the APXS is about 8mm shorter, so we split the difference, leaving the APXS slightly off the soil (we don't want the APXS to touch the soil, as that can mess up the instrument permanently) but close enough to still get good data. (Originally, we didn't overdrive at all, and the 8mm standoff significantly reduced the quality of the APXS data we gathered in the early part of the mission.) It so happens with this new target that the usual 4mm overdrive is just a little too much -- it causes the IDD's elbow joint to extend just a tiny amount beyond its limit. Ashitey and I work out that 3mm is under the limit -- just barely -- and 4mm is over.
Seems like the solution is obvious -- just back off to 3mm. But we don't want to compromise the quality of the APXS data. Another concern is that the 3mm overdrive still leaves us awfully close to the elbow joint's limit, and we're not sure whether we need to leave more margin. The good news is that we probably won't actually come close to the limit -- the simulation is assuming the trench is about 1cm deeper than it probably is, so the position we actually contact will probably be 1cm higher, and we won't come close to the joint limit at all. But we're not sure, and our job is to make sure this stuff works even in the worst case.
Argh. Did I say I've missed this?
So I call Rudi Reider, thisol's APXS PUL, and ask him about the quality difference between 3mm and 4mm. (Rudi, you'll remember, is the German fellow who wears both belts and suspenders.) "You do what you can do technically, because we can work with almost anything," he says. One problem down, one to go.
Art asks me to check into the joint-limit problem with an IDD expert, such as Eric Baumgartner or Bob Bonitz ("not that we don't trust you ..."). I don't see Eric, but to my surprise, Bob's in his office. I describe the problem to him. Bob complains good-naturedly that they cut off his charge number for dealing with this stuff, then thinks, mutters, swears, flips through the flight software source code, and says it will be OK at the 3mm standoff position. The limit in question isn't the joint's physical limit, it's a limit built into the software, so we don't have to worry about the motors stalling as they try to push against the hardstops, or anything like that. As long as we're not commanding the joint past the soft limit, and we're not, the sequence will be fine.
"What would you do?" I ask him.
"Probably just exactly what you did," he says. High praise!
I go back downstairs to finish up the sequence, and Bob shows up a few minutes later to let me know the picture is actually a little better than we'd thought -- the software allows for a little bit of slop, which translates into a little extra margin for this move.
"Thanks, Angry Bob!"
He corrects me: "Frustrated Bob."
In addition to our visit from Frustrated Bob, we're visited by Rob Manning. Rob doesn't work on MER any more, but he stops by once in a while to see how things are going.
"I heard you set the distance record for Spirit," he says, shaking my hand. (That was a while back, but what the hell.) "Congratulations!"
We get to talking about our recent backward drives, and Rob, a veteran of Pathfinder's rover team, points out something I never noticed. The MER rovers use the same rocker/bogie suspension as Sojourner, of course, but -- the part I never noticed -- the suspension was front-to-back reversed on MER. No big deal, as it's constructed to work either way, but it's funny that I'd never heard this or observed it myself until now.
"So Sojourner was driving backward all the time," I say.
"Sojourner would say we're always driving backward," he says.
Rob and Art gab a while about MER and MSL. One thing they agree on is that Cassini will start stealing our thunder (what thunder we have left) when they get to Saturn. "We've gotta make it to those hills before SOI[1]," says Rob. "Well, we're running ahead of schedule," says Art.
"Where will we go after the hills?" Rob asks.
Art says another good target would be the wall of Gusev Crater itself. "We've got two years to get there."
"Two Earth years?"
"Yeah, about -- assuming the rovers can survive that long. If the rovers can survive this winter solstice, that's how long they've got until the next winter solstice on Mars." But he shakes his head. "I don't even want to think about the rovers lasting that long."
I have a sudden vision of myself as a doddering old man, still nursing Spirit around Mars, a few meters at a time .... Well, there are worse ways to spend your life.
Art also relays a bit of gossip. From what he says, NASA HQ does not want another solar-powered rover on Mars. Future rovers will use RTGs[2], and HQ is willing to fight the public battle that's sure to ensue.
That's a good thing, as it will maximize the value of the rovers. We could have gotten a lot more science done with the power improvements (and mass savings) RTGs provide. But, like so many decisions, that one might change with the presidential election. "We'll know in five months whether we have to redesign the things," Art says.
Art still intends to stay on MER rather than jump ship to MSL. When he says so, Rob says, "Well, that's a shame."
"You're saying you want me to work on the project?"
"I think you'd bring a lot to MSL."
Art feigns wounded shock. "I thought you loved me for who I am!"
Rob's lips quirk as he answers tactfully. "I ... appreciate you for who you are."
[1] Saturn Orbit Insertion.
[2] Radioisotope Thermoelectric Generators. You hook a chunk of a radioactive rock, such as plutonium, to a device called a thermocouple, which converts the heat from the rock's radioactive decay into electricity that powers your spacecraft. They're about a million times less dangerous than your car, but some people freak out whenever they hear about a spacecraft making energy from a radioactive rock, because they see it on the launch pad and get visions of mushroom clouds wiping out half of Florida. (Yes, some people do think that would be a bad thing.)
My own personal view is that the best way to sell the use of RTGs to these people is to stress that we're getting that scary old plutonium safely off of this planet and putting it where it can't hurt them any more. Maybe that tack wouldn't work in the real world as well as it does in my imagination, though.
Ashitey has done about a third of the sequencing by the time I arrive, and he works on the second third while I tackle the, uh, third third.
Ashitey also conducts an intense series of cell phone conversations with Eric Baumgartner. Since winter is approaching, bringing shorter days and colder temperatures, our poor solar-powered rovers have to start making do with less energy. To help with this, Eric and others have worked out that we don't have to have the IMU on while using the IDD. The IMU -- Inertial Measurement Unit -- keeps track of changes in the rover's attitude (that is, its roll, pitch, and yaw). For example, if the rover is heading straight down a gentle hill, the IMU might report that we were pitched nose-down at five degrees. Well, there was concern that the rover's attitude might change when it extended the IDD -- doing this changes the vehicle's center of mass slightly, and could cause us to pitch forward a little, for instance. So to keep track of these changes, we've had a rule that we must turn on the IMU when doing any IDD motion. But the IMU is, relatively speaking, a power hog, and measurements have shown that the original concern was overstated. So the rules have changed: under most circumstances, we can leave the IMU off and save some power. So Ashitey was going to leave the IMU off during thisol's IDD sequences.
That, at least, was the plan. To set up for that, we already set some parameters on the rover that tell it not to worry that the IMU is off when the IDD is moving, and we got no flak about setting those parameters because the sequence that set them had already been run on Opportunity. But it turns out today that Opportunity hasn't actually used the IDD since setting the parameters this way -- so, in other words, the effect of the new settings hasn't been tested on the flight vehicles. They've sort of been tested on the testbed vehicles -- only one of them ever had an IMU, and that IMU was removed some time ago. But it's not quite the same thing, and after some discussion, Art wimps out. Because of our current weird timing, we'd lose two or more sols if something goes wrong with this. "We don't want to be the guinea pig on this one," he says.
"Well, how about leaving the IMU off for the overnight sequence only?" Ashitey proposes. It's a reasonable suggestion: this way, the sol-137 daytime IDD sequences would still succeed; only the overnight sequence could possibly fail.
Art muses, frowns, shakes his head. "No, we'd still lose that observation, which we'd have to redo," he points out. "Plus, we'd lose sol 138. It's not worth it."
The IMU commands go back in. Looks like Opportunity will have to be the risk-taker this time.
Meanwhile, I'm having some fun writing an IDD sequence for the first time since -- well, I can't remember. It's been long enough that I've almost forgotten how to do it, or so it seems at first. Turns out, though, that it's like riding a bike: you have to keep pedaling or you'll fall off and hurt yourself. Or something like that. I've kind of missed it.
The IDD work we're doing is all to explore a trench we dug the other sol. One of the targets the scientists want us to poke at is a spot called "Mill Basin," located on the trench's floor. When we show the animation at the Activity Plan Approval Meeting, Art doesn't like the instrument placement on that target. Not our fault, it just puts the instruments a little too close to the trench wall for his taste.
So we work with the scientists to choose another target that's a little closer to the middle of the trench floor, and when we place the instruments there, we have a new problem. When we switch from the MB to the APXS, we normally place the APXS 4mm closer to the soil than the last position of the MB. We have the MB touch the soil, but the APXS is about 8mm shorter, so we split the difference, leaving the APXS slightly off the soil (we don't want the APXS to touch the soil, as that can mess up the instrument permanently) but close enough to still get good data. (Originally, we didn't overdrive at all, and the 8mm standoff significantly reduced the quality of the APXS data we gathered in the early part of the mission.) It so happens with this new target that the usual 4mm overdrive is just a little too much -- it causes the IDD's elbow joint to extend just a tiny amount beyond its limit. Ashitey and I work out that 3mm is under the limit -- just barely -- and 4mm is over.
Seems like the solution is obvious -- just back off to 3mm. But we don't want to compromise the quality of the APXS data. Another concern is that the 3mm overdrive still leaves us awfully close to the elbow joint's limit, and we're not sure whether we need to leave more margin. The good news is that we probably won't actually come close to the limit -- the simulation is assuming the trench is about 1cm deeper than it probably is, so the position we actually contact will probably be 1cm higher, and we won't come close to the joint limit at all. But we're not sure, and our job is to make sure this stuff works even in the worst case.
Argh. Did I say I've missed this?
So I call Rudi Reider, thisol's APXS PUL, and ask him about the quality difference between 3mm and 4mm. (Rudi, you'll remember, is the German fellow who wears both belts and suspenders.) "You do what you can do technically, because we can work with almost anything," he says. One problem down, one to go.
Art asks me to check into the joint-limit problem with an IDD expert, such as Eric Baumgartner or Bob Bonitz ("not that we don't trust you ..."). I don't see Eric, but to my surprise, Bob's in his office. I describe the problem to him. Bob complains good-naturedly that they cut off his charge number for dealing with this stuff, then thinks, mutters, swears, flips through the flight software source code, and says it will be OK at the 3mm standoff position. The limit in question isn't the joint's physical limit, it's a limit built into the software, so we don't have to worry about the motors stalling as they try to push against the hardstops, or anything like that. As long as we're not commanding the joint past the soft limit, and we're not, the sequence will be fine.
"What would you do?" I ask him.
"Probably just exactly what you did," he says. High praise!
I go back downstairs to finish up the sequence, and Bob shows up a few minutes later to let me know the picture is actually a little better than we'd thought -- the software allows for a little bit of slop, which translates into a little extra margin for this move.
"Thanks, Angry Bob!"
He corrects me: "Frustrated Bob."
In addition to our visit from Frustrated Bob, we're visited by Rob Manning. Rob doesn't work on MER any more, but he stops by once in a while to see how things are going.
"I heard you set the distance record for Spirit," he says, shaking my hand. (That was a while back, but what the hell.) "Congratulations!"
We get to talking about our recent backward drives, and Rob, a veteran of Pathfinder's rover team, points out something I never noticed. The MER rovers use the same rocker/bogie suspension as Sojourner, of course, but -- the part I never noticed -- the suspension was front-to-back reversed on MER. No big deal, as it's constructed to work either way, but it's funny that I'd never heard this or observed it myself until now.
"So Sojourner was driving backward all the time," I say.
"Sojourner would say we're always driving backward," he says.
Rob and Art gab a while about MER and MSL. One thing they agree on is that Cassini will start stealing our thunder (what thunder we have left) when they get to Saturn. "We've gotta make it to those hills before SOI[1]," says Rob. "Well, we're running ahead of schedule," says Art.
"Where will we go after the hills?" Rob asks.
Art says another good target would be the wall of Gusev Crater itself. "We've got two years to get there."
"Two Earth years?"
"Yeah, about -- assuming the rovers can survive that long. If the rovers can survive this winter solstice, that's how long they've got until the next winter solstice on Mars." But he shakes his head. "I don't even want to think about the rovers lasting that long."
I have a sudden vision of myself as a doddering old man, still nursing Spirit around Mars, a few meters at a time .... Well, there are worse ways to spend your life.
Art also relays a bit of gossip. From what he says, NASA HQ does not want another solar-powered rover on Mars. Future rovers will use RTGs[2], and HQ is willing to fight the public battle that's sure to ensue.
That's a good thing, as it will maximize the value of the rovers. We could have gotten a lot more science done with the power improvements (and mass savings) RTGs provide. But, like so many decisions, that one might change with the presidential election. "We'll know in five months whether we have to redesign the things," Art says.
Art still intends to stay on MER rather than jump ship to MSL. When he says so, Rob says, "Well, that's a shame."
"You're saying you want me to work on the project?"
"I think you'd bring a lot to MSL."
Art feigns wounded shock. "I thought you loved me for who I am!"
Rob's lips quirk as he answers tactfully. "I ... appreciate you for who you are."
[1] Saturn Orbit Insertion.
[2] Radioisotope Thermoelectric Generators. You hook a chunk of a radioactive rock, such as plutonium, to a device called a thermocouple, which converts the heat from the rock's radioactive decay into electricity that powers your spacecraft. They're about a million times less dangerous than your car, but some people freak out whenever they hear about a spacecraft making energy from a radioactive rock, because they see it on the launch pad and get visions of mushroom clouds wiping out half of Florida. (Yes, some people do think that would be a bad thing.)
My own personal view is that the best way to sell the use of RTGs to these people is to stress that we're getting that scary old plutonium safely off of this planet and putting it where it can't hurt them any more. Maybe that tack wouldn't work in the real world as well as it does in my imagination, though.
2009-05-21
Spirit Sol 136
I'm on shift, but I have nothing to do. Really nothing: we're in that weird time-phasing mode again, where Earth and Mars times don't align very well for our purposes. As a result, this is a restricted sol, meaning the rover planners have nothing to do. It's all remote science -- taking pictures and MTES observations. No driving, no IDD.
I take the opportunity to catch up on random backlogged junk -- reading email, updating some scripts, you name it. If every day were this interesting, I'd shoot myself.
I take the opportunity to catch up on random backlogged junk -- reading email, updating some scripts, you name it. If every day were this interesting, I'd shoot myself.
2009-05-20
Spirit Sol 135
Ashitey's and my schedules got all screwed up by the SpaceOps fiasco, followed by my semi-switching with him yesterday. An email exchange fails to clarify who's on shift today -- both of us were volunteering. Weird; most arguments of that sort involve people trying to get out of work.
Where I think the matter gets left is: Ashitey's working today. But it's not really clear, so I wait until it would be time for him to be there, and then I call Emily Eelkema, thisol's TUL.
"Hey, Emily, is Ashitey there?"
"No ... and we're just about to have our Activity Plan Approval Meeting." She starts to sound concerned. "He'd normally be here by now."
Er ...
"Well, there was some schedule confusion," I tell her. "I'll call him and get back to you. One way or another, I'll make sure you have an RP-2."
Ashitey doesn't answer his cell phone. Oh, goody. So I call Emily and tell her I'll be there in 20 minutes, then I jump into the car and take off.
Twenty minutes later, I walk into the sequencing MSA ... and there's Ashitey, who looks at me in surprise. We both laugh, and I turn around and go home.
Where I think the matter gets left is: Ashitey's working today. But it's not really clear, so I wait until it would be time for him to be there, and then I call Emily Eelkema, thisol's TUL.
"Hey, Emily, is Ashitey there?"
"No ... and we're just about to have our Activity Plan Approval Meeting." She starts to sound concerned. "He'd normally be here by now."
Er ...
"Well, there was some schedule confusion," I tell her. "I'll call him and get back to you. One way or another, I'll make sure you have an RP-2."
Ashitey doesn't answer his cell phone. Oh, goody. So I call Emily and tell her I'll be there in 20 minutes, then I jump into the car and take off.
Twenty minutes later, I walk into the sequencing MSA ... and there's Ashitey, who looks at me in surprise. We both laugh, and I turn around and go home.
2009-05-19
Spirit Sol 134
Ashitey needs to leave early, so he asks me to come in at 7 and handle the CAM for him. I arrive just after Chris hands off to him -- if we'd known, I could have come in a little earlier and Ashitey wouldn't have had to come in at all. Oh, well. At least it's a straightforward drive -- except for some unusual business involving the turn-for-comm at the end, it's in the same mold as everything else we've been doing lately.
"By the way, congratulations on yesterday," Ashitey says.
I just got here. "How did we do yesterday?"
"One hundred and thirteen meters," he says. We shake hands. It's not a record, but it's a good solid sol. Another day, another football field.
We were helped along by a good autonav performance -- our longest autonav ever, in fact, at 78m. At the moment, we're in a less rocky zone, which means autonav has fewer obstacles to avoid, which means it spends more time driving and less time thinking.
Mostly, the day is otherwise normal. The only glitch comes when a script that's an important part of the uplink process turns out to have a bug. I help Cindy Oda work around it. Turns out the script was updated the previous day and wasn't sufficiently tested. Man, I hate those software developers!
"By the way, congratulations on yesterday," Ashitey says.
I just got here. "How did we do yesterday?"
"One hundred and thirteen meters," he says. We shake hands. It's not a record, but it's a good solid sol. Another day, another football field.
We were helped along by a good autonav performance -- our longest autonav ever, in fact, at 78m. At the moment, we're in a less rocky zone, which means autonav has fewer obstacles to avoid, which means it spends more time driving and less time thinking.
Mostly, the day is otherwise normal. The only glitch comes when a script that's an important part of the uplink process turns out to have a bug. I help Cindy Oda work around it. Turns out the script was updated the previous day and wasn't sufficiently tested. Man, I hate those software developers!
2009-05-18
Spirit Sol 133
They found out what caused yestersol's unexpected reboot. It resulted from a bug in the flight software, a type of bug called a "race condition." It happens when two tasks try to use a shared resource in an order the programmer didn't expect. Every once in a while, they interact in an undesirable order -- "racing" for the prize, as it were -- and something goes wrong. In this case, one task running on the rover tried to make a note that the morning bootup should be considered successful. At the same time, another task tried to log a routine note about the spacecraft's operation.
The problem is that both tasks write to a memory area that's normally write-protected, and they don't do it correctly. The first task does it properly -- politely waiting for exclusive access to the shared memory area, unlocking the area, making its note, relocking the area, and then releasing the memory area for use by other tasks. But the second task isn't so polite; it wrongly goes ahead and unlocks/relocks the shared area without first waiting for exclusive access. Normally, this just happens to work -- the vulnerability window is narrow, and we just happen to get away with it. But not this time. This time, our luck ran out, and the bug was exposed: the second task relocked the memory area just before the first task could write to it. Trying to write to a locked (write-protected) memory area causes a fatal error that results in a spacecraft reset.
And what are we going to do about it? Maybe nothing. As Bob Denise, who's filling me in on the details, points out, the two rovers have now been booted a combined total of about 1000 times, and this bug has been exposed only once. To a first approximation, this suggests we'll see this bug once every thousand times we boot the rovers -- so we might well just never see it again.[1] We might create and upload a software patch (which can always introduce unforeseen problems of its own), or we might just live with the apparently small risk of occasionally losing a sol.
In any case, we're proceeding normally thisol. A drive. A good long one, too -- they end up throwing out some science to make more time for it, and it ends up totaling about 3.5 hours of driving. In theory, we could make about 150m, but we'll probably cover more like 100m, possibly less. Still, we'll cover a good chunk of ground, which will help make up for the two days we effectively lost to the anomaly.
The planning for this makes a hash of the SOWG meeting. I'm not really tracking it, because the details they're worrying over don't concern me. Larry Soderblom's the SOWG chair thisol, and he seems to be thrashing a bit as he tries to make sense of all of the science requests, so that he can discard just the right set. I presume he makes the right calls in the end, though I wouldn't want to be in his shoes today.
I haven't been RP-1 in a long time, and today I remember some of the things I haven't missed about it. The main thing is the interruptions -- oh, Lord, the interruptions. I'm trying to evaluate the terrain, choose a path that keeps all parts of the rover's body safe and respects all of its mechanical limits while making efficient progress toward our goal, compensate for limitations in our tools, document my work as I go, make sure I've thought of everything that can possibly go wrong, all of this under time pressure ... and every thirty seconds, someone has an urgent question for me. Are we driving backward again thisol? Is there a parameter we can tweak to cut down on roverdancing? What azimuth should we use for the drive-direction PANCAMs? Have you finished editing the slice file yet? How long are we going to spend on the blind drive? Daddy, why is the sky blue?
I miss a lot of stuff about being an RP-1. But not that part.[2]
I pick out a simple path that takes us more or less where we were going before we were so rudely interrupted. We'll scoot straight ahead about 25m, climbing up onto the next ridge. After that, we'll turn on autonav. But while I'm planning that, Ashitey works out another solution, zig-zagging around some local hollows and cresting the ridge at a lower point off to the right. He shows me his idea and I like it. His solution is better because we can see the terrain beyond the ridge at that point a little more clearly -- with my approach, autonav might get stuck trying to descend the ridge, just as it did a few sols ago, but with Ashitey's, we can have some confidence that autonav will make real progress. The only downside is that his blind-drive segment is shorter, only about 16m, and not as direct. I think about it for a minute and then adopt his path; given the timings, since it improves our chances of a successful autonav, we should make more progress for the day with his solution, even if the first part is less efficient. After looking at it a while, I find a way to extend the blind drive to 34m, making that approach a winner all around.
We were originally planning to do the blind drive backward again. But while reviewing the sequence as part of our handover, Ashitey and I notice an error (which went undetected last time we tried this -- oops). Normally, at the end of the blind drive, we tell the rover to take a couple of pictures of the region in front of it; this helps ensure that it will understand the area it's in at the end of the sol even if the autonav doesn't go anywhere. The problem is, when driving backward, we should tell the rover to take pictures of the area behind the rover, not in front. It's an easy fix, and not a big deal even if we don't fix it, but it makes us nervous about what else we might have overlooked. So we turn the rover around and have it drive forward instead, as it normally does. We already proved our point about driving backward anyway; we made about 14m of progress (regress?) before the anomaly stopped us the other sol. We have little to gain by doing it again, and too much to lose. So we don't.
I also simplify our day in another way. Julie raises a concern about wasting energy roverdancing when the rover isn't going to make any progress anyway. Is there something we can change to make it give up sooner? I suggest replacing the single long-distance autonav waypoint with a series of shorter autonav waypoints, but we'd have to revisit the timeout-handling logic. Maybe there are flight software parameters we could tweak to help out as well, but we'd have to ask Mark Maimone and he's already gone home. And you know what -- we just don't have time to deal with it today. If we've learned anything, it's that we shouldn't try to come up with this stuff under time pressure unless we really have to. So I tell her we'll think about it and not do it today, and she's cool with that. So we don't.
I need to blow off a little steam after this day, so I put something in the uplink report that I've been thinking about for a while:
Anybody who doesn't get it has no business working here anyway, I figure.
Footnotes:
[1] Of course, we have seen that bug again, a few times. We continue to just live with it; it's not a big enough problem to be worth the risks involved in fixing it. I'd like to add, though, that I'm permanently amazed by the team's ability to diagnose these problems -- in this case, by the next day -- from a hundred million miles away. Amazed.
[2] Sometimes it still feels like that, although the protocol is now very much more that it's OK to tell people to bug off and you'll get back to them when you're good and ready. (It helps to not phrase it like that, although some days ....) This is a good change. For a long time, it felt like I was the dad in Vonnegut's story Harrison Bergeron -- the guy who has to wear a device that makes a loud noise in his ear every few seconds, so that he won't be able to unfairly exercise his higher-than-average intelligence.
The problem is that both tasks write to a memory area that's normally write-protected, and they don't do it correctly. The first task does it properly -- politely waiting for exclusive access to the shared memory area, unlocking the area, making its note, relocking the area, and then releasing the memory area for use by other tasks. But the second task isn't so polite; it wrongly goes ahead and unlocks/relocks the shared area without first waiting for exclusive access. Normally, this just happens to work -- the vulnerability window is narrow, and we just happen to get away with it. But not this time. This time, our luck ran out, and the bug was exposed: the second task relocked the memory area just before the first task could write to it. Trying to write to a locked (write-protected) memory area causes a fatal error that results in a spacecraft reset.
And what are we going to do about it? Maybe nothing. As Bob Denise, who's filling me in on the details, points out, the two rovers have now been booted a combined total of about 1000 times, and this bug has been exposed only once. To a first approximation, this suggests we'll see this bug once every thousand times we boot the rovers -- so we might well just never see it again.[1] We might create and upload a software patch (which can always introduce unforeseen problems of its own), or we might just live with the apparently small risk of occasionally losing a sol.
In any case, we're proceeding normally thisol. A drive. A good long one, too -- they end up throwing out some science to make more time for it, and it ends up totaling about 3.5 hours of driving. In theory, we could make about 150m, but we'll probably cover more like 100m, possibly less. Still, we'll cover a good chunk of ground, which will help make up for the two days we effectively lost to the anomaly.
The planning for this makes a hash of the SOWG meeting. I'm not really tracking it, because the details they're worrying over don't concern me. Larry Soderblom's the SOWG chair thisol, and he seems to be thrashing a bit as he tries to make sense of all of the science requests, so that he can discard just the right set. I presume he makes the right calls in the end, though I wouldn't want to be in his shoes today.
I haven't been RP-1 in a long time, and today I remember some of the things I haven't missed about it. The main thing is the interruptions -- oh, Lord, the interruptions. I'm trying to evaluate the terrain, choose a path that keeps all parts of the rover's body safe and respects all of its mechanical limits while making efficient progress toward our goal, compensate for limitations in our tools, document my work as I go, make sure I've thought of everything that can possibly go wrong, all of this under time pressure ... and every thirty seconds, someone has an urgent question for me. Are we driving backward again thisol? Is there a parameter we can tweak to cut down on roverdancing? What azimuth should we use for the drive-direction PANCAMs? Have you finished editing the slice file yet? How long are we going to spend on the blind drive? Daddy, why is the sky blue?
I miss a lot of stuff about being an RP-1. But not that part.[2]
I pick out a simple path that takes us more or less where we were going before we were so rudely interrupted. We'll scoot straight ahead about 25m, climbing up onto the next ridge. After that, we'll turn on autonav. But while I'm planning that, Ashitey works out another solution, zig-zagging around some local hollows and cresting the ridge at a lower point off to the right. He shows me his idea and I like it. His solution is better because we can see the terrain beyond the ridge at that point a little more clearly -- with my approach, autonav might get stuck trying to descend the ridge, just as it did a few sols ago, but with Ashitey's, we can have some confidence that autonav will make real progress. The only downside is that his blind-drive segment is shorter, only about 16m, and not as direct. I think about it for a minute and then adopt his path; given the timings, since it improves our chances of a successful autonav, we should make more progress for the day with his solution, even if the first part is less efficient. After looking at it a while, I find a way to extend the blind drive to 34m, making that approach a winner all around.
We were originally planning to do the blind drive backward again. But while reviewing the sequence as part of our handover, Ashitey and I notice an error (which went undetected last time we tried this -- oops). Normally, at the end of the blind drive, we tell the rover to take a couple of pictures of the region in front of it; this helps ensure that it will understand the area it's in at the end of the sol even if the autonav doesn't go anywhere. The problem is, when driving backward, we should tell the rover to take pictures of the area behind the rover, not in front. It's an easy fix, and not a big deal even if we don't fix it, but it makes us nervous about what else we might have overlooked. So we turn the rover around and have it drive forward instead, as it normally does. We already proved our point about driving backward anyway; we made about 14m of progress (regress?) before the anomaly stopped us the other sol. We have little to gain by doing it again, and too much to lose. So we don't.
I also simplify our day in another way. Julie raises a concern about wasting energy roverdancing when the rover isn't going to make any progress anyway. Is there something we can change to make it give up sooner? I suggest replacing the single long-distance autonav waypoint with a series of shorter autonav waypoints, but we'd have to revisit the timeout-handling logic. Maybe there are flight software parameters we could tweak to help out as well, but we'd have to ask Mark Maimone and he's already gone home. And you know what -- we just don't have time to deal with it today. If we've learned anything, it's that we shouldn't try to come up with this stuff under time pressure unless we really have to. So I tell her we'll think about it and not do it today, and she's cool with that. So we don't.
I need to blow off a little steam after this day, so I put something in the uplink report that I've been thinking about for a while:
Driving a Mars rover ain't like dustin' crops, boy. Without
precise calculations, we'd drive into a ditch or bounce too
close to a 60cm rock, and that would end your trip real quick,
wouldn't it?!
Anybody who doesn't get it has no business working here anyway, I figure.
Footnotes:
[1] Of course, we have seen that bug again, a few times. We continue to just live with it; it's not a big enough problem to be worth the risks involved in fixing it. I'd like to add, though, that I'm permanently amazed by the team's ability to diagnose these problems -- in this case, by the next day -- from a hundred million miles away. Amazed.
[2] Sometimes it still feels like that, although the protocol is now very much more that it's OK to tell people to bug off and you'll get back to them when you're good and ready. (It helps to not phrase it like that, although some days ....) This is a good change. For a long time, it felt like I was the dad in Vonnegut's story Harrison Bergeron -- the guy who has to wear a device that makes a loud noise in his ear every few seconds, so that he won't be able to unfairly exercise his higher-than-average intelligence.
2009-05-17
Spirit Sol 132
All that work we put into the drive, and not much came of it. The rover flight software fataled -- rebooting the rover, and consequently stopping everything -- only a few minutes after the drive started. We only made it about 12.7m.
Nobody knows why the fatal error happened. There was a set of data management sequences running in parallel with the drive, though we have no idea whether there was any connection between that and the error -- we've done this once before, and it worked fine then. Anyway, the rover's apparently healthy; it knows where it is and was able to communicate with Earth right on schedule. They'll just need some time to figure this out.
So they won't need RPs much today -- which doesn't affect me much, since I'm not working today anyway. Chris Leger asks me to fill in for him as RP-1 tomorrow so that he can pick his brother up at the airport (John Wright is in Montreal and Ashitey is already scheduled as RP-2 tomorrow, leaving me as the only other available Spirit RP), so maybe I'll get to do something then. I hope we'll be driving and/or IDDing nextersol -- not only because it will mean the rover's healthy, but also because, since I'm filling in for Chris, I'll be RP-1 for the first time in weeks. It would be nice to have something to do.
But there's nothing for me to do now. They're going to have to devote thisol to gathering data to help diagnose the anomaly -- and, I have to remind myself, I'm not working anyway. So Ashitey and I go explore the Open House a little more. Ashitey's on a quest to get a little card they're handing out -- it's like a baseball card, with a picture of the rover on it. We walk around through all of the MER areas, but we don't find one, and nobody seems to know what he's talking about.
After we return to 264, Ashitey asks Joe Melko, who has one of the cards, where he got his. Joe answers -- in a way that comes across as rather rude -- that they have them over at von Karman auditorium. ("I guess you didn't look hard enough," Joe smirks, not exercising his best social skills.) By this time, they need Ashitey for thisol's sequencing and the upcoming CAM, so he can't go get one. I don't say anything about it to him, but I walk over to von Karman, pick up a card for him, bring it back, and hand it to him. His look of astonished delight makes the short trip more than worthwhile.
As I'm heading to the car, a clearly lost woman with two daughters calls across the street to me. "Do you know where mission operations is?" she asks. Turns out she's looking for 230 -- and she's found it, but she's on the north side of the building, and the entrance she needs is on the south side. "Follow me!" I say, and I lead them around to the other side.
As we're walking, she asks what I do, and I tell her I'm one of the rover drivers for Spirit. She and her daughters are astonished. ("Really? Really? Really?") "Oh, but you look so young," she exclaims. (I never get tired of hearing that. Such vanity!) We talk a little more about the mission, and she shakes her head wonderingly. "You have the dream job," she says.
Yes. Yes, I do.
Nobody knows why the fatal error happened. There was a set of data management sequences running in parallel with the drive, though we have no idea whether there was any connection between that and the error -- we've done this once before, and it worked fine then. Anyway, the rover's apparently healthy; it knows where it is and was able to communicate with Earth right on schedule. They'll just need some time to figure this out.
So they won't need RPs much today -- which doesn't affect me much, since I'm not working today anyway. Chris Leger asks me to fill in for him as RP-1 tomorrow so that he can pick his brother up at the airport (John Wright is in Montreal and Ashitey is already scheduled as RP-2 tomorrow, leaving me as the only other available Spirit RP), so maybe I'll get to do something then. I hope we'll be driving and/or IDDing nextersol -- not only because it will mean the rover's healthy, but also because, since I'm filling in for Chris, I'll be RP-1 for the first time in weeks. It would be nice to have something to do.
But there's nothing for me to do now. They're going to have to devote thisol to gathering data to help diagnose the anomaly -- and, I have to remind myself, I'm not working anyway. So Ashitey and I go explore the Open House a little more. Ashitey's on a quest to get a little card they're handing out -- it's like a baseball card, with a picture of the rover on it. We walk around through all of the MER areas, but we don't find one, and nobody seems to know what he's talking about.
After we return to 264, Ashitey asks Joe Melko, who has one of the cards, where he got his. Joe answers -- in a way that comes across as rather rude -- that they have them over at von Karman auditorium. ("I guess you didn't look hard enough," Joe smirks, not exercising his best social skills.) By this time, they need Ashitey for thisol's sequencing and the upcoming CAM, so he can't go get one. I don't say anything about it to him, but I walk over to von Karman, pick up a card for him, bring it back, and hand it to him. His look of astonished delight makes the short trip more than worthwhile.
As I'm heading to the car, a clearly lost woman with two daughters calls across the street to me. "Do you know where mission operations is?" she asks. Turns out she's looking for 230 -- and she's found it, but she's on the north side of the building, and the entrance she needs is on the south side. "Follow me!" I say, and I lead them around to the other side.
As we're walking, she asks what I do, and I tell her I'm one of the rover drivers for Spirit. She and her daughters are astonished. ("Really? Really? Really?") "Oh, but you look so young," she exclaims. (I never get tired of hearing that. Such vanity!) We talk a little more about the mission, and she shakes her head wonderingly. "You have the dream job," she says.
Yes. Yes, I do.
2009-05-16
Spirit Sol 131
Today's the start of the JPL Open House, or, as I call it, Deep Space Disneyland. It has the feel of some weird combination of Disneyland and a trade show -- complete with Disneyland prices for the food ($4 burgers, for example) and JPL T-shirts and stuff.
But I don't mind that part of it, and neither, it seems, do the attendees. I love Open House, and I really wish I could have volunteered this year -- but I gotta work. Still, I come in early enough to spend a couple of hours walking around and looking at the people. I tend to worry that the space program is supported mainly by older people, with interest fading out among the youth, but I can't see any sign of that by looking at the people around me. Go to Disneyland, come here, it's the same mix; everyone from toddlers to retirees loves the space program. As usual, it's a huge event, with tens of thousands of people coming in -- on everything from motorcycles to tour buses.
Mars in general, and the rovers in particular, are especially popular. JPL filled the 303 cafeteria with Mars stuff, including the ever-popular exhibit where a bunch of kids lie prone on a mat and a volunteer drives a small rover over them. The kids love this. Love it. LOVE it. I'd like to try it myself. I think when you come out the other end of the line, you can go into the ISIL and see the sandbox (our testbed), but I'm not sure.
The line into 303 is huge, and it snakes past a bunch of posters documenting the history of Spirit and Opportunity. One of these posters shows our RATting at Mazatzal. This is my favorite, because it helps me put things in perspective. At the time, I felt so awful about my bad performance at Mazatzal, and in the end it doesn't matter: we got the scientists the data they wanted, and they're even bragging about it to tens of thousands of people today. So things you feel lousy about can work out fine.
As the line progresses, it also winds past an open area where they've put several rover models on display. And they're driving one of the testbed rovers around in it, so that people can see for themselves how the rovers move. Mark Maimone is standing there with a microphone, capably answering questions from the audience. As he finishes answering one question, I go up to him and ask another: "Who are those handsome guys who drive the rovers?" Mark laughs and introduces me to the crowd. I spend a few minutes chatting with a visibly impressed family, and then fade back and let Mark have the show. (Nagin tells me she stepped into that area to answer a couple of questions and didn't get out again until four hours later.)
On second thought, maybe it's just as well I didn't volunteer for Open House this year. I think my ego would explode.
I run into Bob Deen, who has a story of his own about International Legal Affairs (the dorks who spoiled my Canada trip). A little while back, he wanted to release for public use some software he'd developed at JPL. The software got to ILA -- and sat there. And sat there. And sat there. At long last, a specific customer for the software started asking for it, and Bob had to get his section manager involved to get ILA to rubber-stamp the damn software release already.
The whole process took over six months.
Today's downlink is delayed again, just like yesterday. So Art decides we're not even going to try to IDD. "It's a touch-and-go sol without the touch," he says. So it's an "and-go" sol, I guess. I left Open House in plenty of time, but I've got another 30-45 minutes to twiddle my thumbs before the delayed data actually gets here.
During the lull, Larry Soderblom plays an audio clip of John's call-in on the NPR show "Car Talk." I'd been offered this interview, as it happens, and turned it down because it was scheduled for a time when I'd be working. So they offered it to John -- and then rescheduled it, so I could have done it after all. Oh, well. John does better with it than I would have, anyway.[1] The idea was to call in as if we needed some car advice, and slowly reveal that we were talking about a Mars rover. I had come up with an idea that they didn't use. ("How old is your vehicle?" "Less than a year." "And how many miles on it?" "About three hundred million ....")
Anyway, John soon reveals that he's talking about the Mars rover, and they pretend to be surprised, and then they talk about the rover a little bit. One of the hosts asks John, "Why not drive this thing every day like tomorrow something catastrophic could happen?" Mark Adler, listening to the recording, responds immediately: "Mission Manager thinking!"
We might drive backward thisol. Joe Melko raises an old argument, that if the rover's going to collide with something, it's better for the rear of the rover to make contact than the front. This is because the rover has enough strength to push hard enough on the (front-mounted) IDD to fatally damage it. But the body of the rover, which would make contact in a backward-driving collision, might be strong enough to survive.
On the other hand, if we're wrong about that and the body of the rover were pierced, the rover would soon freeze to death. Better to lose the IDD than the whole vehicle. But as an experiment, we decide to drive backward thisol anyway.[2]
At long last, we start to get data. Six whole minutes' worth. Scott Doudrick tells us that's all we're getting for a while: next, there's a priority table change that interrupts the stream for 35 minutes. After which we get 30 minutes' worth of additional data before another 15-minute gap induced by a Deep Space Network station handover. We've really got to start getting our data into the queue ahead of Opportunity.
So, so far we have only the HAZCAM images. John and I are content to wait for the downlink, but Larry Soderblom's getting antsy and prods us to do something. So we make a basic plan: we'll drive blind a few meters, up to the limit of the HAZCAM data, and then autonav after that. We choose a couple of autonav waypoints so that we can turn one of them into a blind waypoint later, if we get the NAVCAM data soon enough.
We also back up again today at the start of the drive, like yesterday. Today it's for a different reason. There's no hazard in front of us, but autonav left a 15cm rock right next to the left front wheel, under the rover's body. This is likely safe, but we don't like it: there's a wheel strut just next to the rock, which could theoretically cause a problem, and it's possible for a rock to damage cabling under there. We can avoid all of this by just backing up and driving around the rock -- and what the heck, we have plenty of time. So we do that.
One thing that's good about this rock: it means we didn't give up anything when we decided to drop the "touch" part of thisol's "touch-and-go" plans. We dropped it for time reasons: because we wouldn't have time to plan the IDD work in addition to the drive. But with that rock in the way, we wouldn't have been able to safely deploy the IDD anyhow.
At long last, we get more data. Yestersol's drive was a decent 53m (40 blind, 13 autonav) -- not great by our current standards, but I remind myself again that we would have killed for a drive like this not so long ago. John has to leave so he can get ready for his early-morning flight to Montreal, so the rest of the sequencing is up to me. I don't mind that at all, of course -- it's almost like being an RP-1 again.
We have NAVCAMs, so I'm able to push the blind drive out about 30m farther. This takes us over a gap in the mesh, but I check out the terrain in the raw images, and it looks perfectly safe. Art asks me to cut the drive short before the gap anyway, though -- he wants to play it extra-safe since we only have one RP available. I know the traverse is fine, but I don't want Art to have to lie awake worrying about it, so I cut it short. A few minutes after that, we get the PANCAM mesh, and the PANCAMs fill in the gap -- so Art drops his objection, and the blind drive is extended once more.
Mark Maimone explains Mars rovers to admiring visitors.
The endless line snakes around the Mars exhibit.
The first thing that made me feel any better about Mazatzal.
Giggling kids get run over by a prototype Mars rover.
Footnotes:
[1] Just as a measure of how well John did, I still occasionally run into people who remember that interview.
[2] The rovers were originally designed to drive about equally well backward as forward -- in some ways, they drive a little better backward. It's amazing how long it took us to get comfortable with the idea of driving backward, though. From my current perspective, I see that we took some time to really learn to think like the rovers.
But I don't mind that part of it, and neither, it seems, do the attendees. I love Open House, and I really wish I could have volunteered this year -- but I gotta work. Still, I come in early enough to spend a couple of hours walking around and looking at the people. I tend to worry that the space program is supported mainly by older people, with interest fading out among the youth, but I can't see any sign of that by looking at the people around me. Go to Disneyland, come here, it's the same mix; everyone from toddlers to retirees loves the space program. As usual, it's a huge event, with tens of thousands of people coming in -- on everything from motorcycles to tour buses.
Mars in general, and the rovers in particular, are especially popular. JPL filled the 303 cafeteria with Mars stuff, including the ever-popular exhibit where a bunch of kids lie prone on a mat and a volunteer drives a small rover over them. The kids love this. Love it. LOVE it. I'd like to try it myself. I think when you come out the other end of the line, you can go into the ISIL and see the sandbox (our testbed), but I'm not sure.
The line into 303 is huge, and it snakes past a bunch of posters documenting the history of Spirit and Opportunity. One of these posters shows our RATting at Mazatzal. This is my favorite, because it helps me put things in perspective. At the time, I felt so awful about my bad performance at Mazatzal, and in the end it doesn't matter: we got the scientists the data they wanted, and they're even bragging about it to tens of thousands of people today. So things you feel lousy about can work out fine.
As the line progresses, it also winds past an open area where they've put several rover models on display. And they're driving one of the testbed rovers around in it, so that people can see for themselves how the rovers move. Mark Maimone is standing there with a microphone, capably answering questions from the audience. As he finishes answering one question, I go up to him and ask another: "Who are those handsome guys who drive the rovers?" Mark laughs and introduces me to the crowd. I spend a few minutes chatting with a visibly impressed family, and then fade back and let Mark have the show. (Nagin tells me she stepped into that area to answer a couple of questions and didn't get out again until four hours later.)
On second thought, maybe it's just as well I didn't volunteer for Open House this year. I think my ego would explode.
I run into Bob Deen, who has a story of his own about International Legal Affairs (the dorks who spoiled my Canada trip). A little while back, he wanted to release for public use some software he'd developed at JPL. The software got to ILA -- and sat there. And sat there. And sat there. At long last, a specific customer for the software started asking for it, and Bob had to get his section manager involved to get ILA to rubber-stamp the damn software release already.
The whole process took over six months.
Today's downlink is delayed again, just like yesterday. So Art decides we're not even going to try to IDD. "It's a touch-and-go sol without the touch," he says. So it's an "and-go" sol, I guess. I left Open House in plenty of time, but I've got another 30-45 minutes to twiddle my thumbs before the delayed data actually gets here.
During the lull, Larry Soderblom plays an audio clip of John's call-in on the NPR show "Car Talk." I'd been offered this interview, as it happens, and turned it down because it was scheduled for a time when I'd be working. So they offered it to John -- and then rescheduled it, so I could have done it after all. Oh, well. John does better with it than I would have, anyway.[1] The idea was to call in as if we needed some car advice, and slowly reveal that we were talking about a Mars rover. I had come up with an idea that they didn't use. ("How old is your vehicle?" "Less than a year." "And how many miles on it?" "About three hundred million ....")
Anyway, John soon reveals that he's talking about the Mars rover, and they pretend to be surprised, and then they talk about the rover a little bit. One of the hosts asks John, "Why not drive this thing every day like tomorrow something catastrophic could happen?" Mark Adler, listening to the recording, responds immediately: "Mission Manager thinking!"
We might drive backward thisol. Joe Melko raises an old argument, that if the rover's going to collide with something, it's better for the rear of the rover to make contact than the front. This is because the rover has enough strength to push hard enough on the (front-mounted) IDD to fatally damage it. But the body of the rover, which would make contact in a backward-driving collision, might be strong enough to survive.
On the other hand, if we're wrong about that and the body of the rover were pierced, the rover would soon freeze to death. Better to lose the IDD than the whole vehicle. But as an experiment, we decide to drive backward thisol anyway.[2]
At long last, we start to get data. Six whole minutes' worth. Scott Doudrick tells us that's all we're getting for a while: next, there's a priority table change that interrupts the stream for 35 minutes. After which we get 30 minutes' worth of additional data before another 15-minute gap induced by a Deep Space Network station handover. We've really got to start getting our data into the queue ahead of Opportunity.
So, so far we have only the HAZCAM images. John and I are content to wait for the downlink, but Larry Soderblom's getting antsy and prods us to do something. So we make a basic plan: we'll drive blind a few meters, up to the limit of the HAZCAM data, and then autonav after that. We choose a couple of autonav waypoints so that we can turn one of them into a blind waypoint later, if we get the NAVCAM data soon enough.
We also back up again today at the start of the drive, like yesterday. Today it's for a different reason. There's no hazard in front of us, but autonav left a 15cm rock right next to the left front wheel, under the rover's body. This is likely safe, but we don't like it: there's a wheel strut just next to the rock, which could theoretically cause a problem, and it's possible for a rock to damage cabling under there. We can avoid all of this by just backing up and driving around the rock -- and what the heck, we have plenty of time. So we do that.
One thing that's good about this rock: it means we didn't give up anything when we decided to drop the "touch" part of thisol's "touch-and-go" plans. We dropped it for time reasons: because we wouldn't have time to plan the IDD work in addition to the drive. But with that rock in the way, we wouldn't have been able to safely deploy the IDD anyhow.
At long last, we get more data. Yestersol's drive was a decent 53m (40 blind, 13 autonav) -- not great by our current standards, but I remind myself again that we would have killed for a drive like this not so long ago. John has to leave so he can get ready for his early-morning flight to Montreal, so the rest of the sequencing is up to me. I don't mind that at all, of course -- it's almost like being an RP-1 again.
We have NAVCAMs, so I'm able to push the blind drive out about 30m farther. This takes us over a gap in the mesh, but I check out the terrain in the raw images, and it looks perfectly safe. Art asks me to cut the drive short before the gap anyway, though -- he wants to play it extra-safe since we only have one RP available. I know the traverse is fine, but I don't want Art to have to lie awake worrying about it, so I cut it short. A few minutes after that, we get the PANCAM mesh, and the PANCAMs fill in the gap -- so Art drops his objection, and the blind drive is extended once more.
Mark Maimone explains Mars rovers to admiring visitors.
The endless line snakes around the Mars exhibit.
The first thing that made me feel any better about Mazatzal.
Giggling kids get run over by a prototype Mars rover.
Footnotes:
[1] Just as a measure of how well John did, I still occasionally run into people who remember that interview.
[2] The rovers were originally designed to drive about equally well backward as forward -- in some ways, they drive a little better backward. It's amazing how long it took us to get comfortable with the idea of driving backward, though. From my current perspective, I see that we took some time to really learn to think like the rovers.
2009-05-15
Spirit Sol 130
I see Tatiana[1] on my way in, and she apologizes for the SpaceOps travel not happening. But I tell her it's no problem, which it isn't, and she looks relieved to hear it.
Jeng commiserates with my non-misery, too. "How did they get it so fucked up?" he asks rhetorically. "Well, I'll tell you one thing, it takes pros to fuck things up this badly."
On my way up the stairs in 264, I see the TUL, Emily Eelkema, coming down. She cocks her head strangely and says, "You'll do."
It turns out that John went to the cafeteria 30 minutes ago and nobody's seen him. He doesn't have a cell phone, either -- somehow he manages to casually resist this. So they can't reach him, but they need to start planning.
In particular, they want to know if it's safe to deploy the IDD. I look at the imagery that's come down so far, and it's not promising. We didn't take the final stutter-step -- not too surprising, as there's a cluster of big rocks in our way -- and the HAFIQ[2], a set of images that's supposed to help us see what was under the rover in just such a case, turns out not to show us what we need to see thisol.
We've been in this fix before, and have been able to use thumbnail images[3] taken during the drive to show that there's nothing scary under the rover. So I go to the SMSA to see if Rich Petras and I can make the case.
Even without this problem, the rover's performance yestersol was a little disappointing. It made it through the blind drive just fine, but then the autonav got spooked by something. We can't see what spooked it, but we know we drove it up onto the top of a ridge, and our best guess is that the far side of the ridge sloped down too abruptly for a safe drive. Effectively, the far side of the ridge was an obstacle. So the rover started doing its little roverdance, making small turns as it steps back and forth, to try to find a way to the goal. In the process of doing this, it turned toward the big cluster of rocks we're currently facing -- another obstacle -- and, boxed in on two sides, never found a way out.
Mark Maimone tries to give me a hard time (jokingly) about the rover not completing its drive, but (equally jokingly) I'm having none of it. The author of the autonav code shouldn't have messed with me. "Oh, so you're saying we got through the blind drive -- that is, the part we planned -- just fine," I tease him, "and it was the autonav that failed. So this is our fault? Did we write the autonav code?" Mark mock-surrenders.
While Rich and I look through the thumbnails, John Wright shows up to do the same. We're pretty sure it's safe, but we're having a hard time working out the position of two possible deployment hazards -- we think they're well under the vehicle, or off to its right, but we're not sure yet. Art asks us for the status, and we tell him we're probably OK but we're still investigating.
"Bottom line, how are you going to convince me it's safe to do this?" he asks.
I shrug. "We'll have to convince ourselves first," I say. I think that was a good answer.
Ray Arvidson walks in to hear the tail of that conversation. "It's OK not to do this if there's any queasiness," he says. That puts a different face on the matter immediately. Rich, John, and I look at each other and shrug. "OK, let's not deploy, then," I say. Ray nods.
I should have expected something like this. Ray doesn't want to stop and IDD anyway; he's just going along with it because of demand from the other scientists. In the same vein, he enlists our help in selling this decision. "Don't be wishy-washy about it, though," he says. "Go into the meeting and say, 'We can't do this.'" That's John's job, and he's comfortable with it, so that's what we do.
The stress level drops rapidly, and we're able to joke about the situation a little more. Noting that the rover completed the blind drive and then autonav decided it wasn't safe to go any farther, Mark Adler says, "I guess the rover drivers knew exactly how far to go." Steve Collins starts doing a Kirk quote: "Risk ... is our business ...," he says, and I finish the quote: "That's why we're out there!" Adler laughs, but he says, "No, Steve, paranoia is our business."
Getting the rover out of here proves to be a small challenge. We can't drive over the rocks now in front of the rover. While they're technically under the rover's traversable limit, the combination of the rocks' own height and the size of the dropoff behind them effectively makes the rocks too tall. We can't drive into the zone that originally stopped the rover's traverse, because we can't see it in any of our post-drive imagery and all we know about it is that the rover thought it was a hazard. So we take the only remaining option: we back up. We can see that the area immediately behind us is safe, so we back into it, then drive over the ridge at a lower point off to our right before continuing. We have to ask MIPL for an additional terrain mesh before we can be sure this path is safe, but we plan the drive anyway, hoping for the best. If this doesn't work, we're out of good options. When the terrain mesh arrives, we use it to check out the drive -- and thankfully, it looks perfectly safe. So that's our plan.
As part of the ongoing cross-pollination between rover teams, Art is going to be Opportunity's Mission Manager during June. "Are you guys disappointed you won't have me to kick around?" he asks.
"No, but you're going to miss us," I reply. "You know what they say: absence makes the Art grow fonder."
Frankly, I'm lucky to be alive.
Footnotes:
[1] My group's secretary, who helps with travel arrangements and the like.
[2] The Hazard-Avoidance Fault Image Queue is the last several images taken by the hazard-avoidance (a.k.a. "autonav") code. If the rover decides to stop, it can be handy to see what led up to its decision; the images in this queue show you that.
[3] Scaled-down versions of the full images taken during the drive. These are small enough that we can often afford to downlink them at a relatively high priority; they don't squeeze out too much other data and can tell us a surprising amount.
Jeng commiserates with my non-misery, too. "How did they get it so fucked up?" he asks rhetorically. "Well, I'll tell you one thing, it takes pros to fuck things up this badly."
On my way up the stairs in 264, I see the TUL, Emily Eelkema, coming down. She cocks her head strangely and says, "You'll do."
It turns out that John went to the cafeteria 30 minutes ago and nobody's seen him. He doesn't have a cell phone, either -- somehow he manages to casually resist this. So they can't reach him, but they need to start planning.
In particular, they want to know if it's safe to deploy the IDD. I look at the imagery that's come down so far, and it's not promising. We didn't take the final stutter-step -- not too surprising, as there's a cluster of big rocks in our way -- and the HAFIQ[2], a set of images that's supposed to help us see what was under the rover in just such a case, turns out not to show us what we need to see thisol.
We've been in this fix before, and have been able to use thumbnail images[3] taken during the drive to show that there's nothing scary under the rover. So I go to the SMSA to see if Rich Petras and I can make the case.
Even without this problem, the rover's performance yestersol was a little disappointing. It made it through the blind drive just fine, but then the autonav got spooked by something. We can't see what spooked it, but we know we drove it up onto the top of a ridge, and our best guess is that the far side of the ridge sloped down too abruptly for a safe drive. Effectively, the far side of the ridge was an obstacle. So the rover started doing its little roverdance, making small turns as it steps back and forth, to try to find a way to the goal. In the process of doing this, it turned toward the big cluster of rocks we're currently facing -- another obstacle -- and, boxed in on two sides, never found a way out.
Mark Maimone tries to give me a hard time (jokingly) about the rover not completing its drive, but (equally jokingly) I'm having none of it. The author of the autonav code shouldn't have messed with me. "Oh, so you're saying we got through the blind drive -- that is, the part we planned -- just fine," I tease him, "and it was the autonav that failed. So this is our fault? Did we write the autonav code?" Mark mock-surrenders.
While Rich and I look through the thumbnails, John Wright shows up to do the same. We're pretty sure it's safe, but we're having a hard time working out the position of two possible deployment hazards -- we think they're well under the vehicle, or off to its right, but we're not sure yet. Art asks us for the status, and we tell him we're probably OK but we're still investigating.
"Bottom line, how are you going to convince me it's safe to do this?" he asks.
I shrug. "We'll have to convince ourselves first," I say. I think that was a good answer.
Ray Arvidson walks in to hear the tail of that conversation. "It's OK not to do this if there's any queasiness," he says. That puts a different face on the matter immediately. Rich, John, and I look at each other and shrug. "OK, let's not deploy, then," I say. Ray nods.
I should have expected something like this. Ray doesn't want to stop and IDD anyway; he's just going along with it because of demand from the other scientists. In the same vein, he enlists our help in selling this decision. "Don't be wishy-washy about it, though," he says. "Go into the meeting and say, 'We can't do this.'" That's John's job, and he's comfortable with it, so that's what we do.
The stress level drops rapidly, and we're able to joke about the situation a little more. Noting that the rover completed the blind drive and then autonav decided it wasn't safe to go any farther, Mark Adler says, "I guess the rover drivers knew exactly how far to go." Steve Collins starts doing a Kirk quote: "Risk ... is our business ...," he says, and I finish the quote: "That's why we're out there!" Adler laughs, but he says, "No, Steve, paranoia is our business."
Getting the rover out of here proves to be a small challenge. We can't drive over the rocks now in front of the rover. While they're technically under the rover's traversable limit, the combination of the rocks' own height and the size of the dropoff behind them effectively makes the rocks too tall. We can't drive into the zone that originally stopped the rover's traverse, because we can't see it in any of our post-drive imagery and all we know about it is that the rover thought it was a hazard. So we take the only remaining option: we back up. We can see that the area immediately behind us is safe, so we back into it, then drive over the ridge at a lower point off to our right before continuing. We have to ask MIPL for an additional terrain mesh before we can be sure this path is safe, but we plan the drive anyway, hoping for the best. If this doesn't work, we're out of good options. When the terrain mesh arrives, we use it to check out the drive -- and thankfully, it looks perfectly safe. So that's our plan.
As part of the ongoing cross-pollination between rover teams, Art is going to be Opportunity's Mission Manager during June. "Are you guys disappointed you won't have me to kick around?" he asks.
"No, but you're going to miss us," I reply. "You know what they say: absence makes the Art grow fonder."
Frankly, I'm lucky to be alive.
Footnotes:
[1] My group's secretary, who helps with travel arrangements and the like.
[2] The Hazard-Avoidance Fault Image Queue is the last several images taken by the hazard-avoidance (a.k.a. "autonav") code. If the rover decides to stop, it can be handy to see what led up to its decision; the images in this queue show you that.
[3] Scaled-down versions of the full images taken during the drive. These are small enough that we can often afford to downlink them at a relatively high priority; they don't squeeze out too much other data and can tell us a surprising amount.
2009-05-14
Spirit Sol 129
The good news is, our big drive on sol 125 set a new record: 123.66m! It helped that we used a new technique for maximizing drive time. At the start of each sol, the rover sends a "beep" signal through its non-directional low-gain antenna to let us know everything is OK. On sol 125, we sent this signal while driving, saving us about five minutes. This has been christened the "New York Drive Style" or "Drive-and-Honk," and will become part of our toolbox for future drive sols.
In the same drive, we broke the 2km mark. We didn't beat Opportunity's single-sol distance record, but that's not too surprising. Anyway, we beat the heck out of our old record, and that's pretty good.
So that's the good news. The bad news is, I can't go to SpaceOps to brag about it. The International Legal Affairs Office at JPL sat on my travel paperwork for a week, so it didn't get to NASA HQ and the State Department in time, so they disapproved it. So John Wright is going to present my paper for me, and I'm going to stay here and drive the rover.
Come to think of it, that's not such a bad deal.
Especially just after we landed, there were a lot of claims about records we'd set or broken. Many of these were reasonable, such as when we broke Sojourner's record for a single-sol traverse. But then there was growing pressure to think up more and more records in an effort to stay newsworthy, and after the first couple of months post-landing, that became harder and harder to do. We quickly moved into a phase of being tongue-in-cheek about it, and there we've stayed. I think Mark Adler's Mission Manager report from sol 123 is a brilliant example:
Sol 123 started off with Pancam and mini-TES observations for
near-field surveys, atmospheric studies, and localization.
Spirit then took a half-hour nap, followed by the day's drive.
This sol we had another 48 meter direct drive, the mid-drive
survey and localization remote sensing, and then 47 m of
driving using auto-navigation. The total was 95.2 meters,
bringing the mission total to 1830 meters. Nextersol, we
should break one Earth nautical mile, which is 1852 meters.
As we drive another kilometer and change to the base of the
hills, we will continue to bring you as many arbitrary
numerical achievements as we can come up with. Think of it as
the games you play on a long road trip so you don't keep
hearing "Are we there yet?". More numerical trivia: a
nautical mile, defined as the length of an arc-minute at the
equator, is almost exactly a kilometer on Mars! Keep that in
mind the next time you're navigating a boat down the Valles
Marineris.
[...]
Plan:
Continue on to the hills. "99 bottles of beer on the wall, ..."
We'd have enough time to sing the whole song today, if we wanted. This is because Opportunity delayed our downlink. Both rovers share the relay passes through Odyssey, and Odyssey is just a FIFO[1] -- that is, whatever got sent to it first, gets sent back to Earth first. Opportunity sent all of its data to Odyssey before we sent any of ours (luck of the draw, more or less), so all of their data will come down before any of ours does.
The upshot is that we're facing a lengthy delay. The downlink starts at 13:00 or so, but we're not going to get any of our data, Art tells me, until 14:30 at the earliest. As it happens, we actually don't get anything until 15:15 -- at very late start, considering that we were scheduled to be done by 18:00 or so.
One thing we do know is that yestersol's data included a slightly worrisome bit of drive data, as Mark Maimone explains to me. The rover keeps an eye on its current tilt as a way of protecting itself from flipping over. Indeed, the rover actually keeps two measurements of its tilt, one an instantaneous measurement taken eight times per second, and the other a "filtered" measurement, which sort of smooths out the instantaneous measurement over time. For an instant during yestersol's drive, the instantaneous measurement of the rover's tilt exceeded 20 degrees -- the limit we've currently set on the rover. Now, the instantaneous measurement can be fooled by the kind of quick short drop the rover experiences when, for instance, it finishes climbing over a rock and the wheels drop to the ground, and we don't want the rover to panic when this happens. So it has a rule that says to stop driving if the instantaneous measurement is over the limit twice in a row. It wasn't -- the measurement settled out by the time the next snapshot was taken -- but we were an eighth of a second away from losing the rest of the drive.
Mark takes the opportunity to complain about overly aggressive drives (his diagnosis of the problem), and while I haven't seen the drive sequence in question, I don't outwardly disagree with him. But it must be said that, in the end, the sequence worked fine. Since the smoothed tilt measurement was only 12 degrees, the 20-degree spike was surely spurious, and could have happened almost any time.
I came in early so I could hang out in the downlink assessment meeting -- I haven't been to one in a long time. (Since the downlink was delayed so long, I didn't need to be early, but of course I didn't know that.) The news in the meeting is good: we're 200m ahead of where we need to be to make it to the hills on schedule. Sometime soon, in the next week or so, they'll choose a spot to stop for a few sols and do some science -- dig a trench, IDD it, and generally hang out for two or three sols. The objective of the trench is said to be "ground truthing" -- confirming orbital measurements with measurements we take on the ground -- and getting MTES observations of the hills from 500m or so out. (I thought at first that it was a little early to be discussing that, then realized that we're only about 1km from the hills now, so we could reach that point as early as next week!) But I think the real reason is that the science teams are getting bored. All this driving is fun for us, but not for many of them.
I'm glad to hear one of the scientists voice what's been a minor annoyance for me: the meeting times are posted with very little notice. We typically haven't been finding out when we need to come in the next day until the day before. Say what you like about Mars time, at least it was predictable. Jim Bell's exact words are: "I would like to make a bitter complaint about people not knowing when meetings are going to be, even the next day." Of course, today was worse than usual because the delayed downlink delayed the meetings as well, and Jim seems to be mollified when Art points this out to him.
Larry Soderbloom leads a discussion of what to do when we get to the hills. The basic choices, which he illustrates by spinning that cool 3-D model around again, are to go uphill into the caldera-like thing or south into the dark basin. He still favors the southern region. It's got an ejecta blanket, a lava flow, or something like that -- whatever it is, the orbiting THEMIS instrument shows a high thermal inertia. Rocks.[2] In addition, the stratigraphy of the plain should be well exposed there -- we might be able to drive up to a rock wall and see sedimentary layers, or anyway that's the geologists' fantasy. There's also a set of seven or so terraces -- huge natural steps carved into a hill. The whole area is "loaded with interesting stuff," in his words.
So Larry's worked out a basic traverse to that region from the "spur" we're currently aiming for; the path is about 625m long, and goes up a hill on the way, so we'll get a pretty good view of the whole area. The route also takes us past some "megaripples," some kind of huge sand-dune-like things.
I've missed those meetings.
Courtesy NASA/JPL-Caltech. A sample orbital map whose colors correspond to different predictions of surface materials. Spirit's been confirming those measurements, providing "ground truth," as she moves across the terrain.
While I was sitting in on the meeting, John was doing the hard part. I go take a look. It's a typical drive: a little zig-zagging around obstacles, then a lengthy autonav segment. Ho-hum.
Things get a little silly during the command approval meeting. We've all been there a long time, and for most of these people it's rather late to be working. John Grant briefly zones out looking at a screen saver in the corner and says, "Man, I'm freaked out by those rainbow-colored worms." I look at him strangely and say, "I don't see any rainbow-colored worms ...."
As has often been the case lately, there's not much science in the sol, mostly driving. When Art asks John Grant if science is happy with the plan, however, John says yes. And he adds: "When I came in Monday morning, all Ray said to me was, 'Keep driving, Grant!'" (He says the last part in a mock-gruff voice. John was clearly talking to Grumpy Ray.) And so we shall.
Footnotes:
[1] Computer lingo: First In, First Out. The line you stand in at the grocery store is a FIFO: the first person in line gets served first, and so on.
[2] Rocks take longer to absorb heat than soil does, but they also take longer to bleed the heat back out when the sun goes down. This property of being slow to absorb and slow to release heat is known as a high thermal inertia. It lets you tell from orbit whether an area is mostly rock or mostly soil: measure the temperature of the same area at different times of day and night, and from there it's easy.
As a side note, I am fascinated and awed that it's possible to figure out stuff like that -- and even more fascinated and awed that we can put a chunk of metal in the sky of a planet a hundred million miles away and have it tell us about that planet's surface from hundreds of kilometers above. We live in an age of wonders.
In the same drive, we broke the 2km mark. We didn't beat Opportunity's single-sol distance record, but that's not too surprising. Anyway, we beat the heck out of our old record, and that's pretty good.
So that's the good news. The bad news is, I can't go to SpaceOps to brag about it. The International Legal Affairs Office at JPL sat on my travel paperwork for a week, so it didn't get to NASA HQ and the State Department in time, so they disapproved it. So John Wright is going to present my paper for me, and I'm going to stay here and drive the rover.
Come to think of it, that's not such a bad deal.
Especially just after we landed, there were a lot of claims about records we'd set or broken. Many of these were reasonable, such as when we broke Sojourner's record for a single-sol traverse. But then there was growing pressure to think up more and more records in an effort to stay newsworthy, and after the first couple of months post-landing, that became harder and harder to do. We quickly moved into a phase of being tongue-in-cheek about it, and there we've stayed. I think Mark Adler's Mission Manager report from sol 123 is a brilliant example:
Sol 123 started off with Pancam and mini-TES observations for
near-field surveys, atmospheric studies, and localization.
Spirit then took a half-hour nap, followed by the day's drive.
This sol we had another 48 meter direct drive, the mid-drive
survey and localization remote sensing, and then 47 m of
driving using auto-navigation. The total was 95.2 meters,
bringing the mission total to 1830 meters. Nextersol, we
should break one Earth nautical mile, which is 1852 meters.
As we drive another kilometer and change to the base of the
hills, we will continue to bring you as many arbitrary
numerical achievements as we can come up with. Think of it as
the games you play on a long road trip so you don't keep
hearing "Are we there yet?". More numerical trivia: a
nautical mile, defined as the length of an arc-minute at the
equator, is almost exactly a kilometer on Mars! Keep that in
mind the next time you're navigating a boat down the Valles
Marineris.
[...]
Plan:
Continue on to the hills. "99 bottles of beer on the wall, ..."
We'd have enough time to sing the whole song today, if we wanted. This is because Opportunity delayed our downlink. Both rovers share the relay passes through Odyssey, and Odyssey is just a FIFO[1] -- that is, whatever got sent to it first, gets sent back to Earth first. Opportunity sent all of its data to Odyssey before we sent any of ours (luck of the draw, more or less), so all of their data will come down before any of ours does.
The upshot is that we're facing a lengthy delay. The downlink starts at 13:00 or so, but we're not going to get any of our data, Art tells me, until 14:30 at the earliest. As it happens, we actually don't get anything until 15:15 -- at very late start, considering that we were scheduled to be done by 18:00 or so.
One thing we do know is that yestersol's data included a slightly worrisome bit of drive data, as Mark Maimone explains to me. The rover keeps an eye on its current tilt as a way of protecting itself from flipping over. Indeed, the rover actually keeps two measurements of its tilt, one an instantaneous measurement taken eight times per second, and the other a "filtered" measurement, which sort of smooths out the instantaneous measurement over time. For an instant during yestersol's drive, the instantaneous measurement of the rover's tilt exceeded 20 degrees -- the limit we've currently set on the rover. Now, the instantaneous measurement can be fooled by the kind of quick short drop the rover experiences when, for instance, it finishes climbing over a rock and the wheels drop to the ground, and we don't want the rover to panic when this happens. So it has a rule that says to stop driving if the instantaneous measurement is over the limit twice in a row. It wasn't -- the measurement settled out by the time the next snapshot was taken -- but we were an eighth of a second away from losing the rest of the drive.
Mark takes the opportunity to complain about overly aggressive drives (his diagnosis of the problem), and while I haven't seen the drive sequence in question, I don't outwardly disagree with him. But it must be said that, in the end, the sequence worked fine. Since the smoothed tilt measurement was only 12 degrees, the 20-degree spike was surely spurious, and could have happened almost any time.
I came in early so I could hang out in the downlink assessment meeting -- I haven't been to one in a long time. (Since the downlink was delayed so long, I didn't need to be early, but of course I didn't know that.) The news in the meeting is good: we're 200m ahead of where we need to be to make it to the hills on schedule. Sometime soon, in the next week or so, they'll choose a spot to stop for a few sols and do some science -- dig a trench, IDD it, and generally hang out for two or three sols. The objective of the trench is said to be "ground truthing" -- confirming orbital measurements with measurements we take on the ground -- and getting MTES observations of the hills from 500m or so out. (I thought at first that it was a little early to be discussing that, then realized that we're only about 1km from the hills now, so we could reach that point as early as next week!) But I think the real reason is that the science teams are getting bored. All this driving is fun for us, but not for many of them.
I'm glad to hear one of the scientists voice what's been a minor annoyance for me: the meeting times are posted with very little notice. We typically haven't been finding out when we need to come in the next day until the day before. Say what you like about Mars time, at least it was predictable. Jim Bell's exact words are: "I would like to make a bitter complaint about people not knowing when meetings are going to be, even the next day." Of course, today was worse than usual because the delayed downlink delayed the meetings as well, and Jim seems to be mollified when Art points this out to him.
Larry Soderbloom leads a discussion of what to do when we get to the hills. The basic choices, which he illustrates by spinning that cool 3-D model around again, are to go uphill into the caldera-like thing or south into the dark basin. He still favors the southern region. It's got an ejecta blanket, a lava flow, or something like that -- whatever it is, the orbiting THEMIS instrument shows a high thermal inertia. Rocks.[2] In addition, the stratigraphy of the plain should be well exposed there -- we might be able to drive up to a rock wall and see sedimentary layers, or anyway that's the geologists' fantasy. There's also a set of seven or so terraces -- huge natural steps carved into a hill. The whole area is "loaded with interesting stuff," in his words.
So Larry's worked out a basic traverse to that region from the "spur" we're currently aiming for; the path is about 625m long, and goes up a hill on the way, so we'll get a pretty good view of the whole area. The route also takes us past some "megaripples," some kind of huge sand-dune-like things.
I've missed those meetings.
Courtesy NASA/JPL-Caltech. A sample orbital map whose colors correspond to different predictions of surface materials. Spirit's been confirming those measurements, providing "ground truth," as she moves across the terrain.
While I was sitting in on the meeting, John was doing the hard part. I go take a look. It's a typical drive: a little zig-zagging around obstacles, then a lengthy autonav segment. Ho-hum.
Things get a little silly during the command approval meeting. We've all been there a long time, and for most of these people it's rather late to be working. John Grant briefly zones out looking at a screen saver in the corner and says, "Man, I'm freaked out by those rainbow-colored worms." I look at him strangely and say, "I don't see any rainbow-colored worms ...."
As has often been the case lately, there's not much science in the sol, mostly driving. When Art asks John Grant if science is happy with the plan, however, John says yes. And he adds: "When I came in Monday morning, all Ray said to me was, 'Keep driving, Grant!'" (He says the last part in a mock-gruff voice. John was clearly talking to Grumpy Ray.) And so we shall.
Footnotes:
[1] Computer lingo: First In, First Out. The line you stand in at the grocery store is a FIFO: the first person in line gets served first, and so on.
[2] Rocks take longer to absorb heat than soil does, but they also take longer to bleed the heat back out when the sun goes down. This property of being slow to absorb and slow to release heat is known as a high thermal inertia. It lets you tell from orbit whether an area is mostly rock or mostly soil: measure the temperature of the same area at different times of day and night, and from there it's easy.
As a side note, I am fascinated and awed that it's possible to figure out stuff like that -- and even more fascinated and awed that we can put a chunk of metal in the sky of a planet a hundred million miles away and have it tell us about that planet's surface from hundreds of kilometers above. We live in an age of wonders.
2009-05-10
Spirit Sol 125
We didn't cover as much ground yestersol as we should have. We should have covered 100m or more, but we only made it 80.28m before the rover got spooked by its shadow again. They updated some parameters a while ago to try to solve that problem, but apparently they still have some fine-tuning to do.
But we're on for the three-hour drive tomorrow. We have a chance -- a small one, mind you, but a chance -- of beating even Opportunity's 140m record. Art wants this so badly he can taste it.
We didn't pick the best terrain to try it in. The way in front of us is blocked by a set of ridges, starting about 40m away, which are mostly too tall to drive over. John has planned a zig-zagging path to take us through them to better terrain beyond, but this requires more precise driving than I feel comfortable with -- particularly at that distance. When he shows me the plan, we spend some time looking around at alternatives as well. I spot a relatively low point at the right-hand end of the PANCAM data, one we could easily drive over. It's 60m away, and flat, straight driving all the way. After that, we can turn on the autonav. I talk John into abandoning the work he's done so far and aiming for this spot instead.
The drive becomes simple, just two waypoints -- one blind, one more for autonav. We'll have 40 minutes of blind driving (taking us 60m), followed by about 2.5 hours on autonav. At best, the autonav might take us 87m, so our best case for nextersol is 147m. Opportunity's record is 140.996.
But power concerns interfere. Our TAP tries hard not to, but in the end he has to cut 10 minutes out of the drive, meaning our best case will be about 142m. If the rover has to stop for any reason -- if it has to avoid a single obstacle along that path -- that will slow it down enough that we won't break the record. In this terrain, it's inconceivable that we'll go so far without autonav finding an obstacle, so mentally I've already given up on breaking the record. I don't care about it, anyway. It's more interesting to better our own personal best, and we're going to do that -- we'll set a new record for Spirit, a good one. It'll be a good drive.
Steve Squyres stops in to wish us luck. "I'd love to see you break the century mark [100m], and beat Opportunity, too." He shakes his head in wonder at how far we've come. "I remember seeing Jennifer Trosper standing up at some early meeting and saying the project can't commit to more than 325m, total."
Then he comes to the real reason he's eager for us to beat Opportunity's record. "When Opportunity's done noodling around at Endurance Crater, we'll need something to light a fire under the science team."
Ah-HAH! He's just using us. Bastard!
[Next post: sol 129, May 14.]
But we're on for the three-hour drive tomorrow. We have a chance -- a small one, mind you, but a chance -- of beating even Opportunity's 140m record. Art wants this so badly he can taste it.
We didn't pick the best terrain to try it in. The way in front of us is blocked by a set of ridges, starting about 40m away, which are mostly too tall to drive over. John has planned a zig-zagging path to take us through them to better terrain beyond, but this requires more precise driving than I feel comfortable with -- particularly at that distance. When he shows me the plan, we spend some time looking around at alternatives as well. I spot a relatively low point at the right-hand end of the PANCAM data, one we could easily drive over. It's 60m away, and flat, straight driving all the way. After that, we can turn on the autonav. I talk John into abandoning the work he's done so far and aiming for this spot instead.
The drive becomes simple, just two waypoints -- one blind, one more for autonav. We'll have 40 minutes of blind driving (taking us 60m), followed by about 2.5 hours on autonav. At best, the autonav might take us 87m, so our best case for nextersol is 147m. Opportunity's record is 140.996.
But power concerns interfere. Our TAP tries hard not to, but in the end he has to cut 10 minutes out of the drive, meaning our best case will be about 142m. If the rover has to stop for any reason -- if it has to avoid a single obstacle along that path -- that will slow it down enough that we won't break the record. In this terrain, it's inconceivable that we'll go so far without autonav finding an obstacle, so mentally I've already given up on breaking the record. I don't care about it, anyway. It's more interesting to better our own personal best, and we're going to do that -- we'll set a new record for Spirit, a good one. It'll be a good drive.
Steve Squyres stops in to wish us luck. "I'd love to see you break the century mark [100m], and beat Opportunity, too." He shakes his head in wonder at how far we've come. "I remember seeing Jennifer Trosper standing up at some early meeting and saying the project can't commit to more than 325m, total."
Then he comes to the real reason he's eager for us to beat Opportunity's record. "When Opportunity's done noodling around at Endurance Crater, we'll need something to light a fire under the science team."
Ah-HAH! He's just using us. Bastard!
[Next post: sol 129, May 14.]
2009-05-09
Spirit Sol 124
Yestersol, we traversed an incredible 180m!
Well, not really. What happened was, we did our turn for comm (turning the rover to the best direction for communication, so that we get more data back) after telling the rover it was in a new site. This happened to be the right thing to do for yesterday's drive, but it's unusual, and it confused the ground software that processes the results. That software saw that the turn took place at position 0,0 (in the new site, but the software in question doesn't know about that), so it thought that we'd driven all the way back to where we'd started -- roughly doubling our actual distance traveled. In reality, we still covered 95.21m, which is yet another new record.
Like most of the records we've been setting, this one won't stand for long. We normally spend at most an hour driving each sol, sometimes two. Nextersol, we'll get about two and a half hours. For the sol after that, if our available power level remains high, they're planning to allocate three hours to driving, which will really let us strut our stuff.
Not that we have anything to complain about. Even with the drive-time allocations we've been getting, we're racking up the meters. In fact, we can now say we're racking up the miles: I missed it, but a couple of sols ago we passed 1.6km, which is roughly one full mile. We're over 1.9km total distance now, and we're about halfway from Bonneville to the Columbia Hills.
The meetings are happening so close together on our new compressed schedule that we're showing the same animation practically back to back -- sometimes literally only 15 or 30 minutes apart. Also, it's now usually the same people at each meeting, and the sequence doesn't change much. So today Art suggests that we skip one of the showings.
"Are you going to feel unloved if we don't show your hard work?" he asks me.
"No," I reply. "I'll feel unloved whether we show it or not."
"Good!" he says, and we move on.
Well, not really. What happened was, we did our turn for comm (turning the rover to the best direction for communication, so that we get more data back) after telling the rover it was in a new site. This happened to be the right thing to do for yesterday's drive, but it's unusual, and it confused the ground software that processes the results. That software saw that the turn took place at position 0,0 (in the new site, but the software in question doesn't know about that), so it thought that we'd driven all the way back to where we'd started -- roughly doubling our actual distance traveled. In reality, we still covered 95.21m, which is yet another new record.
Like most of the records we've been setting, this one won't stand for long. We normally spend at most an hour driving each sol, sometimes two. Nextersol, we'll get about two and a half hours. For the sol after that, if our available power level remains high, they're planning to allocate three hours to driving, which will really let us strut our stuff.
Not that we have anything to complain about. Even with the drive-time allocations we've been getting, we're racking up the meters. In fact, we can now say we're racking up the miles: I missed it, but a couple of sols ago we passed 1.6km, which is roughly one full mile. We're over 1.9km total distance now, and we're about halfway from Bonneville to the Columbia Hills.
The meetings are happening so close together on our new compressed schedule that we're showing the same animation practically back to back -- sometimes literally only 15 or 30 minutes apart. Also, it's now usually the same people at each meeting, and the sequence doesn't change much. So today Art suggests that we skip one of the showings.
"Are you going to feel unloved if we don't show your hard work?" he asks me.
"No," I reply. "I'll feel unloved whether we show it or not."
"Good!" he says, and we move on.
2009-05-08
Spirit Sol 123
I walk into the sequencing room and announce: "Gentlemen, behold -- Krispy Kreme!"
Ah, Aqua Teen Hungerforce ... nobody gets the allusion, but they like the doughnuts, anyway.
It's a good thing I came in early, because they moved the schedule up and didn't think to tell me. So instead of having an hour to get up to speed before the activity plan approval meeting, I arrive just before the meeting starts. But these drive sequences haven't been that complex, and this one's no exception, which helps me catch up quickly. More plowing ahead toward the hills.
As we're walking through the sequence, Chris asks me if there's an option to change some RoSE behavior he dislikes. "No, sorry," I tell him. He shrugs. "Ah, it's OK. RoSE probably has the fewest idiosyncrasies of any tool out there," he says. As if on cue, one of the other tools we're using crashes.
Our Linux Journal article was approved at long last! It's been delayed so long that some of the wording in it is now inappropriate (such as a reference to the mission as "a three-month journey of exploration" -- it's already been longer than that, and the article won't appear for some time yet), and every time I read it I see new things I'd change. But, to hell with it: it's going to see print, and that's going to be cool. Gentlemen, behold: I'm in the Linux Journal!
The other day, I thought I was being excessively concerned about the possibility of a sequence being interrupted during the end-of-drive wheel wiggle. Was all that business about calling Art at home and begging the MER-B uplink team to repackage the uplink too paranoid? Maybe not: yesterday's drive was interrupted during the wheel wiggle. Yesterday was structured differently, so the drive sequence wasn't deactivated then -- it went on to unmask the goal error, and the rover is fine. But this is exactly what I feared could happen to the sequence I was concerned about; if it had, our nightmare scenario would have played out. Score one for paranoia.
Ah, Aqua Teen Hungerforce ... nobody gets the allusion, but they like the doughnuts, anyway.
It's a good thing I came in early, because they moved the schedule up and didn't think to tell me. So instead of having an hour to get up to speed before the activity plan approval meeting, I arrive just before the meeting starts. But these drive sequences haven't been that complex, and this one's no exception, which helps me catch up quickly. More plowing ahead toward the hills.
As we're walking through the sequence, Chris asks me if there's an option to change some RoSE behavior he dislikes. "No, sorry," I tell him. He shrugs. "Ah, it's OK. RoSE probably has the fewest idiosyncrasies of any tool out there," he says. As if on cue, one of the other tools we're using crashes.
Our Linux Journal article was approved at long last! It's been delayed so long that some of the wording in it is now inappropriate (such as a reference to the mission as "a three-month journey of exploration" -- it's already been longer than that, and the article won't appear for some time yet), and every time I read it I see new things I'd change. But, to hell with it: it's going to see print, and that's going to be cool. Gentlemen, behold: I'm in the Linux Journal!
The other day, I thought I was being excessively concerned about the possibility of a sequence being interrupted during the end-of-drive wheel wiggle. Was all that business about calling Art at home and begging the MER-B uplink team to repackage the uplink too paranoid? Maybe not: yesterday's drive was interrupted during the wheel wiggle. Yesterday was structured differently, so the drive sequence wasn't deactivated then -- it went on to unmask the goal error, and the rover is fine. But this is exactly what I feared could happen to the sequence I was concerned about; if it had, our nightmare scenario would have played out. Score one for paranoia.
2009-05-04
Spirit Sol 119
Naturally, the morning drive ran to completion. It didn't even come close to being cut off during the wheel wiggle. So all that work was for nought. Of course, if we hadn't done it, and the drive had been cut off .... At least I was able to sleep last night.
Anyway, the bad stuff didn't happen. Indeed, the drive went very well. We didn't make 100m, but we did set yet another new record, 92.7m.
"We set a new record, but we didn't smash the old one," John says.
"So we made Art happy," I say.
"Yeah," Craig Leff chimes in, "that's the bad news, but ...."
Another good thing that came out of yesterday's worries was a realization that this could have happened to nearly any drive -- in principle, any of our drives could have gotten chopped off this way. We just never thought about it before. Most of the time, we drive only once per sol, so if a drive got deactivated during the wheel wiggle, we'd know there was a problem -- we wouldn't have gotten the final drive imagery, which happens after that. But if that did happen, we might not notice that the goal error was masked, so we might unwittingly send a drive sequence that would execute with the mask on. To avert this possibility, I go talk to the mobility/IDD folks and ask them nicely to start putting the mask's state in their downlink reports.
We got pretty close to the crater rim. Thisol we'll drive in a zigzag path up to the rim, take some pictures of the crater, and drive on. At one point, I look at the drive from overhead, and realize our path forms a capital "M." So I attach a picture of this to my uplink report, with the note, "Today's drive sequences have been brought to you by the letter 'M.'"
Yes, sequences -- plural. We're splitting the sequence again. Art has some questions about this at the walkthrough, and I go through why there won't be a problem. He believes me, but says something to John along the lines of, "What will we do if we see a problem?"
John shrugs. "That's why we have an RP-2 here, for precisely that kind of thing."
"Yeah," Art replies, "then you [gesturing at me] can call us at home and tell us there's a problem."
"Well, if you'd rather I not tell you ..." I shrug.
Art's eyes widen. "No, no!" he exclaims. "You did the right thing!"
Damn straight, I did.
When I'm on my way out, I run into Richard Kornfeld. He was one of our SIEs, but I haven't seen him for a while, so I've assumed he's been working on Opportunity. Nope. He's moved on; he's now doing some kind of system engineering or something for MSL. He asks if I'm thinking of transferring to another project.
"No way," I tell him. "I'm staying until the rovers die or I do."
[Next post: sol 123, May 8.]
Courtesy NASA/JPL-Caltech. Looking ahead. Those hills are getting bigger all the time.
Anyway, the bad stuff didn't happen. Indeed, the drive went very well. We didn't make 100m, but we did set yet another new record, 92.7m.
"We set a new record, but we didn't smash the old one," John says.
"So we made Art happy," I say.
"Yeah," Craig Leff chimes in, "that's the bad news, but ...."
Another good thing that came out of yesterday's worries was a realization that this could have happened to nearly any drive -- in principle, any of our drives could have gotten chopped off this way. We just never thought about it before. Most of the time, we drive only once per sol, so if a drive got deactivated during the wheel wiggle, we'd know there was a problem -- we wouldn't have gotten the final drive imagery, which happens after that. But if that did happen, we might not notice that the goal error was masked, so we might unwittingly send a drive sequence that would execute with the mask on. To avert this possibility, I go talk to the mobility/IDD folks and ask them nicely to start putting the mask's state in their downlink reports.
We got pretty close to the crater rim. Thisol we'll drive in a zigzag path up to the rim, take some pictures of the crater, and drive on. At one point, I look at the drive from overhead, and realize our path forms a capital "M." So I attach a picture of this to my uplink report, with the note, "Today's drive sequences have been brought to you by the letter 'M.'"
Yes, sequences -- plural. We're splitting the sequence again. Art has some questions about this at the walkthrough, and I go through why there won't be a problem. He believes me, but says something to John along the lines of, "What will we do if we see a problem?"
John shrugs. "That's why we have an RP-2 here, for precisely that kind of thing."
"Yeah," Art replies, "then you [gesturing at me] can call us at home and tell us there's a problem."
"Well, if you'd rather I not tell you ..." I shrug.
Art's eyes widen. "No, no!" he exclaims. "You did the right thing!"
Damn straight, I did.
When I'm on my way out, I run into Richard Kornfeld. He was one of our SIEs, but I haven't seen him for a while, so I've assumed he's been working on Opportunity. Nope. He's moved on; he's now doing some kind of system engineering or something for MSL. He asks if I'm thinking of transferring to another project.
"No way," I tell him. "I'm staying until the rovers die or I do."
[Next post: sol 123, May 8.]
Courtesy NASA/JPL-Caltech. Looking ahead. Those hills are getting bigger all the time.
2009-05-03
Spirit Sol 118
This was not our best drive. The rover traversed about 37m (not that long ago, I keep reminding myself, this would have been a fantastic sol's drive), then the autonav failed out, deciding that there was no safe path from its present position. Oddly, according to the rover's on-board navigation maps, the terrain underneath the rover itself is flagged as too dangerous to drive into, which should never happen.
What appears to have happened is this. Going on autonav, the rover drove into a kind of ditch. (This was safe -- the ditch was wide enough, and its walls shallow enough, to enter.) When it was trying to climb out, its pitch changed significantly -- it's going up the ditch wall -- and this changed its evaluation of the immediately surrounding terrain. It decided that the path in front of it was a huge cliff (an illusion caused by its inability to see the side of the ditch it was climbing, only the terrain beyond) and behind it was a huge wall (another illusion -- this was really the floor and rear wall of the ditch). So it stopped where it was. The vehicle is safe and healthy, just a little confused about how ditches work.
For nextersol, John's plan is to floor it. He's planning a 100m drive -- which will be yet another record for us, if we make it. Art and I both tell him, tongue in cheek, that that's a mistake. We should set a 91m record, then a 92m record, and so on, not jump all the way to 100m! Art starts semi-seriously pushing for this, comparing it to the old Soviet weightlifters. They'd be able to break the record by, say, five pounds, but they'd do it by setting five new records, one pound apart. The more he pushes for the idea, the more I think he starts to like it. But John stands firm: "No, I want us to set a big record, and make them all quail in terror ...." Of course, I'm all for this, and in the end I actually extend the drive another 4m or so to ensure we actually make it up onto the rim. Total planned distance: about 106m.
We want to drive as much as possible nextersol, but the rover needs to nap in the middle of the day for power reasons. So we end up splitting the drive sequence into two parts (blind pre-siesta and autonav post-siesta), something we haven't been allowed to do since the Humphrey Bump Incident. I've been pushing to enable this, and it looks like I've finally won. The Curse of the Split Drive has been lifted!
A piece of delayed good news: we nailed the MTES of Potrillo. A week ago, we did that drive where we had to drive 12m before trying to hit a bullseye with the MTES, a hard problem because of the unknown degree of slip and the MTES's narrow field of view. Today's MTES PUL, Laura, cheerfully plunks down a black-and-white NAVCAM image with the MTES field of view shaded in purple. The rock we were aiming for, Potrillo, is in bounds -- it's at the bottom right-hand edge, but it's in there. Our perfect record is unbroken.
During a lull, Larry Soderblom distracts the entire room by showing a gorgeous 3-D model of the Columbia Hills on the room's projection screen. The model was compiled from orbital images taken by MGS. Larry shows us where we're heading, and breaks the news that we're not likely to climb the hills when we get there. "We would be crazy to climb those hills," he says. "We'd get a great view, but there's nothing up there when you get there." However, there's a lot of good stuff near the hills' base, including some cool "steps" about 500m away from our current destination. Best of all, we see formations that might be bedrock -- just the stuff we need to tell Gusev's "water story" -- assuming, as everyone does, that it has such a story to tell.
We finish up and everyone else leaves; I'm still working on my uplink report. And I see a problem with the drive.
At the end of every drive, we perform a "wheel wiggle" -- the rover turns its four steerable wheels all the way in, then straightens them out again. The theory behind this is that it will help settle the rover in its current position, mitigating the possible effect of settling into the soil, falling off a rock, etc. It's important, especially for IDD work, that the rover move as little as possible after sending us its final images. Even a few millimeters of unexpected settling can be a bad thing.
Ordinarily, the rover won't perform any driving maneuvers, not even a wheel wiggle, when it has a goal error (meaning, usually, that it didn't drive all the way to the target we specified; this is fairly common, as we often deliberately set goals we know the rover won't reach). We want the rover to wiggle its wheels even when it didn't reach its goal, though, because the science team might just want to use the IDD wherever we ended up. So, while wiggling the wheels, we "mask" the goal error. We send a command telling the rover to ignore the goal error, then we have it wiggle the wheels, then we send another command saying to pay attention to the goal error again ("unmasking" it).
Here's what we didn't think of. Suppose the first drive sequence gets deactivated while the rover is wiggling its wheels. Then it won't get to the command that says to pay attention to the goal error. That means that when the afternoon drive sequence executes, the rover will ignore any goal errors that may arise. This would not be good.
Now, it's unlikely that this will even happen. We expect that we allowed adequate time for the morning drive, meaning that it won't be running when we go to sleep -- it will have already finished. And the wheel wiggle takes less than a minute, so it's a somewhat narrow window of vulnerability. But we can't be certain, and the fact that the wiggle occurs at the end makes the odds a little worse for us.
Ah, hell. The Curse of the Split Drive is back.
I call Art -- at home, oy vey -- to alert him to the problem. He thinks of something I hadn't: is the mask volatile (that is, forgotten when the rover shuts down) or non-volatile (remembered even across reboots)? If the state is volatile, it doesn't matter; the mask will be forgotten when the rover takes its nap, and when it awakes, everything will be fine. I don't know, so I call Mark Maimone at home, poor guy. Mark greps[1] through the flight software's source code -- and calls me back with bad news. The mask is remembered across reboots. We have a problem to solve.
Solving the problem in the sequence is easy, taking me just a couple of minutes. I remove the commands that mask and unmask the goal error; if the wheel wiggle doesn't happen, we just won't worry about it. The rover's going to drive again in the afternoon, anyway; the odds that we were going to care about this particular wiggle were already slim.[2] The harder problem is: how are we going to get these commands to the rover? Everyone on the Spirit uplink team has already gone home, including the people who know how to convert the commands I've written into the binary form we uplink.
Ah, but the Opportunity uplink team is still here -- this is a late day for them. So I run upstairs with the intention of -- on Art's behalf -- begging their mission manager, Rick Welch, for the use of their Sequence Integration Engineer, Dan Moyers. But when I arrive, they're having their sequence walkthrough. I don't want to interrupt this, so I sit in a corner and try to stay quiet. Internally, I keep hearing Inigo Montoya talking to Miracle Max: "Sir, we are in a terrible hurry ...." But I'm mindful of Miracle Max's reply, and I keep my mouth shut.
The moment they're finished, I ask Rick for the favor. They're running late today, but it turns out this is a good time -- Dan is sitting idle, waiting for some other sequences to be delivered. So Dan and Rick and Bill Bensler wrap up the sequences, sign the paperwork, and run it downstairs for me. The MER-B team saves the day.
I have one more thing to do before I go. I go back downstairs, log in, and write a big, big thank-you to the MER-B team in my uplink report.
Courtesy NASA/JPL-Caltech. The final front HAZCAM from the previous drive. Doesn't look scary to me, but it looked scary to Spirit.
Footnotes:
[1] A geek term meaning "searches."
[2] It seems a better solution would be to unconditionally unmask the goal error at the top of the second sequence. I don't know why I didn't think of that then. Time pressure, maybe.
What appears to have happened is this. Going on autonav, the rover drove into a kind of ditch. (This was safe -- the ditch was wide enough, and its walls shallow enough, to enter.) When it was trying to climb out, its pitch changed significantly -- it's going up the ditch wall -- and this changed its evaluation of the immediately surrounding terrain. It decided that the path in front of it was a huge cliff (an illusion caused by its inability to see the side of the ditch it was climbing, only the terrain beyond) and behind it was a huge wall (another illusion -- this was really the floor and rear wall of the ditch). So it stopped where it was. The vehicle is safe and healthy, just a little confused about how ditches work.
For nextersol, John's plan is to floor it. He's planning a 100m drive -- which will be yet another record for us, if we make it. Art and I both tell him, tongue in cheek, that that's a mistake. We should set a 91m record, then a 92m record, and so on, not jump all the way to 100m! Art starts semi-seriously pushing for this, comparing it to the old Soviet weightlifters. They'd be able to break the record by, say, five pounds, but they'd do it by setting five new records, one pound apart. The more he pushes for the idea, the more I think he starts to like it. But John stands firm: "No, I want us to set a big record, and make them all quail in terror ...." Of course, I'm all for this, and in the end I actually extend the drive another 4m or so to ensure we actually make it up onto the rim. Total planned distance: about 106m.
We want to drive as much as possible nextersol, but the rover needs to nap in the middle of the day for power reasons. So we end up splitting the drive sequence into two parts (blind pre-siesta and autonav post-siesta), something we haven't been allowed to do since the Humphrey Bump Incident. I've been pushing to enable this, and it looks like I've finally won. The Curse of the Split Drive has been lifted!
A piece of delayed good news: we nailed the MTES of Potrillo. A week ago, we did that drive where we had to drive 12m before trying to hit a bullseye with the MTES, a hard problem because of the unknown degree of slip and the MTES's narrow field of view. Today's MTES PUL, Laura, cheerfully plunks down a black-and-white NAVCAM image with the MTES field of view shaded in purple. The rock we were aiming for, Potrillo, is in bounds -- it's at the bottom right-hand edge, but it's in there. Our perfect record is unbroken.
During a lull, Larry Soderblom distracts the entire room by showing a gorgeous 3-D model of the Columbia Hills on the room's projection screen. The model was compiled from orbital images taken by MGS. Larry shows us where we're heading, and breaks the news that we're not likely to climb the hills when we get there. "We would be crazy to climb those hills," he says. "We'd get a great view, but there's nothing up there when you get there." However, there's a lot of good stuff near the hills' base, including some cool "steps" about 500m away from our current destination. Best of all, we see formations that might be bedrock -- just the stuff we need to tell Gusev's "water story" -- assuming, as everyone does, that it has such a story to tell.
We finish up and everyone else leaves; I'm still working on my uplink report. And I see a problem with the drive.
At the end of every drive, we perform a "wheel wiggle" -- the rover turns its four steerable wheels all the way in, then straightens them out again. The theory behind this is that it will help settle the rover in its current position, mitigating the possible effect of settling into the soil, falling off a rock, etc. It's important, especially for IDD work, that the rover move as little as possible after sending us its final images. Even a few millimeters of unexpected settling can be a bad thing.
Ordinarily, the rover won't perform any driving maneuvers, not even a wheel wiggle, when it has a goal error (meaning, usually, that it didn't drive all the way to the target we specified; this is fairly common, as we often deliberately set goals we know the rover won't reach). We want the rover to wiggle its wheels even when it didn't reach its goal, though, because the science team might just want to use the IDD wherever we ended up. So, while wiggling the wheels, we "mask" the goal error. We send a command telling the rover to ignore the goal error, then we have it wiggle the wheels, then we send another command saying to pay attention to the goal error again ("unmasking" it).
Here's what we didn't think of. Suppose the first drive sequence gets deactivated while the rover is wiggling its wheels. Then it won't get to the command that says to pay attention to the goal error. That means that when the afternoon drive sequence executes, the rover will ignore any goal errors that may arise. This would not be good.
Now, it's unlikely that this will even happen. We expect that we allowed adequate time for the morning drive, meaning that it won't be running when we go to sleep -- it will have already finished. And the wheel wiggle takes less than a minute, so it's a somewhat narrow window of vulnerability. But we can't be certain, and the fact that the wiggle occurs at the end makes the odds a little worse for us.
Ah, hell. The Curse of the Split Drive is back.
I call Art -- at home, oy vey -- to alert him to the problem. He thinks of something I hadn't: is the mask volatile (that is, forgotten when the rover shuts down) or non-volatile (remembered even across reboots)? If the state is volatile, it doesn't matter; the mask will be forgotten when the rover takes its nap, and when it awakes, everything will be fine. I don't know, so I call Mark Maimone at home, poor guy. Mark greps[1] through the flight software's source code -- and calls me back with bad news. The mask is remembered across reboots. We have a problem to solve.
Solving the problem in the sequence is easy, taking me just a couple of minutes. I remove the commands that mask and unmask the goal error; if the wheel wiggle doesn't happen, we just won't worry about it. The rover's going to drive again in the afternoon, anyway; the odds that we were going to care about this particular wiggle were already slim.[2] The harder problem is: how are we going to get these commands to the rover? Everyone on the Spirit uplink team has already gone home, including the people who know how to convert the commands I've written into the binary form we uplink.
Ah, but the Opportunity uplink team is still here -- this is a late day for them. So I run upstairs with the intention of -- on Art's behalf -- begging their mission manager, Rick Welch, for the use of their Sequence Integration Engineer, Dan Moyers. But when I arrive, they're having their sequence walkthrough. I don't want to interrupt this, so I sit in a corner and try to stay quiet. Internally, I keep hearing Inigo Montoya talking to Miracle Max: "Sir, we are in a terrible hurry ...." But I'm mindful of Miracle Max's reply, and I keep my mouth shut.
The moment they're finished, I ask Rick for the favor. They're running late today, but it turns out this is a good time -- Dan is sitting idle, waiting for some other sequences to be delivered. So Dan and Rick and Bill Bensler wrap up the sequences, sign the paperwork, and run it downstairs for me. The MER-B team saves the day.
I have one more thing to do before I go. I go back downstairs, log in, and write a big, big thank-you to the MER-B team in my uplink report.
Courtesy NASA/JPL-Caltech. The final front HAZCAM from the previous drive. Doesn't look scary to me, but it looked scary to Spirit.
Footnotes:
[1] A geek term meaning "searches."
[2] It seems a better solution would be to unconditionally unmask the goal error at the top of the second sequence. I don't know why I didn't think of that then. Time pressure, maybe.
2009-05-02
Spirit Sol 117
Well, our new distance record didn't last very long. The moment I arrive, John informs me that they just set a new record. They're not sure of the exact number yet, but it's somewhere between 90m and 92m. (90.49, it turns out.) There was an uplink error on sol 115, which kept us from sending any sequences to the rover, so sol 116 -- the sol that's just ending -- was a repeat of 115. As a result, this record-setting drive sequence ended up being worked on (over two days) by Chris, John, and Ashitey -- all the Spirit drivers except me. Sigh.
He's having a little trouble with thisol's drive, though: the software keeps placing the waypoint icons several meters below the ground. According to RSVP, Spirit will immediately descend about 10m, then drive normally underground for a few hours. We feel this is unlikely, so we investigate a little. It turns out that due to a sequencing error in the record-setting drive sequence, we took the final front HAZCAM images at the wrong point during the drive -- we took them before telling the rover it had reached a new site, rather than after. As a result, when the data from those images were merged in with the other data, they screwed up the height map -- the part of the data that tells RSVP where the ground is. When we switch to using updated data with a corrected height map, everything works fine.
One of the geologists, Rob Sullivan, tells me excitedly that the trench we dug the other day worked great. "It's nice to see the grins on the instrument people's faces," he says. What did they find? "They found different stuff than usual," he says. They don't know what or why yet, "but that's what you hope for." If they're happy, I'm happy.
Our next target is another crater about 70m away -- should be a simple, one-sol drive, given the way we've been driving lately. The only fly in the ointment is that our data is limited. For some reason, we didn't get any PANCAM images to support the drive, so we're only able to use NAVCAMs; this limits us to about 15m of blind driving, after which the rover will go as far as it can on its own -- slower, of course. As a result, we might have to take two sols to do this one-sol drive.
Art thinks the rovers will last until September 19th. Why then? Because September 18th is the winter solstice on Mars. If we can survive that, our energy picture starts to get better again.
Saina is leaving next week for her dad's funeral back home in Iran. She hasn't been home in 10 years, she says, and she expects a lot of changes. Iran is a developing nation, and "it's different every year."
And speaking of comings and goings, Jennifer Trosper has now left. She didn't just leave the mission, she's left JPL -- or, to be more precise, she's gone to D.C. to represent JPL at NASA. (Her husband got a transfer.) This opened a mission manager role -- which is now filled by Art. Heaven help us!
Courtesy NASA/JPL-Caltech. A front HAZCAM image from sol 114, showing us poking at the trench we successfully dug when I was last on shift (sol 113).
He's having a little trouble with thisol's drive, though: the software keeps placing the waypoint icons several meters below the ground. According to RSVP, Spirit will immediately descend about 10m, then drive normally underground for a few hours. We feel this is unlikely, so we investigate a little. It turns out that due to a sequencing error in the record-setting drive sequence, we took the final front HAZCAM images at the wrong point during the drive -- we took them before telling the rover it had reached a new site, rather than after. As a result, when the data from those images were merged in with the other data, they screwed up the height map -- the part of the data that tells RSVP where the ground is. When we switch to using updated data with a corrected height map, everything works fine.
One of the geologists, Rob Sullivan, tells me excitedly that the trench we dug the other day worked great. "It's nice to see the grins on the instrument people's faces," he says. What did they find? "They found different stuff than usual," he says. They don't know what or why yet, "but that's what you hope for." If they're happy, I'm happy.
Our next target is another crater about 70m away -- should be a simple, one-sol drive, given the way we've been driving lately. The only fly in the ointment is that our data is limited. For some reason, we didn't get any PANCAM images to support the drive, so we're only able to use NAVCAMs; this limits us to about 15m of blind driving, after which the rover will go as far as it can on its own -- slower, of course. As a result, we might have to take two sols to do this one-sol drive.
Art thinks the rovers will last until September 19th. Why then? Because September 18th is the winter solstice on Mars. If we can survive that, our energy picture starts to get better again.
Saina is leaving next week for her dad's funeral back home in Iran. She hasn't been home in 10 years, she says, and she expects a lot of changes. Iran is a developing nation, and "it's different every year."
And speaking of comings and goings, Jennifer Trosper has now left. She didn't just leave the mission, she's left JPL -- or, to be more precise, she's gone to D.C. to represent JPL at NASA. (Her husband got a transfer.) This opened a mission manager role -- which is now filled by Art. Heaven help us!
Courtesy NASA/JPL-Caltech. A front HAZCAM image from sol 114, showing us poking at the trench we successfully dug when I was last on shift (sol 113).
Subscribe to:
Posts (Atom)
Posts
