Bad news: in about ten days, Khaled will be a Rover Planner no more. He's leaving, taking over the role of coordinating the flight software uploads.
As politely as I can, I ask him the only question that comes to mind: "Are you crazy?"
"I've been asking myself that," he admits with a mournful tone. "It's certainly not going to be as much fun as this job. But this one doesn't meet those ECAP goals I keep hearing, so ...."[1]
Well, the guy's gotta do what he's gotta do. Andy isn't going to be happy to hear this -- more headaches for him. I'm not too happy about it myself, honestly. Khaled's a fine RP; he came up to speed quickly, knows the rovers well, and is thorough, reliable, and conscientious. He'll be hard to replace.
One thing I won't miss is the smell of his wife's cooking, when he brings in his leftovers for lunch nearly every day. Not that her cooking smells bad. Quite the contrary, it always smells delicious. The problem with this is that it makes my own hunger impossible to ignore -- whenever I get a whiff of Khaled's lunch, I have to go buy lunch from the cafeteria. And that's never as good as his. Darn him!
Thisol's sequencing is not much more complex than yestersol's. Khaled and I put the extra time to good use by preparing the drive sequences. He'd already put together a zeroth draft for the first couple of upcoming drive sols (510 and 511), and we go over them and make some changes together.
Gee, he's not only thorough, reliable, and conscientious, he also works ahead, making things simpler for the rest of us. And now he's leaving.
Bastard.[2]
Footnotes:
[1] An ECAP -- Employee Contribution Assessment and Planning -- is JPL's annual employee performance review.
[2] Khaled has come and gone a couple of times now. Right now, he's off working on Moonrise, but I hope he'll be back again soon. He's damn good.
2010-06-30
2010-06-29
Opportunity Sol 508 (Spirit Sol 529)
This is one of the simplest days I've had in a long time: just a tool change from APXS to MB, continuing our exploration of the soil near Purgatory Ripple. And I'm not even RP-1 (Jeng is), so I just have to review it.
Yawn.
One-way light-time to Mars has been falling. Right now it's eight minutes, twenty-seven seconds -- we're just about equidistant from Mars and the Sun. A big Isosceles triangle in space. 8:27 -- that's 507 seconds, one less than the sol number we're planning. That's the first time those numbers have crossed. "The rovers haven't been this close to Earth since December 2003," Steve remarks.
The CAM goes smoothly; another one down. "One more day and we hit the gas!" Steve exclaims. "I can't wait to get outta here!"
Yawn.
One-way light-time to Mars has been falling. Right now it's eight minutes, twenty-seven seconds -- we're just about equidistant from Mars and the Sun. A big Isosceles triangle in space. 8:27 -- that's 507 seconds, one less than the sol number we're planning. That's the first time those numbers have crossed. "The rovers haven't been this close to Earth since December 2003," Steve remarks.
The CAM goes smoothly; another one down. "One more day and we hit the gas!" Steve exclaims. "I can't wait to get outta here!"
2010-06-28
Opportunity Sol 507 (Spirit Sol 528)
Ah, that feels better. The drive went just exactly as planned; the old track, tailings, and wide swaths of undisturbed soil are within the IDD work volume. I've never been so excited about an 80cm drive.
Nor has most of the rest of the team. I forget, they actually started their approach to this spot well before I got here -- it took something like a week to do the drive away, turn-in-place, and drive back -- and today was the last chance to get it. But we got it. Brenda says Steve is "happy as a clam," and he is. "Take your time getting SAP set up, Brenda," he says. "I'm just gonna sit here and gaze in wonder at this reachability map -- nice job, Rover Planners!"
I feel a little silly about taking a bow (figuratively!). It's just 80cm; I'm supposed to be able to do this with my eyes closed. Well, whatever.
I take a moment to look at the drive telemetry. After all that time we spent worrying about what to set them to, I'm curious about what the bogie angles were during this drive. The left bogie started at 14 degrees -- the value that aborted our previous drive -- and we knew it was going to increase as we finished climbing North Ripple, but how much did it increase?
It went to 15 degrees. That's it. All that time we spent quibbling about the drive, arguing over whether 20 degrees was enough headroom, and so on ... and we needed about one extra degree.
Well, at least it worked. It did execute nicely, and now we're ready to swing the IDD out and attack the soil. We'll start with a 2x2 MI mosaic around a patch of undisturbed soil near the RF wheel, then do another such mosaic, followed by an APXS placement, in the track.
This is fairly straightforward IDD work, though it's making Jeng nervous. Maybe it's that he doesn't yet trust his ability to sequence the IDD, or maybe it's the early deadline -- this is what we call a "tight" sol, with uplink in just a few hours, leaving little room for error. But it's an undramatic day; with me looking over his shoulder, Jeng cranks out the IDD sequence in plenty of time.
At the Activity Plan Approval Meeting, Paolo points out that the near side of the track, the spot we've picked as our representative undisturbed-soil location, is darker than the far side. This means that the far side has lighter-colored dust, which generally means smaller, finer dust particles. This is the stuff we sometimes call "foo-foo dust," and there's a going hypothesis that that's what Purgatory Ripple is made of. So, Paolo suggests, maybe we should be IDDing there instead?[1]
Rob Sullivan answers the question. "What we think we're seeing there is the result of a recent wind event -- in the last few sols, a gust of wind came through here and blew away a lot of the material we dredged up when we exited Purgatory Ripple. So a lot of that dust probably blew in from that event. The nearby patch is more likely to be the native material of Purgatory Ripple."
"So your interpretation is right," Steve Squyres chimes in. "That lighter-colored stuff probably is foo-foo dust -- but in this case, that's a bad thing. But keep makin' suggestions!"
The rest of the day goes ahead pretty nominally. At the CAM, Dan resurrects the Picture-of-the-Day tradition to show a true-color PANCAM image of the trench we (inadvertently) made in Purgatory Ripple. It's enormous, as wide as a wheel and at least that deep.
"How did we ever get outta that?" he wonders aloud.
"We have really good rover drivers," Wendy Calvin says.
"And a whole lotta patience," adds Steve.
Courtesy NASA/JPL-Caltech. At last. That's what we've been trying to do.
Footnotes:
[1] A characteristically brilliant observation. At this point, Paolo had been on the project for only a few weeks -- a couple of months, at most -- but he was already showing the keen insights and attention to detail that would later make him a top-notch rover driver.
Nor has most of the rest of the team. I forget, they actually started their approach to this spot well before I got here -- it took something like a week to do the drive away, turn-in-place, and drive back -- and today was the last chance to get it. But we got it. Brenda says Steve is "happy as a clam," and he is. "Take your time getting SAP set up, Brenda," he says. "I'm just gonna sit here and gaze in wonder at this reachability map -- nice job, Rover Planners!"
I feel a little silly about taking a bow (figuratively!). It's just 80cm; I'm supposed to be able to do this with my eyes closed. Well, whatever.
I take a moment to look at the drive telemetry. After all that time we spent worrying about what to set them to, I'm curious about what the bogie angles were during this drive. The left bogie started at 14 degrees -- the value that aborted our previous drive -- and we knew it was going to increase as we finished climbing North Ripple, but how much did it increase?
It went to 15 degrees. That's it. All that time we spent quibbling about the drive, arguing over whether 20 degrees was enough headroom, and so on ... and we needed about one extra degree.
Well, at least it worked. It did execute nicely, and now we're ready to swing the IDD out and attack the soil. We'll start with a 2x2 MI mosaic around a patch of undisturbed soil near the RF wheel, then do another such mosaic, followed by an APXS placement, in the track.
This is fairly straightforward IDD work, though it's making Jeng nervous. Maybe it's that he doesn't yet trust his ability to sequence the IDD, or maybe it's the early deadline -- this is what we call a "tight" sol, with uplink in just a few hours, leaving little room for error. But it's an undramatic day; with me looking over his shoulder, Jeng cranks out the IDD sequence in plenty of time.
At the Activity Plan Approval Meeting, Paolo points out that the near side of the track, the spot we've picked as our representative undisturbed-soil location, is darker than the far side. This means that the far side has lighter-colored dust, which generally means smaller, finer dust particles. This is the stuff we sometimes call "foo-foo dust," and there's a going hypothesis that that's what Purgatory Ripple is made of. So, Paolo suggests, maybe we should be IDDing there instead?[1]
Rob Sullivan answers the question. "What we think we're seeing there is the result of a recent wind event -- in the last few sols, a gust of wind came through here and blew away a lot of the material we dredged up when we exited Purgatory Ripple. So a lot of that dust probably blew in from that event. The nearby patch is more likely to be the native material of Purgatory Ripple."
"So your interpretation is right," Steve Squyres chimes in. "That lighter-colored stuff probably is foo-foo dust -- but in this case, that's a bad thing. But keep makin' suggestions!"
The rest of the day goes ahead pretty nominally. At the CAM, Dan resurrects the Picture-of-the-Day tradition to show a true-color PANCAM image of the trench we (inadvertently) made in Purgatory Ripple. It's enormous, as wide as a wheel and at least that deep.
"How did we ever get outta that?" he wonders aloud.
"We have really good rover drivers," Wendy Calvin says.
"And a whole lotta patience," adds Steve.
Courtesy NASA/JPL-Caltech. At last. That's what we've been trying to do.
Footnotes:
[1] A characteristically brilliant observation. At this point, Paolo had been on the project for only a few weeks -- a couple of months, at most -- but he was already showing the keen insights and attention to detail that would later make him a top-notch rover driver.
2010-06-25
Opportunity Sol 505 (Spirit Sol 525)
Six fucking centimeters. That's it. Six fucking centimeters is all the distance we made yestersol.
It's my fault. In the drive sequence, we retained most of the limits that had been set for the previous drive. We were just passing over the ground they had meant to pass over, so we figured the same limits would be fine.
Frankly, this was just lazy. And it's what stopped us. As it happened, Opportunity's left bogie articulated just a little more than we expected it to as it climbed the ripple that's under us now (North Ripple), and that terminated the drive. After six centimeters.
We really, really should have caught this in advance. The bogie limit was 14 degrees, and if we'd checked the telemetry, we'd have noticed that it was already just over 11 degrees -- not a lot of headroom there. The really painful part is that I thought of that while we were doing the sequencing, and then got distracted by something and never got back to checking. Oy vey.[1]
And this changes our plans for the weekend. We'd originally intended to IDD the far track -- the one we were driving to -- but since we didn't make it there, that's out. So it'll be a touch-and-go sol, where we IDD whatever happens to be close to us and then try to drive the remaining 69cm or so to the far track.
Assuming we're capable of that much.
Both Jeff Favretto and Steve Squyres are philosophical about the drive result. "This is just a consequence of using conservative limits," Jeff says. And conservative limits are exactly what we promised the Mars Program Office we'd use, adds Steve. But as usual, things aren't that simple. Apparently, the Program management has been putting some pressure on John Callas (acting project manager while Jim Erickson is out of town). He's getting it from both sides: some people are complaining that we're being reckless cowboys; others, that we're too conservative. They want us to be successful without making any mistakes, which is fine to a point, but they've pushed it far enough that it amounts to demanding that we be lucky.[2] I'm glad I don't have his job.[3]
Squyres might have been getting some of that "too-conservative" flack as well. "I want you to drive safely," he tells me, "but I should say that if we don't make it to the target after this attempt, we're just going to IDD whatever we've got and then get out of here." "So, no pressure?" I ask. "Yeah," he grins, "no pressure."
The IDD work is fairly straightforward, and I get it more or less knocked off in a couple of hours. It initially looks like one of the targets the science team chose was too close to our right front wheel, but the simulation says it's OK, so I move on.
The bulk of the sol's work is in arguing over the drive. Not in implementing it, just in arguing about it. I go through every single parameter limit and check it against both the current state and the RSVP-simulated maximum, trying to pick sensible numbers for everything. We're left with a small set of numbers to argue about, the suspension limits -- most crucially, what should be the proper settings for the bogie limits that stopped us on 503? This turns into, I kid you not, a couple of hours' discussion at least.[4]
Maybe this is all ridiculous. The distance of this drive is about equal to a single wheel rotation -- how much trouble could we get into, even if we abandoned the limits altogether? But at this point we're so damn paranoid that we're ready to entertain the ridiculous.
The number we eventually pick for the max bogie limit is 25 degrees. That's about twice what we're at now, and just 5 degrees shy of the hardstop. It's almost impossible for us to actually hit it, and all things considered, I can feel OK about that. If we actually do reach that limit, something really strange will be happening, and we'll want the drive to stop. Which is what the limit is for.
Our paranoia doesn't end there. Steve asked me earlier in the day for a list of everything that could possibly stop the drive, and one thing that could stop it is for the IDD work to fail -- that would prevent the drive from even starting. So I look back at that, and the IDD target that concerned me before starts to look even more worrisome.
RSVP can show something called the "collision volumes," which RSVP visualizes as a set of purple boxes that are slightly bigger than the rover parts. They're like the rover is wearing bulky translucent purple padding all over. The on-board flight software uses these collision volumes to detect a pending self-collision; the idea is to be conservative about the possibility of collisions during (especially) IDD work -- if the padded version of the rover doesn't collide with itself, then the real thing won't. RSVP also tells us whether these collisions will occur in the sequence we've planned, and since the simulation is happy, we should expect that the real thing will be okay.
And yet ... when we're inspecting the nearby IDD target, the collision volumes come so close to intersecting that we have to really work to show that they're not. What if the flight vehicle is just a tiny bit different from this simulation? It shouldn't be, but as I said, we're pretty paranoid now, and the difference would have to be only a tiny one -- maybe small enough that it would have gone undetected until now -- to screw us up once again.
I'm just being paranoid. I'm sure I'm just being paranoid. But I'm not the only one, and we end up deciding to turn off those collision checks during the close approach. We're not coming so close that there's a risk of an actual collision, just a risk that the collision volumes -- the padding -- could collide, and turning them off means we can be sure it won't happen.
I won't feel good about any of this until Monday.
[Next post: sol 528 (Opportunity sol 507), June 28.]
Footnotes:
[1] This exact incident, if I recall aright, is why checking the suspension history from the previous drive is something I now do every sol. It's bad enough to make a dumb mistake, much worse to repeat it.
[2] Fortunately, this was the last time that Mars Program ever made such a stupid demand. cough
[3] Still true. And how.
[4] In a way, this is one of the things I really love about my job: the obsessive focus on getting these things right. We might spend an hour arguing about a degree, or a millimeter. Sometimes those discussions are horrendously painful when I'm in the middle of them, but when I take a step back, I love being in an environment where people are that passionate about even the little things.
It's my fault. In the drive sequence, we retained most of the limits that had been set for the previous drive. We were just passing over the ground they had meant to pass over, so we figured the same limits would be fine.
Frankly, this was just lazy. And it's what stopped us. As it happened, Opportunity's left bogie articulated just a little more than we expected it to as it climbed the ripple that's under us now (North Ripple), and that terminated the drive. After six centimeters.
We really, really should have caught this in advance. The bogie limit was 14 degrees, and if we'd checked the telemetry, we'd have noticed that it was already just over 11 degrees -- not a lot of headroom there. The really painful part is that I thought of that while we were doing the sequencing, and then got distracted by something and never got back to checking. Oy vey.[1]
And this changes our plans for the weekend. We'd originally intended to IDD the far track -- the one we were driving to -- but since we didn't make it there, that's out. So it'll be a touch-and-go sol, where we IDD whatever happens to be close to us and then try to drive the remaining 69cm or so to the far track.
Assuming we're capable of that much.
Both Jeff Favretto and Steve Squyres are philosophical about the drive result. "This is just a consequence of using conservative limits," Jeff says. And conservative limits are exactly what we promised the Mars Program Office we'd use, adds Steve. But as usual, things aren't that simple. Apparently, the Program management has been putting some pressure on John Callas (acting project manager while Jim Erickson is out of town). He's getting it from both sides: some people are complaining that we're being reckless cowboys; others, that we're too conservative. They want us to be successful without making any mistakes, which is fine to a point, but they've pushed it far enough that it amounts to demanding that we be lucky.[2] I'm glad I don't have his job.[3]
Squyres might have been getting some of that "too-conservative" flack as well. "I want you to drive safely," he tells me, "but I should say that if we don't make it to the target after this attempt, we're just going to IDD whatever we've got and then get out of here." "So, no pressure?" I ask. "Yeah," he grins, "no pressure."
The IDD work is fairly straightforward, and I get it more or less knocked off in a couple of hours. It initially looks like one of the targets the science team chose was too close to our right front wheel, but the simulation says it's OK, so I move on.
The bulk of the sol's work is in arguing over the drive. Not in implementing it, just in arguing about it. I go through every single parameter limit and check it against both the current state and the RSVP-simulated maximum, trying to pick sensible numbers for everything. We're left with a small set of numbers to argue about, the suspension limits -- most crucially, what should be the proper settings for the bogie limits that stopped us on 503? This turns into, I kid you not, a couple of hours' discussion at least.[4]
Maybe this is all ridiculous. The distance of this drive is about equal to a single wheel rotation -- how much trouble could we get into, even if we abandoned the limits altogether? But at this point we're so damn paranoid that we're ready to entertain the ridiculous.
The number we eventually pick for the max bogie limit is 25 degrees. That's about twice what we're at now, and just 5 degrees shy of the hardstop. It's almost impossible for us to actually hit it, and all things considered, I can feel OK about that. If we actually do reach that limit, something really strange will be happening, and we'll want the drive to stop. Which is what the limit is for.
Our paranoia doesn't end there. Steve asked me earlier in the day for a list of everything that could possibly stop the drive, and one thing that could stop it is for the IDD work to fail -- that would prevent the drive from even starting. So I look back at that, and the IDD target that concerned me before starts to look even more worrisome.
RSVP can show something called the "collision volumes," which RSVP visualizes as a set of purple boxes that are slightly bigger than the rover parts. They're like the rover is wearing bulky translucent purple padding all over. The on-board flight software uses these collision volumes to detect a pending self-collision; the idea is to be conservative about the possibility of collisions during (especially) IDD work -- if the padded version of the rover doesn't collide with itself, then the real thing won't. RSVP also tells us whether these collisions will occur in the sequence we've planned, and since the simulation is happy, we should expect that the real thing will be okay.
And yet ... when we're inspecting the nearby IDD target, the collision volumes come so close to intersecting that we have to really work to show that they're not. What if the flight vehicle is just a tiny bit different from this simulation? It shouldn't be, but as I said, we're pretty paranoid now, and the difference would have to be only a tiny one -- maybe small enough that it would have gone undetected until now -- to screw us up once again.
I'm just being paranoid. I'm sure I'm just being paranoid. But I'm not the only one, and we end up deciding to turn off those collision checks during the close approach. We're not coming so close that there's a risk of an actual collision, just a risk that the collision volumes -- the padding -- could collide, and turning them off means we can be sure it won't happen.
I won't feel good about any of this until Monday.
[Next post: sol 528 (Opportunity sol 507), June 28.]
Footnotes:
[1] This exact incident, if I recall aright, is why checking the suspension history from the previous drive is something I now do every sol. It's bad enough to make a dumb mistake, much worse to repeat it.
[2] Fortunately, this was the last time that Mars Program ever made such a stupid demand. cough
[3] Still true. And how.
[4] In a way, this is one of the things I really love about my job: the obsessive focus on getting these things right. We might spend an hour arguing about a degree, or a millimeter. Sometimes those discussions are horrendously painful when I'm in the middle of them, but when I take a step back, I love being in an environment where people are that passionate about even the little things.
2010-06-24
Opportunity Sol 504 (Spirit Sol 524)
Yestersol was a two-sol plan and tomorrow's a three-sol weekend plan; we just have a pre-planning meeting today. Assuming none of our paranoid limit checks triggers and we actually make it to our goal, what's on tap for the weekend is simply some light IDD work: a little MI, a little APXS, a little MB. Nothing out of the ordinary.
I hope that will mean we have time to start planning the drive away from Purgatory. I haven't given any thought to it, and I just noticed that I'm scheduled all next week. I've been off this rover for a while, but I want to look like I know what I'm doing.
I hope that will mean we have time to start planning the drive away from Purgatory. I haven't given any thought to it, and I just noticed that I'm scheduled all next week. I've been off this rover for a while, but I want to look like I know what I'm doing.
2010-06-23
Opportunity Sol 503 (Spirit Sol 523)
It's my first sol driving the rovers since I came back from jury duty, and I've switched to Opportunity -- my first sol on this rover since, well, since before they encountered Purgatory Ripple.
Yestersol's drive was a 20-degree turn in place followed by a whopping 50cm drive forward, in two 25cm arcs. But only one of the two arcs completed. Opportunity sensed that she'd slipped 32% on that arc, and we'd set a limit of 30%, so she stopped. Opportunity is scared of bogging down again.
But Khaled and Paolo analyze the drive and conclude that it's safe for us to proceed. What happened was that as we did the turn in place, the right-front wheel -- whose steering actuator is broken -- was mostly just dragged along, digging itself a small trench in the process. Then when we tried to step forward, we slipped a touch more than expected as we climbed out of that trench. But the rover's safe, on top of the soil rather then underneath it, and the way ahead is clear.
Our goal here is simply to drive back to a part of the tracks we left as we exited Purgatory. Yestersol's drive was supposed to leave us about 50cm from them, but, well ... so today we'll drive a whopping 75cm; if it goes well, we'll be right where we need to be to IDD the tracks over the weekend. Then next week we'll pack up and leave.
Khaled and I do spend a fair amount of time working on the drive -- everyone's paranoid about re-approaching this ripple -- but we're ready in plenty of time. Jeff Favretto notices the level of effort we're putting in, and during one of the meetings, he just grins and holds his hands a little more than shoulder-width apart. "Seventy-five centimeters," he grins, shaking his head. "This is how far we're going. It takes all that ...."
Meanwhile, Spirit's exhibiting a little bit of unusual behavior. During her last drive, the left bogie angles changed in unusual ways, indicating that the middle wheel kept lifting well into the air and then dropping back down. One hypothesis is that something's stuck on the outside of the wheel. If this is true, we'll never be able to see it: they're not visible from any of the cameras.[1] Jeff thinks he knows what's stuck in the wheel: "It's that dinosaur bone we've been looking for all this time!"
Courtesy NASA/JPL-Caltech. Post-drive front HAZCAM image showing the track created by the right front wheel when we turned in place on the previous drive.
Footnotes:
[1] As you can see, we hadn't then thought of using the MI to view the middle wheels. We didn't think of that trick until much later, when Spirit was embedded in Troy.
Yestersol's drive was a 20-degree turn in place followed by a whopping 50cm drive forward, in two 25cm arcs. But only one of the two arcs completed. Opportunity sensed that she'd slipped 32% on that arc, and we'd set a limit of 30%, so she stopped. Opportunity is scared of bogging down again.
But Khaled and Paolo analyze the drive and conclude that it's safe for us to proceed. What happened was that as we did the turn in place, the right-front wheel -- whose steering actuator is broken -- was mostly just dragged along, digging itself a small trench in the process. Then when we tried to step forward, we slipped a touch more than expected as we climbed out of that trench. But the rover's safe, on top of the soil rather then underneath it, and the way ahead is clear.
Our goal here is simply to drive back to a part of the tracks we left as we exited Purgatory. Yestersol's drive was supposed to leave us about 50cm from them, but, well ... so today we'll drive a whopping 75cm; if it goes well, we'll be right where we need to be to IDD the tracks over the weekend. Then next week we'll pack up and leave.
Khaled and I do spend a fair amount of time working on the drive -- everyone's paranoid about re-approaching this ripple -- but we're ready in plenty of time. Jeff Favretto notices the level of effort we're putting in, and during one of the meetings, he just grins and holds his hands a little more than shoulder-width apart. "Seventy-five centimeters," he grins, shaking his head. "This is how far we're going. It takes all that ...."
Meanwhile, Spirit's exhibiting a little bit of unusual behavior. During her last drive, the left bogie angles changed in unusual ways, indicating that the middle wheel kept lifting well into the air and then dropping back down. One hypothesis is that something's stuck on the outside of the wheel. If this is true, we'll never be able to see it: they're not visible from any of the cameras.[1] Jeff thinks he knows what's stuck in the wheel: "It's that dinosaur bone we've been looking for all this time!"
Courtesy NASA/JPL-Caltech. Post-drive front HAZCAM image showing the track created by the right front wheel when we turned in place on the previous drive.
Footnotes:
[1] As you can see, we hadn't then thought of using the MI to view the middle wheels. We didn't think of that trick until much later, when Spirit was embedded in Troy.
2010-06-10
Spirit Sol 511
"My Ph.D. thesis was on writing a program that would design robots for you," Chris says. "You'd tell it what you want, and it would design a robot to do it. That way, I could spend more time climbing." (He's an avid rock climber.)
"See, I'd have done it the other way around. I'd have designed a robot that would go climbing for me, so I could spend my time designing robots," I say.
"That's the difference between you and me," he deadpans.
It's Friday, and that means it's another three-sol plan. I did essentially all of the IDD work for all three sols yesterday so that I could focus on the drive today, which should make today easy.
Ha.
Since we're on tight sols, our uplink's at 17:40, and for a while there it really doesn't look like we're going to make it. In the end we finish with less than half an hour to spare. We haven't cut things that close in a while. I am awfully glad I did the IDD sequences up front, though; it was only two or three hours' work, but we really didn't have those two or three hours today.
This might be my last sol on Spirit, at least for the next month or so. I'm potentially on jury duty next week, and then I head upstairs to the Land of Opportunity. Which will be different from the last time I was up there. I came downstairs not long before they ran into what they're now calling "Purgatory Ripple," and they extracted themselves already. But the effects of Purgatory Ripple will outlast the actual experience of our being embedded in it. We're going to spend some time investigating the ripple itself, and we're going to drive differently from now on -- probably much less aggressively, but certainly differently.
How differently? That's still an open question. Khaled's working out some of the details, and he comes by to talk to me and Chris about it. They've got a list of proposals going, including my two sneaky tricks. (First trick: periodically stop and try to turn in place. If you can't turn, you're stuck; preclude further driving. Second trick: periodically try to run visodom. If it converges, you must not have moved significantly between the two images, meaning that you're stuck; preclude further driving.) As we talk, I come up with a couple of other creative ideas. One of them is to abuse the IMU: if it doesn't show that your local "down vector" is changing with respect to time, you're probably not climbing ripples and hence may be stuck; preclude driving. Similarly, we could check the suspension limits: if the suspension isn't flexing over time, stop driving.[1]
Nearly all of our ideas are reactive measures: try to detect when you're bogging down and keep from making a bad situation worse. What we'd really like are predictive measures, where we avoid getting stuck in the first place. We're short on those.
But we'd better do at least some driving. Recent email traffic prompted by the 1000-sol news has pointed out that we're close to a couple more milestones: 95,303 images so far (nearly 100,000) and a total distance of 9752.26m driven so far (nearly 10,000). I'm so proud of these rovers.
Footnotes:
[1] The end result was to use VO in a cleverer way: take a picture, move 20cm, and update VO. If VO reports that you've slipped more than 40%, preclude further driving. If it doesn't converge, try again, and preclude driving if you slip more than 40% or if it still doesn't converge.
But that's expensive: it takes at least six minutes, as many as 10, and we have to stop and do it every 20m or so on Opportunity. So, as we speak, there's an MIT grad student working to make an IMU-based classifier do the work for us -- an essentially free check that will notice relatively quickly if we're embedded and stop driving.
[Next post: sol 523 (Opportunity sol 503), June 23. Damn jury service.]
"See, I'd have done it the other way around. I'd have designed a robot that would go climbing for me, so I could spend my time designing robots," I say.
"That's the difference between you and me," he deadpans.
It's Friday, and that means it's another three-sol plan. I did essentially all of the IDD work for all three sols yesterday so that I could focus on the drive today, which should make today easy.
Ha.
Since we're on tight sols, our uplink's at 17:40, and for a while there it really doesn't look like we're going to make it. In the end we finish with less than half an hour to spare. We haven't cut things that close in a while. I am awfully glad I did the IDD sequences up front, though; it was only two or three hours' work, but we really didn't have those two or three hours today.
This might be my last sol on Spirit, at least for the next month or so. I'm potentially on jury duty next week, and then I head upstairs to the Land of Opportunity. Which will be different from the last time I was up there. I came downstairs not long before they ran into what they're now calling "Purgatory Ripple," and they extracted themselves already. But the effects of Purgatory Ripple will outlast the actual experience of our being embedded in it. We're going to spend some time investigating the ripple itself, and we're going to drive differently from now on -- probably much less aggressively, but certainly differently.
How differently? That's still an open question. Khaled's working out some of the details, and he comes by to talk to me and Chris about it. They've got a list of proposals going, including my two sneaky tricks. (First trick: periodically stop and try to turn in place. If you can't turn, you're stuck; preclude further driving. Second trick: periodically try to run visodom. If it converges, you must not have moved significantly between the two images, meaning that you're stuck; preclude further driving.) As we talk, I come up with a couple of other creative ideas. One of them is to abuse the IMU: if it doesn't show that your local "down vector" is changing with respect to time, you're probably not climbing ripples and hence may be stuck; preclude driving. Similarly, we could check the suspension limits: if the suspension isn't flexing over time, stop driving.[1]
Nearly all of our ideas are reactive measures: try to detect when you're bogging down and keep from making a bad situation worse. What we'd really like are predictive measures, where we avoid getting stuck in the first place. We're short on those.
But we'd better do at least some driving. Recent email traffic prompted by the 1000-sol news has pointed out that we're close to a couple more milestones: 95,303 images so far (nearly 100,000) and a total distance of 9752.26m driven so far (nearly 10,000). I'm so proud of these rovers.
Footnotes:
[1] The end result was to use VO in a cleverer way: take a picture, move 20cm, and update VO. If VO reports that you've slipped more than 40%, preclude further driving. If it doesn't converge, try again, and preclude driving if you slip more than 40% or if it still doesn't converge.
But that's expensive: it takes at least six minutes, as many as 10, and we have to stop and do it every 20m or so on Opportunity. So, as we speak, there's an MIT grad student working to make an IMU-based classifier do the work for us -- an essentially free check that will notice relatively quickly if we're embedded and stop driving.
[Next post: sol 523 (Opportunity sol 503), June 23. Damn jury service.]
2010-06-09
Spirit Sol 510
The radiation source in the MB has weakened over time, like a slowly fading light bulb. The MB still works fine, but you have to integrate longer to get the same quality data, much as you have to leave a camera's shutter open longer in dim light.
As a result, thisol we're just continuing to let the MB integrate. The rover will also take pictures and stuff, but there are no RP sequences thisol.
Which doesn't mean I have nothing to do. I go ahead and get the weekend's IDD sequences built (this is even more fun than usual, I discover, when there's no time pressure), and Chris and I -- mostly Chris -- look ahead at the weekend drive. This should make tomorrow much simpler, as we'll be able to focus on planning the drive, and we'll be able to work on it together. And we've already done some of the reconnaissance.
The big deal for the day is that we're planning sol 510 and Opportunity is planning sol 490. Hence, the two rovers have a combined total of ONE THOUSAND SOLS ON MARS.
And counting.
As a result, thisol we're just continuing to let the MB integrate. The rover will also take pictures and stuff, but there are no RP sequences thisol.
Which doesn't mean I have nothing to do. I go ahead and get the weekend's IDD sequences built (this is even more fun than usual, I discover, when there's no time pressure), and Chris and I -- mostly Chris -- look ahead at the weekend drive. This should make tomorrow much simpler, as we'll be able to focus on planning the drive, and we'll be able to work on it together. And we've already done some of the reconnaissance.
The big deal for the day is that we're planning sol 510 and Opportunity is planning sol 490. Hence, the two rovers have a combined total of ONE THOUSAND SOLS ON MARS.
And counting.
2010-06-08
Spirit Sol 509
It's a tight sol, with the uplink due in just a few hours, so we're starting at the ungodly hour of 07:00. I'm also working with John Wright (he's RP-1 and I'm RP-2), and who the heck knows when was the last time that happened.
All that really happened while I was gone was some IDD work on the magnets and solar panels, followed by a drive to Backstay, a pointy, very roughly tetrahedral rock just a few meters away from the end of our last drive. Thisol we're starting an IDD campaign on Backstay, which campaign might or might not include using the RAT.
The one good thing about tight sols is that they're usually relatively simple. Then again, this is because our time is relatively limited, so it doesn't make things that much easier, really. And the uncertainty about the RAT adds some complexity.
The problem is that the question of whether we can safely use Spirit's RAT still hasn't been resolved. Or anyway, not everyone thinks it has, which comes to pretty much the same thing. Alistair, the RAT PUL, says at the SOWG meeting that he got a thumbs-up from Eric Baumgartner, but nobody else seems to have seen the email. Later on, the email shows up, but it seems Eric's approval was based on a false premise -- he thought a test had been run that in fact hadn't. So Ashitey's back in the testbed all day, working feverishly against our tight schedule to get the right test run before we have to uplink.
To cope with this uncertainty, John and I put the RAT-related commands in a separate sequence and work out a way for the backbone sequence to run correctly whether the moves in the RAT subsequence are performed or not. That way we can cleanly drop all the RAT stuff just by not delivering that sequence. The cost is that we have to simulate both cases carefully, more or less doubling our validation work, but we'd pretty much have to do that anyhow; this is the cleanest way.
And then it turns out we might not be able to use the RAT on this target after all: the PUL selected a target using the reachability map, but the reachability map doesn't know that we've restricted the allowed range of motion for the turret joint. And it turns out that the target the PUL selected, which is in a fairly narrow reachability band, causes us to violate that limit. Later he finds another target, though, one that's reachable and satisfies the science needs, so we're back on.
Except that we still haven't heard from Ashitey. Time passes. In the end, we get the go. Backstay is in for a brushing.
All that really happened while I was gone was some IDD work on the magnets and solar panels, followed by a drive to Backstay, a pointy, very roughly tetrahedral rock just a few meters away from the end of our last drive. Thisol we're starting an IDD campaign on Backstay, which campaign might or might not include using the RAT.
The one good thing about tight sols is that they're usually relatively simple. Then again, this is because our time is relatively limited, so it doesn't make things that much easier, really. And the uncertainty about the RAT adds some complexity.
The problem is that the question of whether we can safely use Spirit's RAT still hasn't been resolved. Or anyway, not everyone thinks it has, which comes to pretty much the same thing. Alistair, the RAT PUL, says at the SOWG meeting that he got a thumbs-up from Eric Baumgartner, but nobody else seems to have seen the email. Later on, the email shows up, but it seems Eric's approval was based on a false premise -- he thought a test had been run that in fact hadn't. So Ashitey's back in the testbed all day, working feverishly against our tight schedule to get the right test run before we have to uplink.
To cope with this uncertainty, John and I put the RAT-related commands in a separate sequence and work out a way for the backbone sequence to run correctly whether the moves in the RAT subsequence are performed or not. That way we can cleanly drop all the RAT stuff just by not delivering that sequence. The cost is that we have to simulate both cases carefully, more or less doubling our validation work, but we'd pretty much have to do that anyhow; this is the cleanest way.
And then it turns out we might not be able to use the RAT on this target after all: the PUL selected a target using the reachability map, but the reachability map doesn't know that we've restricted the allowed range of motion for the turret joint. And it turns out that the target the PUL selected, which is in a fairly narrow reachability band, causes us to violate that limit. Later he finds another target, though, one that's reachable and satisfies the science needs, so we're back on.
Except that we still haven't heard from Ashitey. Time passes. In the end, we get the go. Backstay is in for a brushing.
2010-06-02
Spirit Sol 503
When I arrive we're having problems with the OSS, the network filesystem that we use for, well, pretty much everything related to uplink and downlink. One effect of this is that SAP hangs at startup, so the scientists can't do their science planning. Mark Powell, one of the SAP developers, is scrambling for a fix, but he doesn't seem to be having much luck.
Meanwhile, Steve reveals that he and Dave DesMarais have been going over the science results and have concluded that it's time to move on. We've going to finish up IDD work in this location thisol, then drive back to Methuselah and on around to the south side of Husband Hill toward the South Basin. This will take us uphill, though possibly not to the summit. We'll have to see. I'll be back on Opportunity before much longer, anyway, so the struggle to reach the summit will be someone else's for a while.
Mark has just about given up. I decide I should try to help him. I suggest setting up a temporary OSS under somebody's home directory -- all they need is a file structure, and SAP can be given the root of that structure at startup. He looks startled. "Worth a try," he shrugs. A few minutes later he has this working, and Alicia Fallacaro is whipping up a science plan. She beams gratefully at me. "Scott saves the day!" she says.
When I see the IDD target they want to poke at, I almost regret helping them get up and running. The target -- "Nantucket," continuing the Moby Dick theme -- is in a particularly awkward region of the IDD work volume, tight up against the left front quarter of the rover itself. I muse aloud that during the nominal mission this would have freaked me out, which seems to surprise Al Herrera. "I thought you were always cool, calm, and collected," he says. "That's because you only see the back of my head," I tell him. "If there were a mirror set up on the monitor so you could see my face, you'd know better." I'm only half kidding.
Nantucket's a real bitch of a target for a 2x2 MI preceded by a MB touch. There's almost no room to switch from the MB to the MI, for one thing; for another thing, the IDD really wants to collide with itself as we move between the stacks. I find a way to make it work, but the tolerances are so tight that Chris has a heart attack when he looks at it. We show Squyres, who decides to split the MI work -- half on Nantucket, half on a nearby rock we call "Starbuck." We use an MB-touch on Starbuck, since it's far enough away for us to do this comfortably. On Nantucket, we'll just use a less reliable MI poke, which will save us from having to do the tool change. (Poking a rock with the MI is enough like harpooning a whale that we should have named this target after one of the Pequod's harpooners. "Queequeg" was already taken, but we could have named it "Tashtego" or "Daggoo." Ah, well.)
While I've been working on this complex IDD sequence, Chris has been cranking out a comparably difficult drive. I don't like sols like these, just because we don't really get enough time to review each other's work, and that bites us this time. At the CAM, at the very last moment, I realize that we made a mistake: we do some visodom, then some blind driving, then some autonav -- but we forgot to clear the nav map between the blind and autonav segments. This means the rover will have an inaccurate picture of the world around it -- potentially a dangerously inaccurate picture, since it could drive over ground it thought was safe but was in fact host to a deadly obstacle.
So we need to rebundle. All I need to do is insert a single command that tells the rover to clear the nav map before the autonav segment of the drive, but here I make another mistake: Jim Erickson asks whether that's really all we need to do, and I reply with insufficient authority in my voice. He starts to look nervous, and though I realize my mistake, it's too late now. There's no help for it but to call another expert.
Mark Maimone's in China, and all of Jeff Biesiadecki's contact numbers lead to voice mail. Fortunately, Khaled did some testing of exactly such situations in the testbed, and I'm able to reach him. So his vote plus mine makes Jim look happy again -- well, as happy as he ever looks -- and we commit to the change.
[Next post: sol 509, June 8.]
Meanwhile, Steve reveals that he and Dave DesMarais have been going over the science results and have concluded that it's time to move on. We've going to finish up IDD work in this location thisol, then drive back to Methuselah and on around to the south side of Husband Hill toward the South Basin. This will take us uphill, though possibly not to the summit. We'll have to see. I'll be back on Opportunity before much longer, anyway, so the struggle to reach the summit will be someone else's for a while.
Mark has just about given up. I decide I should try to help him. I suggest setting up a temporary OSS under somebody's home directory -- all they need is a file structure, and SAP can be given the root of that structure at startup. He looks startled. "Worth a try," he shrugs. A few minutes later he has this working, and Alicia Fallacaro is whipping up a science plan. She beams gratefully at me. "Scott saves the day!" she says.
When I see the IDD target they want to poke at, I almost regret helping them get up and running. The target -- "Nantucket," continuing the Moby Dick theme -- is in a particularly awkward region of the IDD work volume, tight up against the left front quarter of the rover itself. I muse aloud that during the nominal mission this would have freaked me out, which seems to surprise Al Herrera. "I thought you were always cool, calm, and collected," he says. "That's because you only see the back of my head," I tell him. "If there were a mirror set up on the monitor so you could see my face, you'd know better." I'm only half kidding.
Nantucket's a real bitch of a target for a 2x2 MI preceded by a MB touch. There's almost no room to switch from the MB to the MI, for one thing; for another thing, the IDD really wants to collide with itself as we move between the stacks. I find a way to make it work, but the tolerances are so tight that Chris has a heart attack when he looks at it. We show Squyres, who decides to split the MI work -- half on Nantucket, half on a nearby rock we call "Starbuck." We use an MB-touch on Starbuck, since it's far enough away for us to do this comfortably. On Nantucket, we'll just use a less reliable MI poke, which will save us from having to do the tool change. (Poking a rock with the MI is enough like harpooning a whale that we should have named this target after one of the Pequod's harpooners. "Queequeg" was already taken, but we could have named it "Tashtego" or "Daggoo." Ah, well.)
While I've been working on this complex IDD sequence, Chris has been cranking out a comparably difficult drive. I don't like sols like these, just because we don't really get enough time to review each other's work, and that bites us this time. At the CAM, at the very last moment, I realize that we made a mistake: we do some visodom, then some blind driving, then some autonav -- but we forgot to clear the nav map between the blind and autonav segments. This means the rover will have an inaccurate picture of the world around it -- potentially a dangerously inaccurate picture, since it could drive over ground it thought was safe but was in fact host to a deadly obstacle.
So we need to rebundle. All I need to do is insert a single command that tells the rover to clear the nav map before the autonav segment of the drive, but here I make another mistake: Jim Erickson asks whether that's really all we need to do, and I reply with insufficient authority in my voice. He starts to look nervous, and though I realize my mistake, it's too late now. There's no help for it but to call another expert.
Mark Maimone's in China, and all of Jeff Biesiadecki's contact numbers lead to voice mail. Fortunately, Khaled did some testing of exactly such situations in the testbed, and I'm able to reach him. So his vote plus mine makes Jim look happy again -- well, as happy as he ever looks -- and we commit to the change.
[Next post: sol 509, June 8.]
2010-06-01
Spirit Sol 502
At the SOWG meeting, there's a controversy about the RAT. Fortunately, I've been warned about this in email from Ashitey: he says we're still worried about the badly worn RAT causing the IDD to damage itself through vibration, even when just brushing (everyone agrees grinding is out) and that the Honeybee guys are wrongly suggesting there's no problem.
Which is exactly what happens at the SOWG: when Steve asks about it, the Honeybee guy says there's no problem. "We've seen lots of lost-contact events even back to the beginning of the mission," he argues, suggesting that the events we see now don't imply the arm is actually vibrating dangerously. Steve doesn't just take the RAT guy's word for it, though; he wants more imaging as part of a diagnostic that will eliminate all doubt. Since that's exactly what Ashitey asked me to ask for, I spot my cue and pipe up. "Yeah, Steve, we'd like HAZCAM, PANCAM, and NAVCAM of the RAT brush and teeth before we stow at this location." I've already looked through past sols and found the last time we did this (sol 426), which will make it easy to do again, tomorrow or whenever.
"Sounds like we're pretty much on the same page, then," Steve replies.
The RAT guy tries to tell Steve that the instrument is safe to use, but Steve says, "I'm worried about the whole rover, not just the RAT." This is why Steve is so awesome[1]: lots of science team leaders would have thought it's their job to stand up for the science side, no matter what (or so I'm told). Not many would realize that the science team is best served by preserving the goddamned vehicle. Or so I'm told. Anyway, we're damned fortunate that Steve's not of that stripe.
Chris had to take his dog to the vet this morning (she's got bad allergies), so I've swapped with him: reversing the official plan, I'm RP-1, and he'll be in later as RP-2. But it hardly matters; the two-sol plan is so simple that I'm already done when he arrives. Or rather, I should have been done. At the activity plan approval meeting, it turns out I've sequenced the wrong thing, moving the IDD to the wrong target at one point. This is not really my fault, as the official plan had it wrong, and that's what I was following. But I should have seen it coming because of the Curse of the Premature Animation. (Your line: "I used to have a problem with premature animation, but now I just think about baseball.")
That curse is this. Whenever I'm so confident that I've finished early that I go ahead and build the uplink report and/or the official MPEG animation before the walkthrough -- much less before the APAM, as I did today -- something will have to change in the sequence that invalidates that work. This happens every time, and today was no exception.
Fortunately, it's easy to fix: almost before they can finish describing the problem, I've already fixed the sequence and have redone the on-the-fly animation. But you'd think I'd know by now that I'm cursed. Cursed. Cursed!
Maybe next time I can avert the curse by thinking about baseball. It's worth a try.
Footnotes:
[1] I mean, of course, that it's one of the reasons. One of many, many, many.
Which is exactly what happens at the SOWG: when Steve asks about it, the Honeybee guy says there's no problem. "We've seen lots of lost-contact events even back to the beginning of the mission," he argues, suggesting that the events we see now don't imply the arm is actually vibrating dangerously. Steve doesn't just take the RAT guy's word for it, though; he wants more imaging as part of a diagnostic that will eliminate all doubt. Since that's exactly what Ashitey asked me to ask for, I spot my cue and pipe up. "Yeah, Steve, we'd like HAZCAM, PANCAM, and NAVCAM of the RAT brush and teeth before we stow at this location." I've already looked through past sols and found the last time we did this (sol 426), which will make it easy to do again, tomorrow or whenever.
"Sounds like we're pretty much on the same page, then," Steve replies.
The RAT guy tries to tell Steve that the instrument is safe to use, but Steve says, "I'm worried about the whole rover, not just the RAT." This is why Steve is so awesome[1]: lots of science team leaders would have thought it's their job to stand up for the science side, no matter what (or so I'm told). Not many would realize that the science team is best served by preserving the goddamned vehicle. Or so I'm told. Anyway, we're damned fortunate that Steve's not of that stripe.
Chris had to take his dog to the vet this morning (she's got bad allergies), so I've swapped with him: reversing the official plan, I'm RP-1, and he'll be in later as RP-2. But it hardly matters; the two-sol plan is so simple that I'm already done when he arrives. Or rather, I should have been done. At the activity plan approval meeting, it turns out I've sequenced the wrong thing, moving the IDD to the wrong target at one point. This is not really my fault, as the official plan had it wrong, and that's what I was following. But I should have seen it coming because of the Curse of the Premature Animation. (Your line: "I used to have a problem with premature animation, but now I just think about baseball.")
That curse is this. Whenever I'm so confident that I've finished early that I go ahead and build the uplink report and/or the official MPEG animation before the walkthrough -- much less before the APAM, as I did today -- something will have to change in the sequence that invalidates that work. This happens every time, and today was no exception.
Fortunately, it's easy to fix: almost before they can finish describing the problem, I've already fixed the sequence and have redone the on-the-fly animation. But you'd think I'd know by now that I'm cursed. Cursed. Cursed!
Maybe next time I can avert the curse by thinking about baseball. It's worth a try.
Footnotes:
[1] I mean, of course, that it's one of the reasons. One of many, many, many.
Subscribe to:
Posts (Atom)
Posts
