The weekend's drive went splendidly, racking up 32m. And we're poised to do it again, with a broad lane continuing before us.
Except for the speed bump. About 12m away from us is a decision point, where the trough sort of splits into two other troughs. Each of the branches (Left Fork and Right Fork) has a ripple in front of it, a ripple with geometry that's ominously similar to Purgatory's. Alternatively, we're sitting more or less at another fork in the road -- but the third choice that provides (Right-of-Right Fork) appears to end in a cul-de-sac with bad-looking stuff at the end. But we don't have to take any of those three paths. We could start ripple-hopping to the east, hoping to find a better trough.
Thus begins the Big Debate. Right-of-Right Fork gets thrown out pretty quickly, as just a way to store up trouble. Of Left Fork vs. Right Fork, Rob Sullivan's analysis shows that Left Fork's ripple is shorter but steeper, and angle is more worrisome than height, so that's a loser.
Which leaves us with Right Fork and ripple-hopping. The eventual decision is to prefer Right Fork: ripple-hopping affords no immediate progress toward our goal, is not guaranteed to provide a better alternative soon, and has dangers of its own. Besides, we're going to come across ripples like these sooner or later on the way to Erebus and, beyond it, Victoria. We might as well start learning to deal with them.
But still, that ripple ....
At least Jeng tells me something that makes me feel better about tackling it. The problem at Purgatory, he says, wasn't just the geometry of Purgatory itself. It was exacerbated by the fact that when we hit Purgatory, we were just descending another ripple (later called "North Ripple"). This meant the rover was initially angled down and thus "wanted" to bury itself in Purgatory. For this ripple, we'd be pretty much flat as we encountered it, which is a much more favorable geometry for climbing it.
So Jeng and I plan a route along the trough, while Rob Sullivan goes off and does more analysis and Beth Dewell calls Jim Erickson and John Callas. Jim and John are nervous about this, and also nervous about the fact that our planned path extends 20m beyond the ripple. This would mean that if our new slip-detection approach doesn't work, we could spin our wheels for 20m worth of driving in the ripple -- about half the damage we did at Purgatory, but much more than we'd be comfortable with. "You'd have a new Project Manager if you did that," Jim says grimly. "You say that like it's a bad thing," jokes John -- his deputy -- but Jim is serious. Though it didn't cost us a rover, Purgatory was a costly mistake, yet a forgivable one because it had never happened before and really was not foreseeable. But if we did it twice -- "Even I would wonder about my fitness for the job if it happened again," Jim says.
In the end they accept the drive as it stands, trusting our argument about the robustness of the slip check. Really, it's hard to get the slip checks to pass even when we're genuinely not slipping, depending as they do on the finicky mechanism of visodom used in a very difficult environment. If anything, the checks are conservative. At their suggestion, we mitigate the risk further by splitting the post-drive segment so that it effectively checks for stuckage an extra time.
Still, I'd feel a lot more comfortable if we were risking only the rover, and not also risking Jim's career.[1]
So we're done. Or not. While all this has been going on, Rob Sullivan's continued analysis turns up two other possible dangers past the initial ripple. A second ripple lies behind the first one and to its left, while a third lies a bit farther on and to the right. So even once we pass the first ripple, we've got some zig-zagging to do. I sigh, roll up my sleeves, and start hacking the sequence some more.
It's an awfully good thing Rob found those ripples -- it would really have sucked to sail past the first one, only to get stuck another couple of meters in. Our terrain mesh thisol is pretty poor for some reason, so the farther ripples didn't show up well; only his experienced geologist's eye gave us warning of the danger.
As if all that weren't enough, Rob saves the day once more. We decided to take "cleat movies" of our progress over the ripple -- small HAZCAM images subframed to the wheels, taken after each 50cm of commanded motion. Normally these are at a relatively low priority. But then we realize that taking these at a low priority is a mistake: if we do get bogged down, we won't get the images for another day, and that will be an awfully tense day. So we'd really like to get them sooner, but by the time we realize this, it's so late on the East Coast that the person who's normally responsible for this has had to go home. But Rob knows the job, so with his usual good humor and great attitude, he stays late and fixes them for us. He even sticks around through the CAM to make sure everything is ready for the spacecraft.
You know how there was a Fifth Beatle? Rob is, like, the (n+1)th Rover Planner. I love that guy.[2]
There were many obstacles thisol, figuratively and literally, but we got some good news, too. We had another cleaning event -- the dust devils worked their magic -- and we now have another 100 Watt-hours to play with each sol. Thanks, Mars!
[Next post: sol 551 (Opportunity sol 530), July 22.]
Footnotes:
[1] But then, this is why Jim was such an awesome guy: even feeling that his career was on the line, he trusted us unless he had an articulable, rational reason not to.
[2] Much later, I even gave him an official Rover Planner T-shirt. Yes, there really are official Rover Planner T-shirts. And no, you can't buy one.
2010-07-19
2010-07-16
Opportunity Sol 525 (Spirit Sol 546)
I envy Cooper and Jeng. Yestersol they were pushed pretty hard to improve our rate of travel while remaining compliant with the Rules of the Road, and they came up with a good one (along with Mark Maimone). Essentially, instead of using visodom at 40cm intervals throughout the drive, they drive 5m at a normal rate and then did a clever slip check using visodom over the next 40cm. This does two or three visodom updates every 5.4m of travel, rather than the 13 or so we were doing over the same distance. This let them greatly increase our traverse distance; now we can attain the full 30m/sol we're currently allowed by the Rules, and do it in only an hour and a half or so.[1]
I hate myself for not thinking of this. And I came in in such a good mood. Well, Jeng and I are going to pull the same trick thisol, and any sol where we have a nice long trough to drive in.
But maybe we won't drive south along the trough thisol. According to the orbital mesh, we're only 25m or so from the Erebus Highway. Driving there would mean driving at an azimuth of about 140deg -- across the ripples, not along the troughs, which would significantly limit our distance each sol.
I start looking into whether this is a sensible approach. What I find is that the images from our present location don't seem to agree with the orbital mesh. In our current images, we can't see anything that looks like Erebus Highway closer than 100m or so, putting it at least 6 driving sols away. (More actual sols than that, since we can't drive every sol over weekends and we're coming into restricted sols soon.) And even that's a gamble: it might be farther away, and it might not even speed up our traverse once we do get there. Meanwhile, we believe we can drive 30m per sol straight south toward Erebus itself. We talk it over as a team and decide on the direct route. But we're going to keep our eyes open, and if we see nearby Erebus Highway material, we'll consider going for it. If nothing else, we eventually cross something that looks like Erebus Highway -- Rob Sullivan calls it the Erebus Parking Lot -- even if we just head straight south.
So that's the plan.
Somehow we missed a major milestone. That is, a kilometerstone. Recently, the rovers' combined odometry crossed the 10km mark. And climbing.
[Next post: sol 549 (Opportunity sol 528), July 19.]
Footnotes:
[1] A close descendant of this technique is used to check Opportunity's slip to this day. We also check slip only every 20m or so now, rather than every 5m, so the expense is even further reduced.
I hate myself for not thinking of this. And I came in in such a good mood. Well, Jeng and I are going to pull the same trick thisol, and any sol where we have a nice long trough to drive in.
But maybe we won't drive south along the trough thisol. According to the orbital mesh, we're only 25m or so from the Erebus Highway. Driving there would mean driving at an azimuth of about 140deg -- across the ripples, not along the troughs, which would significantly limit our distance each sol.
I start looking into whether this is a sensible approach. What I find is that the images from our present location don't seem to agree with the orbital mesh. In our current images, we can't see anything that looks like Erebus Highway closer than 100m or so, putting it at least 6 driving sols away. (More actual sols than that, since we can't drive every sol over weekends and we're coming into restricted sols soon.) And even that's a gamble: it might be farther away, and it might not even speed up our traverse once we do get there. Meanwhile, we believe we can drive 30m per sol straight south toward Erebus itself. We talk it over as a team and decide on the direct route. But we're going to keep our eyes open, and if we see nearby Erebus Highway material, we'll consider going for it. If nothing else, we eventually cross something that looks like Erebus Highway -- Rob Sullivan calls it the Erebus Parking Lot -- even if we just head straight south.
So that's the plan.
Somehow we missed a major milestone. That is, a kilometerstone. Recently, the rovers' combined odometry crossed the 10km mark. And climbing.
[Next post: sol 549 (Opportunity sol 528), July 19.]
Footnotes:
[1] A close descendant of this technique is used to check Opportunity's slip to this day. We also check slip only every 20m or so now, rather than every 5m, so the expense is even further reduced.
2010-07-14
Opportunity Sol 523 (Spirit Sol 544)
I was worried about yestersol's drive -- needlessly, as it turns out. (I can't believe I'm still doing that.) It went damn near perfectly, putting us about 10cm from our goal.
Now that we're here, it looks like the way ahead of us is not scary after all. Quite the contrary: with only the FHAZ and a single NCAM wedge to go by, we can drive about 20m down a nice smooth lane, with only a bit of leftward curve to follow the trough.
John Callas has brought a couple of blind students in to observe. I have some time before the SOWG starts, so I talk with them a little. They're eager and enthusiastic (who can blame them?). John told me about them the other day, floating the idea of having one or more of them shadow a rover driver for a while. I told him I'd be happy to help with this, but wasn't sure exactly how we'd achieve that -- it's a very visual job. But we'll figure something out. In the meanwhile, they're going to help make our Web pages more accessible to blind people, or something.
Since the way ahead of us is clear, the SOWG grants us even more time to drive -- three and a half hours thisol. We'll go about 20m. The only thing I dislike about this decision is that it makes it even worse if the drive fails, because so much science gets pushed out to make way for it. But I don't think they really care that much; there's very little good science they can do before we get to the Erebus Highway anyhow, so they're actually doing themselves a favor by letting us drive as much as we can each sol.
When we look at the planned drive in an overhead map, it looks as if we could potentially make it to the Erebus Highway in just two or three more sols. We won't -- if only because we've run out of tricks for working around the data-products limit and might need to shorten the next drive to get some crap downlinked -- but we're getting close. For several sols, we've glimpsed it -- maybe -- in the distance, in the NCAMs. Very, very close.
[Next post: sol 546 (Opportunity sol 525), July 16.]
Now that we're here, it looks like the way ahead of us is not scary after all. Quite the contrary: with only the FHAZ and a single NCAM wedge to go by, we can drive about 20m down a nice smooth lane, with only a bit of leftward curve to follow the trough.
John Callas has brought a couple of blind students in to observe. I have some time before the SOWG starts, so I talk with them a little. They're eager and enthusiastic (who can blame them?). John told me about them the other day, floating the idea of having one or more of them shadow a rover driver for a while. I told him I'd be happy to help with this, but wasn't sure exactly how we'd achieve that -- it's a very visual job. But we'll figure something out. In the meanwhile, they're going to help make our Web pages more accessible to blind people, or something.
Since the way ahead of us is clear, the SOWG grants us even more time to drive -- three and a half hours thisol. We'll go about 20m. The only thing I dislike about this decision is that it makes it even worse if the drive fails, because so much science gets pushed out to make way for it. But I don't think they really care that much; there's very little good science they can do before we get to the Erebus Highway anyhow, so they're actually doing themselves a favor by letting us drive as much as we can each sol.
When we look at the planned drive in an overhead map, it looks as if we could potentially make it to the Erebus Highway in just two or three more sols. We won't -- if only because we've run out of tricks for working around the data-products limit and might need to shorten the next drive to get some crap downlinked -- but we're getting close. For several sols, we've glimpsed it -- maybe -- in the distance, in the NCAMs. Very, very close.
[Next post: sol 546 (Opportunity sol 525), July 16.]
2010-07-13
Opportunity Sol 522 (Spirit Sol 543)
Surprisingly, visodom performed well through the entire drive yestersol. We'd been worried visodom wouldn't work reliably with nothing but our own tracks for it to look at, but so far, so good.
The drive itself went where it was supposed to. We're at the end of one trough, and thisol our plan is to hop into the next one (about 5m worth of driving, in a lazy "S") and drive about 10m along it, to a point where the terrain seems to fall off slightly. If we could see it better in the NCAM coverage, we might be able to drive farther -- or so I think. Later the PCAMs arrive, and they show that, yep, the terrain appears to fall off a bit there. Oh, well; we're using up all the time we've got anyway. (Visodom works, but it's slow! This 15m of driving will take us about 2.5 hours. We'll actually spend maybe 10 minutes turning the wheels; all the rest of that is visodom overhead.)
It's Rob Sullivan's first sol as SOWG chair. He does a great job, but at one point, when he's having trouble reaching consensus on what science should be slotted into the tiny amount of time left over after driving, he passes on a bit of wisdom: "One of the things Steve told me was that you've failed as SOWG chair if you have to decide rather than reach consensus." They reach consensus.
We're settling on a drive template, which makes things easier. Between the enormous setup block and the smaller teardown block, we cut the drive into a series of segments each about 5m long. Between segments, we reseed visodom -- slow, but necessary to work around a flight software bug that causes visodom's estimated roll and pitch errors to accumulate, which can abort the drive. And at the start of each segment, we zero the failure count and choose a new max failure count for that segment. This helps us play by the Rules of the Road, which demand that we catch slip -- if we've bogged down entirely, visodom will fail to converge and the error limit will stop us. (We can't just choose a single count large enough for the entire drive, because then we can dig in a lot more before stopping. The approach we've come up with uses several smaller counts instead, making the trigger more reliable.)
I also remember something we forgot on yestersol's drive: at the end of the drive, we have to straighten the wheels. The end of yestersol's drive included a last-minute comm turn, which left the wheels cocked in. There's nothing inherently wrong with this, but if another steering actuator should fail on this vehicle, we want it to "fail straight," since that's the least bad configuration for continued driving. I've made quite a point of this in my "Driving a Mars Rover for Dummies" talk -- which I've given twice publicly now -- so I'd feel like twice as much of a dumbass if I forgot this on some sol when it really mattered!
The drive itself went where it was supposed to. We're at the end of one trough, and thisol our plan is to hop into the next one (about 5m worth of driving, in a lazy "S") and drive about 10m along it, to a point where the terrain seems to fall off slightly. If we could see it better in the NCAM coverage, we might be able to drive farther -- or so I think. Later the PCAMs arrive, and they show that, yep, the terrain appears to fall off a bit there. Oh, well; we're using up all the time we've got anyway. (Visodom works, but it's slow! This 15m of driving will take us about 2.5 hours. We'll actually spend maybe 10 minutes turning the wheels; all the rest of that is visodom overhead.)
It's Rob Sullivan's first sol as SOWG chair. He does a great job, but at one point, when he's having trouble reaching consensus on what science should be slotted into the tiny amount of time left over after driving, he passes on a bit of wisdom: "One of the things Steve told me was that you've failed as SOWG chair if you have to decide rather than reach consensus." They reach consensus.
We're settling on a drive template, which makes things easier. Between the enormous setup block and the smaller teardown block, we cut the drive into a series of segments each about 5m long. Between segments, we reseed visodom -- slow, but necessary to work around a flight software bug that causes visodom's estimated roll and pitch errors to accumulate, which can abort the drive. And at the start of each segment, we zero the failure count and choose a new max failure count for that segment. This helps us play by the Rules of the Road, which demand that we catch slip -- if we've bogged down entirely, visodom will fail to converge and the error limit will stop us. (We can't just choose a single count large enough for the entire drive, because then we can dig in a lot more before stopping. The approach we've come up with uses several smaller counts instead, making the trigger more reliable.)
I also remember something we forgot on yestersol's drive: at the end of the drive, we have to straighten the wheels. The end of yestersol's drive included a last-minute comm turn, which left the wheels cocked in. There's nothing inherently wrong with this, but if another steering actuator should fail on this vehicle, we want it to "fail straight," since that's the least bad configuration for continued driving. I've made quite a point of this in my "Driving a Mars Rover for Dummies" talk -- which I've given twice publicly now -- so I'd feel like twice as much of a dumbass if I forgot this on some sol when it really mattered!
2010-07-12
Opportunity Sol 521 (Spirit Sol 542)
I come in early to get a jump on things. This was a mistake -- the downlink's been delayed (Spirit's data was queued ahead of ours on ODY), so there's nothing for me to get a jump on. At least it's being delayed for a good reason. Spirit has an earlier start and a tighter deadline, and they're also downlinking part of a huge panorama they took over the Independence Day weekend. I see part of this as it's coming down, and it's clear the full version will be spectacular.
Two hours later, our data starts flowing. The news isn't great, but it's okay. We'd planned about a 16m traverse, and we actually made about 10m progress. We'd have made more, but visual odometry -- which we rely upon for our slip checks -- produced a couple of erroneous results, making the rover think it was slipping a lot more than it actually was. As it happens, the rover thought it was slipping 32.5%, just barely over the 30% limit then in effect. So it stopped short.
Thisol we're just carrying on. The rover hasn't dug in and isn't in any apparent danger, so there's no reason not to just shoot for another 15m or so. The project issued new Rules of the Road that allow us to accept up to 40% slip, but this makes me a little nervous. The good slip estimates -- the ones produced during the larger portion of the drive, where visodom was working normally -- were in the neighborhood of 5% or less. So wouldn't it be better to actually lower the allowed slip percentage? We could set it to, say, 20% -- well above what we actually expect a good update to produce -- so that if visodom suddenly produces a bogus update that exceeds that threshold, the rover will stop moving instead of thinking it's somewhere it's not and driving into a ripple instead of along the trough.
But Jeng talks me out of this. The rover can actually climb these ripples, he points out. As long as it keeps heading more or less south -- there's essentially no chance of anything else, bogus update or no bogus update -- it'll be doing the right thing. And if it should get bogged down, visodom will fail to converge (because it doesn't expect the terrain not to change, an interesting result I'd never thought of). If that happens, the visodom failure-count limit will be exceeded, and the rover will stop shortly after.
The other thing that worries me is that we don't really know why the bogus updates occurred. We didn't get much data down; in particular, the data that would let us debug this problem is still aboard Opportunity. So if it was related to the unusual stuff we're doing with visodom -- looking over our right shoulder at the tracks we're leaving behind ourselves -- we'll just get more bogus updates at the start of this drive and fail again, probably without making much real progress. But with no debugging data available until tomorrow and no other real options anyway, there's little else we can do but cross our fingers and continue along the Blue Line toward Erebus.
Two hours later, our data starts flowing. The news isn't great, but it's okay. We'd planned about a 16m traverse, and we actually made about 10m progress. We'd have made more, but visual odometry -- which we rely upon for our slip checks -- produced a couple of erroneous results, making the rover think it was slipping a lot more than it actually was. As it happens, the rover thought it was slipping 32.5%, just barely over the 30% limit then in effect. So it stopped short.
Thisol we're just carrying on. The rover hasn't dug in and isn't in any apparent danger, so there's no reason not to just shoot for another 15m or so. The project issued new Rules of the Road that allow us to accept up to 40% slip, but this makes me a little nervous. The good slip estimates -- the ones produced during the larger portion of the drive, where visodom was working normally -- were in the neighborhood of 5% or less. So wouldn't it be better to actually lower the allowed slip percentage? We could set it to, say, 20% -- well above what we actually expect a good update to produce -- so that if visodom suddenly produces a bogus update that exceeds that threshold, the rover will stop moving instead of thinking it's somewhere it's not and driving into a ripple instead of along the trough.
But Jeng talks me out of this. The rover can actually climb these ripples, he points out. As long as it keeps heading more or less south -- there's essentially no chance of anything else, bogus update or no bogus update -- it'll be doing the right thing. And if it should get bogged down, visodom will fail to converge (because it doesn't expect the terrain not to change, an interesting result I'd never thought of). If that happens, the visodom failure-count limit will be exceeded, and the rover will stop shortly after.
The other thing that worries me is that we don't really know why the bogus updates occurred. We didn't get much data down; in particular, the data that would let us debug this problem is still aboard Opportunity. So if it was related to the unusual stuff we're doing with visodom -- looking over our right shoulder at the tracks we're leaving behind ourselves -- we'll just get more bogus updates at the start of this drive and fail again, probably without making much real progress. But with no debugging data available until tomorrow and no other real options anyway, there's little else we can do but cross our fingers and continue along the Blue Line toward Erebus.
2010-07-09
Opportunity Sol 518 (Spirit Sol 539)
Somewhat to my surprise, the drive executed quite well. We made the whole 15m, right along the intended path, so we're just continuing along the Blue Line today.
The drive even went faster than we'd thought, executing in only an hour and a half, as opposed to the two and a half we expected. But it turns out this is due to unintentional cheating. When we used to charge along the plains, we used step-skipping -- if autonav saw a sufficiently benign path, the rover could take a couple of extra steps without updating its world map. We don't want to be doing that now, but we forgot to turn it off, so the rover didn't pay as close attention to the drive as it was supposed to. That's why it went faster. Unfortunately, we'll have to turn off step-skipping as of thisol.
We also continue to face a data-products crunch. The problem right now is not the amount of data on the rover, but the number of files, a separate limit. The kind of driving we've been doing generates huge numbers of data products, and our downlink has been fairly poor, so today we have a choice to make. We can either cut back the verbosity of the drive data -- undesirable, because then if something goes wrong, we may lack the data to diagnose it; or delete yestersol's output without downlinking it -- but then we won't know why visodom didn't converge in some cases where we expected it to; or not drive -- but nobody wants that; or find a bunch of other stuff we can delete. They go with the last one.
Thisol brings a revision to the Rules of the Road. In short, they're relaxing. We can now accept 40% slip, rather than the 25% that had been the previous rule (or the 30% we'd actually been targeting). And tilt limits and max drive distances are no longer explicitly specified in most cases; we're now supposed to use "appropriate" figures. This is good and bad -- on the one hand, it means the RPs will take more of the heat if we make a poor choice. But on the upside, we get to make the choices, which is how I think it should be. And it's a reflection of our overall excellent performance since egressing Purgatory. So, hell, you look at it the right way, it's a compliment. Can't complain about that.
[Next post: sol 542 (Opportunity sol 521), July 12.]
The drive even went faster than we'd thought, executing in only an hour and a half, as opposed to the two and a half we expected. But it turns out this is due to unintentional cheating. When we used to charge along the plains, we used step-skipping -- if autonav saw a sufficiently benign path, the rover could take a couple of extra steps without updating its world map. We don't want to be doing that now, but we forgot to turn it off, so the rover didn't pay as close attention to the drive as it was supposed to. That's why it went faster. Unfortunately, we'll have to turn off step-skipping as of thisol.
We also continue to face a data-products crunch. The problem right now is not the amount of data on the rover, but the number of files, a separate limit. The kind of driving we've been doing generates huge numbers of data products, and our downlink has been fairly poor, so today we have a choice to make. We can either cut back the verbosity of the drive data -- undesirable, because then if something goes wrong, we may lack the data to diagnose it; or delete yestersol's output without downlinking it -- but then we won't know why visodom didn't converge in some cases where we expected it to; or not drive -- but nobody wants that; or find a bunch of other stuff we can delete. They go with the last one.
Thisol brings a revision to the Rules of the Road. In short, they're relaxing. We can now accept 40% slip, rather than the 25% that had been the previous rule (or the 30% we'd actually been targeting). And tilt limits and max drive distances are no longer explicitly specified in most cases; we're now supposed to use "appropriate" figures. This is good and bad -- on the one hand, it means the RPs will take more of the heat if we make a poor choice. But on the upside, we get to make the choices, which is how I think it should be. And it's a reflection of our overall excellent performance since egressing Purgatory. So, hell, you look at it the right way, it's a compliment. Can't complain about that.
[Next post: sol 542 (Opportunity sol 521), July 12.]
2010-07-08
Opportunity Sol 517 (Spirit Sol 538)
I've got my clock/radio tuned to the news station, so I'm literally awakened by the news that my favorite city in the world, London, was viciously attacked by barbarous cowards today. Bombs on the subway, and on one of those iconic red double-decker buses. Dozens of people are known dead, hundreds more probably dead. Maybe a thousand injured. Nobody really knows just how bad it is yet, but it's bad.
Not for the first time, I think to myself that I'm making a mistake. As much as I love it, I should leave Mars and go work in counter-terrorism in some capacity. I have a moral responsibility to act. I can't solve it all myself, but I should do my tiny little part rather than fuck around with an interplanetary Erector Set.
I usually tell myself that my job actually is important, that people need something to live for and we're part of providing that. But on a day like today, that justification -- I should call it a rationalization -- sounds awfully damn hollow.
I don't know what to do. I love you, London. I go to work.
Our drive onto the on-ramp, sol before yestersol, went fine. Yesterday Cooper and Jeff took us to the other end of the on-ramp; today we drive onto the freeway -- or East Path, or the Blue Line, as I've started calling it (from the blue lines in Rob Sullivan's images, designating the path we eventually chose).
Today's plan is to make about 15m total distance, about 10m of which will actually be along the Blue Line. Over the weekend, we might do some IDD work on the magnets or the solar panels -- or we might just do one drive followed by light remote sensing, to let us clear flash a bit.
Flash is a big worry, because our downlinks haven't been that great. Thisol we're looking at a meager 46 Mbits, for example. And since the drive and associated post-drive imaging consume about 35 Mbits of that -- maybe more -- there's not a lot for the science team to squabble over. The plan that comes out of the SOWG is heavy on the drive, light on science. And that's even after we help out by cutting back the motor/IMU data-collection rate from 8 Hz to to 4 Hz, nice fellas that we are.
While Khaled works on the drive, I scramble around, looking into a number of issues we need to resolve for thisol's plan. First priority is to meet with the Spirit rover planners to understand why they considered the limit-cycle check (which stops drives if the vehicle detects excessive slip) a hair-trigger check on their drives. I had formed the impression that the limit-cycle check wasn't very reliable, but it turns out I was wrong; it's reliable, they just didn't want to use it to detect slip much under 100%. This is good news; it means we can use it on Opportunity to detect slip in the neighborhood of 30%, which is all we're allowed.[1]
Next thing is to look into the suspension and tilt limits. I graph the data for the last several drives to try to understand how we're articulating when driving in this terrain. The idea is to set limits that are a bit higher than we expect, and it turns out that we're already setting them pretty close to the right numbers.
Finally, I have to scare up a Mobility/IDD person to do some longer-term analysis. I've generally been wanting them to do more of this. We ought to know more about how the vehicle has been performing -- how close IDD placements have been to predictions, how well visual odometry has been performing, and so on. During the nominal mission, those guys spent a lot of time at such tasks as these, but not any more. So I figure I should set them tasks as they occur to me; there's plenty they could be doing, but they don't always know what we want. But it turns out that the Mobility/IDD guy on shift today is Marcel -- who's from London, and has family there. I let it go.
Khaled and I spend some time working on the drive together, and then he works out the tricky visodom pointing with Mark while I take care of a visitor -- another friend of Susan Kurtik's, someone's foster child or something.[2] Like most kids, she gets into playing around with the rover, zooming the camera around and dragging it all over the terrain.
It's been a hectic day, and it doesn't stop at the CAM. Turns out they messed up an ODY comm window request or something, and instead of the meager 46 Mbits we thought we were getting, we're going to have only 27 Mbits. Oops. That's not even enough for the critical drive data. People start scrambling, and luckily there's an ODY ACE on console who's able to command the spacecraft to accept more data. So we're back up to 46. Lucky us.
A hundred million miles away, a long way from all this pain and death and worry, a plucky little robot trundles slowly toward its next destination. And so it goes.
Footnotes:
[1] Briefly put, the limit-cycle check periodically looks to see how far the rover has actually progressed versus the distance commanded. When you're driving with visodom, this is an easy-to-sequence slip check -- though expensive because visodom driving itself is expensive. Later, as our paranoia levels subsided slightly, we switched to a much less expensive slip-check technique.
[2] Susan Kurtik, as you'll recall, is the woman who hired me at JPL in the first place, recruiting me from the University of Illinois at Urbana-Champaign. Whatever Susan asks me for, Susan gets.
Not for the first time, I think to myself that I'm making a mistake. As much as I love it, I should leave Mars and go work in counter-terrorism in some capacity. I have a moral responsibility to act. I can't solve it all myself, but I should do my tiny little part rather than fuck around with an interplanetary Erector Set.
I usually tell myself that my job actually is important, that people need something to live for and we're part of providing that. But on a day like today, that justification -- I should call it a rationalization -- sounds awfully damn hollow.
I don't know what to do. I love you, London. I go to work.
Our drive onto the on-ramp, sol before yestersol, went fine. Yesterday Cooper and Jeff took us to the other end of the on-ramp; today we drive onto the freeway -- or East Path, or the Blue Line, as I've started calling it (from the blue lines in Rob Sullivan's images, designating the path we eventually chose).
Today's plan is to make about 15m total distance, about 10m of which will actually be along the Blue Line. Over the weekend, we might do some IDD work on the magnets or the solar panels -- or we might just do one drive followed by light remote sensing, to let us clear flash a bit.
Flash is a big worry, because our downlinks haven't been that great. Thisol we're looking at a meager 46 Mbits, for example. And since the drive and associated post-drive imaging consume about 35 Mbits of that -- maybe more -- there's not a lot for the science team to squabble over. The plan that comes out of the SOWG is heavy on the drive, light on science. And that's even after we help out by cutting back the motor/IMU data-collection rate from 8 Hz to to 4 Hz, nice fellas that we are.
While Khaled works on the drive, I scramble around, looking into a number of issues we need to resolve for thisol's plan. First priority is to meet with the Spirit rover planners to understand why they considered the limit-cycle check (which stops drives if the vehicle detects excessive slip) a hair-trigger check on their drives. I had formed the impression that the limit-cycle check wasn't very reliable, but it turns out I was wrong; it's reliable, they just didn't want to use it to detect slip much under 100%. This is good news; it means we can use it on Opportunity to detect slip in the neighborhood of 30%, which is all we're allowed.[1]
Next thing is to look into the suspension and tilt limits. I graph the data for the last several drives to try to understand how we're articulating when driving in this terrain. The idea is to set limits that are a bit higher than we expect, and it turns out that we're already setting them pretty close to the right numbers.
Finally, I have to scare up a Mobility/IDD person to do some longer-term analysis. I've generally been wanting them to do more of this. We ought to know more about how the vehicle has been performing -- how close IDD placements have been to predictions, how well visual odometry has been performing, and so on. During the nominal mission, those guys spent a lot of time at such tasks as these, but not any more. So I figure I should set them tasks as they occur to me; there's plenty they could be doing, but they don't always know what we want. But it turns out that the Mobility/IDD guy on shift today is Marcel -- who's from London, and has family there. I let it go.
Khaled and I spend some time working on the drive together, and then he works out the tricky visodom pointing with Mark while I take care of a visitor -- another friend of Susan Kurtik's, someone's foster child or something.[2] Like most kids, she gets into playing around with the rover, zooming the camera around and dragging it all over the terrain.
It's been a hectic day, and it doesn't stop at the CAM. Turns out they messed up an ODY comm window request or something, and instead of the meager 46 Mbits we thought we were getting, we're going to have only 27 Mbits. Oops. That's not even enough for the critical drive data. People start scrambling, and luckily there's an ODY ACE on console who's able to command the spacecraft to accept more data. So we're back up to 46. Lucky us.
A hundred million miles away, a long way from all this pain and death and worry, a plucky little robot trundles slowly toward its next destination. And so it goes.
Footnotes:
[1] Briefly put, the limit-cycle check periodically looks to see how far the rover has actually progressed versus the distance commanded. When you're driving with visodom, this is an easy-to-sequence slip check -- though expensive because visodom driving itself is expensive. Later, as our paranoia levels subsided slightly, we switched to a much less expensive slip-check technique.
[2] Susan Kurtik, as you'll recall, is the woman who hired me at JPL in the first place, recruiting me from the University of Illinois at Urbana-Champaign. Whatever Susan asks me for, Susan gets.
2010-07-06
Opportunity Sol 515 (Spirit Sol 536)
It's the fifth of July. Jim Erickson had asked me to check out the drive results yesterday so that we'd have a coherent story going into the planning meeting today. I did that, and I come in half an hour early today, to boot, but somehow we still don't have a coherent story.
When Steve Squyres polls the RPs (and we all know how painful that can be), we're not ready. "We're still arguing," is all I say on mike. He laughs and tells us we can have a few more minutes.
Our overall plan had been to drive north to positions where we could evaluate two south-leading paths. One of these is West Path, which tries to take us west around Purgatory Ripple; the other is East Path. The first half of that drive, which took us to where we could evaluate West Path, occurred over the weekend. We were going to do the second half thisol, but -- we eventually agree -- we seem to have good enough imaging for both paths from our present position. (Steve is visibly overjoyed to hear this -- he really didn't want to spend another sol just figuring out where we'd go.)
So much for the easy part. But the real kicker is, do we go east or west? Since the part of Erebus we're heading for is southeast of us, it would seem like a no-brainer to choose East Path -- if we choose West Path, we're gambling that we'll eventually find a way to ripple-hop east, and we don't like to gamble. But the ripples to the east of our current position look more fearsome than their westward counterparts, making the gamble look more attractive.
Just as Steve gets back to us, though, we find a pretty good eastward solution. So we have an answer, albeit just in the nick of time. East Path wins.
As hard as it was to reach that answer, that was cake compared to the next part. The East Path on-ramp is fairly narrow, not much wider than the rover's wheel base, and it's "guarded" by a low ripple. The most appealing way to enter the on-ramp (the way that takes us directly across the lowest part of the ripple) puts us at an extremely poor comm heading -- possibly in the PMA exclusion zone, which we're supposed to try to avoid because it means the rover's trying to radiate a signal through the mast. (Which works, but it further reduces the quality of an already poor comm pass.)
We solve it by doing some stuff we're not supposed to do: we drive straight north until we're just west of the on-ramp, then turn in place and scoot directly backward (east) across the ripple. The bad things we're doing include crossing the ripple at an oblique angle, turning in place (Opportunity has a bad steering actuator in one wheel, which hinders turning in place), and driving backward instead of forward across the ripple (so if we should get stuck, we'll have a little less strength if we have to drive out forward rather than backward). But this puts us at a much better comm heading, which we need; and anyway, this is a relatively low ripple, so we shouldn't bog down in it. I just hope those aren't famous last words.
We still have one more problem. According to our new Rules of the Road, we're not supposed to drive more than 5m in one sol unless we're monitoring slip to ensure that we're not slipping more than 25%. With the way we've written the sequence, we can't check our slip precisely enough to meet this requirement. Our choices are to rewrite the sequence in a way that will make it less likely to correctly execute the drive, or cut the drive short, so that we won't make it onto the on-ramp -- or waive the rule.
John Callas, who's the acting project manager today, is extremely reluctant to waive the rule. "It's not how we said we were going to drive here," he points out. "If we're not going to do what we said we were going to do, we should be willing to change what we said we were going to do."
He's got a real point. Cindy Oda tries to get him to let us slide: "Can you just let us do it this way today, and tomorrow we'll do it whatever way you want to?" But I don't want to put John in that position, and he doesn't like it either: "So you've already got your hand in the cookie jar, so why not just let you have this cookie?" he grins.
Still, it looks to me as if John is feeling the pressure from the team, but he needs a face-saving way out. In the end, Jake Matijevic rides to the rescue. "This isn't the place to apply the Rules of the Road," he points out gently. "They were meant for terrain we haven't been over before. Here, we're just following along our old tracks, except at the very end, which is only a couple of meters of driving -- a distance we'd be willing to tolerate anyway." This gets John off the hook, and it has the extra added benefit of being true. So we're heading for the on-ramp at last.
[Next post: sol 538 (Opportunity sol 517), July 8.]
When Steve Squyres polls the RPs (and we all know how painful that can be), we're not ready. "We're still arguing," is all I say on mike. He laughs and tells us we can have a few more minutes.
Our overall plan had been to drive north to positions where we could evaluate two south-leading paths. One of these is West Path, which tries to take us west around Purgatory Ripple; the other is East Path. The first half of that drive, which took us to where we could evaluate West Path, occurred over the weekend. We were going to do the second half thisol, but -- we eventually agree -- we seem to have good enough imaging for both paths from our present position. (Steve is visibly overjoyed to hear this -- he really didn't want to spend another sol just figuring out where we'd go.)
So much for the easy part. But the real kicker is, do we go east or west? Since the part of Erebus we're heading for is southeast of us, it would seem like a no-brainer to choose East Path -- if we choose West Path, we're gambling that we'll eventually find a way to ripple-hop east, and we don't like to gamble. But the ripples to the east of our current position look more fearsome than their westward counterparts, making the gamble look more attractive.
Just as Steve gets back to us, though, we find a pretty good eastward solution. So we have an answer, albeit just in the nick of time. East Path wins.
As hard as it was to reach that answer, that was cake compared to the next part. The East Path on-ramp is fairly narrow, not much wider than the rover's wheel base, and it's "guarded" by a low ripple. The most appealing way to enter the on-ramp (the way that takes us directly across the lowest part of the ripple) puts us at an extremely poor comm heading -- possibly in the PMA exclusion zone, which we're supposed to try to avoid because it means the rover's trying to radiate a signal through the mast. (Which works, but it further reduces the quality of an already poor comm pass.)
We solve it by doing some stuff we're not supposed to do: we drive straight north until we're just west of the on-ramp, then turn in place and scoot directly backward (east) across the ripple. The bad things we're doing include crossing the ripple at an oblique angle, turning in place (Opportunity has a bad steering actuator in one wheel, which hinders turning in place), and driving backward instead of forward across the ripple (so if we should get stuck, we'll have a little less strength if we have to drive out forward rather than backward). But this puts us at a much better comm heading, which we need; and anyway, this is a relatively low ripple, so we shouldn't bog down in it. I just hope those aren't famous last words.
We still have one more problem. According to our new Rules of the Road, we're not supposed to drive more than 5m in one sol unless we're monitoring slip to ensure that we're not slipping more than 25%. With the way we've written the sequence, we can't check our slip precisely enough to meet this requirement. Our choices are to rewrite the sequence in a way that will make it less likely to correctly execute the drive, or cut the drive short, so that we won't make it onto the on-ramp -- or waive the rule.
John Callas, who's the acting project manager today, is extremely reluctant to waive the rule. "It's not how we said we were going to drive here," he points out. "If we're not going to do what we said we were going to do, we should be willing to change what we said we were going to do."
He's got a real point. Cindy Oda tries to get him to let us slide: "Can you just let us do it this way today, and tomorrow we'll do it whatever way you want to?" But I don't want to put John in that position, and he doesn't like it either: "So you've already got your hand in the cookie jar, so why not just let you have this cookie?" he grins.
Still, it looks to me as if John is feeling the pressure from the team, but he needs a face-saving way out. In the end, Jake Matijevic rides to the rescue. "This isn't the place to apply the Rules of the Road," he points out gently. "They were meant for terrain we haven't been over before. Here, we're just following along our old tracks, except at the very end, which is only a couple of meters of driving -- a distance we'd be willing to tolerate anyway." This gets John off the hook, and it has the extra added benefit of being true. So we're heading for the on-ramp at last.
[Next post: sol 538 (Opportunity sol 517), July 8.]
2010-07-01
Opportunity Sol 510 (Spirit Sol 531)
Cooper had a car accident. I figured it was only a matter of time before this happened to one of us, and I'm glad it wasn't I. The car is a wreck, but his daughter, who was in the car with him, was fine, and so is he. Except that he thinks he has a cracked rib. So he's leaving early.
So anyway.
Thisol is the last sol we'll spend around Purgatory Ripple. We've got two MI stacks to do, then we take off. One of the MI targets, "Cleat Tab," is pretty easy. The other, "Torment," lives up to its name. (I wanted to call it "Cleat Fanta," but I was overruled.) As we exited Purgatory Ripple, the front and rear wheels left separate but overlapping tracks, so that the tracks have a sort of inner wall and an outer wall. Torment is on the interior of the outer wall and we're trying to shoot a stack normal to it, which means at about a 45-degree angle to the surrounding terrain. This in itself is not hard; the problem is that by the time the IDD is close enough to get a focused image, the MI poker will contact the inner wall. This is something we want to avoid.
If we were closer to the target, we could solve it by flipping the wrist into the other configuration, which would put the MI poker on the high side of the terrain. But to reach the target, we have to stretch the arm out so far that that wrist configuration would cause a self-collision.
In the end, we just have to come straight down on it -- not normal to the wall -- and hope for the best.
While I'm doing that, Cooper and Paolo are solving a problem with the drive. The drive itself is quite simple: we just back up 2m across North Ripple, turning as we go to set ourselves up for a northward drive along a trough between two ripples. (Our ultimate goal is southeast of here; we're taking a few steps north so that we can get into position to evaluate candidate routes that would take us around Purgatory Ripple without crossing too many other ripples along the way.)
The problem has to do with the turn. We need to track our position with visual odometry so that we can tell whether we're slipping too much -- it's one of the Rules of the Road for this vehicle in the post-Purgatory era -- but in this sandy, rock-free terrain, the only features visodom can converge on are our own tracks. And we're turning so much during the backup that our tracks won't be in the cameras' field of view the whole time.
Unless we change the camera pointing in midstream. Visual odometry works by comparing successive pairs of images -- you take an image, step, take another image, and compare the two. Since it only needs two pictures at any given time, the current one and the one from the previous step, you can "reseed" it at any point. It's time-consuming, but it works. And we overestimated the drive time, so we'll have enough time for one such reseed. So Cooper and Paolo figure out a pair of pointings -- one for the first half of the drive, and one for the second -- and that's what we go with.
That's all. Except for one more problem that comes up at the CAM. "What if," Emily asks -- "what if the poker does contact the terrain after all?"
I have to admit it's a good question. "We don't expect any contact, but all the moves are in free-space mode. So if we contact the terrain, it shouldn't damage the arm, but it'll fault out the IDD sequence and keep us from stowing, which will also mean we don't drive."
We generally sequence this way. For a time, there was a war between what Jeff Biesiadecki termed the "Baumgartian" and "Bonitzian" views of this problem.[1] In the Baumgartian view, unexpected contact with the terrain is something that should immediately stop motion: the terrain isn't where you think it is, so you should stop and reassess everything from the ground. So you use free-space mode unless you're actually expecting contact. In the Bonitzian view, it's acceptable for a sequence to tolerate early contact -- as long as you've written it carefully, so that it will proceed safely whether you have early contact or not.
For a long time I was a Bonitzian, but somewhere in there I converted to the Baumgartian camp, as I think most of us did. Even leaving the safety arguments aside, the Baumgartian approach makes sequencing faster and simpler, since you don't have to validate that the sequence always works correctly in the face of any unexpected contact.
So this sequence was Baumgartian. But in this case, the Baumgartian downside -- faulting out the sequence, and blowing the drive with it -- would be worse than usual. Not as bad as harming the vehicle, but still pretty bad. The Mars Program office has been breathing down our necks to get us moving again (I'm told), so nobody really wants to risk the drive. Luckily, the fix -- converting to Bonitzian style -- is easy in this case: I simply change the MI stack moves so that they use guarded mode rather than free-space mode. This way, if the MI poker senses contact with the terrain, it won't approach the terrain any more closely but will generally go on with the sequence. We can't always make that change so easily, but in this case it happens to work out. Score one for the Bonitzians.[2]
[Next post: sol 536 (Opportunity sol 515), July 6.]
Footnotes:
[1] This is after Eric Baumgartner and Bob Bonitz, two of the original rover drivers -- both of them deeply involved in the development of the arm. Though long gone from the project, they remain important influences to this day.
[2] But this was one of the last scores for the Bonitzians. Today, we're all Baumgartians. Mainly, this is due to a project-wide change in attitude about sols: they're clearly more plentiful than we once thought, so we don't go as far out of our way to save them as we used to. In that world, the main advantage of Bonitzianism -- it can enable you to autonomously recover and go on to do other activities -- disappears, and the main advantage of Baumgartianism -- it's more paranoid about unexpected behavior -- wins out.
So anyway.
Thisol is the last sol we'll spend around Purgatory Ripple. We've got two MI stacks to do, then we take off. One of the MI targets, "Cleat Tab," is pretty easy. The other, "Torment," lives up to its name. (I wanted to call it "Cleat Fanta," but I was overruled.) As we exited Purgatory Ripple, the front and rear wheels left separate but overlapping tracks, so that the tracks have a sort of inner wall and an outer wall. Torment is on the interior of the outer wall and we're trying to shoot a stack normal to it, which means at about a 45-degree angle to the surrounding terrain. This in itself is not hard; the problem is that by the time the IDD is close enough to get a focused image, the MI poker will contact the inner wall. This is something we want to avoid.
If we were closer to the target, we could solve it by flipping the wrist into the other configuration, which would put the MI poker on the high side of the terrain. But to reach the target, we have to stretch the arm out so far that that wrist configuration would cause a self-collision.
In the end, we just have to come straight down on it -- not normal to the wall -- and hope for the best.
While I'm doing that, Cooper and Paolo are solving a problem with the drive. The drive itself is quite simple: we just back up 2m across North Ripple, turning as we go to set ourselves up for a northward drive along a trough between two ripples. (Our ultimate goal is southeast of here; we're taking a few steps north so that we can get into position to evaluate candidate routes that would take us around Purgatory Ripple without crossing too many other ripples along the way.)
The problem has to do with the turn. We need to track our position with visual odometry so that we can tell whether we're slipping too much -- it's one of the Rules of the Road for this vehicle in the post-Purgatory era -- but in this sandy, rock-free terrain, the only features visodom can converge on are our own tracks. And we're turning so much during the backup that our tracks won't be in the cameras' field of view the whole time.
Unless we change the camera pointing in midstream. Visual odometry works by comparing successive pairs of images -- you take an image, step, take another image, and compare the two. Since it only needs two pictures at any given time, the current one and the one from the previous step, you can "reseed" it at any point. It's time-consuming, but it works. And we overestimated the drive time, so we'll have enough time for one such reseed. So Cooper and Paolo figure out a pair of pointings -- one for the first half of the drive, and one for the second -- and that's what we go with.
That's all. Except for one more problem that comes up at the CAM. "What if," Emily asks -- "what if the poker does contact the terrain after all?"
I have to admit it's a good question. "We don't expect any contact, but all the moves are in free-space mode. So if we contact the terrain, it shouldn't damage the arm, but it'll fault out the IDD sequence and keep us from stowing, which will also mean we don't drive."
We generally sequence this way. For a time, there was a war between what Jeff Biesiadecki termed the "Baumgartian" and "Bonitzian" views of this problem.[1] In the Baumgartian view, unexpected contact with the terrain is something that should immediately stop motion: the terrain isn't where you think it is, so you should stop and reassess everything from the ground. So you use free-space mode unless you're actually expecting contact. In the Bonitzian view, it's acceptable for a sequence to tolerate early contact -- as long as you've written it carefully, so that it will proceed safely whether you have early contact or not.
For a long time I was a Bonitzian, but somewhere in there I converted to the Baumgartian camp, as I think most of us did. Even leaving the safety arguments aside, the Baumgartian approach makes sequencing faster and simpler, since you don't have to validate that the sequence always works correctly in the face of any unexpected contact.
So this sequence was Baumgartian. But in this case, the Baumgartian downside -- faulting out the sequence, and blowing the drive with it -- would be worse than usual. Not as bad as harming the vehicle, but still pretty bad. The Mars Program office has been breathing down our necks to get us moving again (I'm told), so nobody really wants to risk the drive. Luckily, the fix -- converting to Bonitzian style -- is easy in this case: I simply change the MI stack moves so that they use guarded mode rather than free-space mode. This way, if the MI poker senses contact with the terrain, it won't approach the terrain any more closely but will generally go on with the sequence. We can't always make that change so easily, but in this case it happens to work out. Score one for the Bonitzians.[2]
[Next post: sol 536 (Opportunity sol 515), July 6.]
Footnotes:
[1] This is after Eric Baumgartner and Bob Bonitz, two of the original rover drivers -- both of them deeply involved in the development of the arm. Though long gone from the project, they remain important influences to this day.
[2] But this was one of the last scores for the Bonitzians. Today, we're all Baumgartians. Mainly, this is due to a project-wide change in attitude about sols: they're clearly more plentiful than we once thought, so we don't go as far out of our way to save them as we used to. In that world, the main advantage of Bonitzianism -- it can enable you to autonomously recover and go on to do other activities -- disappears, and the main advantage of Baumgartianism -- it's more paranoid about unexpected behavior -- wins out.
2010-06-30
Opportunity Sol 509 (Spirit Sol 530)
Bad news: in about ten days, Khaled will be a Rover Planner no more. He's leaving, taking over the role of coordinating the flight software uploads.
As politely as I can, I ask him the only question that comes to mind: "Are you crazy?"
"I've been asking myself that," he admits with a mournful tone. "It's certainly not going to be as much fun as this job. But this one doesn't meet those ECAP goals I keep hearing, so ...."[1]
Well, the guy's gotta do what he's gotta do. Andy isn't going to be happy to hear this -- more headaches for him. I'm not too happy about it myself, honestly. Khaled's a fine RP; he came up to speed quickly, knows the rovers well, and is thorough, reliable, and conscientious. He'll be hard to replace.
One thing I won't miss is the smell of his wife's cooking, when he brings in his leftovers for lunch nearly every day. Not that her cooking smells bad. Quite the contrary, it always smells delicious. The problem with this is that it makes my own hunger impossible to ignore -- whenever I get a whiff of Khaled's lunch, I have to go buy lunch from the cafeteria. And that's never as good as his. Darn him!
Thisol's sequencing is not much more complex than yestersol's. Khaled and I put the extra time to good use by preparing the drive sequences. He'd already put together a zeroth draft for the first couple of upcoming drive sols (510 and 511), and we go over them and make some changes together.
Gee, he's not only thorough, reliable, and conscientious, he also works ahead, making things simpler for the rest of us. And now he's leaving.
Bastard.[2]
Footnotes:
[1] An ECAP -- Employee Contribution Assessment and Planning -- is JPL's annual employee performance review.
[2] Khaled has come and gone a couple of times now. Right now, he's off working on Moonrise, but I hope he'll be back again soon. He's damn good.
As politely as I can, I ask him the only question that comes to mind: "Are you crazy?"
"I've been asking myself that," he admits with a mournful tone. "It's certainly not going to be as much fun as this job. But this one doesn't meet those ECAP goals I keep hearing, so ...."[1]
Well, the guy's gotta do what he's gotta do. Andy isn't going to be happy to hear this -- more headaches for him. I'm not too happy about it myself, honestly. Khaled's a fine RP; he came up to speed quickly, knows the rovers well, and is thorough, reliable, and conscientious. He'll be hard to replace.
One thing I won't miss is the smell of his wife's cooking, when he brings in his leftovers for lunch nearly every day. Not that her cooking smells bad. Quite the contrary, it always smells delicious. The problem with this is that it makes my own hunger impossible to ignore -- whenever I get a whiff of Khaled's lunch, I have to go buy lunch from the cafeteria. And that's never as good as his. Darn him!
Thisol's sequencing is not much more complex than yestersol's. Khaled and I put the extra time to good use by preparing the drive sequences. He'd already put together a zeroth draft for the first couple of upcoming drive sols (510 and 511), and we go over them and make some changes together.
Gee, he's not only thorough, reliable, and conscientious, he also works ahead, making things simpler for the rest of us. And now he's leaving.
Bastard.[2]
Footnotes:
[1] An ECAP -- Employee Contribution Assessment and Planning -- is JPL's annual employee performance review.
[2] Khaled has come and gone a couple of times now. Right now, he's off working on Moonrise, but I hope he'll be back again soon. He's damn good.
2010-06-29
Opportunity Sol 508 (Spirit Sol 529)
This is one of the simplest days I've had in a long time: just a tool change from APXS to MB, continuing our exploration of the soil near Purgatory Ripple. And I'm not even RP-1 (Jeng is), so I just have to review it.
Yawn.
One-way light-time to Mars has been falling. Right now it's eight minutes, twenty-seven seconds -- we're just about equidistant from Mars and the Sun. A big Isosceles triangle in space. 8:27 -- that's 507 seconds, one less than the sol number we're planning. That's the first time those numbers have crossed. "The rovers haven't been this close to Earth since December 2003," Steve remarks.
The CAM goes smoothly; another one down. "One more day and we hit the gas!" Steve exclaims. "I can't wait to get outta here!"
Yawn.
One-way light-time to Mars has been falling. Right now it's eight minutes, twenty-seven seconds -- we're just about equidistant from Mars and the Sun. A big Isosceles triangle in space. 8:27 -- that's 507 seconds, one less than the sol number we're planning. That's the first time those numbers have crossed. "The rovers haven't been this close to Earth since December 2003," Steve remarks.
The CAM goes smoothly; another one down. "One more day and we hit the gas!" Steve exclaims. "I can't wait to get outta here!"
2010-06-28
Opportunity Sol 507 (Spirit Sol 528)
Ah, that feels better. The drive went just exactly as planned; the old track, tailings, and wide swaths of undisturbed soil are within the IDD work volume. I've never been so excited about an 80cm drive.
Nor has most of the rest of the team. I forget, they actually started their approach to this spot well before I got here -- it took something like a week to do the drive away, turn-in-place, and drive back -- and today was the last chance to get it. But we got it. Brenda says Steve is "happy as a clam," and he is. "Take your time getting SAP set up, Brenda," he says. "I'm just gonna sit here and gaze in wonder at this reachability map -- nice job, Rover Planners!"
I feel a little silly about taking a bow (figuratively!). It's just 80cm; I'm supposed to be able to do this with my eyes closed. Well, whatever.
I take a moment to look at the drive telemetry. After all that time we spent worrying about what to set them to, I'm curious about what the bogie angles were during this drive. The left bogie started at 14 degrees -- the value that aborted our previous drive -- and we knew it was going to increase as we finished climbing North Ripple, but how much did it increase?
It went to 15 degrees. That's it. All that time we spent quibbling about the drive, arguing over whether 20 degrees was enough headroom, and so on ... and we needed about one extra degree.
Well, at least it worked. It did execute nicely, and now we're ready to swing the IDD out and attack the soil. We'll start with a 2x2 MI mosaic around a patch of undisturbed soil near the RF wheel, then do another such mosaic, followed by an APXS placement, in the track.
This is fairly straightforward IDD work, though it's making Jeng nervous. Maybe it's that he doesn't yet trust his ability to sequence the IDD, or maybe it's the early deadline -- this is what we call a "tight" sol, with uplink in just a few hours, leaving little room for error. But it's an undramatic day; with me looking over his shoulder, Jeng cranks out the IDD sequence in plenty of time.
At the Activity Plan Approval Meeting, Paolo points out that the near side of the track, the spot we've picked as our representative undisturbed-soil location, is darker than the far side. This means that the far side has lighter-colored dust, which generally means smaller, finer dust particles. This is the stuff we sometimes call "foo-foo dust," and there's a going hypothesis that that's what Purgatory Ripple is made of. So, Paolo suggests, maybe we should be IDDing there instead?[1]
Rob Sullivan answers the question. "What we think we're seeing there is the result of a recent wind event -- in the last few sols, a gust of wind came through here and blew away a lot of the material we dredged up when we exited Purgatory Ripple. So a lot of that dust probably blew in from that event. The nearby patch is more likely to be the native material of Purgatory Ripple."
"So your interpretation is right," Steve Squyres chimes in. "That lighter-colored stuff probably is foo-foo dust -- but in this case, that's a bad thing. But keep makin' suggestions!"
The rest of the day goes ahead pretty nominally. At the CAM, Dan resurrects the Picture-of-the-Day tradition to show a true-color PANCAM image of the trench we (inadvertently) made in Purgatory Ripple. It's enormous, as wide as a wheel and at least that deep.
"How did we ever get outta that?" he wonders aloud.
"We have really good rover drivers," Wendy Calvin says.
"And a whole lotta patience," adds Steve.
Courtesy NASA/JPL-Caltech. At last. That's what we've been trying to do.
Footnotes:
[1] A characteristically brilliant observation. At this point, Paolo had been on the project for only a few weeks -- a couple of months, at most -- but he was already showing the keen insights and attention to detail that would later make him a top-notch rover driver.
Nor has most of the rest of the team. I forget, they actually started their approach to this spot well before I got here -- it took something like a week to do the drive away, turn-in-place, and drive back -- and today was the last chance to get it. But we got it. Brenda says Steve is "happy as a clam," and he is. "Take your time getting SAP set up, Brenda," he says. "I'm just gonna sit here and gaze in wonder at this reachability map -- nice job, Rover Planners!"
I feel a little silly about taking a bow (figuratively!). It's just 80cm; I'm supposed to be able to do this with my eyes closed. Well, whatever.
I take a moment to look at the drive telemetry. After all that time we spent worrying about what to set them to, I'm curious about what the bogie angles were during this drive. The left bogie started at 14 degrees -- the value that aborted our previous drive -- and we knew it was going to increase as we finished climbing North Ripple, but how much did it increase?
It went to 15 degrees. That's it. All that time we spent quibbling about the drive, arguing over whether 20 degrees was enough headroom, and so on ... and we needed about one extra degree.
Well, at least it worked. It did execute nicely, and now we're ready to swing the IDD out and attack the soil. We'll start with a 2x2 MI mosaic around a patch of undisturbed soil near the RF wheel, then do another such mosaic, followed by an APXS placement, in the track.
This is fairly straightforward IDD work, though it's making Jeng nervous. Maybe it's that he doesn't yet trust his ability to sequence the IDD, or maybe it's the early deadline -- this is what we call a "tight" sol, with uplink in just a few hours, leaving little room for error. But it's an undramatic day; with me looking over his shoulder, Jeng cranks out the IDD sequence in plenty of time.
At the Activity Plan Approval Meeting, Paolo points out that the near side of the track, the spot we've picked as our representative undisturbed-soil location, is darker than the far side. This means that the far side has lighter-colored dust, which generally means smaller, finer dust particles. This is the stuff we sometimes call "foo-foo dust," and there's a going hypothesis that that's what Purgatory Ripple is made of. So, Paolo suggests, maybe we should be IDDing there instead?[1]
Rob Sullivan answers the question. "What we think we're seeing there is the result of a recent wind event -- in the last few sols, a gust of wind came through here and blew away a lot of the material we dredged up when we exited Purgatory Ripple. So a lot of that dust probably blew in from that event. The nearby patch is more likely to be the native material of Purgatory Ripple."
"So your interpretation is right," Steve Squyres chimes in. "That lighter-colored stuff probably is foo-foo dust -- but in this case, that's a bad thing. But keep makin' suggestions!"
The rest of the day goes ahead pretty nominally. At the CAM, Dan resurrects the Picture-of-the-Day tradition to show a true-color PANCAM image of the trench we (inadvertently) made in Purgatory Ripple. It's enormous, as wide as a wheel and at least that deep.
"How did we ever get outta that?" he wonders aloud.
"We have really good rover drivers," Wendy Calvin says.
"And a whole lotta patience," adds Steve.
Courtesy NASA/JPL-Caltech. At last. That's what we've been trying to do.
Footnotes:
[1] A characteristically brilliant observation. At this point, Paolo had been on the project for only a few weeks -- a couple of months, at most -- but he was already showing the keen insights and attention to detail that would later make him a top-notch rover driver.
2010-06-25
Opportunity Sol 505 (Spirit Sol 525)
Six fucking centimeters. That's it. Six fucking centimeters is all the distance we made yestersol.
It's my fault. In the drive sequence, we retained most of the limits that had been set for the previous drive. We were just passing over the ground they had meant to pass over, so we figured the same limits would be fine.
Frankly, this was just lazy. And it's what stopped us. As it happened, Opportunity's left bogie articulated just a little more than we expected it to as it climbed the ripple that's under us now (North Ripple), and that terminated the drive. After six centimeters.
We really, really should have caught this in advance. The bogie limit was 14 degrees, and if we'd checked the telemetry, we'd have noticed that it was already just over 11 degrees -- not a lot of headroom there. The really painful part is that I thought of that while we were doing the sequencing, and then got distracted by something and never got back to checking. Oy vey.[1]
And this changes our plans for the weekend. We'd originally intended to IDD the far track -- the one we were driving to -- but since we didn't make it there, that's out. So it'll be a touch-and-go sol, where we IDD whatever happens to be close to us and then try to drive the remaining 69cm or so to the far track.
Assuming we're capable of that much.
Both Jeff Favretto and Steve Squyres are philosophical about the drive result. "This is just a consequence of using conservative limits," Jeff says. And conservative limits are exactly what we promised the Mars Program Office we'd use, adds Steve. But as usual, things aren't that simple. Apparently, the Program management has been putting some pressure on John Callas (acting project manager while Jim Erickson is out of town). He's getting it from both sides: some people are complaining that we're being reckless cowboys; others, that we're too conservative. They want us to be successful without making any mistakes, which is fine to a point, but they've pushed it far enough that it amounts to demanding that we be lucky.[2] I'm glad I don't have his job.[3]
Squyres might have been getting some of that "too-conservative" flack as well. "I want you to drive safely," he tells me, "but I should say that if we don't make it to the target after this attempt, we're just going to IDD whatever we've got and then get out of here." "So, no pressure?" I ask. "Yeah," he grins, "no pressure."
The IDD work is fairly straightforward, and I get it more or less knocked off in a couple of hours. It initially looks like one of the targets the science team chose was too close to our right front wheel, but the simulation says it's OK, so I move on.
The bulk of the sol's work is in arguing over the drive. Not in implementing it, just in arguing about it. I go through every single parameter limit and check it against both the current state and the RSVP-simulated maximum, trying to pick sensible numbers for everything. We're left with a small set of numbers to argue about, the suspension limits -- most crucially, what should be the proper settings for the bogie limits that stopped us on 503? This turns into, I kid you not, a couple of hours' discussion at least.[4]
Maybe this is all ridiculous. The distance of this drive is about equal to a single wheel rotation -- how much trouble could we get into, even if we abandoned the limits altogether? But at this point we're so damn paranoid that we're ready to entertain the ridiculous.
The number we eventually pick for the max bogie limit is 25 degrees. That's about twice what we're at now, and just 5 degrees shy of the hardstop. It's almost impossible for us to actually hit it, and all things considered, I can feel OK about that. If we actually do reach that limit, something really strange will be happening, and we'll want the drive to stop. Which is what the limit is for.
Our paranoia doesn't end there. Steve asked me earlier in the day for a list of everything that could possibly stop the drive, and one thing that could stop it is for the IDD work to fail -- that would prevent the drive from even starting. So I look back at that, and the IDD target that concerned me before starts to look even more worrisome.
RSVP can show something called the "collision volumes," which RSVP visualizes as a set of purple boxes that are slightly bigger than the rover parts. They're like the rover is wearing bulky translucent purple padding all over. The on-board flight software uses these collision volumes to detect a pending self-collision; the idea is to be conservative about the possibility of collisions during (especially) IDD work -- if the padded version of the rover doesn't collide with itself, then the real thing won't. RSVP also tells us whether these collisions will occur in the sequence we've planned, and since the simulation is happy, we should expect that the real thing will be okay.
And yet ... when we're inspecting the nearby IDD target, the collision volumes come so close to intersecting that we have to really work to show that they're not. What if the flight vehicle is just a tiny bit different from this simulation? It shouldn't be, but as I said, we're pretty paranoid now, and the difference would have to be only a tiny one -- maybe small enough that it would have gone undetected until now -- to screw us up once again.
I'm just being paranoid. I'm sure I'm just being paranoid. But I'm not the only one, and we end up deciding to turn off those collision checks during the close approach. We're not coming so close that there's a risk of an actual collision, just a risk that the collision volumes -- the padding -- could collide, and turning them off means we can be sure it won't happen.
I won't feel good about any of this until Monday.
[Next post: sol 528 (Opportunity sol 507), June 28.]
Footnotes:
[1] This exact incident, if I recall aright, is why checking the suspension history from the previous drive is something I now do every sol. It's bad enough to make a dumb mistake, much worse to repeat it.
[2] Fortunately, this was the last time that Mars Program ever made such a stupid demand. cough
[3] Still true. And how.
[4] In a way, this is one of the things I really love about my job: the obsessive focus on getting these things right. We might spend an hour arguing about a degree, or a millimeter. Sometimes those discussions are horrendously painful when I'm in the middle of them, but when I take a step back, I love being in an environment where people are that passionate about even the little things.
It's my fault. In the drive sequence, we retained most of the limits that had been set for the previous drive. We were just passing over the ground they had meant to pass over, so we figured the same limits would be fine.
Frankly, this was just lazy. And it's what stopped us. As it happened, Opportunity's left bogie articulated just a little more than we expected it to as it climbed the ripple that's under us now (North Ripple), and that terminated the drive. After six centimeters.
We really, really should have caught this in advance. The bogie limit was 14 degrees, and if we'd checked the telemetry, we'd have noticed that it was already just over 11 degrees -- not a lot of headroom there. The really painful part is that I thought of that while we were doing the sequencing, and then got distracted by something and never got back to checking. Oy vey.[1]
And this changes our plans for the weekend. We'd originally intended to IDD the far track -- the one we were driving to -- but since we didn't make it there, that's out. So it'll be a touch-and-go sol, where we IDD whatever happens to be close to us and then try to drive the remaining 69cm or so to the far track.
Assuming we're capable of that much.
Both Jeff Favretto and Steve Squyres are philosophical about the drive result. "This is just a consequence of using conservative limits," Jeff says. And conservative limits are exactly what we promised the Mars Program Office we'd use, adds Steve. But as usual, things aren't that simple. Apparently, the Program management has been putting some pressure on John Callas (acting project manager while Jim Erickson is out of town). He's getting it from both sides: some people are complaining that we're being reckless cowboys; others, that we're too conservative. They want us to be successful without making any mistakes, which is fine to a point, but they've pushed it far enough that it amounts to demanding that we be lucky.[2] I'm glad I don't have his job.[3]
Squyres might have been getting some of that "too-conservative" flack as well. "I want you to drive safely," he tells me, "but I should say that if we don't make it to the target after this attempt, we're just going to IDD whatever we've got and then get out of here." "So, no pressure?" I ask. "Yeah," he grins, "no pressure."
The IDD work is fairly straightforward, and I get it more or less knocked off in a couple of hours. It initially looks like one of the targets the science team chose was too close to our right front wheel, but the simulation says it's OK, so I move on.
The bulk of the sol's work is in arguing over the drive. Not in implementing it, just in arguing about it. I go through every single parameter limit and check it against both the current state and the RSVP-simulated maximum, trying to pick sensible numbers for everything. We're left with a small set of numbers to argue about, the suspension limits -- most crucially, what should be the proper settings for the bogie limits that stopped us on 503? This turns into, I kid you not, a couple of hours' discussion at least.[4]
Maybe this is all ridiculous. The distance of this drive is about equal to a single wheel rotation -- how much trouble could we get into, even if we abandoned the limits altogether? But at this point we're so damn paranoid that we're ready to entertain the ridiculous.
The number we eventually pick for the max bogie limit is 25 degrees. That's about twice what we're at now, and just 5 degrees shy of the hardstop. It's almost impossible for us to actually hit it, and all things considered, I can feel OK about that. If we actually do reach that limit, something really strange will be happening, and we'll want the drive to stop. Which is what the limit is for.
Our paranoia doesn't end there. Steve asked me earlier in the day for a list of everything that could possibly stop the drive, and one thing that could stop it is for the IDD work to fail -- that would prevent the drive from even starting. So I look back at that, and the IDD target that concerned me before starts to look even more worrisome.
RSVP can show something called the "collision volumes," which RSVP visualizes as a set of purple boxes that are slightly bigger than the rover parts. They're like the rover is wearing bulky translucent purple padding all over. The on-board flight software uses these collision volumes to detect a pending self-collision; the idea is to be conservative about the possibility of collisions during (especially) IDD work -- if the padded version of the rover doesn't collide with itself, then the real thing won't. RSVP also tells us whether these collisions will occur in the sequence we've planned, and since the simulation is happy, we should expect that the real thing will be okay.
And yet ... when we're inspecting the nearby IDD target, the collision volumes come so close to intersecting that we have to really work to show that they're not. What if the flight vehicle is just a tiny bit different from this simulation? It shouldn't be, but as I said, we're pretty paranoid now, and the difference would have to be only a tiny one -- maybe small enough that it would have gone undetected until now -- to screw us up once again.
I'm just being paranoid. I'm sure I'm just being paranoid. But I'm not the only one, and we end up deciding to turn off those collision checks during the close approach. We're not coming so close that there's a risk of an actual collision, just a risk that the collision volumes -- the padding -- could collide, and turning them off means we can be sure it won't happen.
I won't feel good about any of this until Monday.
[Next post: sol 528 (Opportunity sol 507), June 28.]
Footnotes:
[1] This exact incident, if I recall aright, is why checking the suspension history from the previous drive is something I now do every sol. It's bad enough to make a dumb mistake, much worse to repeat it.
[2] Fortunately, this was the last time that Mars Program ever made such a stupid demand. cough
[3] Still true. And how.
[4] In a way, this is one of the things I really love about my job: the obsessive focus on getting these things right. We might spend an hour arguing about a degree, or a millimeter. Sometimes those discussions are horrendously painful when I'm in the middle of them, but when I take a step back, I love being in an environment where people are that passionate about even the little things.
2010-06-24
Opportunity Sol 504 (Spirit Sol 524)
Yestersol was a two-sol plan and tomorrow's a three-sol weekend plan; we just have a pre-planning meeting today. Assuming none of our paranoid limit checks triggers and we actually make it to our goal, what's on tap for the weekend is simply some light IDD work: a little MI, a little APXS, a little MB. Nothing out of the ordinary.
I hope that will mean we have time to start planning the drive away from Purgatory. I haven't given any thought to it, and I just noticed that I'm scheduled all next week. I've been off this rover for a while, but I want to look like I know what I'm doing.
I hope that will mean we have time to start planning the drive away from Purgatory. I haven't given any thought to it, and I just noticed that I'm scheduled all next week. I've been off this rover for a while, but I want to look like I know what I'm doing.
2010-06-23
Opportunity Sol 503 (Spirit Sol 523)
It's my first sol driving the rovers since I came back from jury duty, and I've switched to Opportunity -- my first sol on this rover since, well, since before they encountered Purgatory Ripple.
Yestersol's drive was a 20-degree turn in place followed by a whopping 50cm drive forward, in two 25cm arcs. But only one of the two arcs completed. Opportunity sensed that she'd slipped 32% on that arc, and we'd set a limit of 30%, so she stopped. Opportunity is scared of bogging down again.
But Khaled and Paolo analyze the drive and conclude that it's safe for us to proceed. What happened was that as we did the turn in place, the right-front wheel -- whose steering actuator is broken -- was mostly just dragged along, digging itself a small trench in the process. Then when we tried to step forward, we slipped a touch more than expected as we climbed out of that trench. But the rover's safe, on top of the soil rather then underneath it, and the way ahead is clear.
Our goal here is simply to drive back to a part of the tracks we left as we exited Purgatory. Yestersol's drive was supposed to leave us about 50cm from them, but, well ... so today we'll drive a whopping 75cm; if it goes well, we'll be right where we need to be to IDD the tracks over the weekend. Then next week we'll pack up and leave.
Khaled and I do spend a fair amount of time working on the drive -- everyone's paranoid about re-approaching this ripple -- but we're ready in plenty of time. Jeff Favretto notices the level of effort we're putting in, and during one of the meetings, he just grins and holds his hands a little more than shoulder-width apart. "Seventy-five centimeters," he grins, shaking his head. "This is how far we're going. It takes all that ...."
Meanwhile, Spirit's exhibiting a little bit of unusual behavior. During her last drive, the left bogie angles changed in unusual ways, indicating that the middle wheel kept lifting well into the air and then dropping back down. One hypothesis is that something's stuck on the outside of the wheel. If this is true, we'll never be able to see it: they're not visible from any of the cameras.[1] Jeff thinks he knows what's stuck in the wheel: "It's that dinosaur bone we've been looking for all this time!"
Courtesy NASA/JPL-Caltech. Post-drive front HAZCAM image showing the track created by the right front wheel when we turned in place on the previous drive.
Footnotes:
[1] As you can see, we hadn't then thought of using the MI to view the middle wheels. We didn't think of that trick until much later, when Spirit was embedded in Troy.
Yestersol's drive was a 20-degree turn in place followed by a whopping 50cm drive forward, in two 25cm arcs. But only one of the two arcs completed. Opportunity sensed that she'd slipped 32% on that arc, and we'd set a limit of 30%, so she stopped. Opportunity is scared of bogging down again.
But Khaled and Paolo analyze the drive and conclude that it's safe for us to proceed. What happened was that as we did the turn in place, the right-front wheel -- whose steering actuator is broken -- was mostly just dragged along, digging itself a small trench in the process. Then when we tried to step forward, we slipped a touch more than expected as we climbed out of that trench. But the rover's safe, on top of the soil rather then underneath it, and the way ahead is clear.
Our goal here is simply to drive back to a part of the tracks we left as we exited Purgatory. Yestersol's drive was supposed to leave us about 50cm from them, but, well ... so today we'll drive a whopping 75cm; if it goes well, we'll be right where we need to be to IDD the tracks over the weekend. Then next week we'll pack up and leave.
Khaled and I do spend a fair amount of time working on the drive -- everyone's paranoid about re-approaching this ripple -- but we're ready in plenty of time. Jeff Favretto notices the level of effort we're putting in, and during one of the meetings, he just grins and holds his hands a little more than shoulder-width apart. "Seventy-five centimeters," he grins, shaking his head. "This is how far we're going. It takes all that ...."
Meanwhile, Spirit's exhibiting a little bit of unusual behavior. During her last drive, the left bogie angles changed in unusual ways, indicating that the middle wheel kept lifting well into the air and then dropping back down. One hypothesis is that something's stuck on the outside of the wheel. If this is true, we'll never be able to see it: they're not visible from any of the cameras.[1] Jeff thinks he knows what's stuck in the wheel: "It's that dinosaur bone we've been looking for all this time!"
Courtesy NASA/JPL-Caltech. Post-drive front HAZCAM image showing the track created by the right front wheel when we turned in place on the previous drive.
Footnotes:
[1] As you can see, we hadn't then thought of using the MI to view the middle wheels. We didn't think of that trick until much later, when Spirit was embedded in Troy.
2010-06-10
Spirit Sol 511
"My Ph.D. thesis was on writing a program that would design robots for you," Chris says. "You'd tell it what you want, and it would design a robot to do it. That way, I could spend more time climbing." (He's an avid rock climber.)
"See, I'd have done it the other way around. I'd have designed a robot that would go climbing for me, so I could spend my time designing robots," I say.
"That's the difference between you and me," he deadpans.
It's Friday, and that means it's another three-sol plan. I did essentially all of the IDD work for all three sols yesterday so that I could focus on the drive today, which should make today easy.
Ha.
Since we're on tight sols, our uplink's at 17:40, and for a while there it really doesn't look like we're going to make it. In the end we finish with less than half an hour to spare. We haven't cut things that close in a while. I am awfully glad I did the IDD sequences up front, though; it was only two or three hours' work, but we really didn't have those two or three hours today.
This might be my last sol on Spirit, at least for the next month or so. I'm potentially on jury duty next week, and then I head upstairs to the Land of Opportunity. Which will be different from the last time I was up there. I came downstairs not long before they ran into what they're now calling "Purgatory Ripple," and they extracted themselves already. But the effects of Purgatory Ripple will outlast the actual experience of our being embedded in it. We're going to spend some time investigating the ripple itself, and we're going to drive differently from now on -- probably much less aggressively, but certainly differently.
How differently? That's still an open question. Khaled's working out some of the details, and he comes by to talk to me and Chris about it. They've got a list of proposals going, including my two sneaky tricks. (First trick: periodically stop and try to turn in place. If you can't turn, you're stuck; preclude further driving. Second trick: periodically try to run visodom. If it converges, you must not have moved significantly between the two images, meaning that you're stuck; preclude further driving.) As we talk, I come up with a couple of other creative ideas. One of them is to abuse the IMU: if it doesn't show that your local "down vector" is changing with respect to time, you're probably not climbing ripples and hence may be stuck; preclude driving. Similarly, we could check the suspension limits: if the suspension isn't flexing over time, stop driving.[1]
Nearly all of our ideas are reactive measures: try to detect when you're bogging down and keep from making a bad situation worse. What we'd really like are predictive measures, where we avoid getting stuck in the first place. We're short on those.
But we'd better do at least some driving. Recent email traffic prompted by the 1000-sol news has pointed out that we're close to a couple more milestones: 95,303 images so far (nearly 100,000) and a total distance of 9752.26m driven so far (nearly 10,000). I'm so proud of these rovers.
Footnotes:
[1] The end result was to use VO in a cleverer way: take a picture, move 20cm, and update VO. If VO reports that you've slipped more than 40%, preclude further driving. If it doesn't converge, try again, and preclude driving if you slip more than 40% or if it still doesn't converge.
But that's expensive: it takes at least six minutes, as many as 10, and we have to stop and do it every 20m or so on Opportunity. So, as we speak, there's an MIT grad student working to make an IMU-based classifier do the work for us -- an essentially free check that will notice relatively quickly if we're embedded and stop driving.
[Next post: sol 523 (Opportunity sol 503), June 23. Damn jury service.]
"See, I'd have done it the other way around. I'd have designed a robot that would go climbing for me, so I could spend my time designing robots," I say.
"That's the difference between you and me," he deadpans.
It's Friday, and that means it's another three-sol plan. I did essentially all of the IDD work for all three sols yesterday so that I could focus on the drive today, which should make today easy.
Ha.
Since we're on tight sols, our uplink's at 17:40, and for a while there it really doesn't look like we're going to make it. In the end we finish with less than half an hour to spare. We haven't cut things that close in a while. I am awfully glad I did the IDD sequences up front, though; it was only two or three hours' work, but we really didn't have those two or three hours today.
This might be my last sol on Spirit, at least for the next month or so. I'm potentially on jury duty next week, and then I head upstairs to the Land of Opportunity. Which will be different from the last time I was up there. I came downstairs not long before they ran into what they're now calling "Purgatory Ripple," and they extracted themselves already. But the effects of Purgatory Ripple will outlast the actual experience of our being embedded in it. We're going to spend some time investigating the ripple itself, and we're going to drive differently from now on -- probably much less aggressively, but certainly differently.
How differently? That's still an open question. Khaled's working out some of the details, and he comes by to talk to me and Chris about it. They've got a list of proposals going, including my two sneaky tricks. (First trick: periodically stop and try to turn in place. If you can't turn, you're stuck; preclude further driving. Second trick: periodically try to run visodom. If it converges, you must not have moved significantly between the two images, meaning that you're stuck; preclude further driving.) As we talk, I come up with a couple of other creative ideas. One of them is to abuse the IMU: if it doesn't show that your local "down vector" is changing with respect to time, you're probably not climbing ripples and hence may be stuck; preclude driving. Similarly, we could check the suspension limits: if the suspension isn't flexing over time, stop driving.[1]
Nearly all of our ideas are reactive measures: try to detect when you're bogging down and keep from making a bad situation worse. What we'd really like are predictive measures, where we avoid getting stuck in the first place. We're short on those.
But we'd better do at least some driving. Recent email traffic prompted by the 1000-sol news has pointed out that we're close to a couple more milestones: 95,303 images so far (nearly 100,000) and a total distance of 9752.26m driven so far (nearly 10,000). I'm so proud of these rovers.
Footnotes:
[1] The end result was to use VO in a cleverer way: take a picture, move 20cm, and update VO. If VO reports that you've slipped more than 40%, preclude further driving. If it doesn't converge, try again, and preclude driving if you slip more than 40% or if it still doesn't converge.
But that's expensive: it takes at least six minutes, as many as 10, and we have to stop and do it every 20m or so on Opportunity. So, as we speak, there's an MIT grad student working to make an IMU-based classifier do the work for us -- an essentially free check that will notice relatively quickly if we're embedded and stop driving.
[Next post: sol 523 (Opportunity sol 503), June 23. Damn jury service.]
2010-06-09
Spirit Sol 510
The radiation source in the MB has weakened over time, like a slowly fading light bulb. The MB still works fine, but you have to integrate longer to get the same quality data, much as you have to leave a camera's shutter open longer in dim light.
As a result, thisol we're just continuing to let the MB integrate. The rover will also take pictures and stuff, but there are no RP sequences thisol.
Which doesn't mean I have nothing to do. I go ahead and get the weekend's IDD sequences built (this is even more fun than usual, I discover, when there's no time pressure), and Chris and I -- mostly Chris -- look ahead at the weekend drive. This should make tomorrow much simpler, as we'll be able to focus on planning the drive, and we'll be able to work on it together. And we've already done some of the reconnaissance.
The big deal for the day is that we're planning sol 510 and Opportunity is planning sol 490. Hence, the two rovers have a combined total of ONE THOUSAND SOLS ON MARS.
And counting.
As a result, thisol we're just continuing to let the MB integrate. The rover will also take pictures and stuff, but there are no RP sequences thisol.
Which doesn't mean I have nothing to do. I go ahead and get the weekend's IDD sequences built (this is even more fun than usual, I discover, when there's no time pressure), and Chris and I -- mostly Chris -- look ahead at the weekend drive. This should make tomorrow much simpler, as we'll be able to focus on planning the drive, and we'll be able to work on it together. And we've already done some of the reconnaissance.
The big deal for the day is that we're planning sol 510 and Opportunity is planning sol 490. Hence, the two rovers have a combined total of ONE THOUSAND SOLS ON MARS.
And counting.
2010-06-08
Spirit Sol 509
It's a tight sol, with the uplink due in just a few hours, so we're starting at the ungodly hour of 07:00. I'm also working with John Wright (he's RP-1 and I'm RP-2), and who the heck knows when was the last time that happened.
All that really happened while I was gone was some IDD work on the magnets and solar panels, followed by a drive to Backstay, a pointy, very roughly tetrahedral rock just a few meters away from the end of our last drive. Thisol we're starting an IDD campaign on Backstay, which campaign might or might not include using the RAT.
The one good thing about tight sols is that they're usually relatively simple. Then again, this is because our time is relatively limited, so it doesn't make things that much easier, really. And the uncertainty about the RAT adds some complexity.
The problem is that the question of whether we can safely use Spirit's RAT still hasn't been resolved. Or anyway, not everyone thinks it has, which comes to pretty much the same thing. Alistair, the RAT PUL, says at the SOWG meeting that he got a thumbs-up from Eric Baumgartner, but nobody else seems to have seen the email. Later on, the email shows up, but it seems Eric's approval was based on a false premise -- he thought a test had been run that in fact hadn't. So Ashitey's back in the testbed all day, working feverishly against our tight schedule to get the right test run before we have to uplink.
To cope with this uncertainty, John and I put the RAT-related commands in a separate sequence and work out a way for the backbone sequence to run correctly whether the moves in the RAT subsequence are performed or not. That way we can cleanly drop all the RAT stuff just by not delivering that sequence. The cost is that we have to simulate both cases carefully, more or less doubling our validation work, but we'd pretty much have to do that anyhow; this is the cleanest way.
And then it turns out we might not be able to use the RAT on this target after all: the PUL selected a target using the reachability map, but the reachability map doesn't know that we've restricted the allowed range of motion for the turret joint. And it turns out that the target the PUL selected, which is in a fairly narrow reachability band, causes us to violate that limit. Later he finds another target, though, one that's reachable and satisfies the science needs, so we're back on.
Except that we still haven't heard from Ashitey. Time passes. In the end, we get the go. Backstay is in for a brushing.
All that really happened while I was gone was some IDD work on the magnets and solar panels, followed by a drive to Backstay, a pointy, very roughly tetrahedral rock just a few meters away from the end of our last drive. Thisol we're starting an IDD campaign on Backstay, which campaign might or might not include using the RAT.
The one good thing about tight sols is that they're usually relatively simple. Then again, this is because our time is relatively limited, so it doesn't make things that much easier, really. And the uncertainty about the RAT adds some complexity.
The problem is that the question of whether we can safely use Spirit's RAT still hasn't been resolved. Or anyway, not everyone thinks it has, which comes to pretty much the same thing. Alistair, the RAT PUL, says at the SOWG meeting that he got a thumbs-up from Eric Baumgartner, but nobody else seems to have seen the email. Later on, the email shows up, but it seems Eric's approval was based on a false premise -- he thought a test had been run that in fact hadn't. So Ashitey's back in the testbed all day, working feverishly against our tight schedule to get the right test run before we have to uplink.
To cope with this uncertainty, John and I put the RAT-related commands in a separate sequence and work out a way for the backbone sequence to run correctly whether the moves in the RAT subsequence are performed or not. That way we can cleanly drop all the RAT stuff just by not delivering that sequence. The cost is that we have to simulate both cases carefully, more or less doubling our validation work, but we'd pretty much have to do that anyhow; this is the cleanest way.
And then it turns out we might not be able to use the RAT on this target after all: the PUL selected a target using the reachability map, but the reachability map doesn't know that we've restricted the allowed range of motion for the turret joint. And it turns out that the target the PUL selected, which is in a fairly narrow reachability band, causes us to violate that limit. Later he finds another target, though, one that's reachable and satisfies the science needs, so we're back on.
Except that we still haven't heard from Ashitey. Time passes. In the end, we get the go. Backstay is in for a brushing.
2010-06-02
Spirit Sol 503
When I arrive we're having problems with the OSS, the network filesystem that we use for, well, pretty much everything related to uplink and downlink. One effect of this is that SAP hangs at startup, so the scientists can't do their science planning. Mark Powell, one of the SAP developers, is scrambling for a fix, but he doesn't seem to be having much luck.
Meanwhile, Steve reveals that he and Dave DesMarais have been going over the science results and have concluded that it's time to move on. We've going to finish up IDD work in this location thisol, then drive back to Methuselah and on around to the south side of Husband Hill toward the South Basin. This will take us uphill, though possibly not to the summit. We'll have to see. I'll be back on Opportunity before much longer, anyway, so the struggle to reach the summit will be someone else's for a while.
Mark has just about given up. I decide I should try to help him. I suggest setting up a temporary OSS under somebody's home directory -- all they need is a file structure, and SAP can be given the root of that structure at startup. He looks startled. "Worth a try," he shrugs. A few minutes later he has this working, and Alicia Fallacaro is whipping up a science plan. She beams gratefully at me. "Scott saves the day!" she says.
When I see the IDD target they want to poke at, I almost regret helping them get up and running. The target -- "Nantucket," continuing the Moby Dick theme -- is in a particularly awkward region of the IDD work volume, tight up against the left front quarter of the rover itself. I muse aloud that during the nominal mission this would have freaked me out, which seems to surprise Al Herrera. "I thought you were always cool, calm, and collected," he says. "That's because you only see the back of my head," I tell him. "If there were a mirror set up on the monitor so you could see my face, you'd know better." I'm only half kidding.
Nantucket's a real bitch of a target for a 2x2 MI preceded by a MB touch. There's almost no room to switch from the MB to the MI, for one thing; for another thing, the IDD really wants to collide with itself as we move between the stacks. I find a way to make it work, but the tolerances are so tight that Chris has a heart attack when he looks at it. We show Squyres, who decides to split the MI work -- half on Nantucket, half on a nearby rock we call "Starbuck." We use an MB-touch on Starbuck, since it's far enough away for us to do this comfortably. On Nantucket, we'll just use a less reliable MI poke, which will save us from having to do the tool change. (Poking a rock with the MI is enough like harpooning a whale that we should have named this target after one of the Pequod's harpooners. "Queequeg" was already taken, but we could have named it "Tashtego" or "Daggoo." Ah, well.)
While I've been working on this complex IDD sequence, Chris has been cranking out a comparably difficult drive. I don't like sols like these, just because we don't really get enough time to review each other's work, and that bites us this time. At the CAM, at the very last moment, I realize that we made a mistake: we do some visodom, then some blind driving, then some autonav -- but we forgot to clear the nav map between the blind and autonav segments. This means the rover will have an inaccurate picture of the world around it -- potentially a dangerously inaccurate picture, since it could drive over ground it thought was safe but was in fact host to a deadly obstacle.
So we need to rebundle. All I need to do is insert a single command that tells the rover to clear the nav map before the autonav segment of the drive, but here I make another mistake: Jim Erickson asks whether that's really all we need to do, and I reply with insufficient authority in my voice. He starts to look nervous, and though I realize my mistake, it's too late now. There's no help for it but to call another expert.
Mark Maimone's in China, and all of Jeff Biesiadecki's contact numbers lead to voice mail. Fortunately, Khaled did some testing of exactly such situations in the testbed, and I'm able to reach him. So his vote plus mine makes Jim look happy again -- well, as happy as he ever looks -- and we commit to the change.
[Next post: sol 509, June 8.]
Meanwhile, Steve reveals that he and Dave DesMarais have been going over the science results and have concluded that it's time to move on. We've going to finish up IDD work in this location thisol, then drive back to Methuselah and on around to the south side of Husband Hill toward the South Basin. This will take us uphill, though possibly not to the summit. We'll have to see. I'll be back on Opportunity before much longer, anyway, so the struggle to reach the summit will be someone else's for a while.
Mark has just about given up. I decide I should try to help him. I suggest setting up a temporary OSS under somebody's home directory -- all they need is a file structure, and SAP can be given the root of that structure at startup. He looks startled. "Worth a try," he shrugs. A few minutes later he has this working, and Alicia Fallacaro is whipping up a science plan. She beams gratefully at me. "Scott saves the day!" she says.
When I see the IDD target they want to poke at, I almost regret helping them get up and running. The target -- "Nantucket," continuing the Moby Dick theme -- is in a particularly awkward region of the IDD work volume, tight up against the left front quarter of the rover itself. I muse aloud that during the nominal mission this would have freaked me out, which seems to surprise Al Herrera. "I thought you were always cool, calm, and collected," he says. "That's because you only see the back of my head," I tell him. "If there were a mirror set up on the monitor so you could see my face, you'd know better." I'm only half kidding.
Nantucket's a real bitch of a target for a 2x2 MI preceded by a MB touch. There's almost no room to switch from the MB to the MI, for one thing; for another thing, the IDD really wants to collide with itself as we move between the stacks. I find a way to make it work, but the tolerances are so tight that Chris has a heart attack when he looks at it. We show Squyres, who decides to split the MI work -- half on Nantucket, half on a nearby rock we call "Starbuck." We use an MB-touch on Starbuck, since it's far enough away for us to do this comfortably. On Nantucket, we'll just use a less reliable MI poke, which will save us from having to do the tool change. (Poking a rock with the MI is enough like harpooning a whale that we should have named this target after one of the Pequod's harpooners. "Queequeg" was already taken, but we could have named it "Tashtego" or "Daggoo." Ah, well.)
While I've been working on this complex IDD sequence, Chris has been cranking out a comparably difficult drive. I don't like sols like these, just because we don't really get enough time to review each other's work, and that bites us this time. At the CAM, at the very last moment, I realize that we made a mistake: we do some visodom, then some blind driving, then some autonav -- but we forgot to clear the nav map between the blind and autonav segments. This means the rover will have an inaccurate picture of the world around it -- potentially a dangerously inaccurate picture, since it could drive over ground it thought was safe but was in fact host to a deadly obstacle.
So we need to rebundle. All I need to do is insert a single command that tells the rover to clear the nav map before the autonav segment of the drive, but here I make another mistake: Jim Erickson asks whether that's really all we need to do, and I reply with insufficient authority in my voice. He starts to look nervous, and though I realize my mistake, it's too late now. There's no help for it but to call another expert.
Mark Maimone's in China, and all of Jeff Biesiadecki's contact numbers lead to voice mail. Fortunately, Khaled did some testing of exactly such situations in the testbed, and I'm able to reach him. So his vote plus mine makes Jim look happy again -- well, as happy as he ever looks -- and we commit to the change.
[Next post: sol 509, June 8.]
2010-06-01
Spirit Sol 502
At the SOWG meeting, there's a controversy about the RAT. Fortunately, I've been warned about this in email from Ashitey: he says we're still worried about the badly worn RAT causing the IDD to damage itself through vibration, even when just brushing (everyone agrees grinding is out) and that the Honeybee guys are wrongly suggesting there's no problem.
Which is exactly what happens at the SOWG: when Steve asks about it, the Honeybee guy says there's no problem. "We've seen lots of lost-contact events even back to the beginning of the mission," he argues, suggesting that the events we see now don't imply the arm is actually vibrating dangerously. Steve doesn't just take the RAT guy's word for it, though; he wants more imaging as part of a diagnostic that will eliminate all doubt. Since that's exactly what Ashitey asked me to ask for, I spot my cue and pipe up. "Yeah, Steve, we'd like HAZCAM, PANCAM, and NAVCAM of the RAT brush and teeth before we stow at this location." I've already looked through past sols and found the last time we did this (sol 426), which will make it easy to do again, tomorrow or whenever.
"Sounds like we're pretty much on the same page, then," Steve replies.
The RAT guy tries to tell Steve that the instrument is safe to use, but Steve says, "I'm worried about the whole rover, not just the RAT." This is why Steve is so awesome[1]: lots of science team leaders would have thought it's their job to stand up for the science side, no matter what (or so I'm told). Not many would realize that the science team is best served by preserving the goddamned vehicle. Or so I'm told. Anyway, we're damned fortunate that Steve's not of that stripe.
Chris had to take his dog to the vet this morning (she's got bad allergies), so I've swapped with him: reversing the official plan, I'm RP-1, and he'll be in later as RP-2. But it hardly matters; the two-sol plan is so simple that I'm already done when he arrives. Or rather, I should have been done. At the activity plan approval meeting, it turns out I've sequenced the wrong thing, moving the IDD to the wrong target at one point. This is not really my fault, as the official plan had it wrong, and that's what I was following. But I should have seen it coming because of the Curse of the Premature Animation. (Your line: "I used to have a problem with premature animation, but now I just think about baseball.")
That curse is this. Whenever I'm so confident that I've finished early that I go ahead and build the uplink report and/or the official MPEG animation before the walkthrough -- much less before the APAM, as I did today -- something will have to change in the sequence that invalidates that work. This happens every time, and today was no exception.
Fortunately, it's easy to fix: almost before they can finish describing the problem, I've already fixed the sequence and have redone the on-the-fly animation. But you'd think I'd know by now that I'm cursed. Cursed. Cursed!
Maybe next time I can avert the curse by thinking about baseball. It's worth a try.
Footnotes:
[1] I mean, of course, that it's one of the reasons. One of many, many, many.
Which is exactly what happens at the SOWG: when Steve asks about it, the Honeybee guy says there's no problem. "We've seen lots of lost-contact events even back to the beginning of the mission," he argues, suggesting that the events we see now don't imply the arm is actually vibrating dangerously. Steve doesn't just take the RAT guy's word for it, though; he wants more imaging as part of a diagnostic that will eliminate all doubt. Since that's exactly what Ashitey asked me to ask for, I spot my cue and pipe up. "Yeah, Steve, we'd like HAZCAM, PANCAM, and NAVCAM of the RAT brush and teeth before we stow at this location." I've already looked through past sols and found the last time we did this (sol 426), which will make it easy to do again, tomorrow or whenever.
"Sounds like we're pretty much on the same page, then," Steve replies.
The RAT guy tries to tell Steve that the instrument is safe to use, but Steve says, "I'm worried about the whole rover, not just the RAT." This is why Steve is so awesome[1]: lots of science team leaders would have thought it's their job to stand up for the science side, no matter what (or so I'm told). Not many would realize that the science team is best served by preserving the goddamned vehicle. Or so I'm told. Anyway, we're damned fortunate that Steve's not of that stripe.
Chris had to take his dog to the vet this morning (she's got bad allergies), so I've swapped with him: reversing the official plan, I'm RP-1, and he'll be in later as RP-2. But it hardly matters; the two-sol plan is so simple that I'm already done when he arrives. Or rather, I should have been done. At the activity plan approval meeting, it turns out I've sequenced the wrong thing, moving the IDD to the wrong target at one point. This is not really my fault, as the official plan had it wrong, and that's what I was following. But I should have seen it coming because of the Curse of the Premature Animation. (Your line: "I used to have a problem with premature animation, but now I just think about baseball.")
That curse is this. Whenever I'm so confident that I've finished early that I go ahead and build the uplink report and/or the official MPEG animation before the walkthrough -- much less before the APAM, as I did today -- something will have to change in the sequence that invalidates that work. This happens every time, and today was no exception.
Fortunately, it's easy to fix: almost before they can finish describing the problem, I've already fixed the sequence and have redone the on-the-fly animation. But you'd think I'd know by now that I'm cursed. Cursed. Cursed!
Maybe next time I can avert the curse by thinking about baseball. It's worth a try.
Footnotes:
[1] I mean, of course, that it's one of the reasons. One of many, many, many.
Subscribe to:
Posts (Atom)
Posts
