Yikes. Cooper had an emergency appendectomy this weekend. He's consequently unable to come to work, so I volunteer to fill in for him. This puts me back on Opportunity for the first time in quite a while -- today, and for the next couple of days.
The Opportunity team has just about decided to drive out of Purgatory. They haven't even tried moving yet, but testbed testing has been encouraging. They've been able to simulate most of the important conditions of the real vehicle, and they keep managing to drive it out.
Indeed, we're back in the testbed again today. Jeng and I whip up a candidate sequence, whose only mobility command is to steer the wheels in preparation for the first stage of the drive. This single command is surrounded (fore and aft) by a zillion imaging commands. We're taking this very, very slowly.
Then we walk down to the testbed, where we meet up with Frank and Mark. We won't be driving the testbed rover in the special soil they cooked up to simulate Opportunity's current site, due to medical concerns: the fine particulates are dangerous to breathe for long periods, and we haven't been formally trained on the respirators. (I don't honestly see what's to know. It's a face mask. You strap it on. But their intent is to keep us healthy, and I can't find too much fault with that. They're insane, but they mean well.[1])
As usual, it just takes forever to get anything done. A couple of hours go by, and we haven't run the sequence yet. Both Frank's and my ESD certifications have lapsed, so we can't man the kill switch anyway, even once the rover gets going. We mostly stand there chit-chatting -- about the rovers, about LISP, about camera phones, about whatever. An employee I've never met brings a couple of guests by, and I go over and talk with them for a while about Opportunity's current status and what we're working on. "That's the second time I've seen you do that," Frank says after they depart. "You're like a rover ambassador."[2]
Then, just before things really get going, I have to leave. I'm working the Open House this weekend, and they have an orientation meeting this afternoon. Which is mostly useless, but one of the speakers makes a good point: "Not everyone has the privilege of working somewhere where everyone wants to go there and see what they do." True, and that's why I'm doing the Open House: for the sake of all those poor slobs who want to be me.
But seriously, I am impressed with the scale of the effort. As another speaker puts it, "This place was not set up to be Disneyland." But we turn it into Disneyland for a couple of days a year, and that's pretty darn cool, really.[3] I'm only too happy to be a part of it again. (Also, she mentions offhandedly that they used to have an open house on the last Sunday of every month! I can't imagine it was anything like the scale of the present undertaking, but even so, that must have been a very different time.)
I make it back to the testbed just after the sequence finishes running, which pretty much officially means I wasted my entire day. We then have a long conference call with the rest of the uplink team, to set up for tomorrow. They'll be doing additional testing on this sequence tomorrow in parallel with the uplink process, but it's clear that at least some people think we're jumping through a ridiculous number of hoops: as Jeff Biesiadecki puts it, "The rover would have to catch on fire for us not to send this sequence."[4]
So I guess we're going ahead with that, then.
I go back to my office and review the sequence once again, and I do catch a couple of minor errors -- one of which would have prevented Opportunity from moving at all. So I guess it wasn't a complete waste after all.
Footnotes:
[1] I eventually had the face-mask training, as part of the Spirit extrication effort. Turns out it's a face mask, and you strap it on.
OK, so there's a little more to it than that. But not much.
[2] To this day, that was the nicest way anyone's ever called me an attention whore.
[3] This is probably why I started calling it "Deep Space Disneyland."
[4] What sequence do we send if that happens?!
2010-05-10
2010-05-06
Spirit Sol 477
Yestersol's drive was too complex. I was thinking about it last night, and my hopes sank lower and lower. We should have broken it into multiple sols. We shouldn't have tried to thread the needle at the end. I should have looked more carefully for obstacles.
Of course, it went splendidly. (If I learned anything from experience, I'd have been expecting this. But I never do.) Dan Moyers tells me the good news as I walk into the SOWG meeting. "Excellent!" I reply. "I was all ready to blame Chris if it went wrong."
As has become our Friday custom, we're planning the rover's activities for the whole weekend. There's lots of IDD work, and lots of remote sensing. (In keeping with the nautical theme of Jib Sheet, most of the remote-sensing targets get names such as "gunwale," "keel," etc. They're taking some pictures of the tracks as well, and Charles Budney suggests the name "wake" for that target.) We have a limited time -- not only are we supposed to stick to a 10-hour day anyway, but uplink's at 19:00, which pretty much enforces that limit thisol. As a result of this tight schedule, Dan rejects a request from the atmospheres team -- it'll just be too hard to fit it in in the time we've got, given the complexity of the overall plan.
As you can imagine, the atmospheres team isn't too happy about that, and they push back hard. But Dan sticks to his guns, and Emily backs him up. "The resource we need to optimize here is team planning time," she says. One of the scientists turns to me off mike and says, "So the engineering team is no longer competent to --" but he's interrupted, and he never finishes the thought. Just as well.
In the room behind us, the Opportunity team is having its own debate, joined by some higher management. Opportunity's been stuck on a dune for the last few weeks, buried up to the tops of the wheels -- almost up to the bottom of the WEB -- and management's been paranoid about allowing us to drive it out. I notice that Jay Torres was sitting by a window that afforded a view into that room, so I ask him if he saw any of the presentation. He shakes his head no, as Chris chimes in, "The content is, we can be a lander or we can drive backward a couple of meters."
"Which we knew three weeks ago," I say.
"Yeah," he replies, "but now our ass is covered."
Yeah, that probably sums it up.
Back downstairs, Emily asks our new TDL what he thought of the acrimony in the SOWG meeting. "I can see why they keep the scientists and engineers in physically separate locations," he says diplomatically.
"Well, it used to be better," Emily says. "But the vehicles have gotten harder to operate. We used to have no power, so we had no concerns about thermal issues or planning time -- there was no power to do anything. But now we're power-rich again, and we'd forgotten there were other constraints."
This is true, but I think it's incomplete. Another part of the problem is that the scientists just aren't here any more. They're voices on a phone, and that makes it easier for both sides to degenerate into an us-versus-them mentality. Worse, they simply don't get to see the whole process any more -- most of them sign off when they're done with their part of the planning, but we keep working for hours. If they can't see what effect their decisions have on the sequencing team, how can they hope to make wise decisions about what can and can't get done? I point this out to Emily, and semi-jokingly suggest a solution: "We should get Squyres to require all of the science team to stay online until we're done. You want us to do some observation that's going to take us two extra hours to fit into the plan? Fine, but you stay here until we leave."
I think Emily likes the idea, but I doubt she'll pitch it to Steve. I might. If the problem gets worse, I might have to.
But at least the rover planners have an easy sol. It's just some light IDD work. Chris is done with the sequences maybe half an hour after the SOWG meeting ends. We hand off and I make some minor changes, but nothing particularly urgent.
Since we're so far ahead of the game, I consider going ahead with the time-consuming process of building an MPEG movie of the planned IDD work for the weekend, which I customarily attach to the sol's uplink report. But when we're this far ahead, something always happens to change the plan, and today's no exception. When we show the animation at the activity plan approval meeting, we find out that the science team changed its mind about which IDD targets they wanted us to use, and somehow the word didn't make it to us. No big deal; that's why we do that. I have it fixed within 15 minutes or so.
But I was right not to go ahead and build the movie early. Huh. Seems like sometimes I do learn from experience, after all.
[Next post: sol 480 (Opportunity sol 460), May 10.]
Of course, it went splendidly. (If I learned anything from experience, I'd have been expecting this. But I never do.) Dan Moyers tells me the good news as I walk into the SOWG meeting. "Excellent!" I reply. "I was all ready to blame Chris if it went wrong."
As has become our Friday custom, we're planning the rover's activities for the whole weekend. There's lots of IDD work, and lots of remote sensing. (In keeping with the nautical theme of Jib Sheet, most of the remote-sensing targets get names such as "gunwale," "keel," etc. They're taking some pictures of the tracks as well, and Charles Budney suggests the name "wake" for that target.) We have a limited time -- not only are we supposed to stick to a 10-hour day anyway, but uplink's at 19:00, which pretty much enforces that limit thisol. As a result of this tight schedule, Dan rejects a request from the atmospheres team -- it'll just be too hard to fit it in in the time we've got, given the complexity of the overall plan.
As you can imagine, the atmospheres team isn't too happy about that, and they push back hard. But Dan sticks to his guns, and Emily backs him up. "The resource we need to optimize here is team planning time," she says. One of the scientists turns to me off mike and says, "So the engineering team is no longer competent to --" but he's interrupted, and he never finishes the thought. Just as well.
In the room behind us, the Opportunity team is having its own debate, joined by some higher management. Opportunity's been stuck on a dune for the last few weeks, buried up to the tops of the wheels -- almost up to the bottom of the WEB -- and management's been paranoid about allowing us to drive it out. I notice that Jay Torres was sitting by a window that afforded a view into that room, so I ask him if he saw any of the presentation. He shakes his head no, as Chris chimes in, "The content is, we can be a lander or we can drive backward a couple of meters."
"Which we knew three weeks ago," I say.
"Yeah," he replies, "but now our ass is covered."
Yeah, that probably sums it up.
Back downstairs, Emily asks our new TDL what he thought of the acrimony in the SOWG meeting. "I can see why they keep the scientists and engineers in physically separate locations," he says diplomatically.
"Well, it used to be better," Emily says. "But the vehicles have gotten harder to operate. We used to have no power, so we had no concerns about thermal issues or planning time -- there was no power to do anything. But now we're power-rich again, and we'd forgotten there were other constraints."
This is true, but I think it's incomplete. Another part of the problem is that the scientists just aren't here any more. They're voices on a phone, and that makes it easier for both sides to degenerate into an us-versus-them mentality. Worse, they simply don't get to see the whole process any more -- most of them sign off when they're done with their part of the planning, but we keep working for hours. If they can't see what effect their decisions have on the sequencing team, how can they hope to make wise decisions about what can and can't get done? I point this out to Emily, and semi-jokingly suggest a solution: "We should get Squyres to require all of the science team to stay online until we're done. You want us to do some observation that's going to take us two extra hours to fit into the plan? Fine, but you stay here until we leave."
I think Emily likes the idea, but I doubt she'll pitch it to Steve. I might. If the problem gets worse, I might have to.
But at least the rover planners have an easy sol. It's just some light IDD work. Chris is done with the sequences maybe half an hour after the SOWG meeting ends. We hand off and I make some minor changes, but nothing particularly urgent.
Since we're so far ahead of the game, I consider going ahead with the time-consuming process of building an MPEG movie of the planned IDD work for the weekend, which I customarily attach to the sol's uplink report. But when we're this far ahead, something always happens to change the plan, and today's no exception. When we show the animation at the activity plan approval meeting, we find out that the science team changed its mind about which IDD targets they wanted us to use, and somehow the word didn't make it to us. No big deal; that's why we do that. I have it fixed within 15 minutes or so.
But I was right not to go ahead and build the movie early. Huh. Seems like sometimes I do learn from experience, after all.
[Next post: sol 480 (Opportunity sol 460), May 10.]
2010-05-05
Spirit Sol 476
As we work our way back down the ridge to Larry's Lookout, our next stop is Jib Sheet. Thisol we're sequencing a drive to take us most of the way there; wherever we end up, we'll IDD over the weekend, and then complete the Jib Sheet approach early next week.
It's a complex drive. To get there, we'll need to drive in a big "U." It's a total distance of about 25m, a much longer drive than we're normally able to accomplish in this challenging terrain. In our favor, the terrain does flatten out considerably as we go, which will make the vehicle's performance more predictable. But the length of the drive and the nature of the overall terrain will make this a long and difficult plan.
We're not sure for a while exactly where we want the rover to end up, or how it will get there, or even whether we'll drive forward or backward. Dan Moyers sees this and decides to have a little fun with TUL-in-training Rich. "We need the final heading so we can do comm predicts," he says. "Ask the rover planners what our final heading will be."
But Rich is on the ball. "Dan wants to know our final heading."
"Sixty," Chris says without turning around. "Plus or minus 180."
It's mostly Chris's show today. I do the stow and work out the best plan for the first couple of meters of the drive, just enough to get us on the road. Chris does most of the work of planning out the route, though I throw in my opinion now and then. For instance, he's hoping to take a shortcut between two larger rocks, letting autonav find a safe path. I do some analysis to work out that the rover won't be able to find a safe path without loosening the autonav settings, and neither one of us wants to do that. So we take the long path.
Our accidental IDD animation keeps turning up new surprises. I composed and posted the full two-sol version of the animation the other day, and Brenda Franklin's been looking over it carefully. Her scrutiny was rewarded with a neat surprise: in the second-to-last frame on the second sol, a huge gust of wind -- maybe a dust devil -- kicks up a huge cloud of dust on a far hill. "Definitely lemonade with whiskey," she says.
(I've had trouble with animated GIFs and Blogger. You should be able to see the animation at twitpic.)
Courtesy NASA/JPL-Caltech. The dust devil is in the upper-left corner, near the horizon. Subtle, but it's there.
It's a complex drive. To get there, we'll need to drive in a big "U." It's a total distance of about 25m, a much longer drive than we're normally able to accomplish in this challenging terrain. In our favor, the terrain does flatten out considerably as we go, which will make the vehicle's performance more predictable. But the length of the drive and the nature of the overall terrain will make this a long and difficult plan.
We're not sure for a while exactly where we want the rover to end up, or how it will get there, or even whether we'll drive forward or backward. Dan Moyers sees this and decides to have a little fun with TUL-in-training Rich. "We need the final heading so we can do comm predicts," he says. "Ask the rover planners what our final heading will be."
But Rich is on the ball. "Dan wants to know our final heading."
"Sixty," Chris says without turning around. "Plus or minus 180."
It's mostly Chris's show today. I do the stow and work out the best plan for the first couple of meters of the drive, just enough to get us on the road. Chris does most of the work of planning out the route, though I throw in my opinion now and then. For instance, he's hoping to take a shortcut between two larger rocks, letting autonav find a safe path. I do some analysis to work out that the rover won't be able to find a safe path without loosening the autonav settings, and neither one of us wants to do that. So we take the long path.
Our accidental IDD animation keeps turning up new surprises. I composed and posted the full two-sol version of the animation the other day, and Brenda Franklin's been looking over it carefully. Her scrutiny was rewarded with a neat surprise: in the second-to-last frame on the second sol, a huge gust of wind -- maybe a dust devil -- kicks up a huge cloud of dust on a far hill. "Definitely lemonade with whiskey," she says.
(I've had trouble with animated GIFs and Blogger. You should be able to see the animation at twitpic.)
Courtesy NASA/JPL-Caltech. The dust devil is in the upper-left corner, near the horizon. Subtle, but it's there.
2010-04-30
Spirit Sol 471
It's a commonplace among engineers that you do your best work in the shower. Your mind and body are relaxed, and you see solutions that you just don't see under other circumstances.
That's not to say you always like the answers you get. While I'm getting ready in the morning, I realize we completely screwed up the huge MI mosaic of Keystone. It's the largest MI mosaic of a single object ever; it took about three hours for the IDD to do the work; and I single-handedly managed to get nearly all of the images out of focus. I've got my eyes closed, I'm putting on the shampoo, and suddenly I visualize the relationship between the rock face and the motion of the IDD as it moves the MI across the surface, and it becomes blindingly clear that the angle is completely wrong for the majority of the surface.
See, the MI has a very shallow field of view. You can think of it as being like a small square sheet of paper, held about 4.6cm (call it two inches) in front of the lens. The "piece of paper" is maybe 3cm on a side and 3mm thick -- maybe instead of a single sheet of paper, you should think of it as a pad of just a few Post-It notes. Anything not within that volume will be either invisible or unfocused. If you look at a surface at too oblique an angle, most of the "notepad" will be either above or below that surface. A small amount of the image will be in focus, but most won't.
We held the paper at the wrong angle.
Some small fraction of the MIs will be in focus, but most won't. We basically just blew two or three sols, four or five hours of IDD actuator life, and hundreds of megabits of downlink. And it's all my fault.
I get to JPL a little early, anxious to know how bad the situation is. Nobody says anything in particular to me, which is usually a bad sign. I fire up mphoto (the image browser application I wrote during solar conjunction), bring up the MI thumbnails, and click on one of them. It happens to be one of the in-focus images, and looks pretty good. Then I click on another one. It's in focus, too. I click on a few more. They're all in focus.
Um. Evidently, I was mistaken. Happily, it would seem that I do better IDD work in the sequencing room than in the shower after all.
And I realize I'm now the holder of another record (not that any of us holds these individually, of course; it's always a team effort). Not only am I the single-sol distance-record holder on both sides of Mars, I've now also done the most MIs of a single object ever.
And they're even in focus!
Sadly, the MI sky flats don't work out again -- there's no time to do them in the weekend plan. When they officially sink the idea, I hit the mike and say, "Screw you guys, I'm doing it anyway!" Which provokes a laugh, and Ken says, "Go, Scott!" But he's philosophical about the lost chance: "I waited 400 sols, I can wait a little longer."
I did make one mistake in the MI sequence. We were performing the MI mosaic in six columns of four stacks apiece, covering the broad rectangular face of Keystone. Now, normally, we take an image of the IDD each time we place an instrument, taking higher-quality images when we place the MI to give us data points for calibrating the HAZCAMs. (When the MI is down, the MB is up, right where the cameras can see it, and its shiny contact ring is easy to pick out in the images.) But we didn't want to spend time or downlink volume taking those images for every stack, since there were so many. Instead, we want to image the IDD only once per column.
So I came up with a clever way to do that. When we're driving, we sometimes use the OK-to-IDD flag as a simple marker -- for instance, if we get to a certain point in the drive, we drop the OK-to-IDD flag. Then later on, the sequence checks if it's OK to IDD; if not, we know we got that far in the sequence and can do something special for that case. So I figured, why not turn that around? In the backbone sequence, we'll drop the OK-to-drive flag as we start each column, then make four calls to the helper sequence (each call to the helper sequence performs one stack and moves to set up for the next stack). The helper sequence will check whether it's OK to drive and, if not, will take the IDD image and then re-raise the flag so that the next call to the helper sequence won't take the image. That way, we'll take the image for only the first stack in each quartet.
Like so many solutions, this one is simple, elegant, and wrong. What I stupidly forgot was that when the IDD is unstowed, we're never OK to drive. So instead of imaging the IDD placement on every fourth call to the helper, we imaged it on every call -- sixteen IDD placement images, not four. And we reused that code when we completed the mosaic, so we're going to repeat the mistake. Oops.
So we got a whole bunch of images of the IDD moving across the surface. High-quality images. Hey ... on a whim, I make a simple animation and project it on the shared screen in the sequencing room. It's awesome -- not only do you see the IDD doing its thing, you even see the rover's own shadow marching away as the sun passes overhead. Steve sees the movie and loves it. "Maybe we should do more things like this," he remarks.
"Yeah, if only all our mistakes turned out this well," I reply.
Brenda Franklin also pipes up. "When life gives you lemons, make lemonade. This is lemonade. This is lemonade with whiskey!"
"Hey, you know what you should do?" Steve says. "You should send this to Guy Webster!" (That's JPL's main press dude for MER.) "When we release the MI mosaic, we can release this movie to go along with it."
Which is a great idea, and displays Steve's flair for outreach. So I put together a version of the movie for release -- actually, two versions, one from each eye -- and email the press people about it. We'll see what happens.[1]
I finish up the sequencing and hand over to Chris, then hang out to work on some other stuff. A little later, Steve comes on the teleconference line again. This time he's raving about the MIs. You can practically hear him jumping up and down with excitement. "Those MIs just look great!" he exclaims. "This is gonna be one of the premier data sets of the mission. I just talked to Ken, and they're going crazy in Flagstaff!" (This is not some strange expression; Flagstaff -- the University of Arizona -- is where Ken and his team are.)
I couldn't be happier, especially since this is the exact diametrical polar opposite of what I expected to hear when I came in today. I tell Steve I'll pass his message along to Chris and Ashitey, and I do.
Steve's got one more thing to mention. He was unable to cover the SOWG meeting this morning because he was teaching a class -- "Astrophysics 102 -- 300 students. At the start of every class, I give them a five-minute update on what's going on on Mars. It's the only part they pay attention for."
[Next post: sol 476, May 5.]
Courtesy NASA/JPL-Caltech. An accidental animation of the first sol's work on Keystone: best mistake ever. Follow the link for a full-sized version.
Courtesy NASA/JPL-Caltech. The mosaic of MI images on Keystone. Again, follow the link to get a full-sized version.
[1] As always, Steve's instincts were spot on. They loved it. They released the animation at the same time as the Keystone press release, and it's shown up more places than I can count since then. Best mistake ever.
That's not to say you always like the answers you get. While I'm getting ready in the morning, I realize we completely screwed up the huge MI mosaic of Keystone. It's the largest MI mosaic of a single object ever; it took about three hours for the IDD to do the work; and I single-handedly managed to get nearly all of the images out of focus. I've got my eyes closed, I'm putting on the shampoo, and suddenly I visualize the relationship between the rock face and the motion of the IDD as it moves the MI across the surface, and it becomes blindingly clear that the angle is completely wrong for the majority of the surface.
See, the MI has a very shallow field of view. You can think of it as being like a small square sheet of paper, held about 4.6cm (call it two inches) in front of the lens. The "piece of paper" is maybe 3cm on a side and 3mm thick -- maybe instead of a single sheet of paper, you should think of it as a pad of just a few Post-It notes. Anything not within that volume will be either invisible or unfocused. If you look at a surface at too oblique an angle, most of the "notepad" will be either above or below that surface. A small amount of the image will be in focus, but most won't.
We held the paper at the wrong angle.
Some small fraction of the MIs will be in focus, but most won't. We basically just blew two or three sols, four or five hours of IDD actuator life, and hundreds of megabits of downlink. And it's all my fault.
I get to JPL a little early, anxious to know how bad the situation is. Nobody says anything in particular to me, which is usually a bad sign. I fire up mphoto (the image browser application I wrote during solar conjunction), bring up the MI thumbnails, and click on one of them. It happens to be one of the in-focus images, and looks pretty good. Then I click on another one. It's in focus, too. I click on a few more. They're all in focus.
Um. Evidently, I was mistaken. Happily, it would seem that I do better IDD work in the sequencing room than in the shower after all.
And I realize I'm now the holder of another record (not that any of us holds these individually, of course; it's always a team effort). Not only am I the single-sol distance-record holder on both sides of Mars, I've now also done the most MIs of a single object ever.
And they're even in focus!
Sadly, the MI sky flats don't work out again -- there's no time to do them in the weekend plan. When they officially sink the idea, I hit the mike and say, "Screw you guys, I'm doing it anyway!" Which provokes a laugh, and Ken says, "Go, Scott!" But he's philosophical about the lost chance: "I waited 400 sols, I can wait a little longer."
I did make one mistake in the MI sequence. We were performing the MI mosaic in six columns of four stacks apiece, covering the broad rectangular face of Keystone. Now, normally, we take an image of the IDD each time we place an instrument, taking higher-quality images when we place the MI to give us data points for calibrating the HAZCAMs. (When the MI is down, the MB is up, right where the cameras can see it, and its shiny contact ring is easy to pick out in the images.) But we didn't want to spend time or downlink volume taking those images for every stack, since there were so many. Instead, we want to image the IDD only once per column.
So I came up with a clever way to do that. When we're driving, we sometimes use the OK-to-IDD flag as a simple marker -- for instance, if we get to a certain point in the drive, we drop the OK-to-IDD flag. Then later on, the sequence checks if it's OK to IDD; if not, we know we got that far in the sequence and can do something special for that case. So I figured, why not turn that around? In the backbone sequence, we'll drop the OK-to-drive flag as we start each column, then make four calls to the helper sequence (each call to the helper sequence performs one stack and moves to set up for the next stack). The helper sequence will check whether it's OK to drive and, if not, will take the IDD image and then re-raise the flag so that the next call to the helper sequence won't take the image. That way, we'll take the image for only the first stack in each quartet.
Like so many solutions, this one is simple, elegant, and wrong. What I stupidly forgot was that when the IDD is unstowed, we're never OK to drive. So instead of imaging the IDD placement on every fourth call to the helper, we imaged it on every call -- sixteen IDD placement images, not four. And we reused that code when we completed the mosaic, so we're going to repeat the mistake. Oops.
So we got a whole bunch of images of the IDD moving across the surface. High-quality images. Hey ... on a whim, I make a simple animation and project it on the shared screen in the sequencing room. It's awesome -- not only do you see the IDD doing its thing, you even see the rover's own shadow marching away as the sun passes overhead. Steve sees the movie and loves it. "Maybe we should do more things like this," he remarks.
"Yeah, if only all our mistakes turned out this well," I reply.
Brenda Franklin also pipes up. "When life gives you lemons, make lemonade. This is lemonade. This is lemonade with whiskey!"
"Hey, you know what you should do?" Steve says. "You should send this to Guy Webster!" (That's JPL's main press dude for MER.) "When we release the MI mosaic, we can release this movie to go along with it."
Which is a great idea, and displays Steve's flair for outreach. So I put together a version of the movie for release -- actually, two versions, one from each eye -- and email the press people about it. We'll see what happens.[1]
I finish up the sequencing and hand over to Chris, then hang out to work on some other stuff. A little later, Steve comes on the teleconference line again. This time he's raving about the MIs. You can practically hear him jumping up and down with excitement. "Those MIs just look great!" he exclaims. "This is gonna be one of the premier data sets of the mission. I just talked to Ken, and they're going crazy in Flagstaff!" (This is not some strange expression; Flagstaff -- the University of Arizona -- is where Ken and his team are.)
I couldn't be happier, especially since this is the exact diametrical polar opposite of what I expected to hear when I came in today. I tell Steve I'll pass his message along to Chris and Ashitey, and I do.
Steve's got one more thing to mention. He was unable to cover the SOWG meeting this morning because he was teaching a class -- "Astrophysics 102 -- 300 students. At the start of every class, I give them a five-minute update on what's going on on Mars. It's the only part they pay attention for."
[Next post: sol 476, May 5.]
Courtesy NASA/JPL-Caltech. An accidental animation of the first sol's work on Keystone: best mistake ever. Follow the link for a full-sized version.
Courtesy NASA/JPL-Caltech. The mosaic of MI images on Keystone. Again, follow the link to get a full-sized version.
[1] As always, Steve's instincts were spot on. They loved it. They released the animation at the same time as the Keystone press release, and it's shown up more places than I can count since then. Best mistake ever.
2010-04-29
Spirit Sol 470
We're in that weird phase where we're planning in the blind. The sequences we wrote yestersol haven't even been uplinked to the rover yet, much less executed, so clearly we haven't gotten the downlink yet. And today's are going to look similar to yesterday's, so we could blow two whole days if we did something wrong yesterday. So I ask Chris -- RP-2 today -- to look them over. In theory, if he finds a problem, we could fix it and uplink the result instead of what we planned yesterday. Not that our mission manager would let us get away with that. But Chris doesn't find anything anyway, so it's moot. At least I'll be able to sleep a little better.
In theory.
Yesterday we did a 4x4 subset of the total 4x6 mosaic. Today we pick up where we left off, doing the remaining 2x6 subset of the mosaic. Then we brush off a spot with the RAT and place the APXS there. Overall, not such a bad day -- yesterday was harder.
The hard part is to fit in the MI sky flats. This is where we take microscopic images of the sky. Yes, you read that right. It's a calibration thing. The problem is that there are restrictions on how we take the sky flats, and it's awkward to satisfy them all at once. First, the MI must be aimed at least 90 degrees away from the sun. Second, the sky-flat images must be taken late in the day, after about 14:00. Third, the MI needs an unobstructed view of the sky, so no part of the rover body or the terrain can be in the way. Fourth, it must be aimed about 40 degrees above the horizon.
The first and second constraints combine to force the MI to aim to the rover's right. So now the problem is to find a solution where the MI is aimed to the right, 40 degrees above the horizon, without any terrain or rover body in view. There doesn't seem to be any way to do this. In one wrist configuration, the rover deck blocks our view when the MI is close to the WEB and the IDD self-collides when the MI is pushed farther from the WEB. In the other wrist configuration, we can choose between having the rover deck in the way or running into a hardstop -- with the arm otherwise in the needed pose, the 40-degree elevation is in a "blind spot" where the turret can't aim the MI.
This means we can't do the sky flats, which makes me feel very bad. The MI PUL, Ken, is such a nice guy, and it's been literally 400 sols since he's had this calibration done. Then at last they make room for it in the plan, and I can't find a solution for him. It's not my fault that the solution's not there to be found, but I still feel bad about it.
But otherwise, the sol goes well. We're practically drunk on power -- 937 Watt-hours thisol, more than we had when we landed. Thanks, dust devils! And the sequencing proceeds without any major hiccups.
At the end of the day, Steve fills us in on the science team's current thinking about where we want to go from here. First, images taken from the other side -- that is, the current side -- of Larry's Lookout (a landmark partway up Husband Hill, which we passed on the way up) have apparently made the scientists even more interested in it, so we're going to head back downhill to it. Downhill is the wrong direction, I want to say (damn, I want to see the view from that summit!) -- but I'm just the chauffeur, so I wisely keep my mouth shut. The right direction is wherever the science is, and if that's downhill from here, then so be it.
On the way back down to Larry's Lookout, we'll check out another rock near this one, then this area called Jib Sheet. We probably won't get quite close enough to Larry's Lookout to touch it, since there's a steep pile of soft sand in the way, but we'll take highly detailed pictures from as close as we can. (Like Burns's Cliff, Steve points out.) Then we turn back around and start slogging up the hill again.
"Sorry about that," Steve says. "This is the best exposed geology we'll see this side of Husband Hill, so we gotta do it."
That's okay, it really is. The big disappointment from today comes from an unexpected direction, when Chris finds a way we could have done the MI sky flats after all. It's a configuration I thought I'd tried, but obviously not. So now I feel twice as bad about not finding a way to do it, because there was a way to be found. I just didn't find it.
(And writing this, it occurs to me there was always a way anyhow. The IDD's elbow joint is fully double-jointed; we could simply have swung the turret up so that it was above the elbow rather than below it. Then aiming the MI for the sky flats would have been trivial -- the rover deck is never in the way in that pose. Oh, I am dumb sometimes, I truly am.)
I swallow my pride and send Ken and Steve email about this. We couldn't do the sky flats today, but we'll still be here over the weekend. Maybe we'll squeeze it in tomorrow.
In theory.
Yesterday we did a 4x4 subset of the total 4x6 mosaic. Today we pick up where we left off, doing the remaining 2x6 subset of the mosaic. Then we brush off a spot with the RAT and place the APXS there. Overall, not such a bad day -- yesterday was harder.
The hard part is to fit in the MI sky flats. This is where we take microscopic images of the sky. Yes, you read that right. It's a calibration thing. The problem is that there are restrictions on how we take the sky flats, and it's awkward to satisfy them all at once. First, the MI must be aimed at least 90 degrees away from the sun. Second, the sky-flat images must be taken late in the day, after about 14:00. Third, the MI needs an unobstructed view of the sky, so no part of the rover body or the terrain can be in the way. Fourth, it must be aimed about 40 degrees above the horizon.
The first and second constraints combine to force the MI to aim to the rover's right. So now the problem is to find a solution where the MI is aimed to the right, 40 degrees above the horizon, without any terrain or rover body in view. There doesn't seem to be any way to do this. In one wrist configuration, the rover deck blocks our view when the MI is close to the WEB and the IDD self-collides when the MI is pushed farther from the WEB. In the other wrist configuration, we can choose between having the rover deck in the way or running into a hardstop -- with the arm otherwise in the needed pose, the 40-degree elevation is in a "blind spot" where the turret can't aim the MI.
This means we can't do the sky flats, which makes me feel very bad. The MI PUL, Ken, is such a nice guy, and it's been literally 400 sols since he's had this calibration done. Then at last they make room for it in the plan, and I can't find a solution for him. It's not my fault that the solution's not there to be found, but I still feel bad about it.
But otherwise, the sol goes well. We're practically drunk on power -- 937 Watt-hours thisol, more than we had when we landed. Thanks, dust devils! And the sequencing proceeds without any major hiccups.
At the end of the day, Steve fills us in on the science team's current thinking about where we want to go from here. First, images taken from the other side -- that is, the current side -- of Larry's Lookout (a landmark partway up Husband Hill, which we passed on the way up) have apparently made the scientists even more interested in it, so we're going to head back downhill to it. Downhill is the wrong direction, I want to say (damn, I want to see the view from that summit!) -- but I'm just the chauffeur, so I wisely keep my mouth shut. The right direction is wherever the science is, and if that's downhill from here, then so be it.
On the way back down to Larry's Lookout, we'll check out another rock near this one, then this area called Jib Sheet. We probably won't get quite close enough to Larry's Lookout to touch it, since there's a steep pile of soft sand in the way, but we'll take highly detailed pictures from as close as we can. (Like Burns's Cliff, Steve points out.) Then we turn back around and start slogging up the hill again.
"Sorry about that," Steve says. "This is the best exposed geology we'll see this side of Husband Hill, so we gotta do it."
That's okay, it really is. The big disappointment from today comes from an unexpected direction, when Chris finds a way we could have done the MI sky flats after all. It's a configuration I thought I'd tried, but obviously not. So now I feel twice as bad about not finding a way to do it, because there was a way to be found. I just didn't find it.
(And writing this, it occurs to me there was always a way anyhow. The IDD's elbow joint is fully double-jointed; we could simply have swung the turret up so that it was above the elbow rather than below it. Then aiming the MI for the sky flats would have been trivial -- the rover deck is never in the way in that pose. Oh, I am dumb sometimes, I truly am.)
I swallow my pride and send Ken and Steve email about this. We couldn't do the sky flats today, but we'll still be here over the weekend. Maybe we'll squeeze it in tomorrow.
2010-04-28
Spirit Sol 469
It's been nearly two weeks since I was last on shift. The drive we planned last time I was on shift went splendidly, putting us just where we wanted. Once there, as we started climbing up the long way around, the scientists started finding more stuff they just had to take a look at. So Spirit spent a while investigating some crumbly stuff called Methuselah, and now we've just driven up to a rock called Keystone, which they're practically salivating over. (And I'm not the only one who thinks so: "Steve's like a kid in a candy store," Dan Moyers remarks to me during the SOWG meeting.)
Thisol we're starting to IDD Keystone. Just to make sure I don't get bored, this is going to be the biggest MI mosaic of a single object ever, a total of 6x4x5+stereo = 144 images total. Our approach is to sequence as many of them today as the rover will have time to execute, and finish the rest tomorrow.
Before we can even really get started, we have to figure out what we want to image. This is troublesome, because we first have to know the angle of the MI with respect to the roughly rectangular rock surface -- that affects how the image footprints are oriented, which affects how you lay out the mosaic -- and none of our tools does that very well. So we basically just eyeball it, and it turns out it's reasonably close to the same orientation as the overall rectangle of rock itself, which simplifies things.
For the next step in the process, Ashitey and I (mostly Ashitey) pick a candidate center for the mosaic and use an image processing program to sketch out what the mosaic will cover with that as its center. Drawing the rectangle is easy, but figuring out where it lies is hard. No tools show you what the MI image covers (RSVP shows you a predicted MI image, but it's hard to figure out exactly where this lies in the original images), so you have to do a lot of eyeballing and detective work, and fit the results back into the original image. Oy. We email the resulting sketch to Steve, who prints it out, plays around with a ruler, and asks us to shift the whole thing 4.5cm to the left. So we do.
I think I have some ideas for improvements for future missions.
Anyway, with that done, the rest of the work isn't so hard. There are a lot of images in the mosaic, but they fall into groups, a total of 24 stacks of six images (including the stereo image) apiece. I further break down the mosaic into six columns of four stacks each, and then there's only a little overhead work to set up each column. It takes time, but I've had harder sequences to build. In the end, it turns out the rover won't have time to do more than four of the columns thisol, but that's still two-thirds of the total; we'll fill in the rest nextersol.
The only serious issue that arises is a question of using the IMU during the IDD sequence. This is something we do when there's a chance the vehicle's overall attitude would change during the sequence, which is something that could potentially endanger the IDD: if the vehicle tips forward when the arm's extended, suddenly it's that much closer to the ground, and could run into something we thought it would be clear of.
We're supposed to need to turn the IMU on when IDDing only when the rover's preloading the RAT or when we're at a significant pitch -- or anyway, that's what I thought the rules were. Ashitey thinks we need to do it when we're tilted, not just when we're pitched. (That is, he thinks we should do it even when we have zero pitch but lots of roll.) Thisol's pitch is only about 6.5 degrees but the overall tilt is a much higher 14.5 degrees, so it's going to matter which of us is right. John Wright is around, so I bring him in as a tie-breaker, and he votes with Ashitey, darn it. ("That was the wrong answer," I tell him.[1])
Why not just turn on the IMU anyway? Spirit's got power to burn, so it's not that. The problem is that there's ten seconds of overhead in every IDD move when the IMU is on, and what with all the crazy stuff we're doing thisol, the ten seconds per move add up to a whopping 38 minutes of overhead! And we don't have that to spare.
We could make back the 38 minutes by cutting another column, but John comes up with a better idea than that. It's a compromise: turn on the IMU during the unstow and first placement, and thereafter only when we actually contact the surface. Those are the times when there's the greatest risk (not that there's any real risk, but we're paid to be paranoid and we earn every penny) that the attitude would change. And that compromise adds only about six or eight minutes to the run-time, not 38. So that's what we do.
It's another fairly long day, but still less than ten hours. And if these images look as cool as I hope they will, it will have been well worth it.
Courtesy NASA/JPL-Caltech. Poised to examine Keystone.
Footnotes:
[1] Turns out I was right: pitch matters, roll doesn't. Fortunately, with the maturity and distance of five years, I have the class not to say nyaah nyaah.
Thisol we're starting to IDD Keystone. Just to make sure I don't get bored, this is going to be the biggest MI mosaic of a single object ever, a total of 6x4x5+stereo = 144 images total. Our approach is to sequence as many of them today as the rover will have time to execute, and finish the rest tomorrow.
Before we can even really get started, we have to figure out what we want to image. This is troublesome, because we first have to know the angle of the MI with respect to the roughly rectangular rock surface -- that affects how the image footprints are oriented, which affects how you lay out the mosaic -- and none of our tools does that very well. So we basically just eyeball it, and it turns out it's reasonably close to the same orientation as the overall rectangle of rock itself, which simplifies things.
For the next step in the process, Ashitey and I (mostly Ashitey) pick a candidate center for the mosaic and use an image processing program to sketch out what the mosaic will cover with that as its center. Drawing the rectangle is easy, but figuring out where it lies is hard. No tools show you what the MI image covers (RSVP shows you a predicted MI image, but it's hard to figure out exactly where this lies in the original images), so you have to do a lot of eyeballing and detective work, and fit the results back into the original image. Oy. We email the resulting sketch to Steve, who prints it out, plays around with a ruler, and asks us to shift the whole thing 4.5cm to the left. So we do.
I think I have some ideas for improvements for future missions.
Anyway, with that done, the rest of the work isn't so hard. There are a lot of images in the mosaic, but they fall into groups, a total of 24 stacks of six images (including the stereo image) apiece. I further break down the mosaic into six columns of four stacks each, and then there's only a little overhead work to set up each column. It takes time, but I've had harder sequences to build. In the end, it turns out the rover won't have time to do more than four of the columns thisol, but that's still two-thirds of the total; we'll fill in the rest nextersol.
The only serious issue that arises is a question of using the IMU during the IDD sequence. This is something we do when there's a chance the vehicle's overall attitude would change during the sequence, which is something that could potentially endanger the IDD: if the vehicle tips forward when the arm's extended, suddenly it's that much closer to the ground, and could run into something we thought it would be clear of.
We're supposed to need to turn the IMU on when IDDing only when the rover's preloading the RAT or when we're at a significant pitch -- or anyway, that's what I thought the rules were. Ashitey thinks we need to do it when we're tilted, not just when we're pitched. (That is, he thinks we should do it even when we have zero pitch but lots of roll.) Thisol's pitch is only about 6.5 degrees but the overall tilt is a much higher 14.5 degrees, so it's going to matter which of us is right. John Wright is around, so I bring him in as a tie-breaker, and he votes with Ashitey, darn it. ("That was the wrong answer," I tell him.[1])
Why not just turn on the IMU anyway? Spirit's got power to burn, so it's not that. The problem is that there's ten seconds of overhead in every IDD move when the IMU is on, and what with all the crazy stuff we're doing thisol, the ten seconds per move add up to a whopping 38 minutes of overhead! And we don't have that to spare.
We could make back the 38 minutes by cutting another column, but John comes up with a better idea than that. It's a compromise: turn on the IMU during the unstow and first placement, and thereafter only when we actually contact the surface. Those are the times when there's the greatest risk (not that there's any real risk, but we're paid to be paranoid and we earn every penny) that the attitude would change. And that compromise adds only about six or eight minutes to the run-time, not 38. So that's what we do.
It's another fairly long day, but still less than ten hours. And if these images look as cool as I hope they will, it will have been well worth it.
Courtesy NASA/JPL-Caltech. Poised to examine Keystone.
Footnotes:
[1] Turns out I was right: pitch matters, roll doesn't. Fortunately, with the maturity and distance of five years, I have the class not to say nyaah nyaah.
2010-04-14
Spirit Sol 455
The good news is that we're not dug in. The bad news is that we're not uphill.
Instead, we experienced alarmingly high levels of side-slip, gravity pulling us downhill as we tried to drive across the slope. The backoff worked, but then, only a couple of meters or so into the uphill segment, the sequence detected that we came too close to an obstacle and shut us down.
Ashitey tells me something I don't want to hear. "We can't make it," he says. "We should go back downhill and try to go around."
I hate it, but he's right. Straight uphill doesn't work. Crossing at a 45-degree angle doesn't work. Shallower angles -- either east or west -- take us into impassable slopes on one side, or rocks and an uncrossable berm on the other. The only way out is down.
Ashitey sets up a meeting among all the concerned parties. We gather in the SOWG room and, in short, agree that Ashitey's right. We'll have to go down.
Twelve meters or so downhill, there's what looks like a graded path around the side. My big concern is that we can't see all the way along the grade, so for all we know, we'll follow it to a certain point and then find that that way is blocked as well. But there's no way up from here, so it's the right next thing to try.
It's a complex drive. Ashitey and I are dumb enough to make it even harder on ourselves by adding a cross-slope leg after the downhill leg. But at least if it works, we'll have made real progress. If you can call a retreat progress. I guess we'll have made real regress, or something like that.
We also manage to pack in a couple of visits. The first is by science
fiction writer Bruce Sterling, whom Frank invited to come see us drive
a Mars rover. This is so wild I can't describe it.
Sterling's not what I expected. I don't know what I expected, but he's not it. He's short, dumpy, long-haired, laconic -- and creative, that part's no surprise. He takes one look at the six-wheeled rover with solar-panel wings and calls it a bug, which is a comparison that somehow never occurred to me, and I'm practically an avocational entomologist.
Our other visitor -- or group of visitors -- is brought around by Eric Baumgartner. They're a quartet of science fair winners from his kid's school, all around eleven or twelve years old. Plus two teachers. Ashitey and I do a good job with them, I think. He makes sure they look through the 3-D glasses, which appeals to everyone but especially kids. I don't think I'm generally very good with kids, but I remember being their age and being excited about science, and I'd have died for a trip like this. They're just like that -- a little intimidated at first, but grooving on it. I manage to get them to laugh and loosen up a little, and then they're just a bunch of smart kids having a blast. I love my job.
The long day goes on. We work apace. Jim Erickson shows up, with his usual grim look, and subtly chides us for adding the cross-slope leg. He's not the type to second-guess our decisions, but he's got a point to make and he makes it. We're putting in a lot of late nights, which exhausts the team. We've got to keep it simple. The extended mission is a marathon, not a sprint.
At this point, it's too late to gain anything by cutting the cross-slope leg. Since it's a sunk cost, we decide to leave it in for now, but we'll take it out if it ends up sucking up any more of our time. (It doesn't: the cross-slope leg makes it into the final sequence.)
We reach that point in the day where we're just waiting for the TAP/SIE to put everything in its final form. Today we put the time to good use. I've been worried about the fact that nobody really pays attention in the walkthroughs any more, which is partly because the complexity of the sequences, and the amount of stuff you have to know about the rover in order to follow along, is really overwhelming. So I break out a few pieces of the sequence and start going over them in detail -- an impromptu rover-driving course. When I see that people's brains are getting full, I declare success and move on to other things. It's only a little progress, but it's progress.
You can tell it's late, by the way. Jeff Favretto, following up on Erickson's earlier comments, says something about the value of getting out by 8:00 so we can spend time with our wives. (It's now nearly 10:00, and we'll be there another hour. We started twelve hours ago.) "I'm single, but if I don't leave by 8, I'll never get a girlfriend," Dan Moyers chimes in. And Favretto quips: "Dan, you'll have to leave a lot earlier than 8 if you want to get a girlfriend."
Ouch!
On a lighter note, Steve Squyres's book (Roving Mars) is coming out soon, and he's asked people on the project to help him make sure that the name of everyone involved in the project is listed in an appendix. This includes everyone from the project managers down to off-site contractors who sewed the parachutes. He had a good start already, but for the past week or so he's been beefing up the list, adding names as he's sent them.
So far he's got four thousand.
[Next post: sol 469, April 28.]
Instead, we experienced alarmingly high levels of side-slip, gravity pulling us downhill as we tried to drive across the slope. The backoff worked, but then, only a couple of meters or so into the uphill segment, the sequence detected that we came too close to an obstacle and shut us down.
Ashitey tells me something I don't want to hear. "We can't make it," he says. "We should go back downhill and try to go around."
I hate it, but he's right. Straight uphill doesn't work. Crossing at a 45-degree angle doesn't work. Shallower angles -- either east or west -- take us into impassable slopes on one side, or rocks and an uncrossable berm on the other. The only way out is down.
Ashitey sets up a meeting among all the concerned parties. We gather in the SOWG room and, in short, agree that Ashitey's right. We'll have to go down.
Twelve meters or so downhill, there's what looks like a graded path around the side. My big concern is that we can't see all the way along the grade, so for all we know, we'll follow it to a certain point and then find that that way is blocked as well. But there's no way up from here, so it's the right next thing to try.
It's a complex drive. Ashitey and I are dumb enough to make it even harder on ourselves by adding a cross-slope leg after the downhill leg. But at least if it works, we'll have made real progress. If you can call a retreat progress. I guess we'll have made real regress, or something like that.
We also manage to pack in a couple of visits. The first is by science
fiction writer Bruce Sterling, whom Frank invited to come see us drive
a Mars rover. This is so wild I can't describe it.
Sterling's not what I expected. I don't know what I expected, but he's not it. He's short, dumpy, long-haired, laconic -- and creative, that part's no surprise. He takes one look at the six-wheeled rover with solar-panel wings and calls it a bug, which is a comparison that somehow never occurred to me, and I'm practically an avocational entomologist.
Our other visitor -- or group of visitors -- is brought around by Eric Baumgartner. They're a quartet of science fair winners from his kid's school, all around eleven or twelve years old. Plus two teachers. Ashitey and I do a good job with them, I think. He makes sure they look through the 3-D glasses, which appeals to everyone but especially kids. I don't think I'm generally very good with kids, but I remember being their age and being excited about science, and I'd have died for a trip like this. They're just like that -- a little intimidated at first, but grooving on it. I manage to get them to laugh and loosen up a little, and then they're just a bunch of smart kids having a blast. I love my job.
The long day goes on. We work apace. Jim Erickson shows up, with his usual grim look, and subtly chides us for adding the cross-slope leg. He's not the type to second-guess our decisions, but he's got a point to make and he makes it. We're putting in a lot of late nights, which exhausts the team. We've got to keep it simple. The extended mission is a marathon, not a sprint.
At this point, it's too late to gain anything by cutting the cross-slope leg. Since it's a sunk cost, we decide to leave it in for now, but we'll take it out if it ends up sucking up any more of our time. (It doesn't: the cross-slope leg makes it into the final sequence.)
We reach that point in the day where we're just waiting for the TAP/SIE to put everything in its final form. Today we put the time to good use. I've been worried about the fact that nobody really pays attention in the walkthroughs any more, which is partly because the complexity of the sequences, and the amount of stuff you have to know about the rover in order to follow along, is really overwhelming. So I break out a few pieces of the sequence and start going over them in detail -- an impromptu rover-driving course. When I see that people's brains are getting full, I declare success and move on to other things. It's only a little progress, but it's progress.
You can tell it's late, by the way. Jeff Favretto, following up on Erickson's earlier comments, says something about the value of getting out by 8:00 so we can spend time with our wives. (It's now nearly 10:00, and we'll be there another hour. We started twelve hours ago.) "I'm single, but if I don't leave by 8, I'll never get a girlfriend," Dan Moyers chimes in. And Favretto quips: "Dan, you'll have to leave a lot earlier than 8 if you want to get a girlfriend."
Ouch!
On a lighter note, Steve Squyres's book (Roving Mars) is coming out soon, and he's asked people on the project to help him make sure that the name of everyone involved in the project is listed in an appendix. This includes everyone from the project managers down to off-site contractors who sewed the parachutes. He had a good start already, but for the past week or so he's been beefing up the list, adding names as he's sent them.
So far he's got four thousand.
[Next post: sol 469, April 28.]
2010-04-13
Spirit Sol 454
We've recovered from the weekend's fault, and the downlink shows we're in pretty much the state we expected: somewhat dug in. A small potato hovers by the left front wheel, and another possible potato -- the scientists bet strongly it's just a dirt clod, not a rock -- sits in the left rear wheel. So it'll be tricky just to get away.
Fortunately, we have a little more time than expected. On the large scale, that is: "There's nothing magic about sol 500 any more," Steve declares at the SOWG meeting. This had previously been the date by which we had to make it to the summit, before the sun angle started to favor south-facing slopes. The resulting lack of energy on our current north-facing slope would mean we'd face serious consequences. In the benign scenario, we'd have less and less time to drive each day, making the top farther and farther away. In the less benign scenario, we'd freeze to death.
But the dust devil that cleaned the solar panels bought us more time than I'd thought: even with the reduced energy level, we can last well past sol 500 now. The pace is now to be driven by the Martian terrain and scientific discoveries, not a mad scramble for the top.
Which is a good thing, because we haven't exactly been making a lot of progress lately. The next thing we're going to try is cross-slope driving. The last drive failed because it was fighting directly uphill -- the same mistake we made around Wopmay, back on the other side of the planet. This time we're going to try to drive at about a 45-degree angle to the slope instead. Before we can even do that, we have to carefully back away downhill, to extricate ourselves from our dug-in position. If you thought the going had been slow recently, just think what it would be like if we picked up another potato ....
Just a few meters away, the slope breaks, and should be easier going. Once the rover gets that far, the drive aims a little more uphill, to avoid a gap in the terrain mesh (representing an area we can't quite see from our current position, and therefore can't evaluate).
This is the first drive I've written for Spirit in quite some time. I have this fantasy of coming downstairs and magically putting everything back into shape, coming up with some brilliant scheme that puts the rover on the top of this hill in no time. Awing the crowd. But no such thing will happen: the guys who've been doing this are plenty smart already, and if there were any such clever trick, they'd have thought of it. Getting to the top is just going to be a long, tough slog, and it's really possible that we just won't make it.
I have a talk coming up in a few days, down in Anaheim at the USENIX 2005 conference. I'll be talking about how to drive a Mars rover. I spend some time thinking about how to put the audience in the right mindset. This is what I come up with:
A little too dramatic, I think; I decide not to use it. But it sounds cool.
Fortunately, we have a little more time than expected. On the large scale, that is: "There's nothing magic about sol 500 any more," Steve declares at the SOWG meeting. This had previously been the date by which we had to make it to the summit, before the sun angle started to favor south-facing slopes. The resulting lack of energy on our current north-facing slope would mean we'd face serious consequences. In the benign scenario, we'd have less and less time to drive each day, making the top farther and farther away. In the less benign scenario, we'd freeze to death.
But the dust devil that cleaned the solar panels bought us more time than I'd thought: even with the reduced energy level, we can last well past sol 500 now. The pace is now to be driven by the Martian terrain and scientific discoveries, not a mad scramble for the top.
Which is a good thing, because we haven't exactly been making a lot of progress lately. The next thing we're going to try is cross-slope driving. The last drive failed because it was fighting directly uphill -- the same mistake we made around Wopmay, back on the other side of the planet. This time we're going to try to drive at about a 45-degree angle to the slope instead. Before we can even do that, we have to carefully back away downhill, to extricate ourselves from our dug-in position. If you thought the going had been slow recently, just think what it would be like if we picked up another potato ....
Just a few meters away, the slope breaks, and should be easier going. Once the rover gets that far, the drive aims a little more uphill, to avoid a gap in the terrain mesh (representing an area we can't quite see from our current position, and therefore can't evaluate).
This is the first drive I've written for Spirit in quite some time. I have this fantasy of coming downstairs and magically putting everything back into shape, coming up with some brilliant scheme that puts the rover on the top of this hill in no time. Awing the crowd. But no such thing will happen: the guys who've been doing this are plenty smart already, and if there were any such clever trick, they'd have thought of it. Getting to the top is just going to be a long, tough slog, and it's really possible that we just won't make it.
I have a talk coming up in a few days, down in Anaheim at the USENIX 2005 conference. I'll be talking about how to drive a Mars rover. I spend some time thinking about how to put the audience in the right mindset. This is what I come up with:
You're 450 sols into a 90-sol mission. Threats surround you; every path is blocked. You're chasing the sun to the top of a hill and if you don't beat it you'll die. Welcome to Mars.
A little too dramatic, I think; I decide not to use it. But it sounds cool.
2010-04-12
Spirit Sol 453
On my way in I notice we're on the front page. Our funding was extended for 18 more months, making the front page of the Pasadena Star-News. I remember when I'd come in and we were front-page news every day. I buy a copy for nostalgia's sake.
I'm in an hour early, to get a jump on the drive sequence, but Jim Erickson needs a minute of my time. "Did you hear about the problems over the weekend?" he asks. Oh, dear.
Both rovers had bad weekends. Opportunity's drive faulted out before it even drove anywhere. Last week the suspension limits tripped during a drive, and tripped again during the auto-recovery -- an off-nominal circumstance, but one we plan for. For the weekend, they sequenced a drive where they backed off the ripple they tripped on, then continued, ignoring the suspension limits during the backoff. But they didn't explicitly disable the limits, they just assumed the limits were i the "ignored" state for the first part of the drive. However, it turns out that the flag that tells the rover to check the suspension limits (not ignore them) was accidentally saved to EEPROM as the default weeks ago, so Opportunity actually applied the most recently set suspension limits, and since they'd tripped before, and Opportunity hadn't moved, it went right ahead and tripped them again. This was a problem that was just waiting to bite us. And then, possibly because of that error, they mis-timed the solar-array stomp, which meant the rover didn't nap when it was supposed to, and thus is power-poor.
Spirit had problems, too. They didn't properly clean up after last Thursday's fatal. (Which fatal, you will recall, was my damn fault.) The fatal caused Spirit to decide it didn't know where its HGA was pointed, so it didn't listen for the uplink from Earth. As a result, none of our cleanup sequences made it up over the weekend. Which means we can't drive today; we still need to do our imaging and update the rover's position. Lucky for us, Beth Dewell noticed the problem and already got Spirit back into decent shape over the weekend, which is a good thing -- else we'd be down for another day.
So, for those of you keeping score at home: last time I was on Spirit, I jammed a potato in the wheel, which blew about ten or twenty sols. This time I fataled it and blew Thursday through today (Monday) at least. The Opportunity problem is not my fault, but I couldn't feel worse if it were. I am a bad rover driver. I'm glum. I want to kill myself.
But if we're not driving today, I have time to answer Jim's questions, at least. He ticks them off. "These problems we've been having with the vehicles lately: are they problems due to unfamiliarity with the new version of the flight software? Are the RPs overworked? Are they complacent?"
I answer these as best I can. I know some of the problems were due to unfamiliarity with the new flight software; it's a partial explanation. I don't think the RPs are generally overworked. (I personally have a lot more work than I have time for, but this afternoon I'm going to talk to someone about coming in to the RSVP development team to pick some of it up, which will mitigate the problem.)
As for complacency ... well, I want to say no, but the honest answer is probably yes. We're not as paranoid about these vehicles as we used to be. Some of that's understandable and even proper. Some of it, maybe not.
Jim nods thoughtfully and goes away. As if on cue, Frank enters, fresh from the Land of Opportunity. He tells me he wouldn't have caught the problem that led to a fatal, either, but: "I was gonna give you a ribbing about it anyway, but then we had our problem over the weekend, and now I can't give you a hard time."
"The universe smiles on Scott," Al Herrera remarks.
If he's right, it means the problem on Opportunity was my fault. You know when I said I couldn't feel worse? I might have been wrong.
Upstairs, while we're waiting for the SOWG meeting to start, we try to look on the bright side of this extended anomaly. "The batteries will be topped off," Chris points out.
"And now the bar is lowered," I add. "If we can just manage to drive without fataling, we'll be so happy." It's black humor, but it's humor.
Squyres is -- what's the equivalent of "visibly unhappy" for a disembodied voice, "audibly unhappy?" Well, he's unhappy. We've got basically nothing to show for last week, with the ODY problem followed by the fatal, and today will be basically spent on recovery. He pokes at this to see if we can drive anyway, but we really can't do it without more data. He manages to talk Callas into allowing a MTES checkout, so I suppose that's something, from his point of view.
But we're 120 sols into the climb and only halfway up the hill, and we're 50 sols away from our deadline. We're power-rich right now, but in about 50 more sols, the sun's wandering position in the sky starts to favor south-facing slopes over north-facing slopes such as the one we're on. Our friendly, solar-panel-cleaning dust devils might have extended the deadline, but we'll eventually have to cross to the far slope or die. Steve wants to go, go, go, and we've got to just sit here. At least for today.
"We're gonna have a good week," Steve says emphatically at the close of the meeting. "Power of positive thinking."
I'm in an hour early, to get a jump on the drive sequence, but Jim Erickson needs a minute of my time. "Did you hear about the problems over the weekend?" he asks. Oh, dear.
Both rovers had bad weekends. Opportunity's drive faulted out before it even drove anywhere. Last week the suspension limits tripped during a drive, and tripped again during the auto-recovery -- an off-nominal circumstance, but one we plan for. For the weekend, they sequenced a drive where they backed off the ripple they tripped on, then continued, ignoring the suspension limits during the backoff. But they didn't explicitly disable the limits, they just assumed the limits were i the "ignored" state for the first part of the drive. However, it turns out that the flag that tells the rover to check the suspension limits (not ignore them) was accidentally saved to EEPROM as the default weeks ago, so Opportunity actually applied the most recently set suspension limits, and since they'd tripped before, and Opportunity hadn't moved, it went right ahead and tripped them again. This was a problem that was just waiting to bite us. And then, possibly because of that error, they mis-timed the solar-array stomp, which meant the rover didn't nap when it was supposed to, and thus is power-poor.
Spirit had problems, too. They didn't properly clean up after last Thursday's fatal. (Which fatal, you will recall, was my damn fault.) The fatal caused Spirit to decide it didn't know where its HGA was pointed, so it didn't listen for the uplink from Earth. As a result, none of our cleanup sequences made it up over the weekend. Which means we can't drive today; we still need to do our imaging and update the rover's position. Lucky for us, Beth Dewell noticed the problem and already got Spirit back into decent shape over the weekend, which is a good thing -- else we'd be down for another day.
So, for those of you keeping score at home: last time I was on Spirit, I jammed a potato in the wheel, which blew about ten or twenty sols. This time I fataled it and blew Thursday through today (Monday) at least. The Opportunity problem is not my fault, but I couldn't feel worse if it were. I am a bad rover driver. I'm glum. I want to kill myself.
But if we're not driving today, I have time to answer Jim's questions, at least. He ticks them off. "These problems we've been having with the vehicles lately: are they problems due to unfamiliarity with the new version of the flight software? Are the RPs overworked? Are they complacent?"
I answer these as best I can. I know some of the problems were due to unfamiliarity with the new flight software; it's a partial explanation. I don't think the RPs are generally overworked. (I personally have a lot more work than I have time for, but this afternoon I'm going to talk to someone about coming in to the RSVP development team to pick some of it up, which will mitigate the problem.)
As for complacency ... well, I want to say no, but the honest answer is probably yes. We're not as paranoid about these vehicles as we used to be. Some of that's understandable and even proper. Some of it, maybe not.
Jim nods thoughtfully and goes away. As if on cue, Frank enters, fresh from the Land of Opportunity. He tells me he wouldn't have caught the problem that led to a fatal, either, but: "I was gonna give you a ribbing about it anyway, but then we had our problem over the weekend, and now I can't give you a hard time."
"The universe smiles on Scott," Al Herrera remarks.
If he's right, it means the problem on Opportunity was my fault. You know when I said I couldn't feel worse? I might have been wrong.
Upstairs, while we're waiting for the SOWG meeting to start, we try to look on the bright side of this extended anomaly. "The batteries will be topped off," Chris points out.
"And now the bar is lowered," I add. "If we can just manage to drive without fataling, we'll be so happy." It's black humor, but it's humor.
Squyres is -- what's the equivalent of "visibly unhappy" for a disembodied voice, "audibly unhappy?" Well, he's unhappy. We've got basically nothing to show for last week, with the ODY problem followed by the fatal, and today will be basically spent on recovery. He pokes at this to see if we can drive anyway, but we really can't do it without more data. He manages to talk Callas into allowing a MTES checkout, so I suppose that's something, from his point of view.
But we're 120 sols into the climb and only halfway up the hill, and we're 50 sols away from our deadline. We're power-rich right now, but in about 50 more sols, the sun's wandering position in the sky starts to favor south-facing slopes over north-facing slopes such as the one we're on. Our friendly, solar-panel-cleaning dust devils might have extended the deadline, but we'll eventually have to cross to the far slope or die. Steve wants to go, go, go, and we've got to just sit here. At least for today.
"We're gonna have a good week," Steve says emphatically at the close of the meeting. "Power of positive thinking."
2010-04-09
Spirit Sol 450
I have to go give an RSVP demo, but first I have to know how Spirit did on my first attempt to drive it for months. I borrow a Web browser window from Colette Wilklow and quickly bring up the automatically generated Web page that summarizes the results.
The first line says more than eight meters -- not too shabby, for this terrain. Then I notice the next line, the usually more accurate line -- the one that tells you not just how far the wheels spun but how far the rover actually moved. About 87cm. That's all.
Fuck!
Just then my cell phone rings.
"Hello?"
It's Mark Maimone. "Well, I ruined my record," he says. "I fataled a spacecraft."
"Fataled" doesn't mean the rover is dead, but it is a bad thing. It means the rover rebooted. And it's our fault.
After I get back from my demo, I get the story. What happened here, it transpires, is that we screwed up something really stupid. It has to do with a feature in the new release of the flight software. The original flight software let you write sequences that made decisions based on a limited set of parameters -- so that the rover could decide to do one thing or another based on its distance to a defined target, whether the previous command had succeeded or failed, and about a dozen more. The new flight software added a bunch more of these parameters, one of them being the rover's current heading. (So now we can write a sequence that says, for instance, "if you're not already heading more or less east, turn to the east.")
Now, there was only one way to add these new parameters in a way that was backward-compatible with the old command dictionary (a requirement imposed from above), and that was sort of a kludge. The trick was to take one of the old parameters -- one we picked because we never actually used it in flight -- and use a command to redefine that one so that it's an alias for one of the new ones, and then tell the rover to test that. The "if" command itself resets the alias to an undefined state afterward. And when you test this parameter when it's in an undefined state, the flight software fatals.
Now, we did everything right. In the main sequence, that is. What we forgot is that we also had a background sequence running, one that checks periodically to see if the rover is too close to an obstacle, so that it can abort driving. And this sequence is basically a long string of "if you're too close to obstacle #1, stop driving; if you're too close to obstacle #2, stop driving ...." One of these commands just happened to undefine the special parameter set in the main sequence, triggering the fatal.
Ashitey and I didn't realize that any "if" reset the special parameter -- we thought only the ones that used that parameter reset it. Mark Maimone knew better, but he was awfully tired and stressed out yesterday. I fear he's overloaded.
John Callas declares by fiat that the team is too tired to work this weekend. We're just going to clean up after our boneheaded error and do a minimal weekend plan. No driving, no IDD. It's a hint that we should go home and rest up. But do I? Nope: too much other work to do.[1]
I'm not surprised the team is tired. This whole hill-climbing campaign with Spirit looks like it could become frustrating. For the rover, it's a long climb. We've been at it three months, and we're only halfway up. The hell of it is, two decently strong men could pick the damn rover up and carry it to the summit in half an hour.
When Andy's new schedule comes out, it shows I'm not downstairs just temporarily. Looks like I'm back on Spirit at least through the end of May, which probably means for good. I'm surprised that I'm disappointed by this. I'm almost ashamed to admit it, but there was a lot to be said for working on Opportunity. The team's more laid back. It's easier. You drive really far. It's a cheap kind of glory but I'll take it.
But in the end, I'm not too disappointed. This rover and I have been through a lot together already, and this just means there's more adventure ahead.
I can't wait to get started.
[Next post: sol 453, April 12.]
Footnotes:
[1] Easy for me to say now, but that was a bad decision: incompatible with rover safety. Shoulda gone home and slept. Shame on me.
The first line says more than eight meters -- not too shabby, for this terrain. Then I notice the next line, the usually more accurate line -- the one that tells you not just how far the wheels spun but how far the rover actually moved. About 87cm. That's all.
Fuck!
Just then my cell phone rings.
"Hello?"
It's Mark Maimone. "Well, I ruined my record," he says. "I fataled a spacecraft."
"Fataled" doesn't mean the rover is dead, but it is a bad thing. It means the rover rebooted. And it's our fault.
After I get back from my demo, I get the story. What happened here, it transpires, is that we screwed up something really stupid. It has to do with a feature in the new release of the flight software. The original flight software let you write sequences that made decisions based on a limited set of parameters -- so that the rover could decide to do one thing or another based on its distance to a defined target, whether the previous command had succeeded or failed, and about a dozen more. The new flight software added a bunch more of these parameters, one of them being the rover's current heading. (So now we can write a sequence that says, for instance, "if you're not already heading more or less east, turn to the east.")
Now, there was only one way to add these new parameters in a way that was backward-compatible with the old command dictionary (a requirement imposed from above), and that was sort of a kludge. The trick was to take one of the old parameters -- one we picked because we never actually used it in flight -- and use a command to redefine that one so that it's an alias for one of the new ones, and then tell the rover to test that. The "if" command itself resets the alias to an undefined state afterward. And when you test this parameter when it's in an undefined state, the flight software fatals.
Now, we did everything right. In the main sequence, that is. What we forgot is that we also had a background sequence running, one that checks periodically to see if the rover is too close to an obstacle, so that it can abort driving. And this sequence is basically a long string of "if you're too close to obstacle #1, stop driving; if you're too close to obstacle #2, stop driving ...." One of these commands just happened to undefine the special parameter set in the main sequence, triggering the fatal.
Ashitey and I didn't realize that any "if" reset the special parameter -- we thought only the ones that used that parameter reset it. Mark Maimone knew better, but he was awfully tired and stressed out yesterday. I fear he's overloaded.
John Callas declares by fiat that the team is too tired to work this weekend. We're just going to clean up after our boneheaded error and do a minimal weekend plan. No driving, no IDD. It's a hint that we should go home and rest up. But do I? Nope: too much other work to do.[1]
I'm not surprised the team is tired. This whole hill-climbing campaign with Spirit looks like it could become frustrating. For the rover, it's a long climb. We've been at it three months, and we're only halfway up. The hell of it is, two decently strong men could pick the damn rover up and carry it to the summit in half an hour.
When Andy's new schedule comes out, it shows I'm not downstairs just temporarily. Looks like I'm back on Spirit at least through the end of May, which probably means for good. I'm surprised that I'm disappointed by this. I'm almost ashamed to admit it, but there was a lot to be said for working on Opportunity. The team's more laid back. It's easier. You drive really far. It's a cheap kind of glory but I'll take it.
But in the end, I'm not too disappointed. This rover and I have been through a lot together already, and this just means there's more adventure ahead.
I can't wait to get started.
[Next post: sol 453, April 12.]
Footnotes:
[1] Easy for me to say now, but that was a bad decision: incompatible with rover safety. Shoulda gone home and slept. Shame on me.
2010-04-08
Spirit Sol 449
ODY is back online; our broadband connection to Mars has been restored. I still don't know what the heck was wrong with it, but I guess I'll find out.
So we're driving. Our plan is to continue the drive that faulted out this past weekend; we've been sitting still since then because of the relay problem. I'm glad we're going to do some driving; else I'd come in next week and have to be RP-1, without any way to get properly up to speed. Sure, I've read through the drive sequences they've been building, but it's never the same as when the pressure's on.
My brain still hasn't caught up with this side of the world. I go over to MIPL to ask them to build us a new terrain mesh with the extra NAVCAM images that just came down, and unconsciously I go over to talk to the Opportunity support guy. Who's as puzzled as I am until I realize I'm supposed to be talking to the Spirit support guy. Sheesh.
But it's fun sequencing this rover again. I'd almost forgotten what challenging drives were like. Opportunity is just a slow car in a parking lot full of speed bumps. Driving Spirit up a rocky hill is delicate, exacting work. Obstacles to the left of you, obstacles to the right of you -- and a slope that's only too happy to help you slide into them.
One of the innovative techniques the Spirits have developed is a parallel obstacle-check sequence. The way we used to drive through hazardous zones was to first construct a list of hazards visible in the imagery. Then we'd write a sequence that would drive a little way, check if we were too close to the obstacles, drive a little farther, check again, and so on. At each stage, you'd tend to check only for obstacles you thought you'd be likely to be near at that point, because it was just a nightmare to try to check for all of them all of the time. The resulting drive sequences had a lot of repetition and were hard to modify: if you wanted to change your driving path, you'd have to revisit a lot of the places where you stopped and checked for hazards to make sure you were checking for the right ones.
So the Spirits worked out a way to separate those concerns. They now build one sequence that checks whether the rover is near any of the known hazards; this is padded out with wait commands so that it takes one minute to run. Then they have a second sequence that calls the first one ten times, and a third sequence that calls the ten-minute version 24 times (so it runs for 240 minutes -- or four hours, longer than their drives). Finally, this third sequence is activated in parallel with the drive. So the drive can just contain the needed commands to take the rover along the desired path, and the parallel obstacle-check sequence will stop it if it gets too close to any of the hazards. The parallel sequence checks all of the hazards, and checks them every minute, and doesn't interfere with your development of the drive sequences; in all three ways, it's better than the old approach.[1]
There's a lot more to come up to speed on as well -- a lot of fiddling with parameters that we don't need to do on Opportunity, but which are needed on Spirit, mainly to avoid catching another potato. Finally, as gravy, thisol we're doing two things for the first time on Mars: at the end of the drive, we're using autonav with visual odometry (so the rover will autonomously avoid obstacles as well as keeping careful track of its location), and we're using a new flight software feature that lets us make run-time decisions based on the vehicle's heading. If all goes well, we'll come in tomorrow to find we've climbed Husband Hill another 15 meters or so.
There's another difference between the two rovers: Opportunity's drive distances have more digits.
[1] A later flight software update added a notion of "keepout zones," which replaced this elaborate tree of obstacle-checking sequences with just a few commands, one command per obstacle. Incomparably better.
So we're driving. Our plan is to continue the drive that faulted out this past weekend; we've been sitting still since then because of the relay problem. I'm glad we're going to do some driving; else I'd come in next week and have to be RP-1, without any way to get properly up to speed. Sure, I've read through the drive sequences they've been building, but it's never the same as when the pressure's on.
My brain still hasn't caught up with this side of the world. I go over to MIPL to ask them to build us a new terrain mesh with the extra NAVCAM images that just came down, and unconsciously I go over to talk to the Opportunity support guy. Who's as puzzled as I am until I realize I'm supposed to be talking to the Spirit support guy. Sheesh.
But it's fun sequencing this rover again. I'd almost forgotten what challenging drives were like. Opportunity is just a slow car in a parking lot full of speed bumps. Driving Spirit up a rocky hill is delicate, exacting work. Obstacles to the left of you, obstacles to the right of you -- and a slope that's only too happy to help you slide into them.
One of the innovative techniques the Spirits have developed is a parallel obstacle-check sequence. The way we used to drive through hazardous zones was to first construct a list of hazards visible in the imagery. Then we'd write a sequence that would drive a little way, check if we were too close to the obstacles, drive a little farther, check again, and so on. At each stage, you'd tend to check only for obstacles you thought you'd be likely to be near at that point, because it was just a nightmare to try to check for all of them all of the time. The resulting drive sequences had a lot of repetition and were hard to modify: if you wanted to change your driving path, you'd have to revisit a lot of the places where you stopped and checked for hazards to make sure you were checking for the right ones.
So the Spirits worked out a way to separate those concerns. They now build one sequence that checks whether the rover is near any of the known hazards; this is padded out with wait commands so that it takes one minute to run. Then they have a second sequence that calls the first one ten times, and a third sequence that calls the ten-minute version 24 times (so it runs for 240 minutes -- or four hours, longer than their drives). Finally, this third sequence is activated in parallel with the drive. So the drive can just contain the needed commands to take the rover along the desired path, and the parallel obstacle-check sequence will stop it if it gets too close to any of the hazards. The parallel sequence checks all of the hazards, and checks them every minute, and doesn't interfere with your development of the drive sequences; in all three ways, it's better than the old approach.[1]
There's a lot more to come up to speed on as well -- a lot of fiddling with parameters that we don't need to do on Opportunity, but which are needed on Spirit, mainly to avoid catching another potato. Finally, as gravy, thisol we're doing two things for the first time on Mars: at the end of the drive, we're using autonav with visual odometry (so the rover will autonomously avoid obstacles as well as keeping careful track of its location), and we're using a new flight software feature that lets us make run-time decisions based on the vehicle's heading. If all goes well, we'll come in tomorrow to find we've climbed Husband Hill another 15 meters or so.
There's another difference between the two rovers: Opportunity's drive distances have more digits.
[1] A later flight software update added a notion of "keepout zones," which replaced this elaborate tree of obstacle-checking sequences with just a few commands, one command per obstacle. Incomparably better.
2010-04-07
Spirit Sol 448
Drive? IDD? Neither: with ODY still out of commission, if we drive today, we might not get enough data down to do anything else until the weekend. So we're taking a gamble, doing a little remote sensing and basically sitting still. If ODY comes back tonight as expected, we'll be able to do normal work tomorrow.
For today: sit and stare at the ceiling.
For today: sit and stare at the ceiling.
2010-04-06
Spirit Sol 447
So I'm back on Spirit. I'm a couple of minutes late, but it doesn't matter: as I'm walking in, they're saying what I expected them to say. Until we've recovered from the ODY thing, no driving, no IDD. Maybe Saturday.
But at least I'm not the only one who blows drives around here. Like Opportunity, Spirit had a two-sol drive over the weekend. The scraps of data we get back today show that the first sol's drive faulted out (early, too), so the second one didn't execute at all.
This shouldn't make me feel better. But it does, damn it, it does.
But at least I'm not the only one who blows drives around here. Like Opportunity, Spirit had a two-sol drive over the weekend. The scraps of data we get back today show that the first sol's drive faulted out (early, too), so the second one didn't execute at all.
This shouldn't make me feel better. But it does, damn it, it does.
2010-04-05
Opportunity Sol 426 (Spirit Sol 446)
Today's my last day on Opportunity for a while, and it's off to an inauspicious start. Our main relay orbiter, Mars Odyssey (ODY), went into safe mode over the weekend. As a result, we didn't get any of our downlink. Not a sausage.
When I look at it the right way, I see our "no news" as good news: I checked in on the rover over the weekend and was alarmed that there had been no updates. I wasn't sure why that would happen, but of course the explanation that immediately occurred to me was that I had killed it. So it's not that, and that's good.
Of course, maybe ODY went into safe mode and I killed Opportunity. Heck, maybe whatever I did to kill Opportunity took ODY with it. ODY learned that Opportunity was gone and then ODY died from grief, maybe. But what are the odds?
Well, if we assume Opportunity is healthy, maybe we can even drive it. Our uplink isn't affected, so we could send up another all-autonav drive sequence. I have it all worked out in my head, and start to pitch the idea. But I'm brought up short by the fact that there's no room in flash: the drive will generate so much data that it will push out other stuff we might not want to discard, and that's a risk we can't take. Plus, nobody seems to know for sure, but several people suspect there's a rule about that: you have to confirm that the rover is healthy before you uplink mobility sequences. (We have someone do that for weekend drives, for instance -- they don't re-plan in the middle, but they can pull the plug on continued driving.) So that idea's out.
So we can't drive, and without images, we can't IDD, either. So you'd think I'd have the day off. But it doesn't work out like that. Instead, I decide to use this unexpected holiday to try to get other work done. I figure if I can get some stuff done for Phoenix today, I won't feel so bad about ignoring them on days when I want to focus on the rovers.
Meanwhile, the ops team has reconfigured the rover to try to send a little data directly down to Earth. This takes lots of power and doesn't return a lot of data, but it'll be better than what we have now. The data brings good news and bad news.
The good news is, we have data, so Opportunity really is alive. (Maybe ODY is grieving for the pope instead.) The bad news is, the drive faulted out. Analysis shows that the first drive got closer to Voyager Crater than we'd planned -- a lot closer. Rather than being 10m or 15m away, we're right on the lip, with 8 degrees of tilt.
But that drive worked. Up to this point, the rover was happy and safe. (Not quite where we thought it would be, but happy and safe all the same.) But then the next sol's drive, the all-autonav continuation drive, kicked in. In the course of this drive, autonav climbed a little higher on the crater lip, hitting the max tilt limit of 12 degrees, which stopped the drive. It drove only about two meters.
Part of this is just due to a dumb misconfiguration. The autonav software has its own notion of what tilts to stay away from, and our default setting for this limit is 15 degrees. Thus, autonav doesn't know to avoid the 12-degree limit that stops driving. (Both limits are configurable; our mistake was to lower the max-tilt limit and not tell autonav.) If we'd lowered the autonav tilt limit as well, autonav would have known it should stay lower on the crater lip, and the drive might well have worked.
The other problem is that the first drive got much closer to the crater than we thought it would. This would be alarming even if the second drive had succeeded, and somebody (not me, thankfully) gets the responsibility for finding out why it happened.
So now we're trying to debug and recover from a drive failure, and we have no data. On a normal sol, this wouldn't be a problem -- we'd just clear the error, aim downhill, and step on the gas. But with no data, there's not a lot we can do. I work with Emily and Charles to re-prioritize the images already taken, so we'll get what we most need a little sooner. But the big problem is that we've gone from a fire hose to a drinking straw. We'll just have to wait.
As it happens, given the ODY situation, we don't want to drive until Wednesday or so anyway, at the earliest. So there's not any particular pressure. Just for the sake of it, I design a drive that would let us return to a clear zone and proceed to our next destination even if we get no further data, but we'll never send it. I just do it for the exercise.
I also realize one other piece of bad news. It's an unusual distinction, but for some reason, I happen to have crossed more kilometer boundaries with these rovers than anyone else. I don't know if I've done more total driving than anyone else (though I wouldn't doubt it), but I just happen to be on shift when the km number clicks over. Well, when the autonav drive faulted out, Opportunity was less than five meters from reaching the 5km mark. Someone else (Frank, I think) will get the glory this time. I mention this to the team, and Geoff Lake cocks an eyebrow at me. "Has anyone ever told you that you're really focused?" he asks. Only twice a day ....
Meanwhile, it occurs to me that we should exploit our fabulous position. We're poised on the lip of this crater; might as well take some pictures! But Charles Budney just shrugs. "Voyager is going to look like Viking," he says. "We'll take a low-res NAVCAM-360 and move on." That's a crime. This would make a spectacular full-color PANCAM panorama, but try as I might, I can't light a fire under the science team to do it, so I give up. But I'll tell you, it's a sad day when the engineers want science more than the scientists do.
Oh, who am I kidding? Science, hell. I just want pretty pictures.
When I look at it the right way, I see our "no news" as good news: I checked in on the rover over the weekend and was alarmed that there had been no updates. I wasn't sure why that would happen, but of course the explanation that immediately occurred to me was that I had killed it. So it's not that, and that's good.
Of course, maybe ODY went into safe mode and I killed Opportunity. Heck, maybe whatever I did to kill Opportunity took ODY with it. ODY learned that Opportunity was gone and then ODY died from grief, maybe. But what are the odds?
Well, if we assume Opportunity is healthy, maybe we can even drive it. Our uplink isn't affected, so we could send up another all-autonav drive sequence. I have it all worked out in my head, and start to pitch the idea. But I'm brought up short by the fact that there's no room in flash: the drive will generate so much data that it will push out other stuff we might not want to discard, and that's a risk we can't take. Plus, nobody seems to know for sure, but several people suspect there's a rule about that: you have to confirm that the rover is healthy before you uplink mobility sequences. (We have someone do that for weekend drives, for instance -- they don't re-plan in the middle, but they can pull the plug on continued driving.) So that idea's out.
So we can't drive, and without images, we can't IDD, either. So you'd think I'd have the day off. But it doesn't work out like that. Instead, I decide to use this unexpected holiday to try to get other work done. I figure if I can get some stuff done for Phoenix today, I won't feel so bad about ignoring them on days when I want to focus on the rovers.
Meanwhile, the ops team has reconfigured the rover to try to send a little data directly down to Earth. This takes lots of power and doesn't return a lot of data, but it'll be better than what we have now. The data brings good news and bad news.
The good news is, we have data, so Opportunity really is alive. (Maybe ODY is grieving for the pope instead.) The bad news is, the drive faulted out. Analysis shows that the first drive got closer to Voyager Crater than we'd planned -- a lot closer. Rather than being 10m or 15m away, we're right on the lip, with 8 degrees of tilt.
But that drive worked. Up to this point, the rover was happy and safe. (Not quite where we thought it would be, but happy and safe all the same.) But then the next sol's drive, the all-autonav continuation drive, kicked in. In the course of this drive, autonav climbed a little higher on the crater lip, hitting the max tilt limit of 12 degrees, which stopped the drive. It drove only about two meters.
Part of this is just due to a dumb misconfiguration. The autonav software has its own notion of what tilts to stay away from, and our default setting for this limit is 15 degrees. Thus, autonav doesn't know to avoid the 12-degree limit that stops driving. (Both limits are configurable; our mistake was to lower the max-tilt limit and not tell autonav.) If we'd lowered the autonav tilt limit as well, autonav would have known it should stay lower on the crater lip, and the drive might well have worked.
The other problem is that the first drive got much closer to the crater than we thought it would. This would be alarming even if the second drive had succeeded, and somebody (not me, thankfully) gets the responsibility for finding out why it happened.
So now we're trying to debug and recover from a drive failure, and we have no data. On a normal sol, this wouldn't be a problem -- we'd just clear the error, aim downhill, and step on the gas. But with no data, there's not a lot we can do. I work with Emily and Charles to re-prioritize the images already taken, so we'll get what we most need a little sooner. But the big problem is that we've gone from a fire hose to a drinking straw. We'll just have to wait.
As it happens, given the ODY situation, we don't want to drive until Wednesday or so anyway, at the earliest. So there's not any particular pressure. Just for the sake of it, I design a drive that would let us return to a clear zone and proceed to our next destination even if we get no further data, but we'll never send it. I just do it for the exercise.
I also realize one other piece of bad news. It's an unusual distinction, but for some reason, I happen to have crossed more kilometer boundaries with these rovers than anyone else. I don't know if I've done more total driving than anyone else (though I wouldn't doubt it), but I just happen to be on shift when the km number clicks over. Well, when the autonav drive faulted out, Opportunity was less than five meters from reaching the 5km mark. Someone else (Frank, I think) will get the glory this time. I mention this to the team, and Geoff Lake cocks an eyebrow at me. "Has anyone ever told you that you're really focused?" he asks. Only twice a day ....
Meanwhile, it occurs to me that we should exploit our fabulous position. We're poised on the lip of this crater; might as well take some pictures! But Charles Budney just shrugs. "Voyager is going to look like Viking," he says. "We'll take a low-res NAVCAM-360 and move on." That's a crime. This would make a spectacular full-color PANCAM panorama, but try as I might, I can't light a fire under the science team to do it, so I give up. But I'll tell you, it's a sad day when the engineers want science more than the scientists do.
Oh, who am I kidding? Science, hell. I just want pretty pictures.
2010-04-02
Opportunity Sol 423 (Spirit Sol 443)
The previous drive went splendidly, just splendidly. We're right where we wanted to be, a few meters off Viking Crater, with a fabulous view into the hole.
Thisol we're planning two drives, to execute over the weekend. On 423, the first drive takes us from Viking to its twin, Voyager. The rover will take some pictures and then, on sol 424, we'll start an autonav-only drive from Voyager toward Erebus.
Erebus is another 800m away or so. That's a distance that used to take months to cover, but at the rate we're now progressing, it'll take only about a week. The sol-424 drive should eat about 100m of that all by itself, and leave us set up Monday to do a drive-by of a small flat crater we can discern in the orbital imagery.
Since we got the downlink late yesterday afternoon, I spent a few hours working ahead last night, so that when I come in, I'm almost done. Of course, "almost done" in rover-driver terms means "I just need to spend a few more hours obsessing over it." Which is exactly what I do, of course.
Evidently I have a reputation for it. At the CAM, Jeff Favretto asks Cooper -- thisol's RP-2 -- if he wants to go over the sequence in detail. "Or, no," Jeff corrects himself, "you like to just skim through them, right?"
"Yeah," Cooper jokes, "-- unlike Scott, who goes over it and over it until you beg him to stop."
I guess it's not a bad reputation, as these things go. There are worse things you could be known for.
[Next post: sol 446 (Opportunity sol 426), April 5.]
Courtesy NASA/JPL-Caltech. Viking Crater, right where it's supposed to be.
Thisol we're planning two drives, to execute over the weekend. On 423, the first drive takes us from Viking to its twin, Voyager. The rover will take some pictures and then, on sol 424, we'll start an autonav-only drive from Voyager toward Erebus.
Erebus is another 800m away or so. That's a distance that used to take months to cover, but at the rate we're now progressing, it'll take only about a week. The sol-424 drive should eat about 100m of that all by itself, and leave us set up Monday to do a drive-by of a small flat crater we can discern in the orbital imagery.
Since we got the downlink late yesterday afternoon, I spent a few hours working ahead last night, so that when I come in, I'm almost done. Of course, "almost done" in rover-driver terms means "I just need to spend a few more hours obsessing over it." Which is exactly what I do, of course.
Evidently I have a reputation for it. At the CAM, Jeff Favretto asks Cooper -- thisol's RP-2 -- if he wants to go over the sequence in detail. "Or, no," Jeff corrects himself, "you like to just skim through them, right?"
"Yeah," Cooper jokes, "-- unlike Scott, who goes over it and over it until you beg him to stop."
I guess it's not a bad reputation, as these things go. There are worse things you could be known for.
[Next post: sol 446 (Opportunity sol 426), April 5.]
Courtesy NASA/JPL-Caltech. Viking Crater, right where it's supposed to be.
2010-03-30
Opportunity Sol 421 (Spirit Sol 441)
To my surprise, the last time the rover was driven was my last shift. Or rather, that was the last long drive -- they had a 7m bump to get the rover up on a nearby ripple, and they've just been IDDing it since then. Now that I'm back, we're pulling up stakes and moving on.
Well, I've been on the other side of that and didn't like it. But Frank (who's been doing the IDDing, with Jeff) has a good attitude about it, unlike what I had. "Hey, when you guys are driving, I just sit in the back seat and enjoy the ride," he grins. Now why can't I be more like that?
Anyway, the drive is a relatively simple one. We're about 70m from Viking Crater, and they just want us to drive to a position next to it so we can image it on the following sol. Amazingly, we actually have an obstacle to dodge: there's a hollow, like a small crater, about 20m away. From what we can see, it's about 20cm deep, and we can't see its bottom so it's likely a bit deeper. We could probably just drive right through it, but for safety's sake, we treat it as a hazard and drive around. For safety's sake, and, hell, for the sake of variety -- driving across this parking lot has been almost too simple. It's a far cry from driving Spirit, where there are more hazards than dust grains.
The tail end of the drive should leave us in Viking's "ejecta blanket," the rocky field that surrounds a crater. Those are the larger rocks tossed out of the explosion that formed Viking Crater millions of years ago. (The larger rocks naturally aren't thrown as far; they stay close to home.) I want to take advantage of this, so I turn on suspension limits that should stop the drive if we the rover drives onto anything bigger than a brick. That way, we have some chance of ending up with something scientifically interesting right in front of us.
But this choice has another side effect: it creates some uncertainty about the ending position of the drive, which now might end anywhere in the last 10m, where the suspension limits are active. And that means that the post-drive imaging of the crater can't be done by aiming relative to the rover (which is how it was originally sequenced); instead, it will have to be done using site-frame pointing. This will help ensure that we nail the center of the crater, but if we end up with a significant tilt, the whole panorama will be tilted as well, possibly cutting off the sides of the crater. If that happens, it'll be my fault. Oy.
After we're done with this crater, we've got big plans for the weekend. Friday, we're driving from Viking to its twin, Voyager, which we'll also want to image. Then we're driving on, on autonav. Somewhere in there, we're supposed to stop, image a crater, and drive on, all without ground intervention. Which means we need to get that drive just right; otherwise, we won't get images of that mid-drive crater. Or, as Matt puts it, "We'll miss a scientific opportunity." So, no pressure. Brian has a suggestion on what to name that crater if the attempt fails: Scott's Folly. Just how I wanted to be remembered!
It might happen that Scott's Folly will be Someone Else's Folly, though. After another couple of days, I'm going downstairs to work on Spirit for a while, to cover for John Wright's absence. I have a feeling that, very soon, I'm going to miss this parking lot.
[Next post: sol 443 (Opportunity sol 423), April 2.]
Well, I've been on the other side of that and didn't like it. But Frank (who's been doing the IDDing, with Jeff) has a good attitude about it, unlike what I had. "Hey, when you guys are driving, I just sit in the back seat and enjoy the ride," he grins. Now why can't I be more like that?
Anyway, the drive is a relatively simple one. We're about 70m from Viking Crater, and they just want us to drive to a position next to it so we can image it on the following sol. Amazingly, we actually have an obstacle to dodge: there's a hollow, like a small crater, about 20m away. From what we can see, it's about 20cm deep, and we can't see its bottom so it's likely a bit deeper. We could probably just drive right through it, but for safety's sake, we treat it as a hazard and drive around. For safety's sake, and, hell, for the sake of variety -- driving across this parking lot has been almost too simple. It's a far cry from driving Spirit, where there are more hazards than dust grains.
The tail end of the drive should leave us in Viking's "ejecta blanket," the rocky field that surrounds a crater. Those are the larger rocks tossed out of the explosion that formed Viking Crater millions of years ago. (The larger rocks naturally aren't thrown as far; they stay close to home.) I want to take advantage of this, so I turn on suspension limits that should stop the drive if we the rover drives onto anything bigger than a brick. That way, we have some chance of ending up with something scientifically interesting right in front of us.
But this choice has another side effect: it creates some uncertainty about the ending position of the drive, which now might end anywhere in the last 10m, where the suspension limits are active. And that means that the post-drive imaging of the crater can't be done by aiming relative to the rover (which is how it was originally sequenced); instead, it will have to be done using site-frame pointing. This will help ensure that we nail the center of the crater, but if we end up with a significant tilt, the whole panorama will be tilted as well, possibly cutting off the sides of the crater. If that happens, it'll be my fault. Oy.
After we're done with this crater, we've got big plans for the weekend. Friday, we're driving from Viking to its twin, Voyager, which we'll also want to image. Then we're driving on, on autonav. Somewhere in there, we're supposed to stop, image a crater, and drive on, all without ground intervention. Which means we need to get that drive just right; otherwise, we won't get images of that mid-drive crater. Or, as Matt puts it, "We'll miss a scientific opportunity." So, no pressure. Brian has a suggestion on what to name that crater if the attempt fails: Scott's Folly. Just how I wanted to be remembered!
It might happen that Scott's Folly will be Someone Else's Folly, though. After another couple of days, I'm going downstairs to work on Spirit for a while, to cover for John Wright's absence. I have a feeling that, very soon, I'm going to miss this parking lot.
[Next post: sol 443 (Opportunity sol 423), April 2.]
2010-03-23
Opportunity Sol 413 (Spirit Sol 434)
Well, we were hoping for another 200+m drive, but we didn't quite make it. Autonav found some obstacles to avoid -- probably just larger ripples -- and that slowed it down enough that we made only 182m.
"Remember when that used to seem really far?" I ask Jeff rhetorically. "Now it's like, 182 meters -- I'm almost ashamed to show my face."
For a change of pace, thisol we're driving. South. 200m or so. It might be our last chance at a record-length drive before we reach the etched terrain, which the orbital images suggest is coming up fast.
So I get a little worried when our TUL relays the Mechanical team's request to do an engineering checkout drive. She's willing to take time out of the drive to do it, but I want one more shot at the record, as unlikely as it is that we'll make it. So I point out that the engineering checkout sequence hasn't been tested on Earth yet, and that seems to pretty much put the kibosh on it. Instead, the engineering checkout will be delayed until someone else's shift. I'm officially evil now.
As it turns out, we don't end up with enough drive time to set a new record anyhow. With our final allocation, we'll likely be able to cover only 180m or so. Oh, the shame. I'll never be able to show my face around here again.
[Next post: sol 441 (Opportunity sol 421), March 30.]
"Remember when that used to seem really far?" I ask Jeff rhetorically. "Now it's like, 182 meters -- I'm almost ashamed to show my face."
For a change of pace, thisol we're driving. South. 200m or so. It might be our last chance at a record-length drive before we reach the etched terrain, which the orbital images suggest is coming up fast.
So I get a little worried when our TUL relays the Mechanical team's request to do an engineering checkout drive. She's willing to take time out of the drive to do it, but I want one more shot at the record, as unlikely as it is that we'll make it. So I point out that the engineering checkout sequence hasn't been tested on Earth yet, and that seems to pretty much put the kibosh on it. Instead, the engineering checkout will be delayed until someone else's shift. I'm officially evil now.
As it turns out, we don't end up with enough drive time to set a new record anyhow. With our final allocation, we'll likely be able to cover only 180m or so. Oh, the shame. I'll never be able to show my face around here again.
[Next post: sol 441 (Opportunity sol 421), March 30.]
2010-03-22
Opportunity Sol 412 (Spirit Sol 433)
Ah, that's the stuff.
Two hundred nineteen point seven meters.
It's not merely a new record -- we broke the 200m limit, and then some. Steve Squyres calls in to the SOWG meeting to congratulate us. "Two hundred meters, that has to be close to the theoretical maximum for these vehicles," he says. "Pessimist!" I reply.
And not only that, we drove a total of 843m last week, according to Charles. Heck, I'll have to take his word for it -- I lost track. But this is just absolutely amazing stuff, so far beyond what we thought these machines were capable of that it almost seems unreal. We're now only about 650m from the next big target, a pair of craters called "Viking" and "Voyager." At the rate we're going, we'll be there by the weekend. Jeff Favretto brought congratulatory doughnuts again, prompting groans from Saina, who's trying to diet but can't resist the sweets.
Plenty of people are congratulating me, but I almost feel I don't deserve it. Our success is mostly a mix of luck -- nastier terrain would cut these drives short in a hurry -- resources allocated to us by the planning team, and the terrific work done by the rovers' designers and builders. I just happened to be at the wheel, or rather keyboard.
Or maybe I just have a problem with accepting compliments.
In other good news, we got the NCAMs down from the previous drive, and they show James Caird Crater right where it was supposed to be. So we know where we are, or rather where we were 200m ago.
219.7m ago.
Downstairs, DS-1 and MRO continue to invade our space (so to speak), and this time, it's personal. MRO put up a huge sign just across from the 5th-floor elevators. The sign masks the words "Mars Exploration Rover," which have been painted on the wall for over a year. And still are, I suppose, though you can't see them. I know this must have been worked out with MER management already, but I'm still offended by it at first. I take my petty revenge by being exaggeratedly condescending in my mind: "Yes, their little project is important, too. Oh, what's that you've got there, a little orbiter? Yeah, that's creative, never been done before. I'm sure you'll be very proud of it." By the time I reach the sequencing room, I'm laughing to myself about the whole thing.
For the past week or longer, we've been seeing these dark smudges on the horizon. Justin Maki has a theory about what they are: he thinks they're Viking and Voyager craters, with the much larger Erebus beyond. Funny, we didn't expect to see them at all, much less for them to show up the way they are. But heck if he doesn't seem to be on to something. He and Tim Parker go off and start drawing some lines on maps, and it turns out the smudges are at just the right azimuths to be those features. That's not proof, yet, but it shows where the smart money would bet. Justin is smart.
Still more good news: we have an even longer drive time thisol. The range data for planning the drive isn't as good as on previous sols, so we have to cut the blind drive a little shorter, but the total time is more than ever before, so the additional autonav time will roughly balance. My guess is that this drive will compare with the record-busting weekend drive. I mention to Saina that we might possibly set a new record, which means more doughnuts. "Please, no!" she begs.
Yet there's some bad news to mix with the good. Spirit's RAT has started acting funny, bouncing dangerously as it grinds. This can break the IDD, so they've precluded any further use of the instrument until they can figure out what's going on. The leading candidate theory is simply that the RAT's drill bit has worn entirely away, which would mean no more drilling for that rover. (They could still use it to brush off surface dust, but no more seeing beneath a rock's surface.) If that's confirmed, Spirit's RAT would be the first instrument to fail permanently on that rover. (Perhaps joining Opportunity's MTES, though the final chapter in that story is not yet written.) It's sad to see these capabilities disappear, a depressing preview of what must someday happen to the vehicles themselves.
All good things must come to an end, I know. But not yet, damn it! Not yet!
Two hundred nineteen point seven meters.
It's not merely a new record -- we broke the 200m limit, and then some. Steve Squyres calls in to the SOWG meeting to congratulate us. "Two hundred meters, that has to be close to the theoretical maximum for these vehicles," he says. "Pessimist!" I reply.
And not only that, we drove a total of 843m last week, according to Charles. Heck, I'll have to take his word for it -- I lost track. But this is just absolutely amazing stuff, so far beyond what we thought these machines were capable of that it almost seems unreal. We're now only about 650m from the next big target, a pair of craters called "Viking" and "Voyager." At the rate we're going, we'll be there by the weekend. Jeff Favretto brought congratulatory doughnuts again, prompting groans from Saina, who's trying to diet but can't resist the sweets.
Plenty of people are congratulating me, but I almost feel I don't deserve it. Our success is mostly a mix of luck -- nastier terrain would cut these drives short in a hurry -- resources allocated to us by the planning team, and the terrific work done by the rovers' designers and builders. I just happened to be at the wheel, or rather keyboard.
Or maybe I just have a problem with accepting compliments.
In other good news, we got the NCAMs down from the previous drive, and they show James Caird Crater right where it was supposed to be. So we know where we are, or rather where we were 200m ago.
219.7m ago.
Downstairs, DS-1 and MRO continue to invade our space (so to speak), and this time, it's personal. MRO put up a huge sign just across from the 5th-floor elevators. The sign masks the words "Mars Exploration Rover," which have been painted on the wall for over a year. And still are, I suppose, though you can't see them. I know this must have been worked out with MER management already, but I'm still offended by it at first. I take my petty revenge by being exaggeratedly condescending in my mind: "Yes, their little project is important, too. Oh, what's that you've got there, a little orbiter? Yeah, that's creative, never been done before. I'm sure you'll be very proud of it." By the time I reach the sequencing room, I'm laughing to myself about the whole thing.
For the past week or longer, we've been seeing these dark smudges on the horizon. Justin Maki has a theory about what they are: he thinks they're Viking and Voyager craters, with the much larger Erebus beyond. Funny, we didn't expect to see them at all, much less for them to show up the way they are. But heck if he doesn't seem to be on to something. He and Tim Parker go off and start drawing some lines on maps, and it turns out the smudges are at just the right azimuths to be those features. That's not proof, yet, but it shows where the smart money would bet. Justin is smart.
Still more good news: we have an even longer drive time thisol. The range data for planning the drive isn't as good as on previous sols, so we have to cut the blind drive a little shorter, but the total time is more than ever before, so the additional autonav time will roughly balance. My guess is that this drive will compare with the record-busting weekend drive. I mention to Saina that we might possibly set a new record, which means more doughnuts. "Please, no!" she begs.
Yet there's some bad news to mix with the good. Spirit's RAT has started acting funny, bouncing dangerously as it grinds. This can break the IDD, so they've precluded any further use of the instrument until they can figure out what's going on. The leading candidate theory is simply that the RAT's drill bit has worn entirely away, which would mean no more drilling for that rover. (They could still use it to brush off surface dust, but no more seeing beneath a rock's surface.) If that's confirmed, Spirit's RAT would be the first instrument to fail permanently on that rover. (Perhaps joining Opportunity's MTES, though the final chapter in that story is not yet written.) It's sad to see these capabilities disappear, a depressing preview of what must someday happen to the vehicles themselves.
All good things must come to an end, I know. But not yet, damn it! Not yet!
2010-03-19
Opportunity Sol 409 (Spirit Sol 430)
You want the good news first, or the bad news?
The good news is, we set another record: 190m.
The bad news is ... huh, there is no bad news. It's all good news.
Jeff was the first to announce the results. He got up way early (or stayed up way late) and sent out a triumphant email at 5:45. I was up to see it at about 6. And neither of us is exactly a morning person.
To congratulate us, Jeff Favretto -- our mission manager that sol and this -- brings in doughnuts, with a congratulations message hand-written in black marker on the top of the pink box. Charles has the honor of announcing the record as the Spirit SOWG meeting is breaking up to make room for Opportunity, and there is much rejoicing. "So that's where the dust devils are coming from," one of the scientists jokes.
It couldn't be a more appropriate sol for the quote that illustrates our rear-looking track image in the Long-Term Planning report, which (as usual) leads off our SOWG meeting. It's a Dakota proverb: "We will be known by the tracks we leave."
The drive should have left us just next to a crater which, yesterday, Ray dubbed "James Caird Crater." (Caird, a whaler, was an important part of Shackleton's expedition. Caird was a Scotsman, which makes him all right by me.) The problem is, we can't see this crater in the images we have. The minimal downlink only gave us forward-looking images, and the crater would be to the west or southwest of us. That crater was our only landmark, meaning that, in a sense, we have no idea where we are.
Perhaps I shouldn't be so dramatic: we have some idea of where we are. Probably a pretty good idea. We'd just like to be able to prove it.
We took NCAM images after the drive; when they make it to the ground this weekend, we should be able to see James Caird Crater in them. Or so we hope. But it seems a shame to be sitting just a few meters from what looks in the orbital imagery like a pretty darn nifty crater, and not get some good images of it. Plus, solid images with good range data would help localize us really well. So, in the back row, Justin Maki and I start scheming. But to little avail: in the end, we fail to talk the science team into spending the bits on it. They might catch it in a PCAM panorama they're taking before we drive on, but the panorama is probably aimed too high (this one needs to be, for science reasons). So we might never see the crater at all.
Well, that's a shame. But the upside is, if indeed we don't see it, it will be because we drove away. Really, really far away. In order to help avoid a repeat of Spirit's squeaky-wheel problem, we're letting the rover rest Friday (today) and Sunday. But Saturday, we're driving with a vengeance. We would have driven a little farther than 190m on the previous drive, but autonav found some obstacles to avoid early on and lost a little time edging around them. Assuming that doesn't happen again this weekend, we'll break 200m this time for sure.
Once the drive sequence is in the bag, Jeff and I take a little time to brainstorm ways to drive even farther on future sols. We don't see a lot to optimize in the current approach -- all the parameters are cranked up as far as is safe, and the general structure of the drive sequences is about as good as it can get. We blind-drive as far as we safely can given the imagery we've got -- lately, this has been about 100m -- and then we switch to the much slower autonav. So the most obvious way to squeeze more time out of a drive is to extend the length of the blind segment.
But we can't, because we're driving as far as our stereo imaging allows. Unless -- "Hey," I say, "what if we did long-baseline stereo? Take an image-pair in the drive direction, drive laterally a few meters, and take another image-pair." I think this is a great idea, but it doesn't seem to set Jeff afire. Turns out they've done this before on the plains, and it didn't seem to improve the data all that much.
Oh, foo. I thought I was being all creative.
[Next post: sol 433 (Opportunity sol 412), March 22.]
The good news is, we set another record: 190m.
The bad news is ... huh, there is no bad news. It's all good news.
Jeff was the first to announce the results. He got up way early (or stayed up way late) and sent out a triumphant email at 5:45. I was up to see it at about 6. And neither of us is exactly a morning person.
To congratulate us, Jeff Favretto -- our mission manager that sol and this -- brings in doughnuts, with a congratulations message hand-written in black marker on the top of the pink box. Charles has the honor of announcing the record as the Spirit SOWG meeting is breaking up to make room for Opportunity, and there is much rejoicing. "So that's where the dust devils are coming from," one of the scientists jokes.
It couldn't be a more appropriate sol for the quote that illustrates our rear-looking track image in the Long-Term Planning report, which (as usual) leads off our SOWG meeting. It's a Dakota proverb: "We will be known by the tracks we leave."
The drive should have left us just next to a crater which, yesterday, Ray dubbed "James Caird Crater." (Caird, a whaler, was an important part of Shackleton's expedition. Caird was a Scotsman, which makes him all right by me.) The problem is, we can't see this crater in the images we have. The minimal downlink only gave us forward-looking images, and the crater would be to the west or southwest of us. That crater was our only landmark, meaning that, in a sense, we have no idea where we are.
Perhaps I shouldn't be so dramatic: we have some idea of where we are. Probably a pretty good idea. We'd just like to be able to prove it.
We took NCAM images after the drive; when they make it to the ground this weekend, we should be able to see James Caird Crater in them. Or so we hope. But it seems a shame to be sitting just a few meters from what looks in the orbital imagery like a pretty darn nifty crater, and not get some good images of it. Plus, solid images with good range data would help localize us really well. So, in the back row, Justin Maki and I start scheming. But to little avail: in the end, we fail to talk the science team into spending the bits on it. They might catch it in a PCAM panorama they're taking before we drive on, but the panorama is probably aimed too high (this one needs to be, for science reasons). So we might never see the crater at all.
Well, that's a shame. But the upside is, if indeed we don't see it, it will be because we drove away. Really, really far away. In order to help avoid a repeat of Spirit's squeaky-wheel problem, we're letting the rover rest Friday (today) and Sunday. But Saturday, we're driving with a vengeance. We would have driven a little farther than 190m on the previous drive, but autonav found some obstacles to avoid early on and lost a little time edging around them. Assuming that doesn't happen again this weekend, we'll break 200m this time for sure.
Once the drive sequence is in the bag, Jeff and I take a little time to brainstorm ways to drive even farther on future sols. We don't see a lot to optimize in the current approach -- all the parameters are cranked up as far as is safe, and the general structure of the drive sequences is about as good as it can get. We blind-drive as far as we safely can given the imagery we've got -- lately, this has been about 100m -- and then we switch to the much slower autonav. So the most obvious way to squeeze more time out of a drive is to extend the length of the blind segment.
But we can't, because we're driving as far as our stereo imaging allows. Unless -- "Hey," I say, "what if we did long-baseline stereo? Take an image-pair in the drive direction, drive laterally a few meters, and take another image-pair." I think this is a great idea, but it doesn't seem to set Jeff afire. Turns out they've done this before on the plains, and it didn't seem to improve the data all that much.
Oh, foo. I thought I was being all creative.
[Next post: sol 433 (Opportunity sol 412), March 22.]
2010-03-18
Opportunity Sol 408 (Spirit Sol 429)
The regular science team has returned from their conference, so Charles is out as SOWG chair (after a damn honorable term of service) and Ray is back.
And thisol is the first sol we're going to seriously shoot for 200m. Apparently, Steve Squyres has been privately campaigning for this. Another person who likes the idea has a pithy reason: "Just to stick it in MSL's eye."
So they give us a heck of a long time to drive: an hour of blind driving followed by three and a half hours of autonav. In the best possible case, this would give us time to drive 235m or so, with 210m being a more likely outcome.
But this judgment is reduced on appeal, as it were. The PCAM team wants to do some damn imaging or something, and then other stuff comes up, and by the time it all settles out, our 3.5 hours of autonav is down to 2.5.
So we won't break 200m thisol, not even by extending the blind drive. But we might come close. And that'll still be pretty darn good.
And thisol is the first sol we're going to seriously shoot for 200m. Apparently, Steve Squyres has been privately campaigning for this. Another person who likes the idea has a pithy reason: "Just to stick it in MSL's eye."
So they give us a heck of a long time to drive: an hour of blind driving followed by three and a half hours of autonav. In the best possible case, this would give us time to drive 235m or so, with 210m being a more likely outcome.
But this judgment is reduced on appeal, as it were. The PCAM team wants to do some damn imaging or something, and then other stuff comes up, and by the time it all settles out, our 3.5 hours of autonav is down to 2.5.
So we won't break 200m thisol, not even by extending the blind drive. But we might come close. And that'll still be pretty darn good.
2010-03-17
Opportunity Sol 407 (Spirit Sol 428)
I have mixed feelings about the result of yestersol's drive. Frank and Brian were given a record-long amount of time to drive -- four hours and fifteen minutes -- and they put it to good use. The new one-sol record is 183m. Great news for the project, but as the now-eclipsed record holder, I can't help feeling a little sad.
And I won't get to regain the title today; they can't spare enough time to drive. We might get 150m or so, though, which used to seem really good but now seems hardly worth spitting on. Since we can't beat the record anyway, I generously accede when they ask for a couple of unusual procedures -- one where we'll stop in the middle of the autonav segment and do some mid-drive imaging (a first, I think, so at least I get to invent something); and another where we have a second, comm-turn-only, drive sequence. Maybe agreeing to this will make me feel like less of a bastard if I decide to refuse them on some sol where we could set the record.
Aw, who am I kidding? I'd never do that.
Charles Budney is the SOWG chair this week. Normally he's a Mini-TES guy, but he's temporarily filling in for a number of absent SOWG-chair types, who are off at some conference. He's been only too happy to let the rover drivers drive, which, as I tell him, is going to make him one popular SOWG chair.
"Just as well," he says, "since it looks like the Mini-TES is dead." This has been the subject of investigation for the last couple of weeks or so. The normally reliable instrument suddenly started returning errors instead of data. The more stuff they've tried, the more their hope has faded. I know Charles in particular has a pessimistic streak, and there's been no formal announcement yet, so I'm still holding out some hope myself (read: "I am still in denial"). The loss of this instrument would be bad, since it helps us determine chemical composition at a distance -- a very powerful tool in our toolkit. I guess we'll see.
Maybe rover death is the somber theme of the day. Downstairs in the sequencing room, Brian Cooper and Justin Maki start talking about what it will be like when one of the rovers dies. "They'll start making fault trees" -- "Pulling people back from other projects" -- "They'll keep trying for about two weeks" -- "Yeah, two weeks is about right, two weeks in denial" -- "They'll think maybe they saw a glitchy signal" -- "It's sad, but you learn a lot about radio science when something like that happens."
"Chris Salvo was saying it's gotten to be like the rovers are right in our back yards," Justin says. "And he's right, it is. But at any minute, they could be a hundred million miles away again. It could happen like that."
What a comforting thought.
In happier news, the dust devil that recently cleaned off Spirit's solar panels seems to have removed the dirt clod from the magnets as well. That's the dirt clod that's been there since the MB picked up a chunk and wiped it off on the magnets, way back during conjunction. I haven't seen the MB lately, but last I saw, a lot of the dirt that had stuck to it was gone as well.
"I need crater names that start with a 'V,'" Charles announces. "These are supposed to be ships of exploration." He doesn't get enough suggestions that start with "V," so he relaxes that requirement and suggestions start flowing faster. Jay Torres suggests "NCC-1701." Cooper suggests "The Minnow," and that one is definitely going to get used.
"We should use that name for one of the craters where we stop and IDD," I say. "Because the target names are so obvious -- Gilligan, the Skipper, ...."
"And if we don't," Charles says, "The Minnow would be lost."
That one almost physically hurt. What did I ever do to him?
And I won't get to regain the title today; they can't spare enough time to drive. We might get 150m or so, though, which used to seem really good but now seems hardly worth spitting on. Since we can't beat the record anyway, I generously accede when they ask for a couple of unusual procedures -- one where we'll stop in the middle of the autonav segment and do some mid-drive imaging (a first, I think, so at least I get to invent something); and another where we have a second, comm-turn-only, drive sequence. Maybe agreeing to this will make me feel like less of a bastard if I decide to refuse them on some sol where we could set the record.
Aw, who am I kidding? I'd never do that.
Charles Budney is the SOWG chair this week. Normally he's a Mini-TES guy, but he's temporarily filling in for a number of absent SOWG-chair types, who are off at some conference. He's been only too happy to let the rover drivers drive, which, as I tell him, is going to make him one popular SOWG chair.
"Just as well," he says, "since it looks like the Mini-TES is dead." This has been the subject of investigation for the last couple of weeks or so. The normally reliable instrument suddenly started returning errors instead of data. The more stuff they've tried, the more their hope has faded. I know Charles in particular has a pessimistic streak, and there's been no formal announcement yet, so I'm still holding out some hope myself (read: "I am still in denial"). The loss of this instrument would be bad, since it helps us determine chemical composition at a distance -- a very powerful tool in our toolkit. I guess we'll see.
Maybe rover death is the somber theme of the day. Downstairs in the sequencing room, Brian Cooper and Justin Maki start talking about what it will be like when one of the rovers dies. "They'll start making fault trees" -- "Pulling people back from other projects" -- "They'll keep trying for about two weeks" -- "Yeah, two weeks is about right, two weeks in denial" -- "They'll think maybe they saw a glitchy signal" -- "It's sad, but you learn a lot about radio science when something like that happens."
"Chris Salvo was saying it's gotten to be like the rovers are right in our back yards," Justin says. "And he's right, it is. But at any minute, they could be a hundred million miles away again. It could happen like that."
What a comforting thought.
In happier news, the dust devil that recently cleaned off Spirit's solar panels seems to have removed the dirt clod from the magnets as well. That's the dirt clod that's been there since the MB picked up a chunk and wiped it off on the magnets, way back during conjunction. I haven't seen the MB lately, but last I saw, a lot of the dirt that had stuck to it was gone as well.
"I need crater names that start with a 'V,'" Charles announces. "These are supposed to be ships of exploration." He doesn't get enough suggestions that start with "V," so he relaxes that requirement and suggestions start flowing faster. Jay Torres suggests "NCC-1701." Cooper suggests "The Minnow," and that one is definitely going to get used.
"We should use that name for one of the craters where we stop and IDD," I say. "Because the target names are so obvious -- Gilligan, the Skipper, ...."
"And if we don't," Charles says, "The Minnow would be lost."
That one almost physically hurt. What did I ever do to him?
2010-03-12
Opportunity Sol 403 (Spirit Sol 423)
Yesterday we planned sols 402 and 403 (which executes today); today we plan the weekend sols, 404 and 405. Khaled could use the experience, so I have him do the bulk of the RP-1 work today.
Which is easy, since he already did most of it yesterday. I spend most of the day struggling with a tricky m4 problem. Don't get me started. But he calls me over to help out with something weird: the INCONs file ("INCONs" is short for "initial conditions" -- it specifies the rover's last known position, which is how it will start the day) shows the rover with the APXS in the wrong position. Instead of being directly on the target, the instrument is laterally offset by about 3.4cm.
On comparing the numbers from that file to the numbers we expected, we find it's offset by the right amount to be an MI poker offset. This term takes some explanation. When we're trying to get good MI images of a rock surface, often we first touch it with the MI's poker, a small, thin, steel rod affixed near the MI's lens. When it senses contact, we back off a known amount, and we can trust that the resulting images will be in good focus.
But the poker doesn't project directly out of the lens, obviously, which leads to a minor problem: when the rock surface is uneven, the poker might contact a position higher or lower than the area we're trying to image. This could lead to out-of-focus images or, in the worst case, we could smash the MI lens while the poker felt nothing.
So in these cases, we do the poker touch differently. First, we slide the instrument over by a known lateral offset, putting the poker where the lens was. Then we touch and retract. Then we slide back, putting the lens directly over the position the poker touched.
And the offset we're seeing in the APXS placement has just the right magnitude to have been introduced by this procedure -- if, for instance, we slid over but didn't slide back. Nothing else we normally do moves the IDD in this exact way, so it has to be some kind of poker-slide problem. And the scientists aren't going to like it: it would mean that some if not all of our instrument readings have been at the wrong location. Basically, we might have wasted one or more entire sols, and we'll have to do them over.
But there's still a puzzle. While the offset has exactly the right magnitude to have been caused by this error, it's not in the direction we would have used in this case. And when Khaled and I look carefully through all the sequences that have executed since the last known-good placement, we can't find the problem in any of them. So we look again, and we still can't find it.
But it has to be that. And I'm quite prepared to believe it is, since we very nearly made exactly this mistake in the last IDD sequence I built; I caught it in time (because I noticed the simulation looked wrong), but we know the problem could bite us.
So it has to be an MI poker-slide problem. But it isn't. What the hell is going on?
Grasping at straws, we reload the INCONs file -- and suddenly the IDD snaps into the right position. Turns out there was an error in the original file, and the Mobility/IDD team corrected it, probably right after we loaded the erroneous version. Everything's fine. We've just wasted half an hour chasing a ghost.
Well, I'll say this: it sure as hell could have been worse.
[Next post: sol 428 (Opportunity sol 407), March 17.]
Which is easy, since he already did most of it yesterday. I spend most of the day struggling with a tricky m4 problem. Don't get me started. But he calls me over to help out with something weird: the INCONs file ("INCONs" is short for "initial conditions" -- it specifies the rover's last known position, which is how it will start the day) shows the rover with the APXS in the wrong position. Instead of being directly on the target, the instrument is laterally offset by about 3.4cm.
On comparing the numbers from that file to the numbers we expected, we find it's offset by the right amount to be an MI poker offset. This term takes some explanation. When we're trying to get good MI images of a rock surface, often we first touch it with the MI's poker, a small, thin, steel rod affixed near the MI's lens. When it senses contact, we back off a known amount, and we can trust that the resulting images will be in good focus.
But the poker doesn't project directly out of the lens, obviously, which leads to a minor problem: when the rock surface is uneven, the poker might contact a position higher or lower than the area we're trying to image. This could lead to out-of-focus images or, in the worst case, we could smash the MI lens while the poker felt nothing.
So in these cases, we do the poker touch differently. First, we slide the instrument over by a known lateral offset, putting the poker where the lens was. Then we touch and retract. Then we slide back, putting the lens directly over the position the poker touched.
And the offset we're seeing in the APXS placement has just the right magnitude to have been introduced by this procedure -- if, for instance, we slid over but didn't slide back. Nothing else we normally do moves the IDD in this exact way, so it has to be some kind of poker-slide problem. And the scientists aren't going to like it: it would mean that some if not all of our instrument readings have been at the wrong location. Basically, we might have wasted one or more entire sols, and we'll have to do them over.
But there's still a puzzle. While the offset has exactly the right magnitude to have been caused by this error, it's not in the direction we would have used in this case. And when Khaled and I look carefully through all the sequences that have executed since the last known-good placement, we can't find the problem in any of them. So we look again, and we still can't find it.
But it has to be that. And I'm quite prepared to believe it is, since we very nearly made exactly this mistake in the last IDD sequence I built; I caught it in time (because I noticed the simulation looked wrong), but we know the problem could bite us.
So it has to be an MI poker-slide problem. But it isn't. What the hell is going on?
Grasping at straws, we reload the INCONs file -- and suddenly the IDD snaps into the right position. Turns out there was an error in the original file, and the Mobility/IDD team corrected it, probably right after we loaded the erroneous version. Everything's fine. We've just wasted half an hour chasing a ghost.
Well, I'll say this: it sure as hell could have been worse.
[Next post: sol 428 (Opportunity sol 407), March 17.]
2010-03-11
Opportunity Sol 401 (Spirit Sol 422)
Four hundred sols, baby. That's what I'm talking about!
The drive to Vostok went damn near perfectly, which apparently means the curse is lifted. As we suspected from the previous images, Vostok is large but very shallow, kind of like Shaquille O'Neal. (Just kidding, Shaq!) Over the millennia, the crater has almost completely filled with dust and soil. Now, only a broad ring of broken stone, like a circular garden path tens of meters across, shows where a meteor struck once, untold ages ago. At least some of Mars's ancient wounds have healed.
Frank and Brian spent yesterday starting on some IDDing of outcrop that ended up right under us, just as we were hoping. We'll likely continue that work over the next several days.
That's what we're doing today, for example. We're planning two sols today and another two tomorrow, so that we won't have to work Saturday. I think this is a great approach (heck, I advocated it myself yesterday), but it so happens that the two most complex sols out of those four are the two we're planning today. Well, if the work weren't difficult, they could find someone cheaper than I to do it.
What makes the sols complex is that we're RATting on both of them. I haven't done this for a while and had almost forgotten how much there is to it. RATting is usually part of a broad campaign that involves complex placements with all of the instruments, and this time's no exception, so the sol keeps me hopping.
It does give me a chance to play with a new flight software feature that tests directly for RAT contact with the terrain. It's a vast improvement over the old way we had to do it, which required deliberately generating a fault and some tricky logic to determine the cause of the fault and conditionally clear it. So this is good.
But the large workload and short timescale (it would be a tight sol even if we were planning only one) has a predictable effect: I revert to get-the-fuck-out-of-my-way mode, in which I trade politeness for efficiency. (Ahem.) I've been trying not to do this, but old habits die hard, and I have to admit this one is damned effective. Andy's already worried that he's been scheduling me for too many shifts, so he asks later, "Are you okay? You seemed a little testy earlier." I hope my reassurance suffices, because I want him to schedule me more! More! MORE!
Uh, anyway. What with all the work, I miss my chance to appear in an IMAX movie[1] they're filming downstairs; among other things, they're attempting to re-create the landing and egress. But it's probably just as well: the world doesn't need to see my face on a screen the size of a city block. On a screen the size of a thimble, maybe that's something the world needs to see. But a screen the size of a city block, forget about it.
Tomorrow would be simpler than today no matter what, but once today's in the bag (with an hour to spare, mind you; maybe I really don't need to be such a jackass), Khaled and I get to work on tomorrow's sequences. Khaled hasn't built IDD sequences from scratch yet, and doing it with no time pressure is the right way to do it. He's come along really well, and is turning into a fine rover planner.
But he can't have any of my shifts. More. More! MORE!
Courtesy NASA/JPL-Caltech; mosaic image hosted at Wikipedia. This is the entire 360-degree NAVCAM panorama, including (centrally) the broad ring of outcrop outlining what's left of Vostok Crater.
Footnotes:
[1] This, of course, was Roving Mars.
The drive to Vostok went damn near perfectly, which apparently means the curse is lifted. As we suspected from the previous images, Vostok is large but very shallow, kind of like Shaquille O'Neal. (Just kidding, Shaq!) Over the millennia, the crater has almost completely filled with dust and soil. Now, only a broad ring of broken stone, like a circular garden path tens of meters across, shows where a meteor struck once, untold ages ago. At least some of Mars's ancient wounds have healed.
Frank and Brian spent yesterday starting on some IDDing of outcrop that ended up right under us, just as we were hoping. We'll likely continue that work over the next several days.
That's what we're doing today, for example. We're planning two sols today and another two tomorrow, so that we won't have to work Saturday. I think this is a great approach (heck, I advocated it myself yesterday), but it so happens that the two most complex sols out of those four are the two we're planning today. Well, if the work weren't difficult, they could find someone cheaper than I to do it.
What makes the sols complex is that we're RATting on both of them. I haven't done this for a while and had almost forgotten how much there is to it. RATting is usually part of a broad campaign that involves complex placements with all of the instruments, and this time's no exception, so the sol keeps me hopping.
It does give me a chance to play with a new flight software feature that tests directly for RAT contact with the terrain. It's a vast improvement over the old way we had to do it, which required deliberately generating a fault and some tricky logic to determine the cause of the fault and conditionally clear it. So this is good.
But the large workload and short timescale (it would be a tight sol even if we were planning only one) has a predictable effect: I revert to get-the-fuck-out-of-my-way mode, in which I trade politeness for efficiency. (Ahem.) I've been trying not to do this, but old habits die hard, and I have to admit this one is damned effective. Andy's already worried that he's been scheduling me for too many shifts, so he asks later, "Are you okay? You seemed a little testy earlier." I hope my reassurance suffices, because I want him to schedule me more! More! MORE!
Uh, anyway. What with all the work, I miss my chance to appear in an IMAX movie[1] they're filming downstairs; among other things, they're attempting to re-create the landing and egress. But it's probably just as well: the world doesn't need to see my face on a screen the size of a city block. On a screen the size of a thimble, maybe that's something the world needs to see. But a screen the size of a city block, forget about it.
Tomorrow would be simpler than today no matter what, but once today's in the bag (with an hour to spare, mind you; maybe I really don't need to be such a jackass), Khaled and I get to work on tomorrow's sequences. Khaled hasn't built IDD sequences from scratch yet, and doing it with no time pressure is the right way to do it. He's come along really well, and is turning into a fine rover planner.
But he can't have any of my shifts. More. More! MORE!
Courtesy NASA/JPL-Caltech; mosaic image hosted at Wikipedia. This is the entire 360-degree NAVCAM panorama, including (centrally) the broad ring of outcrop outlining what's left of Vostok Crater.
Footnotes:
[1] This, of course, was Roving Mars.
2010-03-09
Opportunity Sol 399 (Spirit Sol 420)
Yestersol's drive seems to have ended more or less where we meant it, about 35m from Vostok. But we have a goal error.
While Frank deals with the SOWG, I look into why we got the error. It turns out we didn't leave quite enough time for the worst-case end-of-drive housekeeping, and the worst case occurred yestersol. We timed out at the very, very, very end of the drive, a few seconds before we finished turning for comm. Literally another couple of seconds would have made the difference.
Oh, well. We're where we meant to be, and facing the right way, with only a little bit of egg on our faces. Thisol's drive is so simple that Frank effectively has it finished before the SOWG meeting ends: turn to 150deg and drive about 35m to Vostok, and that's about it. If it goes how it's supposed to, we'll end up with Vostok right under our wheels. If it goes the way the other recent drives have gone -- hell, we'll probably end up in the Pacific Ocean.
While Frank deals with the SOWG, I look into why we got the error. It turns out we didn't leave quite enough time for the worst-case end-of-drive housekeeping, and the worst case occurred yestersol. We timed out at the very, very, very end of the drive, a few seconds before we finished turning for comm. Literally another couple of seconds would have made the difference.
Oh, well. We're where we meant to be, and facing the right way, with only a little bit of egg on our faces. Thisol's drive is so simple that Frank effectively has it finished before the SOWG meeting ends: turn to 150deg and drive about 35m to Vostok, and that's about it. If it goes how it's supposed to, we'll end up with Vostok right under our wheels. If it goes the way the other recent drives have gone -- hell, we'll probably end up in the Pacific Ocean.
2010-03-08
Opportunity Sol 398 (Spirit Sol 419)
The good news is, we did a record-setting blind drive on sol 396. The bad news is, we didn't mean to.
The problem stemmed from a mistake in a waypoint command. A waypoint is actually a "waydisc" -- an imaginary circle on the terrain. The rover drives until it's inside the circle, or until it runs out of time trying. The mistake they made yesterday was that they moved the circle away but forgot to make it bigger, so the rover had to drive farther to reach it. What was intended to be a multi-stage 100m drive with increasingly paranoid suspension checks turned into a single-stage 125m drive with relatively generous suspension limits.
That could have been bad. It's not that we mind making more distance than expected, it's that the mistaken command sent the rover beyond what we could see well in the available imagery. If there had been a rover-killing obstacle out there, we could have sent the rover to its doom.
Luckily for us, the terrain the rover was driving over had no such hazards. (Admittedly, such hazards are rare in this terrain. But as Andy says, we're paid to be paranoid. And brother, we earn it.) Indeed, it stopped only because it ran out of time; if it had been given a couple more minutes to drive, it would have reached the circle anyhow, despite its greater-than-intended distance, and proceeded to the next command. That would have compounded the problem.
Frank needs no reminder that he made a potentially dangerous mistake. It's actually worse for him, because he almost caught it. He and Khaled noticed yesterday that something was screwy with the simulation, but they found a different error -- the rover's initial position was set incorrectly -- and thought that explained what they were seeing.
We've all had days like this. Yesterday it was Brian's and Frank's (and Khaled's) turn. I've had my turns, and I don't need another; they can have mine. Because one of these days, we're afraid we're going to come in and have to tell a story a lot like this one, except it will end with, "and that's how we killed the rover." Before the day is out, I have an automated check in place that will catch us if we make that mistake again. But Murphy's law is merciless: the number of potential mistakes seems infinite.
I try to cheer Frank up. "Still, you set a new blind-driving record! And it's one we're never going to break, at least not on MER!"
"Just what I needed," he groans, "my stupidity enshrined for all time in the Guinness Book."
I'm not very good at cheering people up.
Because the rover timed out while striving for the waydisc, the sol-396 drive didn't go as far as intended, and the sol-397 drive (an automatic continuation drive meant to execute Sunday) noticed the leftover error and, properly, didn't go anywhere at all. So we haven't quite made it to Vostok yet; today's just another long drive.
And believe me, we check the waypoint.
The problem stemmed from a mistake in a waypoint command. A waypoint is actually a "waydisc" -- an imaginary circle on the terrain. The rover drives until it's inside the circle, or until it runs out of time trying. The mistake they made yesterday was that they moved the circle away but forgot to make it bigger, so the rover had to drive farther to reach it. What was intended to be a multi-stage 100m drive with increasingly paranoid suspension checks turned into a single-stage 125m drive with relatively generous suspension limits.
That could have been bad. It's not that we mind making more distance than expected, it's that the mistaken command sent the rover beyond what we could see well in the available imagery. If there had been a rover-killing obstacle out there, we could have sent the rover to its doom.
Luckily for us, the terrain the rover was driving over had no such hazards. (Admittedly, such hazards are rare in this terrain. But as Andy says, we're paid to be paranoid. And brother, we earn it.) Indeed, it stopped only because it ran out of time; if it had been given a couple more minutes to drive, it would have reached the circle anyhow, despite its greater-than-intended distance, and proceeded to the next command. That would have compounded the problem.
Frank needs no reminder that he made a potentially dangerous mistake. It's actually worse for him, because he almost caught it. He and Khaled noticed yesterday that something was screwy with the simulation, but they found a different error -- the rover's initial position was set incorrectly -- and thought that explained what they were seeing.
We've all had days like this. Yesterday it was Brian's and Frank's (and Khaled's) turn. I've had my turns, and I don't need another; they can have mine. Because one of these days, we're afraid we're going to come in and have to tell a story a lot like this one, except it will end with, "and that's how we killed the rover." Before the day is out, I have an automated check in place that will catch us if we make that mistake again. But Murphy's law is merciless: the number of potential mistakes seems infinite.
I try to cheer Frank up. "Still, you set a new blind-driving record! And it's one we're never going to break, at least not on MER!"
"Just what I needed," he groans, "my stupidity enshrined for all time in the Guinness Book."
I'm not very good at cheering people up.
Because the rover timed out while striving for the waydisc, the sol-396 drive didn't go as far as intended, and the sol-397 drive (an automatic continuation drive meant to execute Sunday) noticed the leftover error and, properly, didn't go anywhere at all. So we haven't quite made it to Vostok yet; today's just another long drive.
And believe me, we check the waypoint.
2010-03-06
Captain's Log, Supplemental
I'm proud and humbled and thrilled and honored to say that this blog was presented with the 3rd Spirit Award from UnmannedSpaceflight.com today. As I've posted there in my thank-you to the members of that excellent forum, I'll try to continue to earn it.
I'm also going to try to figure out how it's possible to be both proud and humbled at the same time. It doesn't seem like one could be. But I am.
I'm also going to try to figure out how it's possible to be both proud and humbled at the same time. It doesn't seem like one could be. But I am.
2010-03-03
Opportunity Sol 394 (Spirit Sol 414)
Cindy Oda's one of the nicest people I know, and she's going through a really rough time. Her mom's teeth are in really bad shape, and they're worried about her getting a bad infection. But they can't put her under to fix it because of some medication she's taking, and they're afraid to take her off it. Meanwhile, her husband, Jeff -- one of the other RPs -- has had to fly back to Illinois to deal with his own ailing dad, who has a heart infection or something like that.
So they're unhappy. Things like this make me almost feel bad about how well things are going in my life right at the moment.
Meanwhile, uh, things are going well in my life. (Sorry.) We're doing another uber-drive thisol. Actually, it's more of a mini-uber-drive -- no autonav. If we trip the suspension limits, we'll just have to stop.
I spent some time last night working out a prototype drive, which of course Khaled and Brian and I end up hacking up somewhat. Last night I didn't have time to analyze the past drives, so I wasn't really sure what limits to specify for this one. As the day wears on, Khaled and I keep thinking of something else we should check, look at the data, and interrupt Brian to tweak something else in the sequence. Tilt, bogie/differential limits at various points -- all of it comes up for review.
What we saw during the drive to Trieste was a tilt high-water mark of 11.5 degrees -- very close to the 12-degree limit that would have stopped the drive altogether. Both the tilt and suspension limits for that drive were higher than we've seen lately, which makes sense: we've been driving south, along the ripples, but the Trieste drive took us due west, straight across the ripples. So we were climbing up and down the ripples all the way. It's a wonder Opportunity didn't get seasick.
Because this drive took us somewhat off our intended southerly course, we're driving back to Vostok at a southeasterly angle. This is the worst of both worlds, as far as expected vehicle articulation is concerned. Not only will our tilt vary just as on the Trieste drive, as we climb up hill and over dale, but also our suspension will articulate more when we drive across the dunes at an angle than when we drove directly over them. This is unfortunate, as it means we may have to choose suspension limits that are hard to distinguish from what would happen if we encountered a real obstacle.
And that's a problem, because our drive-direction imaging is none too good. Usually we have good range data out to at least 80m or so, but thisol it peters out at 65m, and that's if you're being generous. Beyond that, we can't reliably determine a hazard from here, and if we can't tell the vehicle what one looks like in a way that distinguishes it from a ripple, we're in trouble.
In the end we decide, of course, to play it safe. We use more-generous-than-usual limits for the part we can see well -- that is, the part up to 65m -- and the usual paranoid limits beyond 65m. We'll probably make between 35m and 65m of progress, and then we'll have a data point that will be useful for planning the next drive. The next drive is going to be a weekend uber-drive series that should take us all the way to Vostok, or mighty close, so this will be very useful information to have.
Before Opportunity can start her drive to Vostok, though, she has to make it out of the crater. ("Crater," hah. Next to Eagle and Endurance, Naturaliste is a dimple.) Yesterday, when Frank and I were working ahead, I initially sequenced this egress using visual odometry. The strategy was a tried-and-true one: have Opportunity use VO to keep careful and highly accurate track of her progress, and stop when she's out.
Then we came up with the idea of using new R9.1 functionality[1] that can check for tilt. This seemed to let us express more nearly what we wanted -- we could have Opportunity drive until she's flat, assume that means she's out of the crater, and then proceed toward Vostok. We liked this better than the first approach, because we were worried the rover would yaw as she backed up, and end up on flat ground but not in the target zone. But if she only checked for tilt, we'd succeed no matter where she ended up, as long as it was flat. So I rewrote the whole thing to do it that way.
But today, when Khaled and I look at the data more closely, we see some problems with that idea. The area immediately behind Opportunity is flat, but the area just behind that has a largish ripple, one it would be easy for Opportunity to end up on. Given that she might be there, what tilt value do we choose to represent a flatness threshold? Too small, and the limit might be tripped because of the ripple, even though she had in fact egressed. Too large, and she might think she was out of the crater even when she wasn't. Neither of those outcomes would bring us joy.
Well, the reason we switched from a position-based approach to a tilt-based approach was that we were worried about yaw. And when we look at what happened on the way in, we see that Opportunity didn't yaw appreciably. Since we're backing straight up, leaving the same way we came in, she probably will do the same on the way out. In other words, yaw isn't likely to be a worry after all.
So I change it all back. Sigh.
Well, one thing I know for sure: if that's the worst thing that happens to me today -- and it is -- I'm damn lucky.
I'm damn lucky.
[Next post: sol 419 (Opportunity sol 398), March 8.]
Footnotes:
[1] That is, new functionality that had been added as part of the recent (R9.1) flight software upgrade.
So they're unhappy. Things like this make me almost feel bad about how well things are going in my life right at the moment.
Meanwhile, uh, things are going well in my life. (Sorry.) We're doing another uber-drive thisol. Actually, it's more of a mini-uber-drive -- no autonav. If we trip the suspension limits, we'll just have to stop.
I spent some time last night working out a prototype drive, which of course Khaled and Brian and I end up hacking up somewhat. Last night I didn't have time to analyze the past drives, so I wasn't really sure what limits to specify for this one. As the day wears on, Khaled and I keep thinking of something else we should check, look at the data, and interrupt Brian to tweak something else in the sequence. Tilt, bogie/differential limits at various points -- all of it comes up for review.
What we saw during the drive to Trieste was a tilt high-water mark of 11.5 degrees -- very close to the 12-degree limit that would have stopped the drive altogether. Both the tilt and suspension limits for that drive were higher than we've seen lately, which makes sense: we've been driving south, along the ripples, but the Trieste drive took us due west, straight across the ripples. So we were climbing up and down the ripples all the way. It's a wonder Opportunity didn't get seasick.
Because this drive took us somewhat off our intended southerly course, we're driving back to Vostok at a southeasterly angle. This is the worst of both worlds, as far as expected vehicle articulation is concerned. Not only will our tilt vary just as on the Trieste drive, as we climb up hill and over dale, but also our suspension will articulate more when we drive across the dunes at an angle than when we drove directly over them. This is unfortunate, as it means we may have to choose suspension limits that are hard to distinguish from what would happen if we encountered a real obstacle.
And that's a problem, because our drive-direction imaging is none too good. Usually we have good range data out to at least 80m or so, but thisol it peters out at 65m, and that's if you're being generous. Beyond that, we can't reliably determine a hazard from here, and if we can't tell the vehicle what one looks like in a way that distinguishes it from a ripple, we're in trouble.
In the end we decide, of course, to play it safe. We use more-generous-than-usual limits for the part we can see well -- that is, the part up to 65m -- and the usual paranoid limits beyond 65m. We'll probably make between 35m and 65m of progress, and then we'll have a data point that will be useful for planning the next drive. The next drive is going to be a weekend uber-drive series that should take us all the way to Vostok, or mighty close, so this will be very useful information to have.
Before Opportunity can start her drive to Vostok, though, she has to make it out of the crater. ("Crater," hah. Next to Eagle and Endurance, Naturaliste is a dimple.) Yesterday, when Frank and I were working ahead, I initially sequenced this egress using visual odometry. The strategy was a tried-and-true one: have Opportunity use VO to keep careful and highly accurate track of her progress, and stop when she's out.
Then we came up with the idea of using new R9.1 functionality[1] that can check for tilt. This seemed to let us express more nearly what we wanted -- we could have Opportunity drive until she's flat, assume that means she's out of the crater, and then proceed toward Vostok. We liked this better than the first approach, because we were worried the rover would yaw as she backed up, and end up on flat ground but not in the target zone. But if she only checked for tilt, we'd succeed no matter where she ended up, as long as it was flat. So I rewrote the whole thing to do it that way.
But today, when Khaled and I look at the data more closely, we see some problems with that idea. The area immediately behind Opportunity is flat, but the area just behind that has a largish ripple, one it would be easy for Opportunity to end up on. Given that she might be there, what tilt value do we choose to represent a flatness threshold? Too small, and the limit might be tripped because of the ripple, even though she had in fact egressed. Too large, and she might think she was out of the crater even when she wasn't. Neither of those outcomes would bring us joy.
Well, the reason we switched from a position-based approach to a tilt-based approach was that we were worried about yaw. And when we look at what happened on the way in, we see that Opportunity didn't yaw appreciably. Since we're backing straight up, leaving the same way we came in, she probably will do the same on the way out. In other words, yaw isn't likely to be a worry after all.
So I change it all back. Sigh.
Well, one thing I know for sure: if that's the worst thing that happens to me today -- and it is -- I'm damn lucky.
I'm damn lucky.
[Next post: sol 419 (Opportunity sol 398), March 8.]
Footnotes:
[1] That is, new functionality that had been added as part of the recent (R9.1) flight software upgrade.
2010-03-02
Opportunity Sol 393 (Spirit Sol 413)
I show up a few minutes late for the meeting (acceptable, since I'm RP-2 today), and Frank tells me he's already done. Not because he's a genius (though I'm not saying he's not), but because all we're doing thisol is a simple tool change -- one real command, the rest auto-generated boilerplate. (Just to leave my mark on the sequence, I change the 13.5cm backoff to 13.4cm.) It's going to be a slow day.
But that's okay. I expected that, and came in raring to do tomorrow's work, since we already know what it will be. I come up with the idea of combining the crater egress with the next drive toward Vostok (we had originally planned to do these two things in two sols); Ray likes the idea, so I go ahead and sequence it. One hopes this will mean tomorrow's a slow day as well, but you never know.
We're possibly going to rename RSVP for future missions. I suggest the name "R2" -- a nested acronym for "RSVP 2"; I love that it gives us an obvious mascot. Frank and I like it, at least. We'll see if it flies. Or, rather, drives.
But that's okay. I expected that, and came in raring to do tomorrow's work, since we already know what it will be. I come up with the idea of combining the crater egress with the next drive toward Vostok (we had originally planned to do these two things in two sols); Ray likes the idea, so I go ahead and sequence it. One hopes this will mean tomorrow's a slow day as well, but you never know.
We're possibly going to rename RSVP for future missions. I suggest the name "R2" -- a nested acronym for "RSVP 2"; I love that it gives us an obvious mascot. Frank and I like it, at least. We'll see if it flies. Or, rather, drives.
2010-03-01
Opportunity Sol 392 (Spirit Sol 412)
Even though it's because I'm a dumbass (a patriotic dumbass, but a dumbass all the same), the rock we're IDDing is still named "Normandy." So when Albert Yen asks for suggestions for target names, I immediately suggest "Utah," "Omaha," and "Juno." I forget about "Gold" and "Sword," but no matter; the names stick. We'll be IDDing Omaha.
Jeff's father's very sick, so he has to leave town unexpectedly. Cooper gallantly volunteers to fill in, little realizing that he's already committed on Spirit (an unusual occurrence; no wonder he overlooked it in the schedule). So Jeng fills in for him.
It's a simple IDD day: unstow, then MI and MB Omaha; later, we change tools from MB to APXS. I could just about sequence it in my sleep. Because the generally flat rock surface is slightly uneven at the microscale, I get paranoid about the lateral MI move to take the stereo image, and sequence it in an unusual way -- the way we used to do it, by retracting the IDD farther from the surface before moving it over. In the end I decide this is simply too paranoid even for me, so I put it back to the usual way. Unsurprisingly, this prompts another discussion about the RPs and their relative paranoia. Jeng identifies himself as closer to the less-paranoid Wright/Leger end of the scale than to the Ashitey/Biesiadecki/Maimone end (which, it must be said, is where I locate myself). Since John happens to be here to ask, I ask him whether he considers himself or Chris to be the least paranoid RP.
He laughs, then thinks about it. "I am," he says. "But both of us -- well, both Chris and I feel that the rovers are more capable and robust than is generally acknowledged, let's put it that way."
Good spin. But I realize that even though I put myself on the more paranoid end of the scale, I must have some boldness when it comes to driving. After all, I'm currently the long-distance record holder for both rovers. I can't be all that much of a scaredy-cat.
Jeff's father's very sick, so he has to leave town unexpectedly. Cooper gallantly volunteers to fill in, little realizing that he's already committed on Spirit (an unusual occurrence; no wonder he overlooked it in the schedule). So Jeng fills in for him.
It's a simple IDD day: unstow, then MI and MB Omaha; later, we change tools from MB to APXS. I could just about sequence it in my sleep. Because the generally flat rock surface is slightly uneven at the microscale, I get paranoid about the lateral MI move to take the stereo image, and sequence it in an unusual way -- the way we used to do it, by retracting the IDD farther from the surface before moving it over. In the end I decide this is simply too paranoid even for me, so I put it back to the usual way. Unsurprisingly, this prompts another discussion about the RPs and their relative paranoia. Jeng identifies himself as closer to the less-paranoid Wright/Leger end of the scale than to the Ashitey/Biesiadecki/Maimone end (which, it must be said, is where I locate myself). Since John happens to be here to ask, I ask him whether he considers himself or Chris to be the least paranoid RP.
He laughs, then thinks about it. "I am," he says. "But both of us -- well, both Chris and I feel that the rovers are more capable and robust than is generally acknowledged, let's put it that way."
Good spin. But I realize that even though I put myself on the more paranoid end of the scale, I must have some boldness when it comes to driving. After all, I'm currently the long-distance record holder for both rovers. I can't be all that much of a scaredy-cat.
Subscribe to:
Posts (Atom)
Posts
