Martian Chronicles VII: Better is the Enemy

Martian Chronicles VII:

Better is the Enemy

Follow Martian Chronicles, Parts 1 * 2 * 3 * 4 * 5 * 6 * 7 * 8 * 9 * 10 * 11 *12

Three spacecrafts are now hurtling toward the Red Planet to look for evidence that it might once have been wet enough to sustain life. Orbital projections of where Europe’s Mars Express and the two NASA Mars Exploration Rovers (MER) are right now, can be continuously monitored over their half-year journeys. Experiments performed by the MERs will help to determine whether water might have once existed in volume on the red planet. The two Mars Exploration Rovers are targeting what imagery indicates might have been ancient dry lake beds and other geologically interesting sites in early 2004.

A simulated image of the new Mars rover carrying the Athena science instruments.
Credit: NASA

The Martian Chronicles series gives an inside view of what it takes for scientists to deliver a complex mars mission. The journal entries are from Cornell’s Steve Squyres, the Principal Investigator for the Mars Exploration Rovers’ scientific package called Athena. The chronicles begin sequentially from the beginning of July 1999, four years before launch, and will culminate in the dramatic landing of the twin rovers on Mars in January 2004. The expected mission time roaming the red planet is ninety days, from January to April.

The chronicles include an insider’s view of hardware tests and site selection to problem solving and science planning on the surface of Mars.

December 15, 2001

A long-standing problem with the rover’s arm was solved this week. The arm’s job is to place several of our instruments onto martian rocks. Time on Mars is a precious commodity, so to keep from wasting time we’ll often use the arm at night. But it gets really cold at night on Mars. So the motors on the arm need to have heaters on them, allowing the arm to move properly even when it’s really cold out.

The problem is this: What happens if we make a mistake and leave the heaters on in the daytime? The answer, unfortunately, is that the motors could get overheated — cooked, really — and be permanently damaged. Not good. For awhile there, it looked like maybe we were going to have to use much weaker heaters, which would have been safe but which would not have worked well at night. Instead, someone at JPL came up with a very slick design in which the motors are automatically protected from overheating by a thermostat. If a heater goes on at night then everything is fine, but if we leave it on accidentally in the daytime, the thermostat will automatically shut it off. So we can use the arm at night after all.

Oh yes, and I almost forgot… the Mission System Critical Design Review was a big success!

December 22, 2001

Mars Layers
Surface cloud cover

We’ve made good progress with the latest round of Mini-TES testing. After all the problems we had with Mini-TES 2 not working right when it was tilted, we decided a few weeks ago that we had better pull Mini-TES 1 out of storage and make sure that it was okay. We shipped it up to Santa Barbara (carefully!), and the guys there have spent the last couple of weeks checking it out. It’s fine, as it turns out… everything works as it should. So now we know that both instruments for both rovers will behave themselves properly no matter how we tilt them. We don’t want to fly instruments that won’t work right if the rover is parked on a hill!

January 5, 2002

We’ve got an awful lot of testing to do, but sometimes real-world problems get in the way. This was supposed to be the week that we were going to do some important vibration testing on our Mössbauer spectrometer. It didn’t work out that way. Everything needs maintenance and refurbishment from time to time, including vibration test facilities. We’ve had to postpone tests that we were ready for a couple of times recently because the test facilities weren’t ready for us. It’s easy to get a little impatient when this happens. But that’s why we’ve got margin in our schedule!

January 12, 2002

The whole project is about to shift gears pretty soon, in a very big way. In only about a month, we start what’s called "ATLO"… short for "Assembly, Test, and Launch Operations". ATLO is NASA-ese for the process of putting it together, making sure it works, and shooting it off.

And that’s what we’re about to start doing. For a very long time now we’ve been planning, designing, and building bits and pieces of things. The planning is finished, the design is all done, and lots of bits and pieces are built. So starting next month, we begin to put it all together.

This shift to the start of ATLO is one of the biggest changes that a space project goes through, so of course we have to have another review before we do it! Next up is our "ATLO Readiness Review", where we try to convince a review board (and ourselves) that we’re ready to get on with it. The past week has mostly been spent getting ready for this review, and the next one will as well.

Mars Clouds
Receding winter

After all these years, it’s going to feel good to start actually building these things.

January 19, 2002

It’s always something. This week we discovered another one of those dumb mistakes that could really have gotten us into trouble us if we hadn’t caught it now.

One of the things we have to be very careful about is what’s called "Planetary Protection." For us, Planetary Protection means not contaminating the surface of Mars with bacteria from Earth. There are all kinds of good reasons to be careful about this!

One way to do Planetary Protection is to sterilize the hardware we fly, and we do that with much of what we build. But some things, like electronics, are not easy to sterilize. So the other thing we do sometimes is clean our hardware carefully, and then seal it up really tight so nothing bad can possibly get out.

Of course, it’s not that simple. Everything is full of air when we launch from Florida, and the air has to get out as we ascend into space. We can’t let any bacteria leak out with the air, so we put special filters in place that let the air out while keeping the bugs in. And it all works really well.

Mars Layers
Cratering is used to date geological changes

But there’s a problem, as we just learned: These filters don’t just let air through them… they can also let light through them. And we have them in our cameras! Let extra light leak into a camera, and you can take some very bad pictures.

So now we’re designing little "hats" to go over the filters. Air can still get out, but light can’t get in. Problem solved.

But like I said, it’s always something…

January 26, 2002

Murphy’s Law seems to be one of the guiding principles behind space missions, but every now and then something goes right. We had two of ‘em this week.

Several weeks ago, we discovered that we had a problem with Mini-TES number two. The instrument worked fine in most ways, but the results we were getting were more "noisy" than they should have been… a little like too much static in a TV picture. The guys in Santa Barbara tracked this one down and thought that they had fixed it. We did some tests last week, and it turns out that they did more than just fix it… Mini-TES 2 is now even less noisy than Mini-TES 1 is.

The other one had to do with our cameras. For a couple of months now we’ve been wrestling with a problem where the voltage of the electrical power delivered to the cameras wasn’t coming out quite right. We found a fix, but it didn’t look like a very elegant one. Even though it clearly would work, it looked like it could really increase the amount of power it would take to run each camera. And electrical power is really precious on a solar-powered rover! The JPL electronics guys have worked really hard to fix this, and this week we did some tests that showed that in reality the power usage of the cameras will be much lower than we had feared it would be… which means we can use that extra power to take more pictures.

Mars Clouds
Frost covering interior of surface depressions

So this week was a combination of some great design work and a little bit of luck. Occasionally things go your way.

February 2, 2002

We made a major change to our landing site plans this week. A few months ago, we narrowed our choices down to just four places on Mars, and since then all four have been under some really intense study. Unfortunately, that study has turned up what could be a real problem with one of them.

The one that may be dangerous is the one we call Athabasca. It’s a great site, but recently we’ve looked at some radar data that look really scary. You can bounce radar signals off of Mars from Earth, and when you do the signals that come back from Athabasca look like what you’d expect if the surface there were really rough. Not only that, but the roughness seems to be just about the same size as our wheels are. That’s not good!

Mars Layers
Southern pole with dry ice coverage predominantly

To be honest, we’re not 100% certain what the radar data really mean, and we can’t prove beyond a shadow of a doubt that Athabasca’s not perfectly safe. But it looks scary, and we’re not about to take any big chances with these rovers. So Athabasca is off the list, and we’ve replaced it with a site in a place called Isidis. We still have several months before we have to pick the final two sites, and the work continues.

February 9, 2002

We had a great week — Mini-TES 2 passed its vibe test! Vibration testing is one of the crucial events in the life of any piece of space flight hardware. You can read more about vibe testing on our Science Bites page. We had shaken Mini-TES 1 pretty hard some time ago, so we knew that the basic Mini-TES design was solid. This vibe test on Mini-TES 2 was what is sometimes called a "workmanship" test. We already knew it was designed well; this test showed us that this particular Mini-TES is also put together well. The guys in Santa Barbara put it through its paces right after the vibe, and everything checked out fine.

So now it’s on to the next test…

February 16, 2002

Our schedule is so tight that even minor delays seem major, and we had a bunch of them this week. The camera electronics got set back a week because one of the cables in the design was bent a little more sharply than it ought to be, meaning that we had to change things once again to fix it. There’s a minor problem with noise in the APXS electronics that’s had everybody working crazy hours in Germany. And for some weird reason we’ve been having a terrible time lately just trying to ship stuff around. Electronics parts are "in the mail" for what seems like forever on their way to Mainz. Magnets on their way from Copenhagen to LA get hung up in Paris. Calibration targets vanish for a week while being shipped between Arizona and Colorado. It’s maddening. Nothing’s gotten lost permanentely, and if time weren’t so precious, stuff like this wouldn’t matter at all. In fact, the delays really haven’t been all that serious. But when something’s moving as fast as this project is, it can seem like the rest of the world is going in slow motion.

Mars Clouds
Seasonality over Martian sols

February 23, 2002

This is it. On Monday, February 25th, we’re about to begin ATLO: Assembly, Test, and Launch Operations for the rovers.

Every space project has several critical phases in its life, and ATLO is one of them. ATLO is the process of actually putting the flight hardware together, making sure it works, and shooting it off to wherever it’s going. For us, ATLO starts now.

It’s a funny feeling. On the one hand, we barely feel ready for it. Time is precious, and a few more weeks to get better prepared for a job this difficult would be nice. On the other hand, we’ve been waiting for this for so many years, it feels great to finally be doing it! These rovers have been in the planning stages — and have been at the very center of our professional lives — for more than six years now. And seeing the real flight hardware finally start coming together after all that time is a thrill that far outweighs whatever nervousness we might have.

March 2, 2002

Mars Layers
Martian potato-shaped moon, Phobos, from orbit

We had what looked like a pretty serious problem with our Mössbauer Spectrometers this week. We’re now doing thermal testing, which means making sure that the instruments work properly at the temperatures that we’ll actually see on Mars. The Mössbauers were behaving fine at room temperature, but when they got cold they started doing what we call "ringing". Each Mössbauer has a little drive system inside it that vibrates back and forth about 20 times a second. It’s supposed to vibrate in a controlled way, but when it "rings" it’s sort of like what happens when you put a microphone near a speaker and get feedback… it vibrates out of control. And if that happens, we don’t get Mössbauer data.

At mid-week it looked like we might have a pretty bad problem, but the guys in Germany — Goestar and Bodo and Daniel — have really hunkered down and worked hard on it over the past few days and nights. They traced the problem to one part in the electronics, and with a change to that part one of the two instruments is now behaving itself down to temperatures as cold as -120 C. We still have to finish the testing and then fix the other one (once everybody catches up on some sleep!), but right now things are looking pretty good.

March 9, 2002

It’s crunch time for the Mössbauer Spectrometer. The nasty "ringing" problem we had last week has definitely been solved, so it looks like we dodged that bullet. The latest problem is that one of the Mössbauer electronics boards — the part of the instrument that lives inside the rover body — is getting a little flakey on us when we run it at cold temperatures. It doesn’t look like a serious problem, but it’s going to take a little time to fix it.

The Mössbauer sensor heads, though — the parts that go out at the end of the rover’s arm — are ready to go. So we’re going to deliver the instruments to JPL in two parts. This week Goestar is going to come over from Germany with the sensor heads, and spend the week at JPL putting them through a bunch of tests. We’re going to shake them as hard as they’ll be shaken during launch, whack them as hard as they’ll get whacked when we land, vibrate them like they’ll vibrate when the RAT is working, and so on. And once that’s done, the sensor heads will be officially ready to go to Mars.

Mars Clouds
Erosion down the skirt of a runoff channel

Meanwhile, back in Germany, the rest of the guys will be running down the last of the electronics problems. So then, a few weeks from now, we’ll get to do the same thing all over again with the electronics boards.

March 16, 2002

This was Mössbauer week at JPL, and it was pretty wild. Goestar and Bodo brought over the sensor heads for both instruments. (The sensor head is the part that goes out on the rover’s arm.) The big job for the week was to put them through vibration tests that simulate a rocket launch, and other tests that simulate the jolt of an airbag landing on Mars.

These tests are quite something to watch. It’s a lot like what they say about being a pilot: hours of boredom interspersed with moments of sheer terror. The hours of boredom are all the work of setting up for each test, and the moments of sheer terror come when your precious hardware gets subjected to such violent treatment.

Mars Layers
Martian atmosphere and weather awaits the Exploration rovers

The uncanny thing about a vibration test is how much it sounds like a rocket launch. Of course, that makes perfect sense when you think about it. The vibration facility is really nothing other than a gigantic loudspeaker, and the electronics that make it work are nothing other than a gigantic synthesizer. And the song that the synthesizer was playing last week was simply the sound of a Delta rocket heading for space.

We passed nearly all of the tests. Both instruments passed the landing test, and one of them passed the vibration test. The other one had an electronics board that wasn’t quite fastened down right, and it shook loose during vibe. This is why you do tests! We’ll fix it up and test it again in a few weeks.

March 23, 2002

Our landing site situation just changed again. Sometime in the next couple of months we were due to recommend our two final landing site choices to NASA. But now we’ve had a delay, and for once a delay in our schedule is a good thing.

The longer we can wait to pick our landing sites, the more we’ll know about Mars. The Mars Odyssey spacecraft is just beginning to collect data at Mars, and it has started to return some fantastic new data for some of our possible sites. There has been very little time to study the data, and there’s much more data to be taken. So if we can give Odyssey some more time, we’ll know a lot more about Mars when the time comes to choose.

And it turns out that we have some more time. For months we have been very worried that the mass of our spacecraft might be too high. But lately, as we have gotten more and more hardware built and weighed, that has become less of a worry. So with a little breathing room now, we’ve realized we can add a little bit more rocket fuel to the design. And with that extra fuel aboard, we can now target each spacecraft to land just about anywhere we want on the planet. The bottom line is that instead of having to pick our sites soon and stick with them, we’ve got almost another whole year before we have to make up our minds.

Mars Clouds
Clouds over the Tharsis Volcanoes. 13 March 1998 — MOC2-39

March 30, 2002

We had a real good week, for a change. The first piece of good news is that the Mössbauer problem that popped up during vibration testing a couple of weeks ago is easy to fix. Bodo and Goestar have already fixed the one broken instrument in Germany, and the next time they come over they’re going to apply the same fix to the two good instruments already at JPL, just to be on the safe side.

The second piece of good news is that our second Mini-TES is now officially delivered to JPL. If you’re keeping score, that’s three and a half of our twelve payload elements delivered (two Mini-TESes and one and a half Moessbauers — we’ve delivered two Moessbauer sensor heads, but only one of the electronics boards). Lots more to go!

April 6, 2002

Mars Layers
Reticulate geology

We’re working on a little design change to the RAT that should make a big difference in how it works. The Rock Abrasion Tool (a.k.a. "the RAT") is very good at grinding through rock. But all that grinding makes some rock dust, and the dust tends to get all over things. A problem that we’ve been worried about is that there might be some dust left on the rock surface after the RAT is done grinding away at it. Two of our instruments — the APXS and the Mössbauer Spectrometer — wouldn’t be bothered by this. But the Microscopic Imager takes close-up pictures, and those pictures could be pretty useless if there’s lots of dust on the rock.

The old design of the RAT has two grinding wheels. What we’ve found after lots of testing is that the RAT works just about as well with one wheel as it does with two. So we’re keeping one grinding wheel, but considering replacing the other with a brush that whisks away the dust, keeping the surface clean and beautiful. There’s still more testing to be done, but if it works it should make for some great closeup pictures of the insides of martian rocks.

April 13, 2002

When the schedule starts to get tight on a project like this, sometimes you have to start making some tough choices. And our schedule is getting very tight indeed.

There’s a small problem in the electronics of our cameras, and its result is that the pictures are a little bit more "noisy" than we’d like them to be. Imagine a TV picture with some "static" in it. The static is there in our cameras at a level far too low for the eye to see, but we know it’s there.

Being fussy scientists and engineers, we’d love to get rid of that little bit of noise. And we know how to do it. But the changes we’d have to make would take something like five to ten days. And our camera schedule is so tight now that any time we take to make the cameras a little bit better will come out of the time we need for critical testing over the coming months. So we have a choice: a little more noise than we’d like, or five to ten days out of our schedule.

Mars Clouds
Gully-like erosion

There’s a saying in this business that "better is the enemy of good enough". And when the schedule is as tight as ours is, "good enough" starts to look pretty good.

April 20, 2002

So much happened this week that it’s impossible to pick one thing to write about. The electronics boards for the APXS passed their vibration tests in Berlin. We finished fixing the problem with the Mössbauer spectrometer sensor heads that popped up during their vibration tests last month. The motors that we’ve been waiting months for that go in the RAT finally arrived. A problem came up in our camera electronics board that’s going to cost us another two or three week delay that we can barely afford. And we discovered a problem with one of the electronic parts inside Mini-TES 2 that we haven’t figured out how to fix yet. It was quite a week! We’re hoping for a slightly calmer one this week, but we’re not counting on it.

April 27, 2002

We’ve uncovered a very troubling problem with Mini-TES 2: we’re afraid that it might grow whiskers.

Mars Layers
Lava flow or drainage?

What we have learned, just recently, is that one of the electronic parts that we used deep inside Mini-TES 2 was made using some pure tin. And pure tin, it turns out, can do some very strange things. What it does, under certain conditions, is develop microscopic "whiskers"… tiny needle-like growths of metal that protrude out from the tin surface. They’re nasty things to have inside a sensitive piece of electrical equipment. If they grow too far, or if they break off and start rattling around, you can get a short circuit. And then poof, no more instrument.

So what do we do? The thing to do, it turns out, is to open up the part that has the tin in it, and then coat the inside with a kind of spray-on plastic. That’ll help keep tin whiskers from growing, and also will hold any ones that do grow in place so they can’t cause any mischief.

You can imagine that cracking open an instrument that we thought was all done is not exactly high on the list of things we’d like to do! But we have to do it, and Mini-TES 2 is back up in Santa Barbara now, ready to undergo a little minor surgery.

May 4, 2002

The surgery on our Mini-TES 2 instrument is over, and it seems to have gone well. We had to do this to fix the "tin whiskers" problem that I wrote about last week. It was a pretty scary operation. The guys in Santa Barbara had to open up the instrument, and then cut into the steel outer shell of one of the parts on the inside. The critical trick was to keep from getting little metal shavings loose inside the instrument — those could be even worse than the tin whiskers we’re trying to prevent! They cut into it nice and clean, and on Friday they coated the inside of the part with a spray-on plastic that should solve the whisker problem.

We’re not done yet, though. When you do something that severe to a piece of flight hardware, you have to do a lot of testing afterwards to make sure that nothing bad happened. So now we’re in for more vibration testing, more thermal testing, and so forth. We’re not out of the woods yet, but at least the really scary part seems to be over.

May 11, 2002

Good news and bad news on the cameras this week. The good news is that we finally seem to have our electronics boards built. This has been an epic struggle, with one set of boards after another having some kind of problem that has made it impossible for us to fly them. We’ve finally got a good set of boards now — ones that are good enough to go to Mars.

That’s the good news. The bad news is that we’re still having little problems with the electronics. They’re not serious problems, or at least they don’t seem to be. In fact, we’re pretty sure we could fly things just as they are, and still take some very good pictures. But things still aren’t quite right, and "not quite right" is not where we want to be for something as important as our cameras.

Mars Clouds
Mars Exploration Rover A

The problem now isn’t the boards. The problem is somewhere in all the dozens of little electronics parts that go on the boards. We’re going to find it, probably within the next few days. But we’re going to feel an awful lot lot better if we can find it, and fix it, sooner rather than later.

May 18, 2002

We’re finally building cameras! It seemed like this day would never come, but it’s finally here. Most parts of our two cameras — Pancam and the Microscopic Imager, have been coming along fine. The electronics, though, have been giving us fits. We were still troubleshooting problems as recently as a week ago, but Arsham and the rest of the camera electronics guys at JPL have been doing a great job, and now seem to have chased down the last major problems and killed them off.

Mars Layers
Mars Odyssey
Credit: NASA/ JPL

So now it’s time to build a lot of hardware. The first batch of cameras, which are being put together now, will be what we call Engineering Models. They’re essentially identical to the flight cameras, but instead of sending them to Mars we test the heck out of them, to make absolutely certain that the design is sound. We should have them in our hands this coming week. If all goes well, the real flight cameras — the ones that really go to Mars — will start coming together in a little over a week.

May 25, 2002

The big news this week has been all about Mini-TES. The tin whisker problem that we had a few weeks ago now seems to be officially behind us. We had to do a pretty scary repair job on Mini-TES 2 to fix this, but it looks like the fix worked. We just finished putting the instrument through some pretty thorough "thermal vacuum" testing in Santa Barbara, subjecting it to the same kinds of temperatures it will experience on Mars. The news was as good as it possibly could have been — even after the repairs, the instrument works as well as it ever did. So Mini-TES 2 now has a clean bill of health.

Of course, things are never simple. Mini-TES 2 may be healthy, but we’ve known for awhile that we’re also going to have to do a little bit of repair work on Mini-TES 1, and the time for that has come. This isn’t nearly as big a deal as the tin whisker job was; we just have to put one new wire in, to make the design a little safer and more reliable than it is now. But you have to do that sort of thing very carefully, which means a trip to Santa Barbara. So Mini-TES 2 heads back down the coast, Mini-TES 1 heads back up the coast, and we keep sweating just a little bit.

June 1, 2002

Our focus this week has been the mad rush to finish up our first set of APXS instruments and get them delivered to JPL. This is a complicated business, because we have so many instruments to build. On the one hand, JPL needs some instruments right away, to start putting them into the rovers and testing the whole system to see if it works. On the other hand, each APXS that we’re actually going to fly to Mars requires many, many months of testing back in Germany before it’s ready to fly. JPL needs instruments now, but the flight instruments have to stay in Germany until sometime this fall.

So what do we do? We build twice as many instruments. Right now, Rudi and Ralf are scrambling to finish up the first set of instruments, which should go to JPL next week. These will go onto the rovers temporarily. Then, after a day or two to catch their breath, Rudi and Ralf will start assembling the next pair… the ones that’ll actually go to Mars. These will be tested over a long work-filled summer in Mainz, and then get delivered to JPL and swapped with the other ones sometime this fall.

It’s twice as many instruments, and an awful lot of work, but it’s the only way to do it when the schedule is so tight.

Related Web Pages

Mars Vistas: How Earth Will Receive Stunning, High-Resolution Views
JPL Surface Mission
The Pancam Investigation on the NASA 2003 MER Mission
Mars Exploration Rover Homepage