Martian Chronicles IX: Haywires

Follow Martian Chronicles, Parts 1 * 2 * 3 * 4 * 5 * 6 * 7 * 8 * 9 * 10 * 11 *12

Three spacecrafts are now hurtling toward the Red Planet to look for evidence that it might once have been wet enough to sustain life. Orbital projections of where Europe’s Mars Express and the two NASA Mars Exploration Rovers (MER) are right now, can be continuously monitored over their half-year journeys. Experiments performed by the MERs will help to determine whether water might have once existed in volume on the red planet. The two Mars Exploration Rovers are targeting what imagery indicates might have been ancient dry lake beds and other geologically interesting sites in early 2004.

A simulated image of the new Mars rover carrying the Athena science instruments.
Credit: NASA

The Martian Chronicles series gives an inside view of what it takes for scientists to deliver a complex mars mission. The journal entries are from Cornell’s Steve Squyres, the Principal Investigator for the Mars Exploration Rovers’ scientific package called Athena. The chronicles begin sequentially from the beginning of July 1999, four years before launch, and will culminate in the dramatic landing of the twin rovers on Mars in January 2004. The expected mission time roaming the red planet is ninety days, from January to April.

The chronicles include an insider’s view of hardware tests and site selection to problem solving and science planning on the surface of Mars.

October 12, 2002

The MER rovers are fiendishly complicated machines, and literally thousands of things are going to have to go right for this mission to work. All of them have to be tested, and it’s pretty common for things not to work right on the first try. This past week, we’ve been troubleshooting our "step and settle" problem.

The Mini-TES instrument sits at the bottom of a big piece of hardware called the Pancam Mast Assembly, or PMA.

Mini-TES looks up the inside of the PMA tube, using it like a periscope to view the scene around the rover. There are mirrors at the top of the PMA, just like in a periscope, and one of them is supposed to move very quickly as the Mini-TES shifts its gaze from one spot to the next. The time this mirror has to "step and settle" — to move and then settle down so that it’s motionless — is just two tenths of a second. And right now it’s taking a good deal longer than that. If the mirror is still moving when Mini-TES is trying to look out at the world, we get a blurry view and bad data.

The problem seems to be that we’re not giving the motor that moves the mirror enough juice, so we’re going to change that and try it again. We’ll see how it goes. But it’s just one of thousands of things we’ve got to get right before we can take these things to the launch pad.

October 19, 2002

We’re definitely in the troubleshooting phase of the project right now, and for the last couple of weeks annoying little troubles seem to be popping up faster than we can shoot them down. We’re making progress on the step-and-settle problem that came up last week, but we still haven’t solved it yet. And this week a new problem came up on the Mössbauer Spectrometer that we haven’t figured out at all.

The Mössbauer Spectrometer electronics.
Credit: Johannes Gutenberg University

This is a weird one. The Moessbauer has four different "channels": it’s sort of like four instruments rolled into one. When we use it by itself, all four channels work beautifully. But when we hook it up to the rover, only one channel works right. The other three don’t, and that means that we get only one-fourth as much good data as we should.

In principle we can "fix" the problem by changing some software, but we’re not real comfortable with that, at least for now. We don’t understand yet what’s causing the problem, and using a quick-and-dirty solution to fix a problem you don’t understand is rarely the right thing to do. So we’re going to test one idea, and then test another, and keep testing until we get it figured out for real.

It just gets like this sometimes, and when it does you have to remind yourself that none of these problems are major and all of them have solutions. Still, it’ll feel good once we’ve got this batch of problems behind us.

October 26, 2002

Good news: We seem to have solved the "step-and-settle" problem that’s been giving us fits for several weeks. The trouble here was that the mirror at the top of our mast didn’t seem to be moving quickly enough. Our Mini-TES instrument is supposed to tell the mirror what to do. When Mini-TES is ready for the mirror to move, it sends out a signal that means "okay, I’m done taking data for a little while… go ahead and move the mirror." The mirror has two tenths of a second to get to its new position. Mini-TES then sends out a second, different message that says "okay, you’d better be done moving the mirror, because I’m going to start taking data again." And it starts to collect data, assuming that the mirror has stopped moving.

What we thought was going on was that the mirror was moving too slowly, making the data blurry because it was still moving when Mini-TES started to collect data. But what was really going on was that the mirror was ignoring the first message completely, and starting to move when it got the second message! That’ll pretty much guarantee that you’re going to get blurry data, no matter how fast you move the mirror.

Mars Layers
Martian atmosphere and weather awaits the Exploration rovers

We were afraid for a little while that this would be a nasty problem to fix, but Jason Gates found a very elegant solution that involved just putting a couple of tiny "haywires" onto one of the rover’s electronics boards. We still have a little more testing to do, but it looks like one of our unpleasant problems has now been found and fixed.

November 2, 2002

Well, we’re hunting our problems down and killing them off one by one. But some are easier to deal with than others.

Last week we found and fixed the Mini-TES step-and-settle bug. This week, the focus has been on fixing the Mössbauer Spectrometer problem that turned up a couple of weeks ago.

Hunting down a problem like this can be tedious, and you have to be very methodical about it. You might start off with half a dozen different theories about what it could be. Then you have to come up with a test for each theory, and run through all the tests… knocking theories off one by one until you find the culprit.

In this case, the culprit seems to be some rather innocent-looking wires. They’re the wires that run up the rover’s arm, connecting the Mössbauer’s sensor head (which is out on the end of the arm) to the part of the instrument that’s inside the rover. This thing isn’t a normal bundle of wires, though. A normal wire bundle would be too stiff for the arm to bend. Instead, it’s a very special kind of flexible cable. If the flexible cable is in place, the arm bends just fine, but the instrument doesn’t work right. Replace the flexible cable with a normal bundle of wires and the instrument works fine. But we can’t fly it that way, because if we did the arm wouldn’t bend!

So we have to figure out what’s wrong with this flexible cable, and fix it. At least we know where the problem is now.

November 9, 2002

There’s a lot of news this week, both good and bad, but it all pales in comparison to the really big news: We just took one of our rovers for its first drive this week.

This wasn’t the FIDO rover, and it wasn’t an engineering model either. This was the real deal… the MER-2 rover that’s going to Mars. We put it on some blue mats in the Spacecraft Assembly Facility at JPL, and put it through its paces: straight-line drives forward and backward, up and down ramps, turns, and a pirouette-like spin in place.

Extreme Explorers' Hall of Fame
FIDO Rover
Credit: NASA JPL

It’s hard to describe how it felt to watch our first flight rover go for its first drive. It was a very emotional experience. The drive took place five years to the day after NASA first let our science team know that we were going to do this mission. It has been a very long and difficult road since then to get to where we are today.

After the driving was over, it was time for a historic photo op. Check out this picture. It’s a mother-and-child family portrait of JPL Mars rovers, showing both MER-2 and a copy of the famous Sojourner rover from the Mars Pathfinder mission. The MER rovers were built using everything we learned from Sojourner, of course. So it’s the little one in front that’s the mother, and the great big one behind it that’s the brand-new baby.

November 16, 2002

We’re still working what seems like a million problems at once.

We think we’ve solved the problem with the scan mirror for the Mini-TES, which was taking longer to move than it should. We haven’t been able to test the solution yet, though. The APXS has a bug that makes errors pop up in the spectra at random. We’ve found the bug and we know how to fix it, but we haven’t gotten into either rover to do that yet either. We still have a problem with the cable that connects one part of the Mössbauer Spectrometer to the other, and it keeps the instrument from working properly. There are three possible fixes for that one, and we may use any or all of them. And imaging with all of the cameras takes much longer right now than we want it to.

All of this sounds pretty dire, but it’s not. This sort of thing is business as usual on a project as complicated as ours. Every one of these problems is complicated, but every one of them also has a solution. We’ll find them all, and we’ll do everything necessary to make the solutions work.

The idea, of course, is to make it all look easy by the time we get to Mars. But if we do manage to succeed at that a little over a year from now, it’ll be because we’ve overcome all the struggles we’re having now.

November 23, 2002

We dodged a bullet this week. We made a mistake a long time ago, and we just realized it very recently. It was almost too late.

Our Microscopic Imager has to have a dust cover. After all, if you’re going to be waving a sensitive scientific camera around on the surface of the dustiest planet in the solar system, it makes sense to have a cover to keep the lens clean. And the cover has to be transparent. We use a motor to open and close it, and the motor is a very reliable one. But even so, if it fails for some reason, we want to be sure we still can take pictures.

When you use materials in space there are some special things you need to think about. One of them is called "outgassing". Some materials, when they get exposed to the vacuum of space, can outgas — meaning that stuff that would stay totally solid in the Earth’s atmosphere actually evaporates a bit. That can be bad news if the stuff that evaporates condenses again someplace else… like onto the lens of a camera.

Mars Frost
Mesa-like layered geology on Mars

We were dead certain that the transparent material we were making our dust covers out of was immune to outgassing, but we were wrong. We finally did a good test last week just to be on the safe side… with no camera present, of course. And lo and behold, a bunch of crud evaporated off of the cover material and condensed again right next to it, making a real mess. If that happened in flight, we’d have a useless camera on our hands.

So just remove the cover and change to another material, right? Not so simple. Right now the dust cover is on the camera, the camera is on the MER-2 rover’s arm, and the rover is all folded up and almost ready to go into a vacuum chamber for a test! Problem is, when the rover is folded up like that, it takes a couple of days of work to unfold it and get the arm out where you can get at it. And days in the schedule are the most precious thing we have right now. So we really didn’t want to waste two days fixing a problem — especially a problem that never should have happened in the first place.

And then a small miracle happened. Lori Shiraishi and a couple of the other mechanical wizards at JPL looked real hard at the thing, and somehow managed to come up with a special tool that let them snake their way in there, get to the cover, and get it off the camera. If they weren’t mechanical engineers, I think they’d be safe crackers. So the cover is off now, and we can go ahead and put the rover into vacuum chamber and test it safely. In the meantime, we can build some new dust covers. And you can bet they’ll be made of something that we know doesn’t outgas!

November 30, 2002

We’ve just made our last major design decision, and now our RATs have teeth. We settled most of the design details for our Rock Abrasion Tool (RAT) a long time ago. In fact, the two flight RATs are built, tested, and at JPL. But we’ve left one thing open for a long time, and that’s the exact design of the "business end" of the RAT… the grinding heads that will actually contact the rock and grind their way into it.

We’ve known pretty much since the start that we were going to make the grinding heads using diamonds, since diamonds are the hardest materials we can possibly use. But finding the best way to use diamonds has been a very long research project. ("Diamonds are forever", as we put it.) We’ve tried grinding heads encrusted with tiny diamonds. We’ve tried big single diamonds. We’ve tried diamonds coated with nickel. All of these have worked, but they don’t all work equally well. We have no idea how hard martian rocks are going to be, so we have to find the material that works the very best.

And now we seem to have found it. The best grinding heads of all have been ones that are made of a hard resin with lots of fine diamond grit mixed in with it. The great thing about these bits is that they sharpen themselves. Even diamonds wear out after awhile. But the way this resin works, it’s strong enough to hold the diamonds in place only for awhile. Then, after they’ve been used awhile the worn diamonds pull out and the resin wears away… exposing fresh, sharp diamonds underneath. After a lot of testing, we’ve built about a dozen of these bits, and we’ve now shipped them out to JPL to go into the two flight RATs.

December 7, 2002

This week it’s been all about preparation for one of the biggest tests we have in front of us before we launch.

Mars slope
Rugged terrain on Mars

The motto you try to follow in this business is "test as you fly, fly as you test". In other words, test everything on the ground just like you plan to fly it, and then fly it that way. Coming up over the next couple of weeks is one of the most important test-’em-like-we’ll-fly-’em events of the whole MER program. It’s called the "surface thermal vacuum test". In this test we take the whole MER-2 rover, put it into a big space simulation chamber, take the atmosphere down to martian pressure, and take the temperature down to martian temperature. And then we make the rover do just about everything it knows how to do. We can’t drive it, because there isn’t enough room in the chamber. But we do everything else, and that’s a lot. Every instrument gets tested under conditions just like we’ll experience on Mars.

To tell the truth, you’d have to be nuts not to feel just a little nervous before a test as important and complicated as this one. But we’ve been preparing for months, and we think we’re ready. We’ll find out soon.

December 14, 2002

We’re deep into thermal vac with the MER-2 rover now, and so far it’s going great. "Thermal vac" is short for thermal vacuum testing, and it’s one of the toughest tests we have between now and launch.

We started on Thursday with the rover in the test chamber, all folded up the way it’ll be when we land. We took the pressure and temperature in the chamber down to just what they’ll be like on Mars. And then we put the rover to work. Out came the solar panels. Up went the mast and the antenna. The rover stood up, swung its wheels into place, released its arm, and was ready for action.

One of the first things we did once the rover was ready — and one of the first things we’ll do on Mars — was take a whole bunch of pictures. This one shows the Pancam calibration target, also known as the sundial. There are a lot of lights in the test chamber, so the central post doesn’t cast just one shadow here like it will on Mars. But this gives a real sense of what our sundial pictures will look like.

Of course, what we’re really going to Mars to take pictures of is Martian rocks. We didn’t have any Martian rocks to put in the test chamber, but we did cut some nice slabs of a bunch of different Earth rocks and take some pictures of them as well.

Thermal vac isn’t over yet… in fact, in many ways it’s just getting started. We’ll be at it all this coming week too: testing the rover’s arm, taking more pictures, and — especially — putting Mini-TES to work.

December 21, 2002

Thermal vac testing on our first rover is done, and we survived it. We have a very tired team!

Clamshell pod for MER shown in Kennedy storage.
Credit: NASA/KSC

A test like this requires use of a very big, very complicated facility. So you don’t run it just 8 or 12 hours a day… it’s a round-the-clock operation. We’ve all just finished almost two weeks of irregular and very long shifts, but it was worth it. We got some great data, and we sure learned a lot about one of our new rovers.

Over the next few weeks I’ll post here several of the data products that we collected during the test. Here’s a first one for starters. To give us something interesting and challenging to look at in the test chamber, Dick Morris from Johnson Space Center put together a really nice test target with lots of rocks on it. Before we’re done with our testing, we’re hoping to look at every rock on this target with every one of our instruments. The image was taken by Pancam, and shows color close-ups of two of the rocks. We’ve got some spectacular Mini-TES data on this target, which I’ll post next time.

For now, though, I’m going to go sleep for a week!

December 28, 2002

We’re still crunching through all the data that we took during the big MER-2 thermal vacuum test that ended last week. Here’s a picture of a focus target taken by the Microscopic Imager. The whole image is only about 3 centimeters across, which is a little over an inch. And the size of the individual pixels is tiny: about half the thickness of a human hair.

The best thing about this image isn’t how sharp it is. The best thing is how we took it. The Microscopic Imager is on the end of the rover’s arm. First we used a pair of cameras on the front end of the rover to spot this focus target and figure out where it was. Then we used the arm to put the Microscopic Imager where we thought the target was. We snapped a picture and bingo, there was the target. This is just the way we’re going to do it on Mars, and it worked in the test chamber the first time we tried it.

January 4, 2003

For weeks now, we’ve been chasing a nasty problem with our Mössbauer Spectrometer. There may finally be a light at the end of the tunnel.

The problem’s not the instrument… the instrument works just fine when we test it by itself. But the Mössbauer Spectrometer comes in two parts. One part is the "sensor head", which is out at the end of the rover’s arm, and the other is the electronics, deep inside the rover body. The two parts are connected by a long and complicated cable, and it’s the cable that’s been the problem.

The thing you’d want to do is simply replace the cable with one that works better, but that turns out not to be so simple. Half of the cable — the part of it that’s inside the rover — is fairly easy to replace, and we’ve done that already. It helps, but it doesn’t solve the problem. The other half of the cable, which runs up the arm, is very, very difficult to replace. We’d essentially have to take the whole arm apart to do it, and nobody wants to do that to a piece of flight hardware that’s been assembled and tested, and that works beautifully.

So what to do? A team of very talented electrical engineers, both in Germany and at JPL, has been working on that problem for a couple of months now. The solution appears to be to add a tiny little electronics board to the outside of the sensor head. This board improves the signal that runs up the arm enough that the cable can handle it. We tested it this week at JPL with one of our two flight instruments, and it worked. We still have to confirm that it works with the other one, and we also have to make sure that it works at martian temperatures. But the solution to one of our toughest problems now may be in sight.

January 11, 2003

At this stage in a project, it’s all about working out the little details, making sure we’ll be ready to ship the rovers to the Cape when the time comes. And there sure are a lot of details! One of the ones we took care of this week involves our magnet experiment. To do the magnet experiment when we get to Mars, we’ll use the Pancam cameras every few days to take a look at the magnets that are mounted on the rover. Once we see enough martian dust sticking to the magnets, we’ll look at them with the APXS and the Mössbauer Spectrometer to find out what the magnetic stuff in the dust is made of.

It’s simple, but to do that experiment we need to know which direction to point the cameras so that the magnets will be in the field of view. It’d be easy to just wait until the rovers on Mars to figure it out, but time then will be even more precious than it is now. So we swung the cameras around to where we thought the magnets ought to be, and took a few pictures. In the first one, we missed! In the second one we at least managed to get both magnets in the frame. They’re not centered, but it’s good enough for now. We can center them up better once we get to Mars.

And if you see off-center magnet pictures after we land a year from now, you’ll know we didn’t have time to make it pretty, and decided to just go with what we learned this week!

January 18, 2003

There’s been so much news lately that we never got around to posting what’s probably the coolest data product of all from the MER-2 rover testing we did back in December. Here it is. The image on the left is one taken by the rover’s navigation camera, or Navcam. The Navcams are black-and-white cameras, with a wider field of view than Pancam has. On Mars, we’ll use them mostly to figure out where to drive the rovers. In this picture, you can see a special target that was built for this test by Dick Morris, one of our team members from Johnson Space Center in Houston. It’s got a bunch of thin slabs on it, each cut from a different kind of rock.

Mars Frost
Frost in Charitum Montes. 10 June 2003, MOC2-387

The cool thing is the stuff on the right. This is an image of the same target, but put together with data from our Mini-TES instrument. Mini-TES can sense what each rock is made of, and the "false colors" of the pixels in the image show the variation in the composition of the rocks on the target.

But there’s more in the Mini-TES data than you can see in just a simple image. For every pixel in that image, we have more than just a false color… we have a complete infrared spectrum to really tell us in detail what the rock is made of. This plot shows a spectrum for each of the rocks on the target. All those wiggles and squiggles, to a trained spectroscopist, are the distinctive fingerprint of a rock type. We did this as a "blind" test, meaning that the real compositions of the rocks on that target are known only to Dick, and he’s not telling. But with data this good, they won’t be unknown to the rest of the team for long.

January 25, 2003

Time is getting short, and we’re killing off problems as fast as we can. We just got another one.

This was a strange one. During a test we did back in December, something odd happened to our APXS instrument. During a two-hour test, the instrument somehow got reset — that is, it got turned off and back on again — seventeen times. That kind of thing isn’t good, and if it happened on Mars we’d lose data, just like you can lose data on your computer if it gets turned off before you’ve saved something. We couldn’t figure out what was wrong, and it looked like it could be a real problem.

After a lot of sleuthing, the answer became clear. During the same two hours that we were testing the APXS, the rover engineers were testing the rover’s power system. That means they were working with things like the batteries, the electronics that run them, and so forth. And during that time, what we learned was that they had sent seventeen different commands to the power system. That was the clue we needed to solve the mystery. When we dug in deeply, we discovered that there is a nasty bug in the rover design: every time somebody sends a command to the power system, it inadvertently turns the APXS and the Mössbauer Spectrometer off and back on again! It’s such a goofy bug we didn’t believe it at first, but that turned out to be the problem. We’re fixing it with a change to the software, and next month when we do the same test with APXS on the other rover, we’re expecting it to behave itself.

February 1, 2003

The space program is a lot like a family, and our family experienced a terrible loss this week. We’ll return to Mars news next week. For now our thoughts and prayers are with the loved ones of the astronauts who lost their lives in the Columbia tragedy.

February 8, 2003

We passed another big milestone this week — our very last vibe tests. Vibration tests are some of the scariest things you do to space flight hardware. It’s a torture test: You bolt your instrument to a machine that shakes it as hard as the rocket will shake it when you launch it, or even harder. Sometimes the instrument survives the test, and sometimes it doesn’t.

These were the final vibe tests for our flight APXS instruments. We should have done these tests many months ago, of course, but sometimes things don’t work out the way you’d like them to. Several months ago we discovered a very bad mistake that we had made in part of the APXS that detects alpha particles. This "alpha mode", as we call it, is essential for detecting important elements like carbon, and it simply wouldn’t have worked on Mars the way we originally built it. At least we found the problem in time! But it meant that we had to go out and get new and improved alpha detectors, put them in the instrument, and then do all the testing months later than we originally wanted to.

If we hadn’t passed the tests this week, we would have been stuck flying our spare APXS instruments. The spares are okay in most respects, but their alpha detectors won’t work right because of the design mistake we made. So it was with enormous apprehension that we shipped the APXS flight instruments, with the fixed alpha mode included, to Berlin for their final vibe tests. They both passed beautifully, and soon they’ll be ready to go on the rovers. It’s an enormous relief.

Related Web Pages

Mars Vistas: How Earth Will Receive Stunning, High-Resolution Views
JPL Surface Mission
The Pancam Investigation on the NASA 2003 MER Mission
Mars Exploration Rover Homepage