Goldstone, We Had a Problem

Getting Light from the Sun

Coronal Mass Ejection timed to comet NEAT. Credit: SOHO/SWAN and SOHO/LASCO (ESA & NASA)

The problem started on May 4. While passing over one of the smaller (34 meter) dishes of the Deep Space Network, Earth’s main solar monitoring satellite lost its signal.

Called SOHO, the Solar and Heliospheric Observatory had proven to be a huge hit on the internet, as millions could view real-time images of the Sun, discover comets, or check space weather for flares, prominences, or sunspots.

With its twelve instruments, SOHO had also become a crucial part in the scientific plans to study our star. SOHO is a project of international cooperation between ESA and NASA to study the sun, from its deep core to the outer corona, and the solar wind. It was launched in December 1995 on an Atlas IIAS/Centaur rocket. Besides watching the sun, SOHO had become the most prolific discoverer of comets in astronomical history. As of May 2003, more than 620 comets have been found by SOHO.


Blackout for Predictor of Blackouts?

SOHO observations are irreplaceable for scientists who provide advance warning of space storms. Such unpredictable bursts of electromagnetic energy can cripple satellites, if their orbits or positioning are not protected. During SOHO’s blackout periods, space weather forecasting will be set back 20 years, said Joe Kunches, the lead forecaster at NOAA’s Space Environment Center.

With or without warning, recent history had shown how serious the sudden changes in space weather can be. In 1989, a solar storm tripped a protective switches in Canadian Hydro-Québec power company. For nine hours, the entire province of Québec was without power. The problem nearly spread to the United States through an interconnected grid, officials said at the time. In a 1997 solar storm, an AT&T Telestar 401 satellite used to broadcast television shows from networks to local affiliates was blacked out. A more serious breakdown of communications occurred in May 1998 when a space storm disabled PanAmSat’s Galaxy IV. Then, though SOHO was flying, researchers had not yet developed the sophisticated prediction abilities they have today. Among the Galaxy IV casualties: automated teller machines; gas station credit card handling services; 80 percent of all pagers in the United States; news wire service feeds; CNN’s airport network; and some airline weather tracking services.

But when problems started happening in May and June for SOHO, the picture from the Sun took on the character of just such a cascading storm in progress.

Sun and Sand in the Gears

When SOHO’s monitoring computer detected a pointing error, it seemed that its High Gain Antenna (HGA) could not move properly. Used to send pictures and data back to earth, the HGA has a beam about 14 degrees wide and is frequently moved so that it stays centered on Earth. But as SOHO and the Earth go around the Sun, the HGA beam’s position needs to move up and to the side for its periodic adjustments. Therein lied this summer’s problem: SOHO’s High-Gain Antenna could no longer move in a particular direction. A motor or gear assembly was thought to be malfunctioning.

Our dynamic star in background of ancient solar observatory, Stonehenge.

SOHO is located 1.5 million kilometers (one million miles) from Earth. It orbits around the First Lagrangian point, where the combined gravity of the Earth and the sun keep SOHO in an orbit locked to the sun-Earth line. To transmit it high volume of data, the SOHO HGA must rotate to have Earth in its field of view. If the problem was not solved, the Earth would be left outside the HGA beam on a periodic basis of several weeks, with similar blackouts occurring every three months.

In total, the real possibility was an inability to monitor or predict solar storms for about a third of each year.

The anomaly in the HGA was first discovered when engineers detected a discrepancy between the commanded and measured antenna position. In normal conditions, the antenna must be able to move along two axes, vertical and horizontal. The horizontal movement was no longer taking place properly. The last time such a problem wrinkled foreheads was 1998, when a navigation error hindered picture transmissions from the billion dollar observatory. Until SOHO could be restored to its full capabilities two months later, NASA and ESA considered a replacement spacecraft as their only viable alternative, particularly given the proven importance of monitoring the Sun continuously.

Last month as an interim measure to safeguard the spacecraft, its operations were put into a safe-mode until the pointing problem could be looked at in more detail. The science teams began safing their instruments for an off-time of 4-6 weeks. Its low gain antenna, which does not need to be pointed in a specific direction, was used in this protected way to control SOHO, monitor the spacecraft and maintain instrument health. But the pictures were lost.

Sunspots as darkened knots of magnetic energy and rising hot plasma.

To take pictures, the current “trunkline” of solar images depends on a global network of radio dishes, called the Deep Space Network (DSN). To keep continuous data acquisitions, these massive receivers are spread geographically in Madrid, Spain; Canberra, Australia; and Goldstone, California. Smaller dishes bridge the periodic dumping of on-board recorder data, to minimize any losses.

The higher-rate transmissions from the Solar and Heliospheric Observatory (SOHO) were initially interrupted, since these large data sets depended on the movement of the pointing antenna to stay on track. The loss of signal occurred on a 26-meter station of NASA’s Deep Space Network (DSN). Until June 30, 2003, however, the spacecraft continued beaming down its science data, which were successfully picked up by larger 34-meter DSN stations (when available). On June 30, 2003, the 70-meter DSN station in Madrid, Spain, successfully received high-rate science data through SOHO’s omnidirectional on-board low-gain antenna. SOHO normally uses this antenna only for low-rate telemetry in emergencies, and the antenna does not need to be repointed.

Staring at the Sun

European Space Agency (ESA) and NASA engineers asessed several options to recover the situation, or minimize the scientific data loss. The international team of SOHO designers continued to troubleshoot the locked HGA motor. There were several attempts in high speed mode and one attempt in low speed mode to move the antenna by 30 steps, just enough to get another encoder pulse. None was successful.

Solar flares issue strong electromagnetic bursts.

After 8 years in service, the SOHO mission seemed to be showing its age. Other explanations for the behavior of the antenna motor included debris or lubricant thinning, though these have been discounted by the manufacturer.

The next lead the engineers began investigating centered on one puzzling fact: whenever the antenna was not used for more than 24 hours, i.e., when there were only minimal thermal gradients inside the motor, it moved by about 30 steps (for approx. 5 min in low speed mode) before sticking. Temperature problems in some previous planetary probes, like the 1991 Galileo mission to Jupiter, also led to a locked antenna and lessened communication lines back to the DSN. For the SOHO engineers, the hypothesis was that when the motor was uniformly cool enough, a single set of windings may be enough to drive the gears. It was decided to increase the temperature of the antenna slightly to about 10C.

After a spacecraft offpoint maneuver late on the morning of June 18 confirmed that the high gain antenna (HGA) had in fact not moved more than a small amount (in comparison to the steps commanded) since the first offpoint on June 4, the SOHO team decided to switch to the redundant HGA electronics.

The system team commanded an HGA motion about the spacecraft Z axis (i.e., east-west antenna motion) of roughly one degree. Less than one-fifth of that motion took place, as determined from automatic gain control readings at the ground station. The engineers commanded another 131 steps, without any noticable antenna movement.

The similarity of the behavior on both horizontal sides indicated the likelihood of a mechanical problem with the antenna drive motor or mechanisms. In response to their quandary, the team held a teleconference with the manufacturer of the motor (Moog) and, following their recommendation, tried to command the antenna for a rapid thousand-fold boost, in high speed (100 Hz instead of 0.1 Hz) in both directions. This burst ended without success.

Sun Back Online

The solution to bring the Sun back into its continuously monitored status is still ongoing, but likely will combine a clever package of navigation tricks and a greater use of ground station availability to conquer what appears to be a stuck motor gear.

Spectacular science from SOHO, as well as its art, likely restored.

The first patch depends on spreading out a larger Earth network of receivers. The relatively late occurrence of the initial loss of contact means that the effective SOHO’s HGA antenna beam width is larger than anticipated. Also, since the 34-meter stations are much quieter than the smaller stations, the team can use them for longer time periods than expected. Being able to transmit science data through the on-board low-gain antenna using 70- and 34-meter stations therefore means that there will be no hard blackout periods for SOHO science data, given sufficient ground station resources. However, 34- and 70-meter stations are in higher demand than the 26-meter stations that SOHO normally relies on. Minor data losses are therefore inevitable every day during the 2-3 week periods.

The second patch depends on rolling the spacecraft itself, to mimic how the antenna used to swivel horizontally. SOHO scientists expect full high-rate telemetry coverage, even on 26-meter stations, to resume on or about July 14. To achieve this, they will make the spacecraft roll 180 around its Sun-pointing axis in a maneuver currently planned for July 8.

After a number of tests and new insights, SOHO engineers now say there will be no ‘blackout’ periods for SOHO science data. “We’re now talking only moderate fractions per day every day during the 2-3 week periods,” says Bernhard Fleck, ESA’s SOHO Project Scientist.