Short Circuiting the Controller – Missteps in Maintenance and Inspection of Process and Wiring in STS-93

Abstract

During the launch of the Chandra X-Ray Observatory in 1999, an in-flight anomaly occurred a few seconds after liftoff. A power fluctuation caused two Main Engine controllers to drop offline. Fortunately, due to redundancy, the Space Shuttle Columbia was able to successfully reach orbit and avoid an abort. After the successful deployment of Chandra and the safe return of the crew, investigation revealed that the controller failure was due to a wire short in the payload bay. It was suspected that the Kapton insulation on the wire rubbed off against a burred screw head, the result of overtightening of the screw during a maintenance event 4 to 5 years prior to the STS-93 mission. Vibrations led the abraded wire to short during flight. The Space Shuttle Program was grounded for 4 months while a program-wide inspection and wire chafing mitigation effort of all orbiter wiring ensued.

Appendix: SIAT Assessment

The SIAT was chartered by NASA to provide an independent review of the Space Shuttle sub-systems and maintenance practices. During the period from October through December 1999, the team led by Dr. Henry McDonald and comprised of NASA, contractor, and DOD experts reviewed NASA practices, Space Shuttle anomalies, as well as civilian and military aerospace experience.

Wiring In-Flight Anomaly

Corrective Actions:

Maintenance Requirements.

Each Shuttle vehicle contains over 200 miles of wiring throughout the vehicle. As with modern aircraft, Shuttle wiring is a critical system since multiple failures can lead to loss of a vehicle. The primary wiring used in the Shuttle is a nickel-plated copper conductor with 6 mil thick polyimide/FEP insulation (similar to MIL-W-81381, trade name “Kapton”, a wire construction extensively used in aviation from the early 1970’s to mid-1990’s). While this insulation has performed well in many applications, there are known issues related to arc track propagation (carbonization of polyimide and rapid collateral damage to adjacent wiring), mechanical degradation when exposed to certain environments (ultra-violet radiation, high pH materials (>10), sustained long term exposure at elevated temperatures to moisture while under mechanical stress), and insulation cracking when the insulation is nicked and placed under tensile stresses. Polyimide wire insulation performs best in straight runs with minimal bending and flexing. Examination of the Shuttle mid-body would seem to be the ideal application for this type of wiring. The extensive wiring damage found on each vehicle appears to be related to the high and continuous exposure to personnel performing maintenance procedures on various Shuttle systems.

Inspectors have been encouraged not to conduct intrusive inspections to minimize induced wire damage. The most intense inspection has been conducted in the mid-body bays. An examination of the Problem Resolution and Corrective Action system data prior to recent inspections shows the mid-body area to be the fourth highest area with wire damage. The data as of November 18, 1999 shows that since the recent inspections in late August 1999, there have been 485 problem reports written related to wiring in the mid-body area.

Design Issues.

According to an early 1990’s NASA study, the redundancy in 318 criticality-1 (CRIT 1), which is a single failure that could result in loss of life or vehicle, circuits were compromised by placing the redundant circuits in the same wire bundle or clamp. There were 129 CRIT 1/1 areas identified that violate system separation requirements. NASA Standard 8080 requires that critical circuits be physically separated. As an example, six separate areas exist that, if compromised electrically, would result in the loss of all main engine controllers. A review of the data indicates only violations that could be eliminated required a waiver. At the time of this report, a review of criticality-2 (CRIT 2), which is a failure that could result in loss of mission, systems with respect to comprising redundancy was pending.

It is apparent the current wire tray design contributed to STS-93 wiring failures. The use of a wire tray allows wiring to touch metal surfaces, which has resulted in the wiring contacting screw heads and other sharp surfaces. A past and possibility current maintenance practice has changed tray design assumptions. The reuse of tray screws and an occurrence of burred screw heads have created an unexpected chaffing source. There was also considerable configuration variability between vehicles. In some cases, additional chafe protection was added, or screw heads were covered with a protective coating. The wire bundles were permitted to move in clamps and the trays. Typically, critical circuits must be kept physically separated from all surfaces and other wiring.

Arc Tracking.

Damage to wiring or insulation and aging of insulation are a concern to the Shuttle fleet. Several incidents have been recorded over the life of the program.

As the Shuttle fleet continues to age, additional problems are to be expected. Given the life expectancy of the Orbiters, it is essential to plan for maintenance related to aging, not solely for upgrades. As early as 1991, NASA documents reported that arc-tracking was a significant risk on the Shuttle, as identified in the following statement from the 1st. NASA Workshop on Wiring for Space Applications, held at Lewis [Glenn] Research Center in July 1991: “Arc propagation poses a significant and credible threat to mission safety and success in aerospace vehicles [Shuttle]. This workshop was attended by members of the Space Shuttle community including Johnson Space Center and was co-sponsored by NASA Headquarters, Code Q.

Arc tracking susceptibility has not been eliminated, as this is an inherent property of polyimide insulation. Laboratory tests have shown that current circuit breaker technology does not sense arc track events. Intermittent arcing is seen as a varying load by thermal circuit breakers and current spikes can exceed over 1000% of a circuit breaker’s rating without tripping the device. Arc track events have occurred with one- and three-amp circuit breakers; many of the Orbiter circuits are protected by three-amp breakers. Circuit breakers can also fail and not trip during an electrical short.

Human Factors

Findings:

• 1. Communication difficulties exist between all parties particularly in accepting feedback from the workforce, Aerospace Safety Advisory Panel, and independent assessment groups. This factor erodes trust and loyalty within the workforce which are essential for safe work practices.

• 2. Failure to incorporate Human Factors as a critical part of the decision process has increased potential single point and multiple point failures.

• 3. Recent numerous changes and transitions adversely affect work practices, resulting in loss of technical and process-related corporate knowledge (see Issue 7).

• 4. Process improvements made during the transition period to Shuttle Flight Operations Contract have also brought workforce concerns.

• 5. Work stresses, including expanded work assignments and diminished team support, have reduced the capabilities of the downsized workforce. Innovative cross training approaches may be key to regaining competencies and taking advantage of the skill and experiences of an aging workforce.

• 6. The SIAT is concerned that in spite of the Aerospace Safety Advisory Panel recommendations and findings, supported by the SIAT, recurring human factors issues remain unresolved.

• 7. Employee surveys, although limited in current scope, show significant levels of Physical Strain (internalized chronic stress). Internalized chronic stress has been implicated in workers suffering from stress related disease (e.g., gastrointestinal, cardiac, migraines).

Recommendations:

• 1. Communications between the rank-and-file work force, supervisors, engineers and management should be improved.

• 2. Human error management and development of safety metrics, e.g., Kennedy Space Center Shuttle Processing Human Factors team, should be supported aggressively and implemented program wide.

• 3. Selected areas of staffing need to be increased (e.g., the Aerospace Safety Advisory Panel advised 15 critical functional areas are currently staffed one deep).

• 4. The SIAT recommends that the SSP implement the Aerospace Safety Advisory Panel recommendations. Particular attention should be paid to recurring items.

• 5. NASA should expand on the Human Factors research initially accomplished by the SIAT and the Air Force Safety Center. This work should be accomplished through a cooperative effort including both NASA and AFSC. The data should be controlled to protect the privacy of those taking the questionnaires and participating in interviews. Since major failures are infrequent occurrences, NASA needs to include escapes and diving catches (see Appendix 3 of the full report) in their human factors assessments.

• 6. Work teams should be supported through improved employee awareness of stresses and their effect on health and work. Workload and “overtime” pressures should be mitigated by more realistic planning and scheduling; a serious effort to preserve “quality of life” conditions should be made.

• 7. Teamwork and team support should be enhanced to mitigate some of the negative effects of downsizing and transition to Shuttle Flight Operations Contract. Most immediately needed is the provision of relief from deficits in core competencies, with appropriate attention to the need for experience along with skill certification. Further development of the use of cross-training and other innovative approaches to providing on-the-job training in a timely way should be investigated.