• Discuss several different software disasters to provide insights into
  • the types of errors that can occur
  • the costs associated with them

• Mars Climate Orbiter
• Mars Polar Lander
• Patriot Missile Defense System
• Ariane 5
• Therac-25

• Monitor climate changes
• Serve as relay for Mars Polar Lander

• The spacecraft hit the atmosphere earlier than was planned and was thus traveling too fast; this led to the destruction of the spacecraft
• Since the Polar Lander was also lost, the combined cost of the project stands at 327.6 million dollars
• Compare to Mars Observer (lost in 1989): 4 billion dollars!

• undetected errors in ground-based models of the spacecraft
• the operational navigational team was not fully informed on the details of the way that Mars Climate Orbiter was pointed in space
• a final, optional engine firing to raise the spacecraft's path relative to Mars before its arrival was considered but not performed
• the systems engineering function within the project that is supposed to track and double-check all interconnected aspects of the mission was not robust enough
  • this was exacerbated by the first-time handover of a Mars-bound spacecraft by the team that constructed and launched the vehicle to a new multi-mission operations team (this is the “politics” part!)
• some communications channels among project engineering groups were too informal (e.g. not documented!)
• the small mission navigation team was oversubscribed and its work did not receive peer review by independent experts
• personnel were not trained sufficiently in areas such as the relationship between the operation of the mission and its detailed navigational characteristics, or the process of filing formal anomaly reports
• the process to verify / validate certain engineering requirements and the technical interfaces between some project groups, and between the project and its prime mission contractor was inadequate

• No review (e.g. verification)
• insufficient training
• informal processes in place
• formal processes ignored

• the failure was NOT detected by unit testing: which was performed
• it would have been detected by integration testing: which was NOT performed
• to emphasize, an integrated test across subsystems was required to detect the problem

• Give command to deploy spacecraft's legs
  • Legs deploy
    • Test Passed!

• When spacecraft detects “jolt” of landing on the surface of Mars, turn engine off
  • Simulated “jolt” detected, engine shuts off
    • Test Passed!

• Spacecraft enters atmosphere
• Legs deploy and “jolt” the craft
• “Jolt” detected and engine shuts off!

• Spacecraft was still miles above the surface of Mars!
• Spacecraft crashes into Mars and is destroyed

• If the team testing the deployment of the legs had conducted the test while also testing the flight software, the “bug” may have been detected
• Unfortunately, with the “faster, better, cheaper” philosophy of NASA at the time, integration testing was deemed too expensive and was not conducted in a comprehensive fashion

• http://dutita0.twi.tudelft.nl/users/vuik/wi211/disasters.html
• http://www.eiffel.com/doc/manuals/technology/contract/ariane/
• http://www.math.ufl.edu/~cws/3114/ariane-siam.html

• The lives of 28 soldiers when the Scud missile struck a military barracks

• Actual time was reported in seconds by multiplying the internal clock’s value by 1/10
• This calculation was performed using a 24-bit fixed point register

• 1/10 has a non-terminating binary expansion, so its actual value was chopped to fit into a 24-bit register
• This introduces an error equal to 0.000000095
• After running for 100 hours, this error means that the system is 0.34 seconds out of sync with reality

• Scud Missiles travel at 1676 meters per second!
• In 0.34 seconds, they travel half a kilometer (~0.3 of a mile)!
• Without an accurate location, the Patriot missile had no chance to intercept the missile

• Numerical algorithm at an extremely low level led to total system failure and loss of human life
• But there was also a maintenance related problem
  • The error was known, and a fix had been applied in some parts of the software but not others!
  • A complete set of regression tests, if applied after every modification, may have detected the problem
  • In addition, patches were applied inconsistently in the field. A patch required flying a floppy disk to the war zone and having a soldier take the disk to each missile system and apply the update!

• The reused component was more than 10 years old and had flown successfully on numerous Ariane 4 flights
• The problem → certain assumptions changed between the Ariane 4 and the Ariane 5 and the software was not updated in response

• (it predates C++ and Java in this regard)

• The 64-bit value was greater than 2¹⁵ which caused an exception to be generated
• This exception was not caught and it caused the termination of the flight control software 37 seconds into the launch
• The rocket shortly thereafter (3 seconds) lost control and was destroyed

• They reveal that one “vexing” aspect of this disaster is that the error occurred in a software system that was not needed during launch!
• The calculation was supposed to be stopped 9 seconds before launch, but the inertial reference system had been reset during a hold in the countdown and its initialization sequence proceeded during launch.
• This is what caused the rocket to veer off course… the initialization sequence was sending random sequences of 1s and 0s to the flight control software, which was interpreting them as commands to fire various sets of booster jets in completely random patterns!

• Exception handlers had been placed around 4 of 7 variables; unfortunately, the data conversion error occurred in one of the 3 unprotected variables
• Why leave 3 variables unprotected? Performance! If you add exception handling code, you slow the performance of the system
• Plus, the developers had an analysis that showed that overflow could not occur with the 3 unprotected variables
• So they had good reason to leave them unprotected

• The overflow analysis was conducted for the Ariane 4, not the Ariane 5
• Its prediction that overflow could not occur for the three unprotected variables was no longer valid!

• This is not surprising since they invented the technique!

proc foo(actual_value: int)
    assert(actual_value <= maximum_value)