How to write a accident report
ET helps NASA with Probability Risk Assessments (PRA), for upcoming launch
Here is a link to William Harwood's excellent account of shuttle risks in the upcoming flight, scheduled for this Saturday. This link provides context for my comments that follow. http://www.spaceflightnow.com/shuttle/sts121/060629preview/part1.html
About 18 months ago in Houston I reviewed the shuttle Probability Risk Assessment (PRA) material for NASA. PRA works with a list of possible threats, estimates their probablilities and expected losses, and then seeks to assist decision-making for shuttle risk-reduction.
After the PRA group presented their results, I had two major suggestions:
(1) They should prepare a detailed summary matrix (on, of course, 11" by 17" paper), ordering the risks and providing, in a comments column, relevant background for each estimate. Let that intense matrix, backed up by similar more-detailed 11" by 17" arrays of risk estimates, be the main presentation device and analytical tool for making decisions. This was designed to replace their chippy and twiddly PP slides, which made a hash of their good technical work and made it difficult to assess the overall risk context.
(2) The PRA assessments did not take into account a major risk factor in both the Challenger and Columbia accidents: on-ground intellectual failures in engineering analysis. In the case of the Challenger, the analytic process on the day before the accident was seriously deficient, in the sense that--in hindsight to be sure--the Challenger would not have been launched on that very cold day (which compromised the O-rings and caused the accident) if smarter engineering analysis and better decision-making had taken place. In the case of the Columbia, better analysis and decision-making during the flight might have yielded rescue efforts to try to save the crew, which was endangered by damage to the Columbia suffered at launch. I suggested to the PRA group that on-ground analytic problems contributed to something like 1.3 of the 2.0 accidents in the 113 flights. But there was no risk assessment of such in the PRA; that is, about 65% of the directly observed empirical risk in the 113 flights was not accounted for by the PRA model. The shuttle itself was considerably less risky than what was happening on the ground in decision-making about the shuttle.
At the meeting, I also handed out Richard Feynman's famous discussion of shuttle risks, which Feynman prepared as a part of the Challenger investigation in 1987.
The analysis for the upcoming launch of the Discovery in July 2006, as the link above indicates, was an intense evaluation of risks and trade-offs.
On the basis of reading some of the public documentation (and no direct knowledge) for the upcoming flight in the last few weeks, I think that NASA has made a reasonable and well-informed decision for the upcoming flight. It was also a contested decision. I would vote for the launch. The on-ground factors that contributed to 1.3 shuttle losses appear to be mitigated by the thorough analysis for this flight. The current risk number is a cloudy 1 in 100, which is risky but has been acceptable in the past. The cloudy contributions to risk are the recent changes in the foam, which turns Discovery into something of an experiment.
In the Discovery discussions, a telling distinction was made between "programmatic risks" and "crew risks." The programmatic risk is very high right now no matter what happens. Having flown once in 3+ years, the shuttle program might well collapse if unable to fly soon (within a year or so), or if there is another accident even if the crew escaped unharmed. This rescue scenario is itself troublesome, since the rescue launch must quickly take into account what caused the need for the Discovery crew-rescue in the first place.
POWERPOINT FOR DISCOVERY FLIGHT READINESS REVIEW: THE FOAM SLIDES, OR "MAKE THAT CALL NOW, THAT'S 1-800-. "
Here's the PP deck for "STS-121, Flight Readiness Review, External Tank Project (ET-119):"
This pdf file should be up in a separate window to read in parallel with the comments below:
These slides summarize the results of the enormous amount of resources (probably >$1 billion, some estimates are much higher) devoted to the external tank foam problem.
The slides do not display a sense of engineering intelligence or discipline. In the main report, there is a persistent habit
of dequantification and a general absence of units of measurement. The back-up slides are more quantified and at a higher intellectual level. Several of the slides look like they were produced by a designer lacking in scientific training.
The key overview slide (page 3) is a very good idea but a presentation mess. The good idea is to have an intense and fairly detailed summary early in the presentation. But PP's lightweight resolution and lousy design tools compromise the summary slide. Students of PP design might, however, appreciate the 5 sets of orange drop-shadows, 4 wavy-purple color fields, 3 unintentionally 3D blue time-lines, 2 overactive grids, and floating-off-in-space bullets in the highlight box (with an arbitrary change from dots to dashes midstream in the box). All this stuff on one over-produced but importantslide.
In real science, every photograph has a scale of measurement built right in to the photograph. This low-resolution display method makes it impossible to do so. (Even the shuttle close-out photos, just about the most documentary type of photographs one can imagine, have no scales of measurement and no rulers in the pictures.)
The bullet lists tend somewhat to be base-touching grunts, which show effects without causes, actions without actors, verbs without subjects, and nouns without predicates. The branding with 3 logos on every slide (the title slide has 4 logos) is unprofessional, pitchy, turfy. Are we doing engineering analysis or marketing here? Some 20% of the space of every slide (already a a very low resolution display method) is devoted to branding and to the boxed-in awkward and repetitive slide titles. It is as if each and every slide has to remind the viewer what the presentation is about. So the top 20% of every slide is something to skip, perhaps putting some viewers in the mode of skipping and sliding through the rest of the slide. It is as if the top of every slide announces "nothing important here, you've seen it all many times before."
In several slides, the visually most active materials are the cross-hatched exploding 3D arrows linking the external tank to the magnified areas. Why are the arrows pointing anyway? It's just a simple linking line. The idea here of close contextualized imaging of the problem areas is a very good one, but the badly-drawn giant blue arrows are silly, and result in making the dequantified images of the foam problem areas too small.
The typography is poor, with odd hierarchies (underlined bold italic in parentheses at one point). Is "O2" the proper way for NASA contractors and NASA to write the oxygen molecule (even wikipedia uses a subscript)? Does the slide designer know how to write a subscript in PP?
The overlapping statistical graphics on page BU-2 are presented as decoration, not evidence.
The report is 33 slides long; yet about 10 slide-equivalents are essentially content-free (compulsive repetitive branding, twiddly hierarchical organization, empty space, assorted title pages, and so on). This PP fluffed-up material here and quite a bit more could easily be placed in a technical report on 4 pages of an 11" by 17" piece of paper (folded in half), an exercise left to the student.
The tone and style of the presentation seem alienated from professional engineering. It looks like the slides were prepared by a PP designer, assisted now and then by an engineer. Or maybe it is just the PP pitch style diluting the content. At an FRR.
I hope the actual engineering for the shuttle is a lot better than the evidence for the engineering shown in this presentation.
How much does a problematic presentational style signal poor engineering? Is it just PP or a PP designer weakening the quality of evidence? Or are there deeper intellectual failures? The dequantification, the failure to follow professional engineering conventions, the infomercial tone are worrisome. There is no sign of engineering discipline here, except in the back-up slides. Thus the effect of the presentation is to suggest that there just might be some problems with foam engineering and analytical quality. A danger of problematic presentational styles, such as NASA PP, is not only that they enable sloppiness but also that they can place the truth in disrepute.
It is also a shame that all that expensive engineering work winds up being represented in this manner at a Flight Readiness Review.Source: www.edwardtufte.com