Space Forum / Shuttle / April 2005

Considering that launch and landing are *by far* the most risky portions of
spaceflight, "passenger miles traveled" isn't such a useful metric.

Jeff

This ng has plowed the shuttle reliability ground repeatedly since Columbia
was lost 27 months ago.
For what it's worth, here's one of my postings on shuttle reliablility from
Oct 2003, about 9 months after the disaster: I assume that NASA has come up
with new numbers since then, but I haven't done my homework on these new
estimates.
______________________________
So, what's the skinny on shuttle reliability, failure frequency, etc. in
light of the Columbia disaster? I checked some of the reports in my files
and Googled a bit to see if any sense could be made out of all the chatter
about probabilistic risk assessment (PRA), failure statistics, probability
of the next shuttle failure, etc. Here's what I've found so far.

The earliest info I could locate is the post-Challenger PRA calculations
done to satisfy the environmental impact requirements for the Galileo launch
in 1988 (the Galileo spacecraft had more than 50 pounds of plutonium
onboard). These are calculations of shuttle LAUNCH failure frequencies:

Oct88 Galileo PRA excluding 51-L:
5th-%ile     Median     Mean        95th-%ile
1/350         1/78          1/55           1/18

Oct88 Galileo PRA including 51-L:
5th-%ile     Median    Mean        95th-%ile
1/202         1/50         1/36           1/13

The median value is the failure frequency at the 50th percentile (half of
the statistical samples are greater and half are smaller that the median
value). For example, the first case can be interpreted as follows: with 90%
confidence, the shuttle launch failure frequency lies between 1/350 and 1/18
with 1/78 median and 1/55 mean if the 51-L failure is excluded. Also, the
median value is the usual "single-point" number used by statisticians as a
shorthand description of a statistical distribution. Note the relatively
wide spread (a factor of about 20 or more) in the failure frequency values
at the ends of the 90% confidence interval.

SAIC took another look at shuttle LAUNCH failure frequency in 1993:

Apr93 SAIC PRA excluding 51-L:
5th-%ile     Median    Mean        95th-%ile
1/223         1/90          1/73           1/31

Apr93  SAIC PRA including 51-L:
5th-%ile     Median     Mean        95th-%ile
1/118         1/52         1/44           1/21

Apr93 SAIC PRA excluding 51-L and SRB/RSRM failure frequency at 1/327
5th-%ile     Median    Mean        95th-%ile
1/394         1/134       1/83            1/30

For the last estimate, SAIC assumed that the post-Challenger improvements
incorporated in the redesigned solid rocket motor (RSRM) decreased the
failure rate of the SRB/RSRM to 1 in 327 (an educated guess). In the final
report, SAIC made further calculations that reduced the median launch
failure frequency to 1/145, a number frequently cited in late 1990s
literature on shuttle reliability. Also, the spread in the 90% confidence
interval has shrunk (it's a factor of 7 to10 in the SAIC calculations), but
I don't know if this has much significance, considering the approximations
and simplifications used in these calculations.

There have been some reports lately about an unpublished 1999 NASA analysis
of shuttle risk that calculated the LAUNCH failure frequency at 1/556
(median value). This sounds like something from an obscure NASA briefing,
but I haven't been able to locate the briefing charts. Need some help here.

So much for launch risk. What about the risk associated with the entire
shuttle flight regime (launch, orbit, reentry/landing)? In 1997 the National
Research Council was tasked to estimate the risk to the shuttle from orbital
debris and meteoroids. In doing this work, the NRC considered the entire
flight regime and came up with the following:

Without meteoroid and orbital debris
               Ascent    Reentry       Debris      Total
Median    1/248      1/350          na           1/145
Mean       1/219      1/326          na           1/131

With meteoroid and orbital debris
              Ascent    Reentry       Debris      Total
Median     1/248      1/350        1/200       1/84
Mean        1/219      1/326        1/200       1/79

NRC used 1/200 as the failure frequency associated with orbital debris and
meteoroids, a supposedly "worst case" value obtained from NASA, DOD and
industry experts. I don't know if there have been more recent updates for
this number and I don't whether NASA really agrees with it today.
Regardless, the NRC was pretty disturbed by the relatively high failure
frequency (1/84, median) that fell out of this calculation and tried to
light a fire under NASA to pay more attention to meteoroid and orbital
debris risk. In fact, every shuttle flight is analyzed by NASA for orbital
debris/meteoroid risks and the results are presented at a Cargo Integration
Review that's held about 12 months before launch. I assume that the CAIB
will make NASA do other things to mitigate this particular risk (inspecting
the TPS via milsats, using the ISS arms and cameras for closeup TPS
inspection, etc).

In that same unpublished 1999 NASA risk analysis mentioned above, the
failure frequency for a complete shuttle mission was estimated at 1/256
(median), according to media reports following the Columbia disaster.

In March 2001, Michael G. Stamatelatos, NASA's risk assessment manager, gave
a briefing on PRA that contained a chart with the following info for the
shuttle:

                         1997               2002                2007
2012
                      (current)         (objective)        (objective)
(objective)
Vehicle loss       1/148              1/250               1/325
1/500

The 1/148 number looks suspiciously like the 1/145 median value that NRC
calculated, ignoring the risk of orbital debris and meteoroids. Or maybe it'
s just a near coincidence.

I haven't been able to locate any more recent shuttle failure frequency
data, but there's probably more out there somewhere. The article in the
14Feb2003 issue of "Science" (pp. 1001-2), 2 weeks after the Columbia
disaster, mentions the Ulysses PRA, the later 1/145 number, a 1/245 number
from 1998, and has a few quotes from Mike Stametelatos, but otherwise adds
nothing new.  Any help here is appreciated.

Forget about these PRAs for moment. What about the actual shuttle failure
frequency data? Well, there's only two random samples available to date, one
from Challenger (1/25) and one from Columbia (1/88). These two statistical
samples constitute a very meager data set as far as the statisticians are
concerned since they could have come from an infinite number of failure
frequency distributions. With only two samples, it's not possible to
determine the characteristics of the underlying shuttle failure frequency
distribution, i.e. the percentiles, the median value, the mean value,
skewness etc. The only recourse is PRA simulations that can run thousands of
statistical experiments encompassing hundreds of shuttle failures to produce
an estimate of the underlying shuttle failure frequency distribution.

So, what benefits have resulted from all of this PRA effort extending over
the past 15 years? Good question. Well, one question that's probably on a
lot of minds is what's the probability that another crew and vehicle will be
lost before the shuttle program ends. You can take a stab at answering this
by using the median failure frequency from whatever PRA you believe in and
assume an underlying statistical distribution, such as the binomial (=
Bernoulli) distribution. If the 1/148 number is taken as the median shuttle
failure frequency and it's assumed that the shuttle returns to operation in
Jan 2004 and flies through December 2020, then you can calculate the
following:

Launch failure frequency = 1 failure in  148
Launch failure probability = 0.0068
Year in which shuttle program ends 2020
Number of shuttle launches per year
2         4          6
Total number of launches (now to end of program)                   34
68        102
Likelihood of at least 1 launch failure before end of program   20.6% 36.9%
49.9%

If you use the 1/84 number that fell out of the NRC analysis of orbital
debris/meteoroid risk to the orbiter, then these likelihoods become 33.4%,
55.7% and 70.5%, respectively.

Note: see http://www.airsafe.com/risk/shuttle.htm for the rationale behind
this calculation.

Personally, I'm not surprised that there could be a 50/50 chance of losing
another orbiter in the next 100 flights. I think most people have a gut
feeling that this is roughly correct. After all, there are several thousand
Criticality 1 failure points in the shuttle and nothing that NASA does to
fix the problems that caused the Columbia disaster will change this
fundamental fact. And, if you add the unknowns associated with aging of the
orbiter fleet, which is apparently causing some CAIB members to lose sleep,
then that ominous feeling becomes even more intense.

Later
Ray Schmitt

No, number of passengers carried is irrelevant when calculating per
passnger-mile risk (or per passenger-hour risk, or per passenger-trip
risk, for that matter).

While a large vehicle racks up more passenger-miles for every safe
trip,
it also racks up more passenger fatalities for every disaster.  The two
happen in exact proportion to one another, so there's no advantage or
disadvantage to increased passenger count.  To put it a little closer
to the language you used:

For example, one 747 with 400 passengers on board can have just one
accident and generate 400 passenger fatalities at once; to kill that
many people in a car would require lifetimes and countless trips,
not to mention replacing the car hundreds of times.  It's easy for
an airliner to rack up passenger fatalities at a phenomenal rate,
placing cars at an advantage in that regard.

--Rich