The Debate Surrounding The Success Or Failure Of The Patriot Pac-2 Missile

Smith 1

Adam Smith 17.31J, Fall 2004 Paper 2, Choice C (debunking Choice B as it relates to the Patriot case) War with Words about War “The proposal for national missile defense, as articulated now, is the wrong answer to the right question. The right question is: what do we do about rogue states like Saddam's Iraq?” – Richard Butler, former chairman of the UN arms inspection effort in Iraq1 The debate surrounding the success or failure of the Patriot PAC-2 missile in the Gulf War was explosive. Two camps, driven by different motivations, heralded different interpretations of events. One, led by the Army and the Patriot contractor Raytheon, claimed that the ballistic missile defense system enjoyed near-perfect performance. These claims were motivated by efforts to protect their budget, maintain public and ally confidence, and to make the case for more expensive missile defense systems. The other, led by MIT professor Ted Postol, asserted that none of the twenty-nine Patriot engagements were successful. Postol’s camp was motivated by building a good professional reputation, attracting attention, and gaining notoriety. Each of the two groups tried to prove its point by establishing control over metrics and measurement, deeming evidence as classified, and inviting third-party ad-hoc panels. This situation is a classic example of Jasanoff’s constructivist school of thought; two scientific adversaries attacked each other’s arguments using various tactics, allowing the more objectively scientific side to ‘win.’

1

The Commonwealth, July 17/24 2000, p. 5

Smith 2

This paper is divided into three sections. First, we will discuss the motivations of each side in the debate as distinct from objective scientific truth. With those motivations in mind, we will discuss the tactics that each side used in the adversarial debate and how the exchange played out. Finally, we will make observations about the results of the episode.

What is at Stake?

There were three distinct actors in the debate: the Army, the Postol team, and the American Physical Society. Each of these entities had their own goals; these goals ultimately drove the strategies and tactics chosen. The Army2 primarily sought money that they could use to buy equipment and support personnel. There are several factors which influenced its budget for missile defense, most of which were positively effected by the original reports of Patriot’s success. These factors and their causal relationships are illustrated in Figure 1 below. There are two outputs of the chain illustrated: future investments in strategic missile defense and the tactical missile defense budget. The latter was of immediate importance because it promised immediate pay-offs. Indeed, in 1991 the annual budget for tactical missile defense programs was $398M. By fiscal 1992, shortly after the Gulf War, the budget was increased to $858M.3 On the other hand, strategic missile defense was important because, although its pay-offs were further in the future, the budget for such a program would reach 2

This includes the Army’s contractor Raytheon, as well as the defense department as a whole. For simplicity, however, this group is referred to by their leader in this case – the Army. 3 Daniel Golden, “Missile Blower” Boston Globe Magazine, July 19, 1992.

Smith 3

into the tens of billions of dollars.4 It would seem reasonable to look at the Patriot case as an indicator of potential for success of such a strategic system, and thus a tremendous set-back if the Army could not solve the much simpler problem of building a successful Patriot.

Figure 1. Causes and effects determining the budgets for missile defense programs. The reported success rate in the top-right was the most accessible way for the Army to manipulate the budget in the Patriot case. The input of the causal chain, i.e. the variable that is directly manipulable by the Army, is the reported success rate. This was originally expressed in very simple yet unclear terms. Raytheon claimed that the Patriot destroyed just under 90 percent of Scuds in Saudi Arabia and 50 percent in Israel. (It is not clear what 4

Congressional Budget Office, Budgetary and Technical Implications of the Administration's Plan for National Missile Defense, April 2000, available at: http://www.fas.org/spp/starwars/congress/2000_r/000425-cbo-nmd.htm

Smith 4

exactly qualified as a successful destruction, a point that was later questioned. We will discuss this in the next section.) These phenomenally high numbers had positive effects in the causal chain of Figure 1. Public confidence was high, and the Patriot became the symbol of American high technology success in the Gulf War. The favorable public opinion and positive reports resulted in a higher budget for the defense system, as mentioned before. Also, purchases from allies were generated from the reported success; Saudi Arabia, Turkey, and Kuwait ordered a total of 38 Patriots. All of these things resulted in real income for the Raytheon, easing the RTD&E costs for the Army. Thus the group was motivated to defend their claims of success against the attacks that were later made. Ted Postol and his collaborators found discrepancies in the Army’s case, and after the debate began they were fighting for their professional reputations and influential power. If Postol was proven wrong, his future whistle-blowing power would have been significantly reduced as a result of a hit to his credibility. Inversely, if proven correct, then he would gain notoriety and reputation. Finally, the American Physical Society (APS) entered the debate upon request from Postol, who asked the APS to weigh in on the legitimacy of his evidence At issue for the APS was their credibility. If they wrote a report in favor of either side without appropriate substantiation or evidence, they would be open to attack by the party they deemed wrong. Thus, upon agreeing to consider the issue, the APS panel had the responsibility to deliberate on the available evidence and provide clear support for their position.

Smith 5

Tactics in the Debate If the objective truth has the largest impact on the outcome of a debate, the tactics employed by the entities involved have the second largest impact. This case is no different; both the Army and the Postol camp played a series of cards against one another in an attempt to gain the upper hand in scientific and political credibility. Three tactics were used: 1) both sides tried to strategically define the metrics and measurement techniques of system success, 2) the Army attempted to keep evidence hidden, and 3) to help his credibility, Postol invited a third party group (the APS) to comment on scientific evaluations of missile performance.

Defining Metrics and Measurements Each group in this debate tried rigorously to control the terms of the debate. In an oversimplified version of the situation at hand, if two adversaries are drawing different conclusions from the same set of data, then the source of any disparity between those conclusions must be due to the method of inference. In this case, the method of inference has two components: the first is the interpretation of the evidence, resulting in a set of deductions, and the second is the tabulation of deductions into classes (e.g. ‘success’ or ‘fail’). For example, in some Patriot engagements after the Patriot detonated near the Scud during its flight, there was an explosion on the ground at about the time and place that the Scud should have hit had it continued its trajectory. Postol interpreted this explosion as evidence that the Scud was not destroyed, he thus tabulated the

Smith 6

engagement as a ‘failure.’ The Army, however, deduced that it was possible for the explosion to be caused by something else on the ground; the Army included that particular case in the ‘success’ column in their tabulations. There is no objective truth for what qualifies as ‘success’ versus ‘fail’; the groups will always disagree on the definition of a ‘success.’ The next attempt to arrive at an objective truth would involve classifying cases into more descriptive buckets, instead of simply ‘success’ and ‘fail.’ Consider the following four categories:

a. Extensive ground damage b. Clear miss c. Explosion on ground d. No change in Scud trajectory or appearance after Patriot detonation e. Scud intercepted

Roughly speaking, Postol claimed that events A through D in the list above are ‘failures’ while the Army claimed that only event A constitutes a ‘failure.’5 Even in event B, the Army claimed, the engagement was successful because the Scud’s target was not destroyed and the video evidence used to conclude ‘Clear miss’ might not be reliable. Perhaps the decision-maker for whom the analysis is being done should use their own standard for dividing the cases above into successes and failures. In this case, the differences in the adversaries’ tabulation methods are marginalized out as a policy issue; the remaining 5

Need source

Smith 7

scientific task is to interpret engagement evidence into deductions about which of the five cases describes each particular engagement. Even though we have done away with the problem of defining a metric for success, measurement conflicts still remain. The Army gathered evidence primarily by looking at ground explosions where a Scud landed and recovering the Scud warheads. The warhead analysis was classified; we will discuss this later. The ground analysis, however, is failure-positive only when the Scud hit a target and exploded. All other cases are not ruled as failures, so are counted as successes. For example, if the Scud was poorly maintained and came apart on its own due to the physical forces on it during flight, then there would not be ground damage and this means that the Army would have to tabulate the case as a non-failure. It could be that the Scud fell apart on its own, or it could be that the Patriot intercepted it; the extent of the evidence yielded by the Army’s measurement techniques (ground explosion analysis) cannot distinguish these cases. In order to debunk the Army’s case, Postol had to rely on more information-rich evidence sources. The group noted that reporters in the area recorded video footage of several of the engagements, albeit with un-consistent camera settings and filming locations. After obtaining as much footage as possible, Postol meticulously reviewed the materials to identify which engagement each video was of, the location of the camera, etc. Postol then analyzed the dynamics of each engagement, and discovered that more evidence could be found by watching the flight of the Scud with respect to the fireball

Smith 8

created when the Patriot exploded. For example, with reference to the case in which the Scud falls apart in mid-air because of its poor condition, Postol claimed that the videos could distinguish that case from a clear interception case. So, the Army used a less information-rich but more reliable measurement method to show that Scuds often failed, and Postol used a more telling source to posit that Patriots often did not cause the Scuds to fail. The Army, as might be anticipated, attacked Postol’s evidence sources. It claimed that the video was not “fast” enough to offer information about the ScudPatriot dynamics that occurred at velocities faster than the speed of sound. We will discuss Postol’s response to this attack in the section below on third party panels, and will see how the American Physical Society corroborates Postol’s methods. Finally, even if absolute resolution is achieved in the metric and measurement matters, these evaluations are on the engagement-level. The reader might ask: what percentage of engagements should be successful to make the weapon system successful? It is fairly easy to see that this is a policy issue. The only angle that the Army has on this is to design the system to send several Patriots after each incoming missile; i.e. have more than a one-to-one interceptor-to-missile ratio. This increases the probability of an engagement success without additional engineering, but raises the cost of the system. This did not become an issue until later in the debate because the two camps were polarized. Postol said that the threshold did not matter because none of the engagements was successful at all. The Army said the opposite; that the

Smith 9

engagement success rate was so high that any sane threshold would deem the entire system to be worthwhile.

Secrecy The Army classified some of its evidence that it used in measuring success. Based on publicly available sources, it seems that the classified information concerns the Scud warheads recovered from the impact site.6 More specifically, the Army inferred that the Patriots were successful from some warheads which had marks that appeared to be from an exploding Patriot. There was a legitimate need to maintain secrecy around some of the information gained from the Scud impact site analysis. In general, the information is principally held secret so that our enemies do not learn from it how to defeat or weaken our weapon systems. This is a double-edged sword, however, because it also hides information from the public which might reveal performance under what is publicly stated. The Army has an inherent motivation to report positive performance scores, as outlined previously in this paper; therefore, there is a potential for abuse of classification of information for the purposes of avoid public scrutiny. There are two types, then, of analysis that can be done. Classified analysis is performed based on both unclassified and classified information, the totality of evidence available. Because the use of classified information is necessary, the analyzers must have clearance and access. Thus, the danger

6

Ref. Postol’s paper in response to SKZ

Smith 10

with classified analysis is that the entity doing the analysis might be biased, and there can be far fewer checks against a possible bias. Unclassified analysis is based strictly on unclassified information. There are two dangers here. The first danger is that the unclassified analysis might be flawed if the information it is based upon is missing a critical piece of the puzzle. The second danger is that it can be trumped by classified analysis; e.g. the Army could rebuttal against Postol’s unclassified analysis by saying that it has some classified evidence that proves its claims. Once again, there is no natural marketplace of ideas check on the classified analysis. In this case, the GAO intervened. As a body that does not have the biased motivations that the Army has (see previous section), it conducted its own analysis of the classified evidence claims. The result of the study was essentially that the Army was wrong. Although it could not prove foul-play, it showed that the warhead-based inferences were incorrect since the same markings appeared on Scuds which were not attacked by Patriot missiles.

Third-Party Reviews Once the Army’s classified evidence was dispelled as unable to support the Army’s claims, their primary argument against Postol was that the video evidence is unreliable. In response, Postol asked the American Physical Society to commission an ad-hoc panel to evaluate the efficacy of videotape evidence. The APS agreed, and many years later the report was released. The report backed

Smith 11

Postol’s claims and offered detailed and technical support for its position. The Army issued a forty-four page response paper repeating its arguments against the video analysis and highlighting ‘errors’ in the APS report.7 In response to the Army’s response, Postol and Lewis wrote a letter in Science & Global Security responding point-by-point to the Army’s attack on the APS report. It does not appear that the Army responded with a second round reply. Was the APS report objective and correct? The APS has a good reputation, and there does not appear to be any adverse selection in the panel. Furthermore, the report details scientific motivations for each of its claims. Finally, the Army did not attack the legitimacy of the panel itself, although it did attack some of the claims in the report. Those attacks were counter-argued in the Postol report. There is sufficient information in this series of transcripts to build a reasonable scientific argument from the factual analysis done by both parties.

Conclusion The eight year exchange between the two sides of the debate seems to have resulted in some nominal agreement. The Patriot was not nearly as effective as originally claimed, and perhaps was not effective at all. However, there are generalized, higher level lessons that are more pressing. At the outset of the episode, Postol was invited to testify for the House Armed Services Committee about the Patriot. This kind of check was a positive thing for the issue; additional entities evaluating the case will not hurt truth in the 7

Cite SKZ report

Smith 12

end. However, the parties were not motivated purely by truth. As shown in the first section of this paper, each had their own motives which were uncoincidentally different. They also happened to take different stances. The rest plays out under the adversarial framework; each party tried to use factual information to support its claims since factual information is the most convincing, even though their goals might not have been pure truth. Since scientific and factual arguments win out, in the end there should be a convergence to reality. We also saw how other incremental minds came into the debate. The GAO and APS offered analyses of different pieces of the arguments, and allowed a side to gain credibility where it was factually earned, and where the party on its own might not have been able to make the claim because of its ulterior motives.