A Tale of Two Numbers

Arthur Snyder is a physicist at Stanford Linear Accelerator Center. Margaret is a librarian at the Hoover Institution at Stanford University. She has been studying the Kennedy assassination since the late 1960s.

A friend wrote me today to remind me that although David Mantik doesn't have an "open" Internet presence, that he is approachable. I found this out earlier this year, when he and I exchanged email messages...even when I was a little confrontational. David Mantik replied with respect, and even admitted a couple errors in one of his articles.

Apparently Mantik has already privately admitted the errors described in the following article.

While reading Dr. Mantik's article ("How the Film of the Century was Edited") in Assassination Science, I was flabbergasted by the claim that the probability of the eyewitnesses to the "limo stop" being wrong was "10^-17" - a very small number.

Tink [Josiah Thompson] called it loony. Anybody with some experience with eyewitness testimony knows it's never that good.

This is what Dr. Mantik says in Assassination Science:

"As a not entirely hypothetical example, if a single witness has only a 2% chance of being wrong (as in the table above [taken from Elizabeth Loftus' book, Eyewitness Testimony]), and if 10 witnesses report the same event what is the probability that they are all wrong? This subatomic number is (0.02)¹⁰ =10^-17=0.00000000000000001."

10^-17? Wow! While I think it unlikely the Zapruder film has been extensively altered, I don't think it's that unlikely!

So what are these two numbers? What do they mean?

"0.02" is drawn from Table 3.1 in Elizabeth Loftus' book, Eyewitness Testimony. Here is the table:

Saliency	Accuracy	Completeness
0	61	64
1-12	78	81
13-25	81	82
26-50	83	92
50-100	98	98

Loftus adapted this table from a larger table in a paper by Marshall, Marquis and Oskamp (MMO). The paper was entitled, "Effects of Kind of Question and Atmosphere of Interrogation on Accuracy and Completeness of Testimony" (Harvard Law Review 84, p. 1620).

MMO were not attempting to determine the absolute accuracy of witnesses, but to gauge the relative effect of questioning modality and a "supportive" or "challenging" interrogation style.

The MMO experiment consisted of showing a 2 minute movie to 151 subjects. The film is described as "fairly complex" but it is simple enough that I can outline it from memory. They counted about 884 "items" present in their film. The events depicted are much simpler than what happened in Dealey Plaza 35 years ago.

The MMO subjects were questioned immediately after being shown the film using one of four questioning modalities:

1. Moderate guidance.
2. High guidance.
3. Multiple choice.
4. Leading.

How does this compare with the mixture of testimony, interviews and hearsay -- days, months and years later -- used for the witnesses cited in Assassination Science?

The table Dr. Mantik uses was constructed from the answers of those questioned using the multiple choice method! It is difficult to imagine a methodology further removed from the haphazard questioning of the JFK assassination witnesses than multiple choice.

Dr. Mantik draws his 2% "chance of being wrong" (CBW) from the high saliency category in the Loftus table. He does not specify whether he is using the completeness or accuracy number and since they are numerically the same we can't tell.

The phrase "chance of being wrong" as used in Assassination Science is vague. Is he referring to

1. Type I error or false negative (failing to see the limo stop when it really did stop).
2. Type II error or false positive (seeing the limo stop when it didn't).

In MMO's salience study it is the average number of times subjects either missed items in the film ("completeness") or recalled them inaccurately ("accuracy"). The limousine stop is analogous to an "item" in the MMO film, so not seeing the stop would be a "completeness" error. This is what statisticians call a type I error -- a false negative.

MMO did ask a few questions that sought to elicit false positives, but salience refers to items actually in the film, so the most reasonable interpretation of the completeness numbers in this table is as false negatives.

MMO give no information about correlations. E.g., did the subjects typically all get the same items wrong or were the errors evenly distributed among the items? Correlations can change probability calculations drastically.

In the extreme case where all the subjects miss the same 2% of the items, the probability of them all missing a randomly selected item is 2% not (0.02)^N.

The "stop" witnesses are also correlated in another way. They talked to each other. In an extreme case the testimony of Marrion Baker is used both for himself (mistakenly) and for James Chaney (hearsay). The correlation between them is 100%.

The important thing, however, is not the size of the number but what it means:

10^-17=(0.02)¹⁰ is the probability that 10 witnesses each with an independent (uncorrelated) 2% probability of failing to correctly observer an event (e.g., a "limo stop") would ALL fail to correctly observe it! It is the probability they would ALL say it had not stopped when it had stopped.

It is NOT the probability of finding a group of 10 or so witnesses all saying the same wrong thing, e.g., the limo stopped when it did not stop.

Hypothesis Testing

There is a conflict between the eyewitnesses that say the limo "stopped" and the Zapruder film which does not show the limo stopping.

We are considering two hypothesis to explain the conflict:

I. The film has been altered to remove the stop.

II. The witnesses are mistaken.

To compare these two hypothesis we need the probability that this conflict could have arisen under each hypothesis.

For the film alteration hypothesis (I) the probability is small, but not computable.

The relevant probability for the mistaken witnesses hypothesis (II) is the probability that a group of witnesses (large enough to catch our attention) would turn up.

The probability that a specific group of witnesses is mistaken is irrelevant. The relevant question is what is the probability of finding such a group. This probability is much larger.

A Lottery

It's like the lottery. The odds of winning are one in millions. The odds of any particular person winning are very small, but the odds of somebody winning are high!

The limo stop witnesses are the winners in a lottery to attract the attention of alteration advocates. The odds of any particular group winning are small; the odds that some group would win are much higher!

Given that the limo did slow down (losing 1/3 its velocity in a fairly short time), and that the brake lights were on, it is not that surprising that the some people thought it stopped or nearly stopped. The MMO results have no bearing on the probability of this error being made. There is no reason to believe it is as small as 2%.

If there were 250 witnesses in a position to see the limo stop a false positive rate of 2% gives an expected number of false stop reports of 5. The probability of 10 or more false reports would be about 3%. Given the ambiguity of the situation the false positive rate could easily have been larger than 2%.

Ex Post Facto

Any probability analysis that is done after the fact is potentially subject to large selection biases. In an infamous case in particle physics a new particle was "discovered" (the z (8.3)); the chance of it being background was calculated to be 10^-5. It turned out to be selection bias plus a healthy statistical fluctuation.

Two Numbers, Five Mistakes