• Questioning Carl

    “Extraordinary claims require extraordinary evidence”

    Carl Sagan, Cosmos, Ep. 12

    We’ve seen this quote a lot in recent days, thanks to the “speeding neutrino” report. It is a favorite phrase whenever a spectacular new finding generates intense skepticism. Carl Sagan famously invoked it in his TV series Cosmos, when discussing the possibility that the Earth might have been visited by intelligent aliens. The phrase has a nice ring to it. But… Does it make sense?

    I hesitate to question the man who also popularized the phrase “pale blue dot” for which this blog is named! However, inspiring skeptic that he was, I hope that he would have welcomed critical thinking about his own words.

    To be sure, we don’t hold all claims to the same level of scrutiny. If an entomologist says she discovered a new species of ant, we tend to take her at her word that her data are correct and we might not be too bothered if her community took a long time to replicate the observations. In contrast, if she claims to have discovered spiders from Mars she can expect calls for a more rigorous inspection of her evidence and will be pressed to help others replicate the findings.

    Yet, the quality of evidence required to substantiate the claim is the same whether we are talking about ants or alien arachnids. The evidence must include honest and accurate observations, logical arguments, comprehensive consideration of all the data (no ‘cherry-picking’), and, ultimately, reproducibility by others.

    Those of us involved in the recent “arsenic life” controversy saw all this first-hand. Most critics of that study have not called for “extraordinary” evidence. On the contrary, they argue that some of the evidence presented was of poor quality and that independent replication is required – that the ordinary standards needed to substantiate a new claim have not (yet) been met. Although the “extraordinary evidence” aphorism has been invoked by my friend Steve Benner, when you dig into his argument he doesn’t seem to require extraordinary evidence, either. His main point is that chemists have higher standards than others – a debatable point, but not a demand for anything extraordinary. To be sure, the atmospherics surrounding NASA publicity efforts and the response of the blogosphere were indeed extraordinary! But the arsenic claim will rise or fall based on the workings of everyday science. The weirdness lies in the intensity of the spotlights.

    In other words, claims require good evidence. Period. The difference between “ordinary” and “extraordinary” claims lies in the degree to which we, as a community, insist on inspecting the evidence to verify that it is good, not in the nature of the evidence required.

    If you’ve read this far you may be thinking “this is just a game of semantics – of how we define ‘extraordinary’”. I disagree. Here’s why:

    When people say that we should discount stories of alien encounters because “extraordinary claims require extraordinary evidence” they are implying that UFO encounters, tales of alien abduction, etc., cross the bar of “ordinary” evidence. That’s not right! At least in my experience, the problem with these extraordinary claims is not that the evidence is “ordinary”, but that it is very poor evidence. Evidence that fails to meet one or more of the standards listed above – honesty, accuracy, logic, comprehensiveness and, worst of all, reproducibility. To put it another way, “extraordinary claims cannot be substantiated by weak and unreproducible evidence”.

    Back to the case of the speedy neutrinos… Physicists will rake the data over the coals to ensure they are solid. They will insist on independent replication – the gold standard of science. But if the data are found to be solid, and the observation reproducible, then these very ordinary scientific standards will push the research community to accept the claim. That won’t be the end of the story. Surely some clever folks would develop hypotheses that could explain the observations without violating known physics. Those hypotheses would lead to new experiments, and so on, and so on. But the claim made in recent days – that neutrinos can appear to travel faster than light – will nevertheless enter into the scientific cannon as an important finding if it eventually meets the ordinary standards of science.

    Not that I am holding my breath. As a laboratory scientist, I know from experience that confounding observations are usually the result of errors. Often, these errors are devilishly difficult to discover. However, that skepticism doesn’t lead me to demand “extraordinary” evidence. I just need to see this surprising observation tested in the ordinary ways that scientists test new claims.