Why Peer Review Is No Substitute for the Scientific Method

Given the the apparent death of cap and trade legislation in the US Senate, the short-term outcome of the US debate on action that allegedly might reduce climate change may rest primarily on what the USEPA manages to actually do. So it is of some importance what criteria EPA claims to be using in determining the scientific merits of its endangerment finding. In US EPA’s view the UN IPCC reports and other assessments based on it are so satisfactory an assessement of current climate science that no independent EPA analysis was necessary, primarily because of the IPCC’s “rigorous” policy on peer review. EPA cites this review policy as the reason it accepts these reports rather than others, such as the NIPCC report. Recent reports show that as actually carried out the UN IPCC AR4 assessment was much less than rigorous in the application of its peer review guidelines, however. Lost in this exchange, however, is whether the yardsticks being used by the UN and the EPA are reasonable. Both organizations appear to assume that peer review is the important characteristic of valid science included in scientific assessment reports.

I maintain, on the contrary, that the important characteristic should be how well the hypotheses proposed by the UN IPCC corresponds with real world evidence. It is only this crucial correspondence that determines the scientific validity of a hypothesis, not how many or how distinguished the reviewers may be who agree with the relevant hypotheses. This should be evident since any widely held scientific view (such as that the Earth is flat some centuries ago) would have easily qualified as valid science using a peer review standard since the supporters could easily have gotten a large number of favorable reviews of their hypotheses. This is what has happened in the case of the AGW hypothesis. There are enough global warming supporters among climate scientists so that with a little careful selection favorable peer reviews can be obtained for any desired warmist hypothesis. Hence such views can pass the peer review standard whether a hypothesis really stands up to comparisons with real world data or not.

For example, the EPA claims in Response 1-12 to the public comments on the EPA proposed endangerment finding that the 880 page NIPCC report stands in sharp contrast to the IPCC and related reports:

“The [NIPCC] organization does not appear to have established any procedures for author selection and provides no evidence that a transparent and open public or expert review was conducted. Thus, the NIPCC’s approach stands in sharp contrast to the clear, transparent, and open procedures of the IPCC, CCSP, USGCRP, and NRC.”

So although there is some discussion of the arguments raised by the NIPCC report, no real effort appears to have been made to consider using the NIPCC report at least in part on the basis of whether the report had “adequate” peer-review guidelines. According to the EPA, only the IPCC and similar reports including such peer review meet EPA’s “exacting” review standards. How accurate or how closely the NIPCC and other skeptic reports correspond with real world evidence appears not to be of any real importance to the EPA–just how comprehensive the stated review process was supposed to be. Yet when deviations from these standards are detailed EPA maintains that the IPCC conclusions would not have been materially affected rather than admitting that their expressed confidence in the UN procedures was misplaced. This is also an argument that the substantive scientific merits of the non-IPCC assessments do matter, but only when the procedural aspects have not been comprehensively implemented. The reverse should be the case.

The Purposes of Peer Review

The basis for the underlying argument is what is fundamental to the scientific method: Correspondence with real world data or procedural review requirements. In examining this issue it is useful to recall the history of scientific peer review. It was basically introduced so as to decide which papers submitted to printed journals should be included and whether there might be improvements that could be made in those selected, primarily for the purpose of saving then precious journal space. This may have actually been useful in the days when journals were of limited size based on the printing and mailing costs.

Peer review subsequently served an added purpose–to provide a basis for discriminating between the output of various authors/professors and thus providing a basis for conferring academic tenure on some but not on others. The second purpose is still a rationale argument for using peer review, but the first purpose is technologically obsolete since Web publication of added papers is very low cost and may be almost free. Use of Web-based journals has the added advantage that they are normally free to all users rather than limited to the select few who can afford often very expensive subscriptions. And peer review of papers for journal publication has many very important disadvantages, of which the most important is that it often prevents publication of non-conventional ideas that may have great merit. This appears to have been too often the case with regard to the consideration of skeptic contributions to climate science in recent years.

So the extension of journal-based peer review to determining the scientific merit of competing hypotheses is a very important policy issue since it may lead to reducing the importance of comparisons of competing scientific hypotheses against real world data. This is exactly what appears to have happened in the case of the AGW hypothesis of global warming. In fact, warmists have widely cited better peer review as an important reason to support their hypothesis; according to the Climategate emails, leading warmist scientists actively conspired to prevent skeptic-oriented papers from being published in major climate-related journals.

The Fundamental Issue: How Should Scientific Hypotheses Be Judged?

All this highlights the fundamental issue of whether scientific hypotheses should be judged on the basis of whether they have appeared in peer-reviewed journal publications or on the basis of correspondence with observed real world data. I believe very strongly that it is the latter rather than the former that should be used. One important reason is that peer-review is subject to the same “group think” that science should seek to avoid in order to be objective and useful. And that is exactly what has happened in the case of the AGW hypothesis. Despite the absense of any relevant real world data comparisons to support their case, warmists try to use the widespread support (the so-called “consensus”) among sympathetic scientists for their hypothesis to argue that it should be accepted. Obviously if this was the standard, we would still believe that the Earth was flat and that the Earth was the center of the universe, to mention just two widely supported hypotheses disproved by their lack of correspondence with real world data.

It is very unfortunate and may even prove disastrous that EPA and other environmental regulatory institutions appear to have made peer review procedures of much more importance than correspondence with real world data. Scientific assessments need to determine the correspondence between hypotheses on the basis of real world data, not relative “peer review” procedures. This needs to be corrected before immense damage is done to our crucial criteria for judging scientific hypotheses and to our economy as a result of using faulty science for public policy purposes.