Koehler, J. J. (1990). Judgments of evidence quality among scientists as a function of prior beliefs and commitments. Dissertation Abstracts International, 50, 3194.

ABSTRACT

This research is concerned with the influence of scientists' prior beliefs and commitments on their judgments of evidence quality. A descriptive model is proposed which holds that scientific judgments of evidence quality deviate from the judgments of neutral observers by the extent to which the evidence meets the prior beliefs of the judge, the strength with which the beliefs are held, and the extent to which the judge is committed to the beliefs. It predicts that quality judgments about belief-confirming evidence will be more favorable than those made by neutral observers, while judgments about belief-disconfirming evidence will be less favorable. It is further predicted that the magnitude of this deviation will be greatest when prior beliefs are strong and commitments are high. Two methodologies were employed. First, a laboratory experiment was conducted using advanced graduate students in the sciences. Subjects evaluated the quality of studies that agreed or disagreed with their induced expectations and commitments about fictitious scientific controversies. Second, two groups of practicing scientists having strong, opposite beliefs about extrasensory perception (ESP) were surveyed, and asked to make quality judgments about an ESP study that either supported or opposed their beliefs. The data from both the experiment and survey supported most major predictions. Studies that agreed with prior beliefs were judged to be of higher quality than studies that disagreed. As predicted, a strength of prior belief (strong, weak) x agreement (agree, disagree) interaction was found in the experiment indicating that the agreement effect was stronger when prior beliefs were more extreme. Contrary to predictions, a commitment (high, low) x agreement interaction was not found. Finally, the normative issue receives extensive consideration throughout. When asked, most scientists felt that prior beliefs should not influence scientific judgments of evidence quality. However, a Bayesian analysis suggests otherwise and further indicates that their influence ought to increase as the strength with which the beliefs are held increases. Nevertheless it remains questionable whether this approach is appropriate for scientific judgments of quality.