Official websites use. Share sensitive information only on official, secure websites. To whom correspondence may be addressed. Email: breznau. Author contributions: N. Primary meta-analysis of data analysts' results and preparation of metadata for public consumption preformed by N. This open access article is distributed under Creative Commons Attribution License 4.
Will different researchers converge on similar findings when analyzing the same data? Seventy-three independent research teams used identical cross-country survey data to test a prominent social science hypothesis: that more immigration will reduce public support for government provision of social policies.
The choices made by the research teams in designing their statistical tests explain very little of this variation; a hidden universe of uncertainty remains. Considering this variation, scientists, especially those working with the complexities of human societies and behavior, should exercise humility and strive to better account for the uncertainty in their work. Keywords: metascience, many analysts, researcher degrees of freedom, analytical flexibility, immigration and policy preferences.
Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public.
In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. These results call for greater epistemic humility and clarity in reporting scientific findings. Organized scientific knowledge production involves institutionalized checks, such as editorial vetting, peer review, and methodological standards, to ensure that findings are independent of the characteristics or predispositions of any single researcher 1 , 2.
