Teaching in the Garden of Forking Paths: Implementing Many-Analysts Designs in the Classroom

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

The rise of many-analysts studies has underscored substantial variability in research outcomes when different teams independently analyze identical data sets and hypotheses. This paper demonstrates how the many-analysts framework can be effectively integrated into research methods education. We propose a pedagogical approach in which multiple students independently analyze the same research question using identical datasets in their term papers, followed by a meta-analysis of their collective results. This instructional method highlights critical methodological concepts, such as researcher degrees of freedom and epistemic humility. Moreover, this study applies the many-analysts approach specifically to causal inference using observational data with difference-in-differences analyses. The analyses conducted by students revealed a broad range of effect sizes and led to divergent conclusions, thus emphasizing the educational and methodological value of robustness checks involving multiple independent analysts. These findings illustrate how integrating a many-analysts framework in teaching can enhance students' methodological rigor and appreciation of empirical uncertainty.

Article activity feed

  1. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  2. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  3. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  4. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  5. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  6. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  7. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  8. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  9. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  10. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  11. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  12. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology

  13. Observing many students using difference-in-differences designs on the same data and hypothesis reveals a universe of uncertainty https://osf.io/j7nc8 The rise of many-analysts studies has highlighted substantial variability in research outcomes when different teams independently analyze #sociology