How (not) to do Science - Philosophy of Research in Cognitive Science (WS 21/22)

In the early 2010s, the scientific community became aware of a fundamental flaw in their work: Many cognitive and psychological phenomena could not be replicated. Today, this is known as the (still ongoing) replication crisis. During the course, we will explore pitfalls which led to the replication crisis and how to avoid them in future research. For this, we will use and evaluate selected papers on the issues of philosophy of science, experimental designs, (false) use of statistics, open science and others. While the replication crisis is usually attributed to the social and cognitive sciences, many of its pitfalls and solutions are important considerations for other fields as well. Among these are concepts like data management & sharing, researcher degrees of freedom and publication bias.

Session TopicReading MaterialLink to Paper
Karl Popper and demarcationDienes, Z. (2008). Karl Popper and demarcation (ch 1, pp. 1-32). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan.NA
Kuhn & Lakatos: Paradigms and ProgramsDienes, Z. (2008). Kuhn and Lakatos: Paradigms and programs (ch 2, pp. 33-54). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan.NA
Problem of p-ValuesThe Null Ritual What You Always Wanted to Know About Significance Testing but Were Afraid to Ask
Researcher Degrees of FreedomFalse-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant - Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn, 2011
Reproduction Crisis & Open ScienceEstimating the reproducibility of psychological science ; Easing into open science ;
Reproducibility in MLA Step Toward Quantifying Independently Reproducible Machine Learning Research
Replicability CrisisShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences - Stefan Schmidt, 2009
Effects of Low PowerPower failure: why small sample size undermines the reliability of neuroscience
Solution for replicability: preregistrationsThe Value of Preregistration for Psychological Science ; Is Preregistration Worthwile? ;
Theory CrisisThe empirical benefits of conceptual rigor ; Psychology’s Theory Crisis, and Why Formal Modelling Cannot Solve It ;
Generalizability CrisisThe Generalizability Crisis
EthicsEthical principles in machine learning and artificial intelligence: cases from the field and possible ways forward

Note: This course was taught at a University heavily influenced by computer science/ML/AI, thus two sessions (Reproducibility in ML & Ethics) are specifically tailored towards computer science students.