When educational products and programs claim to be supported by research, educators considering the programs must look beyond the claim to understand not just what the research says, but who was involved and who funded it. A working paper by Rebecca Wolf, Jennifer Morrison, Robert Slavin, and Kelsey Risman presented at the March 2019 Society for Research on Educational Effectiveness analyzed more than 150 studies in the What Works Clearinghouse and found that studies conducted or funded by developers of educational interventions showed larger positive effects than independent studies.
Reporting on the study for the Hechinger Report, Jill Barshay explains that, according to study author Rebecca Wolf, these stronger results may appear because developers choose not to publish unfavorable results. Study authors might also exclude students from the study who are unlikely to benefit or create their own assessments to measure success.
Sources: Barshay, J. (2019, March 18). The dark side of education research: Widespread bias. The Hechinger Report.
Wolf, R., Morrison, J., Slavin, R., & Risman, K. (2019). Do developer-commissioned evaluations inflate effect sizes? (Working paper). Baltimore, MD: Johns Hopkins University, School of Education.