Are We Misunderstanding the Evidence? The Hidden Pitfalls of Education Research

June 16, 2025 | The Learning Counsel

A growing number of researchers across social sciences, including education, are grappling with a fundamental challenge: how we interpret the results from research. The way we’ve traditionally analyzed and understood study results has some critical flaws that can lead us astray.”

In a new opinion piece for The Learning Counsel, Rebecca Taylor-Perryman, Managing Director of Research and Data at Leading Educators, weighs the challenges of replicating evidence-based improvement models and the missed opportunities to learn from programs that didn’t show positive evidence.

At the heart of the issue lie some common research practices related to how we determine if a finding is “real.”

The Challenge: Replication

Rebecca writes, “One major issue is the tendency for promising findings to be difficult to replicate. Just as psychology has faced scrutiny over studies that couldn’t be reproduced, education research also struggles with this. Exciting results from one study often don’t hold up when others try to replicate them (Gelbach and Robinson 2021; Open Science Collaborative 2015).”

She adds that it’s rare to hear about the interventions that don’t work. “Negative or null findings often go unpublished, preventing the field from learning valuable lessons about what doesn’t succeed in different contexts. Practitioners are left with confusing and mixed messages,” she cautions.

Measuring how much students grew is simple, but understanding which interventions were responsible for those changes is complex. This lack of a complete picture can lead to significant investment in interventions based on a limited number of positive studies, only to see those investments fall short.”

The Consequence

To be clear, the stakes are high. Rebecca writes, “Think again of the resources sometimes poured into reading programs that don’t align with the well-established science of reading—the consequences for students can be profound. Fortunately, there are ways to navigate this more effectively. But it requires all of us—researchers, practitioners, and policymakers—to become more critical consumers of research.”

Citing Leading Educators’ strong evidence from Harlem, she offers a few ways to raise the bar for evidence quality and how it gets used:

1. Demand multiple studies: Ask vendors and partners for results from several impact studies, not just a single promising one.
2. Consider the context: Inquire about who was included in the study and who was not.
3. Be skeptical of large claims: Exercise caution with interventions reporting very large effect sizes (remember, a moderate effect is generally between 0.05 and 0.20 standard deviations) and ask how those estimates were calculated.
4. Prioritize transparency: Work with partners who are open about the limitations of their data and can explain their impact in ways that resonate with the realities of learning.