Job-embedded, content-focused professional learning increased student learning in Chicago

A July 2022 study by RAND Corporation researchers Kata Mihaly, Isaac M. Opper, and Lucas Greer found that students attending schools randomly assigned to the Chicago Collaborative program had statistically significantly higher test scores than students attending schools randomly assigned to the control group.

These findings challenge the misconception that teacher professional development is ineffective and costly — the content and components matter.

What is an effect size?

Often, student achievement data are reported as changes in proficiency. But because states have different cut-off scores for “proficiency,” it’s difficult to compare outcomes across geographic areas. Instead, all of our evaluations report results in effect sizes.

An effect size is a way to quantify the difference between two groups and determine the efficacy of an intervention. Because the outcomes of an intervention come in many different forms and scales, the effect is usually estimated in standard deviation which allows for comparison across different outcomes and studies.

Using an improvement index

While effect sizes make sense to researchers, they are often less familiar to practitioners. As researcher Robert Slavin writes, “Let’s say a given program had an effect size of +0.30 (or 30% of a standard deviation). Is that large? Small? Is the program worth doing or worth forgetting? There is no simple answer, because it depends on the quality of the study.”

We use an improvement index to address this challenge. The improvement index is the expected change in percentile rank for an average comparison group student if the student had received the intervention. This is a measure used by the What Works Clearinghouse to help readers understand the practical importance of an intervention’s effect. Baird & Pane (2019) compared several options to translate effect sizes and found that the translation to percentiles is the strongest method compared to years of learning, benchmarking (comparing against other estimated effects), and thresholds (likelihood that a student will attain some level of achievement).

Other organizations and the media often translate effects into “additional years of learning.” Be cautious about that! Here’s why:

• Conceptually misleading: The translation assumes that learning is linear and does not take into account that learning rates are highly dependent on student age and whether school is in session.
• Statistical uncertainty: Years of learning perform the worst in terms of statistical uncertainty when compared with other translations of effect sizes.
• Could produce unreasonable values: Highly implausible results are possible, such as many multiples of a year or negative values.
• Results depend on the method: There are many methods to estimate and each method could give substantially different results.