Nudging Teachers, in a Large Field Study, Marginally Boosted Student Math Performance

Fifteen nudges tried out across 140,000 teachers and some 3 million students

In the Organization for Economic Cooperation and Development’s 2022 student assessment, math results for U.S. students were below the global average. The percentage of U.S. 10th graders deemed low performers was more than four times larger than the percentage rated high performers.

A large field study devised by more than two dozen academics, including UCLA Anderson’s Ilana Brody, a Ph.D. student, Hengchen Dai and Hal E. Hershfield, set out to explore whether 15 carefully built nudges —reminders to complete a task — delivered via weekly emails to elementary school teachers might translate into better math progress for their students. 

The study harnessed the vast network of teachers and students with access to Zearn Math, a nonprofit, online math learning platform used in about 25% of U.S. elementary schools.

The results, published in the Proceedings of the National Academy of Sciences, suggest the nudge-the-teacher route, at least as attempted here, produced statistically significant but modest increases in student math progress. “Notably, the impact of behaviorally informed reminders in our mega study was surprisingly small,” the researchers write.

The most effective email exhortation to teachers resulted in a 5.1% improvement in students’ math progress, but after adjusting for the mechanics of simultaneously testing multiple factors (which can overestimate the actual impact of the most effective factor), the researchers estimate the actual improvement was 3.3%. Across the 15 different teacher nudges, the average increase in their students completing online learning modules was 1.9%. That translates to students whose teachers received the most effective behavioral nudge completing an average of 1.81 lessons during the four-week test period, compared with 1.78 lessons completed by students whose teachers received a more mundane nudge reminding them that opening emails from Zearn Math would earn them raffle tickets.

That even the most effective email nudges only “very slightly improved” student progress was a surprise to the researchers. Students whose teachers had received a behaviorally informed weekly email nudge (eg.,: comparing student progress this week to last week or suggesting empathy for student outcomes) slightly outperformed students whose teachers received a standard email (e.g., prompting them to continue reading Zearn Math emails). The effect “was at least 30 times smaller” than the researchers and Zearn Math staff expected. This may reflect a broader challenge in improving education outcomes. The average effect on students’ progress when their teachers received the behaviorally informed reminders was just below the average of effect sizes found in randomized controlled trials commissioned by the U.S. Department of Education.

Nudges at Scale

Though the results suggest that these types of intervention are not suitable for substantially improving young American’s math proficiency, this research delivers on an important complementary goal. 

Behavioral research has come in for constructive criticism that the field’s reliance on smaller-scale lab experiments may not reflect real-world behavior. This study went large and into the field.

University of Pennsylvania’s Angela Duckworth and Katherine Milkman, along with Ahra Ko, research project manager of the Behavior Change for Good interdisciplinary network of researchers, harnessed the input of more than 150 behavioral experts to iterate their way to the 15 nudges.

Zearn Math provided the real-world testing ground: more than 140,000 teachers who are teaching math to nearly 3 million elementary school children using the Zearn Math platform. 

The teachers received one of the 15 email nudges for four consecutive weeks in the fall of 2021. The results from the four-week test period persisted during monitoring over the subsequent eight weeks. That said, the researchers note that the impact of nudges in the educational realm tend to diminish over time, raising the need to test out effective nudges over longer periods.

While results were modest, the research suggests that behaviorally informed emails referencing data about the teachers’ classrooms are a promising direction to pursue. Emails that had these personalized references, like the version that Brody, Dai and Hershfield put together about comparing students’ progress from this week to last week, caused a 2.3% higher lesson completion rate during the four weeks for those teachers’ students compared with teachers who received nudges that did not reference personal data.

The results were consistent across private and public schools, nor was their variance based on socioeconomic factors.