Reading Comprehension Studies Can Mislead Us About What Works

A new meta-analysis takes on the “strategy instruction vs. knowledge-building” question. But that’s a false dichotomy.

---

Nathaniel Hansford, a Canadian teacher and education blogger, is the lead author of a newly published meta-analysis of 71 studies of reading comprehension. It’s the result, he writes, of three and a half years spent screening over 1,500 studies and re-running analyses countless times—which involved “learning far more statistics than [he] ever planned to.” As far as I can tell, Hansford doesn’t have a formal background in statistics or mathematics, so this is an impressive achievement.

But the meta-analysis fails to provide much useful guidance. Do the benefits of comprehension strategy instruction plateau after just a few hours, as some have argued? Maybe yes, maybe no. Ditto, at least according to Hansford, for whether the benefits of building students’ knowledge increase the longer the process continues.

In any event, given that the knowledge-building interventions in the studies didn’t involve coherent knowledge-building curricula—they just incorporated some science or social studies content into literacy instruction—Hansford cautions that “conclusions drawn here should not be interpreted as direct evidence for or against knowledge-building models.”

Still, the meta-analysis contains a strong undercurrent of skepticism about the benefits of knowledge-building for reading comprehension, and it’s something other commentators have picked up on.

On his Substack, David Didau has described Hansford’s finding of relatively strong evidence for reciprocal teaching—which incorporates four comprehension strategies—as “inconvenient for advocates of ‘knowledge-rich’ approaches to education.” Harriet Janetos, another education Substacker, took the release of the meta-analysis as an occasion to criticize the “knowledge-building movement” for “setting up No Strategies Allowed signs” rather than identifying the strategies that are most likely to be helpful.

Speaking as an advocate of knowledge-building, I don’t consider the findings in Hansford’s meta-analysis “inconvenient.” Rather, I find the whole endeavor of questionable value, partly because of the faulty premises of the underlying studies and partly because I believe the meta-analysis understates the evidence for knowledge-building.

I also don’t think it’s fair to say that the knowledge-building movement has set up “No Strategies Allowed” signs. Of course, there’s no official leader of the movement and consequently no single disciplined message. Some knowledge advocates may have argued, or appeared to argue, that all strategy instruction should be verboten.

In any event, it’s understandable that knowledge advocates have targeted skill-and-strategy instruction, given that it routinely displaces subjects like social studies and science from the curriculum. I’ve seen and heard of many instances of skill-and-strategy instruction being way overdone, but I haven’t encountered reports of American kids getting too much knowledge-building.

It’s Not About Choosing Strategies or Knowledge

Still, I and others have argued that it’s not a question of choosing between strategy instruction and knowledge. Rather, it’s about putting a particular text or topic in the foreground and bringing in whatever strategies—or skills or literacy standards—are appropriate to enable students to make meaning from it. That’s different from the typical approach, which is to use a text on some random topic to try to teach a particular strategy.

But back to Hansford’s meta-analysis. One problem is that he puts knowledge-building, or what passes for knowledge-building in the studies he includes, in the same category as strategy instruction. But knowledge-building is what actually enables students to use strategies, or at least to use them for more than the few weeks the typical study lasts.

Take the strategy of “making inferences.” Readers with ample knowledge of the topic are likely to make inferences automatically, while those who lack such knowledge may find it impossible. The same is largely true of the strategies of summarizing and “finding the main idea and key details”—although even students who are knowledgeable about a topic will often need explicit instruction in how to do those things.

The value of strategies is that they get students to pay attention to text they’ve read and ask questions about it. That can boost comprehension. But without a certain level of relevant knowledge, students won’t be able to answer the questions they’re asking, no matter how hard they try.

Failure to Transfer

In his post, Hanford says that “the most surprising result” of the meta-analysis was that the positive effects of content or background knowledge instruction didn’t reliably transfer from tests created by the researchers to standardized measures of progress. This was also true for the other interventions he analyzed. David Didau also called this finding surprising and one that should “give us pause.”

But it’s well known that when researchers use assessments they’ve created themselves, the “effect sizes” are typically larger than when they use assessments created by others. When researchers devise their own tests, they’re likely to measure the skill or content that was taught as part of the study. Standardized or “far transfer” measures, on the other hand, aren’t tied to any particular skills or body of knowledge.

That’s why researchers generally consider standardized measures more reliable indicators of whether an intervention has “worked.” The idea is that those tests are more rigorous because they assess whether students are able to transfer what they’ve learned to another context—which is, of course, the ultimate goal.

But especially when it comes to knowledge-building, it’s difficult if not impossible to see transfer effects in the few weeks that most studies last. Students need to acquire a critical mass of general academic knowledge and vocabulary before they’re equipped to understand texts on unfamiliar topics. The only reliable way to enable them to reach that point is through teaching them about lots of specific topics, over a period of time, because information and vocabulary need a meaningful context in order to stick in long-term memory.¹

Overlooked Studies Showing Transfer

In his meta-analysis, Hansford found that “content instruction” failed to show any statistically significant effects on standardized measures and only “negligible” effect sizes on researcher-created ones. He acknowledges that “some have suggested that content instruction may require long-term implementation to demonstrate effects.” But he claims that “the current evidence base was not sufficient to support or refute this claim.”

In fact, though, we do have evidence from at least two studies supporting that claim. For some reason, neither is included in Hansford’s database.² Both studies found meaningful positive effects on assessments that tested “far transfer,” of the kind assessed by standardized tests, but only after three years.

None of the studies on “content” or knowledge-building that Hansford included in his database lasted more than 45 weeks. Given their short duration, it would have been surprising if any of them had turned up a medium or large effect size on a standardized measure.

Speaking of effect sizes, that’s another problem with the meta-analysis. An effect size is a way of comparing results from studies that use different measures. It can be “statistically significant,” meaning that the result is unlikely to be due just to chance, but that doesn’t necessarily make it meaningful in the real world. So researchers categorize effect sizes as small, medium, or large.

Hansford used his own classification scheme for evaluating the size of results, which appears to be similar to one called Cohen’s d. But as education researcher Matthew Kraft has argued, that framework is unrealistic for education interventions, the vast majority of which fail to produce effects that would even be considered small under its standards. Kraft has proposed a more generous framework for evaluating effect sizes in education studies using standardized measures.³

Hansford dismisses one meta-analysis, which he says is often used to provide support for knowledge-building, as showing only “small” effect sizes on comprehension. But the study’s authors characterize the effect sizes as “large.” The apparent reason for the discrepancy: Hansford used something like the Cohen’s d framework, and the authors of the study used Kraft’s.

The Risks of Relying on These Findings

Educators could misinterpret Hansford’s meta-analysis as a takedown of all knowledge-building curricula, despite his disclaimer to the contrary. Another risk is that they’ll use the findings as a guide to instruction in ways that could backfire. Hansford found relatively strong evidence for “reciprocal teaching,” which involves four strategies: predicting, questioning, clarifying, and summarizing. But that doesn’t mean it will be effective when used in the abstract. Like any other type of strategy instruction, it’s unlikely to work if kids lack the knowledge they need to apply the strategies.⁴

And it’s not always the case that “strategy instruction” is ineffective in the early grades, as Hansford and Didau both advise—at least in the context of a knowledge-building curriculum. I’ve seen first- and second-graders making inferences and predictions and “comparing and contrasting” right and left, when they have the necessary information to work with and the right questions to guide their thinking, even if the activities aren’t labeled “strategy instruction.” With younger children those activities are largely done orally and based on read-alouds, but they lay the groundwork for independent reading comprehension later on.

Meta-analyses and studies of discrete comprehension interventions may also lead teachers to continue to see reading comprehension as something best taught in the abstract, divorced from curriculum content—even if for only a week or two—rather than activities to be woven into instruction in any subject and at any grade level. To make real progress, social studies, science, and even math teachers need to also consider themselves literacy teachers, just as literacy teachers need to see themselves as content teachers. It’s all connected.

“Content” Alone Is Not Enough

At the same time, it’s true that simply teaching “content,” whatever that means, is generally not enough to improve students’ reading comprehension. Knowledge-building requires more than just providing students with information. Ideally, teachers will also ensure that students literally understand the information, are able to retain it in long-term memory and retrieve it when needed, and can reflect on and reason about it.

It’s also crucial to ensure that students become familiar with the complex sentence structure and vocabulary of written language, which can pose a serious barrier to comprehension even for those with content knowledge.

The most effective way to familiarize students with complex syntax and vocabulary is to teach them how to use those things in their own writing. Explicit, manageable writing instruction is also the best way to ensure students retain and understand key information and develop the ability to think about it in complex ways. Unfortunately, Hansford’s meta-analysis doesn’t include the existing studies on links between writing and reading comprehension, not to mention on learning in general.

Rather than more studies and meta-analyses that try to isolate and evaluate discrete reading comprehension “pedagogies,” what educators could really use are studies comparing one actual literacy curriculum to another—not just to some undefined “business as usual.” They could also benefit from studies comparing different approaches to writing instruction.

Ideally, these studies would last three years or more. But given how expensive and difficult it is to conduct studies lasting that long, we’re unlikely to get them. There will always be many more studies of strategy instruction than of knowledge-building—not because strategy instruction in the abstract is better, but because it’s easier to see quick results, making it cheaper to study and more attractive to researchers.

“Reading comprehension isn’t simple,” Hansford advises in his blog post. “And anyone who tells you otherwise probably isn’t looking closely enough.”

Amen. It’s so complex that it can’t be developed and reliably assessed in a few weeks, or even in one or two school years. When translating the results of fairly brief studies into guidance for educators, it’s crucial to bear that fact in mind.

Update, 3.3.26: After hearing from Hansford via Twitter, I changed this post to clarify that Hansford did not use the Cohen’s d framework itself in his study. I also added some information to the footnotes concerning Hansford’s view that Kraft’s alternative framework doesn’t apply to the studies he reviewed and about the apparent omission of two long-term studies from his database.

1. Another problem, as some have pointed out, is that the standardized measures used for assessing comprehension are themselves unreliable. One study found that they disagree almost half the time.

2. One is a study by Kim et al., which found an effect size of .14 on domain-general reading comprehension after three years. Another by May et al., found an effect size of .26 after three years. Hansford has informed me, via Twitter/X, he “can’t comment on any specific studies inclusion status,” so it’s unclear why these studies were not included in his database. A third study that lasted six years also found large effects on standardized tests from a knowledge-building curriculum, but that study has not been peer-reviewed and therefore doesn’t meet Hansford’s selection criteria.

3. Cohen classified any effect size under 0.2 as small, anything between 0.2 and 0.5 as medium, and anything over 0.8 as large. Kraft’s benchmarks are: anything less than 0.05 is small, anything between 0.05 and 0.20 is medium, and anything over 0.20 is large. Via Twitter, Hansford argued that Kraft’s framework is only intended for “large scale, independently funded RCTs on older students, this would not be relevant to our study.” Kraft’s paper says that it is intended for “causal research that evaluates the effect of education interventions on standardized student achievement."

4. In his Substack post, Didau notes that a study done in England showed only negligible effects for reciprocal teaching.