Is the "Buzz" About Knowledge Justified by the Evidence (Part 2)?

The answer is still yes, but we have to be realistic in our expectations.

A bee, presumably buzzing. Photo by Boris Smokrovic on Unsplash

---

Is it true there’s “never been a study showing that increasing knowledge of the world boosts reading comprehension,” as a recent write-up of the evidence claimed? And what evidence should we look to in evaluating that claim?

Part 1 of this post focused on a study of a knowledge-building framework, the Core Knowledge Sequence, led by researcher David Grissmer. That study was found unconvincing in a column by Jill Barshay of the Hechinger Report. In that post, I argued that the Grissmer study did show that a knowledge-building curriculum can boost reading comprehension, albeit within a specific environment.

I’ll now turn to a more recent study that Barshay also finds lacking, along with her general argument that there isn’t enough evidence to support the “buzz” around knowledge-building curricula.

The more recent study—or rather, two combined studies—involved the Core Knowledge Language Arts curriculum, or CKLA. (That’s different from the Core Knowledge Sequence that was the subject of the Grissmer study; the Sequence is less detailed than a curriculum, and it covers subjects beyond Language Arts.)

These studies were designed to follow students from kindergarten through second grade, but—because of factors beyond the researchers’ control, including the pandemic—researchers were only able to collect data after one semester of kindergarten. While CKLA is a literacy curriculum, it covers topics in social studies and science. Kids learned about farms, plants, weather, and Native Americans through read-alouds and discussion.

Researchers gave the children a battery of tests, some of which were designed by the researchers themselves and aligned with what students had been taught. These are known as “proximal” measures. Other tests were standardized or “distal” measures—tests that are not tied to the content in the curriculum—assessing vocabulary and listening comprehension.

The researchers found statistically significant positive effects on the proximal measures but not on the distal ones. “Statistical significance” means that the effects are more likely to be due to the researchers’ intervention than to chance.

Barshay dismissed the study as finding only that “students are more likely to do well on a test of something they have been taught,” adding that “some observers errantly interpreted that as evidence that a knowledge-rich curriculum is beneficial." That hyperlink takes you to a Twitter thread of comments from some prominent researchers and educators who noted the study produced some positive effects.

Did the Study Show Any Benefits?

Barshay’s position seems to be that finding that kids have learned what you taught them is meaningless, especially if you’re comparing them to a control group that hasn’t been taught the same content. She has a point. But that’s not exactly what happened in this study.

The lead researcher, Sonia Cabell of Florida State University, told me that—as noted in the study itself—even the control group schools were committed to knowledge-building; they signed up because they were promised access to CKLA once the study was over. There was some content in their literacy curricula (one was using a district-created curriculum and the other a program called Journeys), and most were also providing science and social studies instruction.

In the treatment group, on the other hand, many teachers were using CKLA instead of teaching science and social studies, even though researchers asked them not to—so the sum total of instructional time the two groups got on science and social studies topics may not have been that different.

As Barshay herself notes, there was least one specific topic that both groups learned about: plants. But when asked to tell a researcher everything they knew about plants, the CKLA children were able to provide more information than the kids in the control group, even though it had been several months since they’d learned about the topic.

Cabell also said that she and her colleagues included, in the assessments they created, some words that were “high mileage” words—general vocabulary that both groups might have been exposed to—along with some words related to the specific content in CKLA. In addition, some words had just been briefly defined during the CKLA read-alouds rather than being a focus of instruction.

“The kids learned what we taught them,” Cabell said, “even if it wasn’t deeply taught.”

Cabell thinks the CKLA group did better because of the way the curriculum is sequenced, with later topics drawing on knowledge that has previously been introduced. She also cites evidence showing that children are better able to acquire vocabulary when it’s been presented in a series of texts organized around a particular topic, like the texts in CKLA.

Knowledge-Building and “Far Transfer”

But what about the lack of positive effects on standardized tests? Barshay is right that ultimately, we want to see evidence of “far transfer”—that is, that the knowledge and vocabulary students have acquired enables them to understand texts on topics they haven’t already learned about. That, essentially, is what standardized reading tests measure.

The problem is, that kind of transfer generally takes three years or more. That’s a key finding of one study Barshay herself cites, which was done by James Kim and his colleagues. If the children in the CKLA study had been able to reach that point after only one semester, it would have been news on the order of “man bites dog.” As it is, the lack of such evidence is merely a “dog bites man” story.

Barshay quotes reading researcher Tim Shanahan as arguing that “if knowledge building does improve reading comprehension, it would take many, many years for it to manifest”—a claim that seems to be offered as an argument against using knowledge-building curricula. But a period of three years, or even four or five, isn’t so long that the prospect of building knowledge to the point of far transfer should be seen as daunting. After all, kids generally go to school for at least 13 years. Rather than suggesting that we should just give up on building knowledge because it takes so long to “manifest,” the evidence indicates that we need to start as early as possible.

Widening or Narrowing Achievement Gaps?

The brevity of the CKLA study may also explain why kids who started out with stronger language skills benefited more than those who were initially lower-achieving. Barshay notes that, “distressingly,” this meant “achievement gaps widened.”

I won’t claim that a curriculum like CKLA can eliminate achievement gaps, but if the study had gone on longer, the gaps might well have narrowed—as they did in the three-year study of a curriculum called Bookworms that I described in Part 1 of this post.

It’s not surprising that the kindergartners who started out with more knowledge were able to retain more new information in the short term. Knowledge sticks best to other related knowledge—like mental Velcro, as researcher Marilyn Adams has famously said. Over time, as the kids who started out with less knowledge acquired more and more of it, they too would likely have accumulated more “Velcro” for new information to stick to.

There’s also non-experimental evidence suggesting that knowledge-building can reduce income-based achievement gaps. In his book Why Knowledge Matters, for example, E.D. Hirsch analyzed data showing that after France switched to an elementary curriculum that de-emphasized knowledge, gaps between socioeconomic groups widened significantly.

Cabell and her fellow researchers didn’t expect to see significant results on standardized measures after a single semester of intervention. They included those measures only because they were expecting the study to last three years. When they initially did see a positive effect on one standardized vocabulary measure, they were surprised. So—unlike most researchers—they checked the words on the standardized test and realized that a few had been taught through CKLA. Once they removed those words from the analysis the effect was no longer significant.

Cabell says the associate editor of the journal where the study was published commended the researchers for their conscientious approach and selected the study as an “editor’s choice.” Ironically, it was that same conscientiousness that deprived them of the results that would have risen to Barshay’s standards.

Putting Standardized Measures in Perspective

I don’t mean to single out Barshay for criticism on this point. Researchers, editors of academic journals, and an influential federal research agency all treat standardized or distal measures as the only valid indicator of success. But some have questioned that assumption.

In an article called “Commercially Developed Tests of Reading Comprehension: Gold Standard or Fool’s Gold?,” two education researchers urged their colleagues to use multiple measures that vary from proximal to distal to better understand the effects of their intervention. That’s the kind of study that was done by James Kim, showing that it took three years for the effects of a knowledge-building curriculum to show up on a distal, or “far transfer,” measure.

That sort of nuanced approach is particularly important in a complex area like reading comprehension. The skills that standardized comprehension tests purport to measure, like making inferences, are highly dependent on whether a reader has knowledge relating to the text, along with a host of other factors. But the assumption has been that students can “master” such skills and then apply them to understand any text. That assumption has seriously distorted reading instruction, and it’s distorted reading research as well.

One possible solution, in addition to using a range of measures as Kim did, would be to use standardized measures that are limited to a particular subject area, like science, rather than standardized reading tests, which have passages on random topics. That would help counter the argument that researcher-created measures are designed only to measure what the researchers have taught. At the same time, it would avoid the problem of trying to test comprehension as a general skill, suggests Hugh Catts, a researcher at Florida State University.

“The notion of achievement in the U.S. is tied to generalized measures of reading,” Catts says. “You’d think if we care about what kids are learning, it would make sense to use measures based on what is being taught in schools.” The goal of reading, Catts adds, isn’t comprehension itself—it’s learning.

The Problem with Waiting for More Experimental Evidence

Barshay is apparently holding out for definitive experimental evidence that knowledge-building curricula boost scores for struggling readers. The problem is that if we wait for that, we’re likely to be waiting a long time. Studies that last three years or more are expensive and extremely rare. A study like the Grissmer one, which lasted 10 years or more, is practically unheard of.

In addition to the expense, it’s hard to exclude other variables in a long-term study. Kids drop out, as happened with the Grissmer study, or it becomes impossible to collect data, as happened with the Cabell study.

“Every study has problems,” says Daniel Willingham, a cognitive psychologist at the University of Virginia who is an author of the Grissmer study. “That’s why you need to look at issues using a number of different techniques—logical, experimental, non-experimental.”

Willingham himself has catalogued the well-established evidence relating to the role of knowledge in comprehension and learning in general, as have a group of international scholars in a new freely downloadable book called Developing Curriculum for Deep Thinking: The Knowledge Revival, which Barshay mentions in her column as part of the recent “buzz” around knowledge. I’ve also written about that evidence, as has education writer Robert Pondiscio in his own response to Barshay’s column. This post is already too long, so I won’t repeat all of that here.

Suffice it to say that it would be beyond unfortunate if people got the mistaken impression that there’s no evidence that knowledge-building can boost reading comprehension. Given national reading scores that have been stagnant or declining for decades, we have plenty of real-world evidence that the dominant approach to comprehension, which foregrounds supposedly general skills, isn’t working. We also have plenty of evidence, of various kinds, that building kids’ knowledge is enormously helpful to their reading abilities.

Let’s not consign even more kids to academic failure while we wait for evidence that is unlikely to arrive anytime soon.