Clearing Up Misconceptions About "the Baseball Study"

Critiques of an iconic experiment on knowledge and reading comprehension miss the mark.

Photo by Eduardo Balderas on Unsplash

---

In the late 1980s, a couple of researchers decided to investigate what was more important to reading comprehension: general comprehension skills—the kinds of skills standardized reading tests seem to measure, like the ability to make inferences—or how much the reader knew about the topic. The researchers, Donna Recht and Lauren Leslie, chose the topic of baseball. The participants in the study were 7^th and 8^th graders, and Recht and Leslie figured there were many kids that age who weren’t generally good readers but did know a lot about baseball.

They divided the students into four groups, depending on how well they had scored on a standardized reading test and how much they knew about baseball. Then they gave them all a passage to read describing half an inning of a baseball game and tested their comprehension of that passage.

What did they find? Knowledge of baseball was the key factor. The students who had been identified as “poor readers” on the standardized test and were also baseball experts did quite well when the topic they were reading about was baseball. In fact, they did significantly better than the “good readers” who knew little or nothing about baseball.

I and many others—such as cognitive psychologist Daniel Willingham—have cited the baseball study to highlight the importance of prior relevant knowledge to reading comprehension. Our argument is that the study suggests that the typical approach to teaching reading comprehension, which puts comprehension skills and strategies in the foreground, is unlikely to work.

The totality of the evidence (including but certainly not limited to the baseball study) suggests it would be far more effective to adopt an approach to comprehension that prioritizes building students’ knowledge. That approach would include bringing in whatever skills or strategies are likely to help students understand the particular text or topic being taught. (I’ll be using “skills” and “strategies” interchangeably, since it’s not clear there’s a significant difference between the two.)

The argument for building knowledge appears to be gaining traction across the country. An increasing number of schools and districts have been adopting a different kind of elementary literacy curriculum—one that puts content in the foreground rather than skills. There are now several such curricula available.

But some within the field of reading education have pushed back on the argument—and some of that pushback has focused on the baseball study.

The Study Lacked “a Third Group”

I recently heard one webinar presenter question the validity of the baseball study because it didn’t include “a third group,” one that had “been taught strategies.” Other studies show, this person argued, that kids with “no knowledge” do better than those with prior knowledge of the topic, as long as they’ve been taught strategies.

I’m not aware of such studies. Yes, there are lots of studies showing benefits from strategy instruction, but the ones I’ve read don’t compare a high-knowledge group that got no such instruction with a low-knowledge group that did.

In any event, it’s true that none of the students in the baseball study got strategy instruction. On the other hand, the goal of strategy instruction is to produce readers who have mastered the skills purportedly measured by comprehension tests—in other words, readers like those categorized as “good” readers in the baseball study. Strategy instruction grew out of the idea that we could teach poor readers to use the kinds of strategies that good readers rely on, perhaps unconsciously.

Maybe, in the baseball study, poor readers with little baseball knowledge would have understood the baseball passage better if they’d gotten strategy instruction—we don’t know, since that’s not what the study was about. But it seems unlikely. The good readers with little baseball knowledge presumably already knew how to use strategies—that, at least in theory, is why they scored well on the standardized test. The study indicates those strategies didn’t help them much when they were reading about baseball. So if we provided the poor low-knowledge readers with the same strategies the good ones already had, it seems that it wouldn’t have made much difference.

That doesn’t mean we should never teach kids comprehension strategies. As I’ve argued before, strategy instruction has its place. It just shouldn’t be in the foreground. And the most effective way to teach strategies like “finding the main idea” is through explicit writing instruction.

Too Few Baseball Studies or Too Many?

One prominent reading researcher, Timothy Shanahan, took on the baseball study in a 2020 blog post titled “Prior Knowledge, or He Isn’t Going to Pick on the Baseball Study.” But pick on it he did.

Shanahan first noted that there is “a substantial and extensive body of research,” going back almost 100 years, “that reveals that readers use their knowledge to understand text.” And he characterized the baseball study as “cool” and “provocative.”

“But,” he continued, “it’s a one-off. There aren’t other studies with this kind of finding.” He found it “interesting that this has not been replicated in more than 30 years…. Kind of hard to generalize very far from that in my opinion.”

But now comes a team of reading researchers who have identified no fewer than 19 “baseball studies,” dating from between 1978 and 2018, 13 of which used the same two measures of baseball knowledge. So it’s hard to see the Recht and Leslie study as a one-off. In fact, this team of researchers argues there have been too many baseball studies.

This team, led by Dan Reynolds and Courtney Hattan, have produced two articles critiquing baseball studies this year. While they say they’re not against building knowledge (none of these critics say they’re against it), they have a lot of complaints about baseball studies. In my view, all of them reflect some fundamental misunderstandings of what these studies actually signify. I’ll address those that I see as the most salient.

Is the Baseball Study Too Influential?

One complaint is that, in Reynolds’ and Hattan’s telling, school districts are rushing to adopt knowledge-building curricula simply after reading a description of the baseball study.

“Journalist Natalie Wexler's popular book [The Knowledge Gap],” they write, “used baseball knowledge and text comprehension to argue that US students are underperforming in reading at a national scale due to low content knowledge in their reading curricula. As a result, some districts have adopted literacy curricula, such as Core Knowledge or Wit and Wisdom, that center domain knowledge in literacy instruction.” (Emphasis added.)

I believe that’s giving the baseball study—and me—a lot more credit than we deserve. Even assuming my book has influenced some curriculum adoptions, I spent less than one page of it describing the baseball study and then referred to it in passing just twice more. I brought in a lot of other evidence on the role of knowledge in comprehension, as have others.

Using Baseball as a “Proxy” for Knowledge?

Reynolds and Hattan also argue that people like me are using “baseball knowledge [as] a widespread proxy for knowledge.” But, they say, these baseball texts actually require only knowledge of “trivia” and specialized vocabulary rather than “conceptual” knowledge. A reader might need to know, for example, that “a can of corn” refers not to something you would buy in a supermarket but rather to a fly ball hit to an outfielder.

The baseball texts used in these studies might look simple, the researchers say, with short sentences and words, but they are “uniquely designed to make non-baseball fans appear to be poor comprehenders of that text.” They argue that the texts kids read in school are more likely to explain arcane terms. In other words, the researchers are crying “foul.”

But, as Reynolds and Hattan acknowledge, these texts read like the transcript of a radio broadcast describing a baseball game. People who are stumped by the baseball passage are likely to have similar difficulty reading a baseball story in a newspaper. The texts used in the studies aren’t “designed” to trip readers up; they come from real life. And real life—including school—is full of text that is easily comprehensible to people who have the background knowledge needed to understand it and completely opaque to those who don’t.

The idea behind these studies isn’t to use baseball knowledge as a “proxy” for all knowledge, Rather, baseball—and other sports—can throw into high relief the way readers (and learners in general) rely on their prior knowledge for comprehension. Yes, texts about baseball assume a lot of familiarity with baseball jargon, but texts about lots of other topics assume familiarity with other jargon. A text on, say, molecular biology is likely to be full of vocabulary incomprehensible to someone who knows nothing about molecular biology.

And, according to Dan Willingham, the effects of prior knowledge have also been observed with less jargon-laden topics like the circulatory system, photocopy technology, and the Vietnam War. Sports topics are particularly striking both because sports writing is full of jargon and because the people who know the jargon aren’t necessarily academic superstars.

What is perhaps less obvious is that we rely on prior knowledge to understand everything we read. That’s particularly true of more academic or cultural kinds of knowledge, because that’s the kind of knowledge that tends to be assumed by complex text. Willingham has noted that general cultural knowledge correlates very highly with general reading comprehension ability—a correlation of .50, which, he says, is about the same as the correlation between the heights of parents and their children. (You can find this in a YouTube video Willingham created years ago, at around 5:25.)

One experiment described by Willingham found that students who scored well on reading tests were also likely to know the answers to questions like these:

•  In what part of the body does the infection called pneumonia occur?

• What is the term for selling domestic merchandise abroad?

• Which mythical Greek hero demonstrated his bravery during his long journey homeward after the Trojan War?

These findings connecting background knowledge to comprehension have been replicated, Willingham has said, but few researchers conduct this kind of study anymore precisely because the effect is so well known.

Are We Just Talking About Trivia?

Still, Reynolds and Hattan might dismiss this kind of knowledge as “trivia” or mere vocabulary knowledge rather than deep conceptual knowledge. I see two responses to that, depending on the text. If the text is full of specialized jargon—as in the baseball study—people who know the jargon are also likely to have acquired deep knowledge of the topic. Vocabulary like that generally doesn’t stick unless it’s attached to something meaningful.

Perhaps because baseball is a non-academic field, it’s easy to dismiss knowledge of baseball vocabulary as trivia. The relationship between vocabulary knowledge and conceptual knowledge may be clearer in other contexts. If you can understand the following article title, for example, your knowledge of molecular biology probably extends to more than just vocabulary: Comparative Molecular Docking of Apigenin and Luteolin Versus Conventional Ligands for TP-53, pRb, APOBEC3H, and HPV-16: Potential Clinical Applications in Preventing Gynecological Malignancies.

On the other hand, when a text assumes general cultural knowledge, the knowledge you need to understand it may be broad rather than deep. You might call it trivia, but it’s useful trivia. If a text refers to pneumonia in passing, you don’t need to be a pulmonologist to understand it, but it helps to have some idea of what pneumonia is. If a sentence says, “Jack had pneumonia, so he wasn’t about to go on a run,” for example, you’ll know the problem was with Jack’s lungs, not his legs. And you won’t be distracted by trying to figure out what pneumonia means; you can just attend to the meaning of the text.

Of course, readers can look up unfamiliar words, but it’s far more efficient to have them already stored in long-term memory. And if a text has lots of unfamiliar words, readers are likely to become confused and frustrated.

Knowledge Means More Than Just Factual Information

But the argument that I and others are making isn’t just—as Reynolds and Hattan imply—that “more knowledge equals more comprehension,” at least not if they’re defining “knowledge” as facts or vocabulary related to the topic. Another important factor is knowledge of syntax, or sentence structure. The syntax of written language is more complex than that of spoken language, using structures like subordinating conjunctions. The same is true of vocabulary like moreover and despite, words we don’t tend to use when we speak.

Academic or complex text is especially likely to use that kind of syntax and vocabulary. If the child baseball experts in the Recht and Leslie study who were poor general readers had been given a PhD dissertation on baseball to read, their baseball knowledge alone might not have been of much help.

To understand complex text, students need to become familiar with both academic vocabulary and complex syntax. And again, writing instruction is probably our best bet for that. If you learn to use a subordinating conjunction or a transition word like moreover in your own writing, you’re far more likely to understand those things when you encounter them in your reading.

It may be true, as Reynolds and Hattan say, that texts used in school are more likely to include explanations of arcane vocabulary than baseball texts do. But as the grade levels go up, the curriculum assumes more and more knowledge. If students haven’t acquired it in previous years, they may be stymied. If you’re a high school junior and you don’t know whether the North or the South fought to keep slavery, or who won the American Revolution (both real examples), you’re probably going to have a hard time understanding your history textbook.

The Curse of Knowledge

It can be hard for educated adults to see how much they’re relying on their general knowledge to understand texts. That’s part of what is sometimes called “the curse of knowledge,” which refers to the difficulty experts have putting themselves in the shoes of non-experts. In his post on the baseball study, for example, Tim Shanahan argued that readers without “a lot of background knowledge” can understand texts if they use strategies—and used himself and his wife as examples.

“Cyndie and I have spent the past couple of years reading classical works of physics, biology, economics, philosophy of science, communications, political science, philosophy and history,” Shanahan wrote, “works that we clearly have insufficient background for. But you know what? We’re able to make sense of them.”

Well, yes—but they both have PhDs, which means they’ve acquired quite a bit of general academic knowledge and vocabulary and familiarity with complex syntax. That too is background knowledge.

Are Baseball Studies Gender-Biased?

Another of Reynolds and Hattan’s objections to baseball studies is that they’re gender-biased because boys are a lot more likely than girls to know about baseball. That may be true (although I know a lot of women who are avid baseball fans), but I doubt that means the studies were unfair to girls, as Reynolds and Hattan suggest. Other studies on more gender-neutral topics—like the Vietnam War—have found similar results. And you would probably get similar results if you used a study that was biased in favor of the knowledge girls are more likely to have, except that there would be more girls in the high-knowledge group.

That’s because the mental processes involved in understanding text, or any information, are essentially the same for all learners, regardless of gender or other variables. If you lack a certain amount of relevant knowledge, your working memory becomes overwhelmed, and your capacity to comprehend is diminished. So by all means, let’s have studies that use topics other than baseball, as Hattan and Reynolds urge. (There do seem to have been enough studies of baseball and comprehension!) But let’s not expect significantly different results.

Does the Baseball Study Mean We Need to Teach Kids About Baseball?

Lastly, when advocates of knowledge-building curricula like me refer to the baseball study, we aren’t arguing for more instruction on baseball, as Reynolds and Hattan seem to imply in one of their articles. “Sports-centered curricula can be engaging for students,” they write, “but teachers and curriculum designers must include more than just male athletes as examples.”

The allusion to sports-centered curricula reminds me of a remark by a friend of mine who came to a book event I did years ago. Sometime later, this person (who happens to be a baseball fan), said, “You know that theory you have that kids need to learn more about baseball in school?”

Nope, that’s not the theory. Rather, the theory is that if kids learn about a lot of different topics, ideally in a logical, coherent order, they will eventually acquire the critical mass of knowledge that will enable them to understand texts on topics they don’t already know about. They will be using strategies like “making inferences” too, although probably not consciously.

But the lesson of the baseball study—or perhaps I should say “studies”—is that even then, there will be a limit to their ability to comprehend. If they encounter a text that uses a lot of vocabulary that’s unfamiliar—whether the topic is molecular biology or cricket—they may well hit a wall. Still, if schools equip students with as much academic knowledge and familiarity with complex syntax as possible, they’ll be in a far better position to understand whatever they read than if they’ve just gotten years of instruction and practice in free-floating comprehension strategies.