If Research Looks Too Good to Be True, Be Cautious

I've made some mistakes, and so have others.

---

A few years ago, a campaign began in the U.K. to advocate for what appeared to be an incredibly simple and cost-effective method of boosting reading outcomes for children: just turn on the captions on children’s TV and video programs.

The campaign went viral, with parallel organizations springing up in the U.S. and India.¹ Celebrities like Jack Black recorded promotional videos extolling the power of same-language subtitles to transform kids’ lives. And the organizations behind the campaigns offered evidence to back their claims.

But a recent paper by a trio of British academics looks into that research and finds it weak. While there’s good evidence that captions can help those who are hard of hearing and adults who are learning an additional language—people who are already fluent readers—there’s little evidence they help children learn to read.

The largest study cited by the caption advocates, done in India, found significant gains in reading ability from having children watch a TV show featuring Bollywood songs with subtitles. But, the authors of the paper say, that study couldn’t demonstrate that watching the TV show caused the gains, since it wasn’t a randomized controlled trial. In addition, the researchers didn’t collect baseline data on the participants’ initial reading skills, and they relied on self-reports of how frequently kids watched the show.

The British academics also conducted their own studies. The first one found that children generally don’t even look at subtitles until they’re able to read pretty fluently. The second found that six weeks of exposure to same-language subtitles doesn’t improve kids’ reading fluency beyond the gains they would experience from typical classroom instruction.

The paper notes that the captions campaign received “substantial press coverage,” including a piece that ran in Forbes by … yours truly. I also wrote about the captions campaign a second time on this Substack.

To be fair (to myself), in my posts I expressed some reservations about the research on captions similar to those put forward in the recent paper. I suggested, for example, that captions alone wouldn’t be enough to teach most children how to decipher words. And like the authors of the paper, I noted that the vocabulary used on TV shows is simpler than that used in books, including children’s books. So, contrary to the caption campaign’s claims, spending 30 minutes reading captions isn’t the same as spending 30 minutes reading a book.

Still, I did say that the evidence on captions was “solid,” and I feel I should correct the record—and perhaps issue a general, if unoriginal, caution: If something appears to be too good to be true, maybe it isn’t.

Do All Learners Progress at the Same Rate?

While we’re on the topic of Mistakes I Have Made, I should mention another apparent one. A couple of years ago, I wrote about a study suggesting that all students learn at the same rate. Yes, the researchers acknowledged, the 7,000 participants—who ranged from late elementary school to college—did start at different places, depending on their background knowledge. But they asserted that no matter where students started, it took about seven “learning opportunities” for any of them to master a typical “knowledge component.”

In my post, I noted that the kind of instruction the participants received, which was delivered by computers, differed from most classroom instruction. They got frequent tests, which probably helped them retain information; immediate feedback; and repeated opportunities to master various tasks. In other words, the instruction aligned with principles of cognitive science. In addition, the “knowledge components” were limited to fields that call largely for remembering discrete bits of information: math, science, and learning a new language. Still I found the research encouraging.²

I also described the study in my book Beyond the Science of Reading. While I again expressed some reservations, I concluded that the study, along with some other evidence, indicates that “relevant background knowledge appears to be more important to rate of learning than innate ability—suggesting that if we do what we can to equalize background knowledge, we can level the playing field significantly.”

But recently I came across a post online, by a math educator, critiquing the study. I’ll confess I didn’t follow every nuance of the critique, but the basic point seems to be that the researchers overlooked the fact that some participants learned much more from the initial period of instruction than others. The critique also questioned whether the participants’ rates of learning were really as similar as the researchers claimed.

I still think we can do a lot to level the playing field by equalizing background knowledge as much as we can. But it makes sense to me that, even then, not everyone will learn at the exact same rate. Individuals vary, for example, in their working memory capacity, and that can certainly affect learning rates.

Real-World Consequences

As far as I know, neither of these studies has had major real-world consequences, but other studies have. Benjamin Bloom’s “two-sigma study,” for example, which appears to show that tutoring can produce outsize learning gains, has influenced major education funders like Mark Zuckerberg and entrepreneurs like Sal Khan. The study has probably also helped fuel the recent widespread enthusiasm for tutoring as the panacea for education woes, sparking massive investment.³

But the emphasis placed on the two-sigma study has been roundly debunked by at least two commentators, Paul T. von Hippel and Daisy Christodoulou. Because of its design, they say, its dramatic findings can’t be extended to typical tutoring situations.

Even when research is pretty solid, acting on it can have unintended real-world consequences. Take, for example, the issue of class size.

The push for reducing class size has come largely from an intuitive sense, shared by many teachers and parents, that smaller classes will result in a better educational experience for students. But advocates have also cited research to support their claims.

There is indeed some research showing that smaller class sizes lead to learning gains, particularly one study done in Tennessee that began in 1985. It found that children in smaller classes had higher test scores than their peers in larger classes, and the effects were lasting.

Still, generally speaking, the research is equivocal. As an article in Chalkbeat put it, summarizing the data, “Students often do better in smaller classes. But there’s no agreement on exactly how much better, and it remains an open question whether or not class size reduction is a particularly good use of funds that could go elsewhere.”

If you reduce the size of classes, for example, you have to hire more teachers. Not only does that cost money, but the influx of inexperienced, less effective teachers could cancel out the gains from smaller classes.⁴ And districts might have to spend money constructing new classrooms and buildings.

New York Class Size Legislation

Nevertheless, after a campaign led largely by teachers, several years ago New York State passed legislation mandating that class sizes be capped at 20 to 25 students, depending on grade level, with the reductions to be phased in over six years.⁵ The legislature didn’t provide funding to implement the mandate.

In New York City, and perhaps elsewhere, relatively affluent and higher-achieving students appear to be benefiting the most from this legislation. That’s because the schools serving lower-income, lower-achieving students often already have small class sizes. They tend to be under-enrolled.

Meanwhile, some higher-achieving schools are scrambling to add classrooms and teachers—and administrators at lower-performing schools worry the more desirable schools will poach their teachers, leaving them with a less experienced, or perhaps just smaller, faculty. Advocates for equity in education have urged city officials to request a pause in implementation of the legislation.⁶

Maybe smaller classes will benefit all students eventually, but the city is now being forced to divert large amounts of money that might have been better spent on other initiatives. Officials estimate the annual cost of hiring additional teachers could be as high as $1.7 billion. Construction costs might cost another $18 billion.

Some of that money could have gone, for example, to helping New York City schools implement newly mandated literacy curricula that take a radically different approach from what educators are used to. Some teachers have complained that the lack of support has left them feeling overwhelmed. Even the smallest class size is unlikely to benefit children if the curriculum is weak or if teachers lack the expertise to implement a strong one well.⁷

So, what does all this add up to? First, we should be cautious about embracing research that appears too good to be true, even if in some cases it might be true. It’s not always easy to evaluate the quality of research, especially for those of us who aren’t trained researchers ourselves. But it’s best to try to approach the data skeptically—especially if we tend to agree with the findings.

Second, research is important, but it shouldn’t be the only guide to policy. Any initiative, no matter how evidence-based, requires trade-offs. And we need to ask if those trade-offs will be worthwhile before embarking on new, and possibly costly, reform efforts.

1. The U.S. organization is now apparently defunct.

2. I wasn’t the only one. At least one other education writer also commented uncritically on the study.

3. High-dosage tutoring can be helpful for math and foundational reading skills like phonics. But if it’s directed at supposedly abstract reading comprehension skills, like “making inferences,” it’s unlikely to work any better than regular classroom instruction focused on such skills. Reading comprehension depends far more on academic knowledge and vocabulary than on “skill.” That probably explains why studies show that reading tutoring is less effective for students at higher grade levels, when texts increasingly assume that the reader has academic background knowledge. In any event, no studies of tutoring have come close to replicating the size of Bloom’s two-sigma effect.

4. Some studies have found that the decline in teacher quality is only temporary. Presumably as new teachers gain more experience, the quality of their teaching improves.

5. Previously, classes had been capped at 30 to 34 students, except for kindergarten, where the maximum was 25.

6. Former mayor Eric Adams came up with a creative way of meeting this year’s requirement that 60 percent of schools meet the new class-size limit: he exempted a bunch of schools from the requirement, including some that didn’t request exemption. That kind of ruse, which was adopted to avoid losing hundreds of millions of dollars in state aid, may become more difficult next year, when the requirement for compliance rises to 80 percent. The newly appointed city schools chancellor has suggested that merging schools could be a way to meet the law’s requirements, but he may encounter parental opposition.

7. Similarly, some of the money spent on tutoring—particularly in reading comprehension—could be better spent on effective whole-class curricula that could eliminate or reduce the need for tutoring.