Literacy Experts Say Some EdReports Ratings Are Misleading

Reviewers are giving some bloated basal readers top marks while failing to put a seal of approval on at least one effective curriculum

---

An organization called EdReports is hugely influential in schools’ adoptions of literacy curricula. But some experts say the organization’s ratings have become unreliable.

EdReports was launched in 2015, with philanthropic support, to help state and local education authorities determine whether a curriculum is truly aligned to the Common Core State Standards in math and literacy. (The organization now also reviews science curricula.) Relying on teams of teacher-reviewers, EdReports issues literacy ratings on a host of criteria categorized under three main “gateways”: text complexity and quality, building knowledge, and usability.

The top rating for each gateway, and each criterion, is green. Curricula that partially meet expectations get yellow. The lowest rating is red. A curriculum needs to get green on each gateway in order to proceed to the next one.

The process of adopting curriculum varies across the country. Some states issue lists of curricula that school districts are either required to choose from or, more frequently, encouraged to use. In other states, decisions are left entirely to local districts—or even individual schools.

Whatever the mechanism, education insiders say, curriculum decisions have often been influenced more by the blandishments and sales pitches of major publishers than by quality—which can be hard to determine, since curricula are usually dense and voluminous. In the wake of the adoption of the Common Core standards by many states after 2010, publishers were slapping stickers saying “Common Core aligned” onto programs that were essentially unchanged. EdReports was intended to provide reliable guidance.

That kind of guidance can be crucial. Curriculum has often been overlooked by policymakers and education reformers, but studies have shown a good one can have at least as large an impact as an effective teacher. And curricula grounded in the science related to reading—including both the evidence for systematic instruction in foundational skills like phonics and for building the kind of knowledge that enables comprehension—have been shown to have positive effects on student achievement.

EdReports’ Influence

By 2022, according to the organization’s annual report, EdReports had been used by over 1,400 districts, representing nearly 16 million students. A number of states now use “all green on EdReports” as a proxy for high quality, as do surveys by organizations like the RAND Corporation on the use of “high-quality instructional materials.” Some states, like Rhode Island, develop lists that include only curricula that have gotten all greens on EdReports. EdReports has also trained curriculum reviewers for states and districts.

Estimates of the organization’s influence vary. “The first line of screening for school systems is EdReports 95% of the time,” says Kareem Weaver, the co-founder and executive director of a literacy-focused nonprofit called Fulcrum. Another literacy consultant, who requested anonymity—I’ll call her Consultant A—says it “depends on where you are.” In some places, the “all green” requirement is written into law, but in a lot of places, “curriculum is still not a part of the conversation.”

Still, where curriculum is part of the conversation, EdReports is likely to pop up. Speaking recently to the Maryland State Board of Education, interim state superintendent Carey Wright—highly regarded for having boosted Mississippi’s reading outcomes when she was superintendent there—said she advises district leaders to “take the time to go to EdReports.” If a curriculum is rated all-green there, she assured board members, “you can take that to the bank, that that is a high-quality piece of instructional material.”

But some literacy experts and education insiders disagree with that assessment. They say EdReports’ ratings were generally reliable in its early years, when it reviewed curricula created by smaller, non-traditional publishers, but things have changed. While its red ratings should still warn people away, its recent green and yellow ratings can be misleading, with enormous consequences.

“What many don’t realize is the extent to which EdReports has influenced other parts of the ecosystem,” says literacy advocate Karen Vaites. “If EdReports reviews have become unreliable, that affects a lot of work in other parts of the ecosystem.”

‘All Greens’ for Overstuffed Programs

One complaint is that EdReports has given all-green ratings to some literacy curricula from large publishers, called basal readers, that don’t deserve them. They include widely used programs like Wonders (McGraw-Hill), Into Reading (Houghton Mifflin Harcourt), and myView (Savvas, formerly Pearson).  These “basals,” are now appearing on many state adoption lists in reliance on EdReports’ high ratings.

Critics say they include more activities and features than any teacher can possibly get through within one school year. While there may be some high-quality texts in the mix, there’s also a lot of time-wasting fluff.

“Today’s basals are designed to excel in curriculum adoption processes, not classrooms,” first-grade teacher and curriculum advocate Abby Boruff has written in a blog post. “They are stuffed with enough manuals, materials, tech gizmos, and bells and whistles to check all the boxes at a surface level, which may also explain how some pulled off all-green ratings from EdReports. ”

Another literacy consultant I spoke with who requested anonymity—Consultant B—said she once heard a representative from a major publisher refer to its basal as “a Vegas buffet.” You wouldn’t want to eat everything, the rep told educators; you need to pick and choose. But, Consultant B told me, many teachers aren’t equipped to recognize which options are “healthy” for their students and which are not.

Several of those I spoke with said that if a basal gets anything less than green from EdReports, the publisher will simply add whatever has been identified as missing and resubmit it for review, leading to all-greens on the next round—and curriculum “bloat.”

Kareem Weaver says that especially in a district with challenging students, relatively inexperienced teachers, and little time allotted for teacher preparation, an overstuffed curriculum can be a disaster. EdReports doesn’t review a curriculum for usability unless it gets all greens on the first two gateways, and Weaver thinks the usability criterion should come first. In his hometown of Oakland, he says, the district adopted a curriculum called EL Education that has rich content but is so voluminous that it can be overwhelming for teachers. As a result, he says, Oakland is making progress, but “every inch is a struggle.”

EdReports says it revised its criteria in 2020 partly to guard against bloat. But, while bloat affects usability, the changes were made to the first two gateways rather than to the usability one. And having EdReports review all curricula for usability, as Weaver suggests, might not solve the problem. EL Education itself made it to the third gateway and got green for usability, as have the all-green basal readers that are overstuffed.

Meredith Liben, a literacy consultant who has worked with EdReports in the past, says the usability gateway was never “fleshed out” to address actual usability, which she calls “a disservice to the field.”

Flaws in the Rubric and the Process

Consultant A said the all-green ratings in general, not just those for usability, don’t seem to align with EdReports’ own criteria. Weaver says EdReports is now “inconsistent at best” and is “providing cover for curriculum options that should not be on the table, because they don’t work for teachers or for students.”

Some observers say the rubric itself is flawed. Kate Crist, a literacy consultant focusing on secondary grades who worked as a reviewer and writer for EdReports from 2015 to 2017, says it was hard for reviewers to agree on the meaning of terms like “text complexity” and “knowledge-building.” There was no mechanism to clarify the definitions, she said.

Another problem, critics say, is that reviewers don’t get enough training. They receive “over” 25 hours, according to the EdReports website, but how much over isn’t clear. In addition, according to Crist and others, reviewers are expected to do a lot of work for minimal pay, leading to high turnover. These issues can produce ratings that appear inconsistent.

“If you have flaws in the tool,” Crist says, “with things getting through that should not get through, and also flaws in the process, that gets you flawed results. I’m actually surprised it took as long as it did for the flaws to get out.”

In her current role, Crist often works with districts that have been advised to use only materials that have gotten all green on EdReports and are disappointed. She says that one official, after reviewing an all-green curriculum the district had purchased, asked her, “Why is this such a mess?”

“If you’re a district and you’ve spent $500,000 or a million on a curriculum,” Crist said, “you’re doing what you’ve been asked to do, but the all-green curriculum is trash. Then you’re stuck trying to figure out how to implement something that has bloat, that doesn’t have enough on foundational skills, whatever. What I’ve seen since 2017 is that EdReports has such a thumb on the scale that it has sort of wreaked havoc.”

Janna Chan, EdReports’ chief external affairs officer, responded in an email: “At EdReports, we pride ourselves on the clarity of our review tools and evidence guides and their consistent application across all materials our expert educators review.” She added that EdReports integrates feedback to refine their criteria, as they did in the 2020 update, but “our highly-calibrated application of the tools remains consistent.”

Effectiveness Isn’t One of EdReports’ Criteria

In addition to usability, many of those I spoke with feel that ratings should take account of evidence of a curriculum’s effectiveness in boosting student outcomes. Weaver says the fact that EdReports doesn’t consider such evidence is “crazy.” While alignment to the standards is important, he says, “I want to know, does it work and is it hard to use?”

Chan said that EdReports invites publishers to share evidence including studies of a curriculum’s effectiveness, or “efficacy.” However, she said, “Efficacy studies are not part of our review tool because EdReports reviews materials as they are written, not as they are implemented.”

Weaver and others also fault EdReports for its yellow ratings of a curriculum called Bookworms K-5 Reading and Writing, which they say has more evidence of effectiveness than any other literacy program and is relatively easy to use. EdReports has reviewed Bookworms three times, but each time has failed to give it the coveted all-greens—for a different reason on each round.

The creator of the Bookworms curriculum, reading researcher Dr. Sharon Walpole, argues that it makes no sense for EdReports to determine that the curriculum is not aligned to the Common Core or similar academic standards when it’s been associated with significant positive results on tests aligned to those standards.

Walpole says she has “taken every report from EdReports very seriously” and has spent years trying to improve the curriculum in ways that are consistent both with reviewer comments and reading research. But that has made little difference, she says, because a new set of EdReports reviewers started from scratch on each round without reading previous reviews to decide what to target.

EdReports’ Chan says that “while EdReports ultimately chooses what to review, we strive to work collaboratively with the publisher to determine whether a review is warranted and, if so, the scope of the review.”

After Walpole got results of the first review, in 2017, she responded by adding a writing component and culminating tasks, as required by the rubric. But she says the next round of ratings, in 2019, was nearly identical to the original one. Once again, Walpole changed features of the curriculum that reviewers criticized, but the scores on the third round, in 2022, were even worse. “I feel like [the process] has been my entire life,” she says, adding that the experience has been frustrating.

Weaver is frustrated too. “The fact that the curriculum with the highest achievement results gets yellow,” says Weaver, referring to Bookworms, “while others that are not usable get green” tells you something. Consultant A says the yellow rating for Bookworms “mystifies me.” And several people told me that states and districts have shied away from Bookworms or are unaware of it because it’s not “all green.”

Overstuffed but Thin on Content

Another problem, say some critics of EdReports, is that even though the basals are overstuffed, they don’t include enough coherent content to develop the kind of knowledge and vocabulary required to develop reading comprehension. Instead of the meaty topics in knowledge-building curricula, like “the American West” or “Early American Civilizations,” the basals focus on broad “themes” like “What Is Friendship?” or “How Do We Build Community?”

“Themes don’t build knowledge logically,” said Consultant B. “They flit.” In addition, she said, the basals often use brief excerpts that don’t have as much potential to build knowledge as the whole texts used by more effective curricula.

I compared various evaluations of third-grade curricula under EdReports’ Indicator 2A, which states, “Texts are organized around a topic/topics … to build students' ability to read and comprehend complex texts independently and proficiently.” Like others I spoke with, I came away unsure why some curricula got full credit and others didn’t.

The myView curriculum, for example, was given four out of four points for a unit called “Environments.” The texts ranged from Grandma and the Great Gourd (tied to the question, “How do people travel in different environments?”) to Why the Sky Is Far Away (‘How do different cultures relate to their environment?”) and Cocoliso (“How can an environment affect lives and relationships?”). The Wonders curriculum got full credit for units organized around “essential questions” that were similarly broad—for example, “How do people from different cultures contribute to a community?”

But when it came to EL Education, “Exploring Literary Classics” was apparently not considered a “topic.” And it seems that a unit in Bookworms called “Patterns in Our World,” which covers concepts like latitude and longitude, was also not seen as “cohesive” enough.

Chan’s email said that knowledge-building “is core to [EdReports’] ELA reviews” and that it is “prioritized throughout our tool.”

David Liben, a literacy expert who played a significant role in developing the Common Core literacy standards, says that in the early days of EdReports, he provided training to its teacher-reviewers. The curricula that he and many others consider effective—including Core Knowledge Language Arts, EL Education, ARC Core, and Wit & Wisdom—got all greens, and basals weren’t being reviewed. (Disclosure: I have received speaking fees from Amplify, a publisher of Core Knowledge Language Arts.)

But then EdReports did its first review of Bookworms, giving it yellows for reasons Liben felt had nothing to do with the standards. That led to a parting of the ways, and Liben stopped training reviewers. Shortly thereafter, he says, publishers of  basals started making changes to their products in an effort to satisfy EdReports’ criteria. “And they started getting all green,” he says.

“I think everyone [who understands curriculum] knows that EdReports is flawed,” says Kate Crist. Consultant A echoed that perception. “We all talk about it,” she said, referring to the literacy community. “We say, ‘what is going on here?’”

Weaver says he’s been contacted by three different EdReports employees, each of whom had concerns about its process. One felt that the quality of the reviewers had declined; another believed the organization should consider student achievement results; and the third “had a lot of concerns,” including that “the usability gateway wasn’t being used enough.”

Few Are Willing to Criticize EdReports Publicly

But few are willing to voice their concerns publicly. I spoke with 12 people for this story, five of whom requested they be quoted anonymously or not at all. Others, including one top state education official, seemed to have concerns but decided they were unable to speak with me.

A prime reason, according to many literacy insiders I spoke with, is that no one wants to antagonize the powerful foundations that have funded EdReports, including the Bill and Melinda Gates Foundation, the William and Flora Hewlett Foundation, and the Charles and Lynn Schusterman Family Philanthropies. Those who work for other education organizations often receive funding from the same sources, or hope to in the future.

“You don’t bite the hand that feeds you,” says Larry Singer, a former CEO of Open Up Resources, which publishes Bookworms as well as EL Education. “You don’t even want to graze it with your teeth. If you’re in a climate where the funding is continuous, and it’s 30% of your budget for 10 years, you can’t risk that.”

Singer says the same group of funders, often led by Schusterman, have sustained networks of organizations and coalitions of state education agencies focused on improving curriculum. That can have benefits, Singer says, but it also poses risks, creating a complex superstructure that rests on what he sees as an increasingly unreliable basis.

Singer says these funders subscribe to a theory of teacher-led decision-making that lies at the root of EdReports’ problems. Teachers should have a role in the evaluation of curriculum, he says, but they need to be guided by experts—to a greater extent than they are during the EdReports process. That makes it even harder to push back, Singer says: if you criticize EdReports, you’re not only criticizing the funders but also their theory of action.

Alternatives to EdReports

One reason there’s so much reliance on EdReports, some say, is that there’s nothing else to turn to for guidance on curriculum decisions. But that’s not entirely true. One educator I spoke with who reviewed literacy curricula for his state’s adoption list told me that authorities there recommended relying instead on two other rubrics: one created by an organization called The Reading League, for foundational reading skills like phonics; and another created by The Knowledge Matters Campaign (KMC), for evaluating whether a curriculum adequately builds knowledge. (I serve on the board of KMC’s nonprofit parent organization.)

Using a rubric, however, involves far more work than consulting a color-coded list. The educator told me he spent eight to ten hours reviewing each curriculum, for no compensation. He suspects some of his fellow reviewers were “just checking the boxes from EdReports.”

KMC doesn’t rate curricula the same way EdReports does, but its website lists eight curricula its own experts have identified as being effective at building knowledge. The list includes Bookworms and another curriculum that has failed to get all greens from EdReports, Fishtank ELA. It does not include any basals.

A new initiative is aiming directly at problems with EdReports ratings. Called the Curriculum Insight Project, it will soon be issuing reports on three “all-green” basal programs it considers flawed: Into Reading, Benchmark Advance, and myView. In addition, the project will report on two balanced literacy programs that received the lowest ratings from EdReports but are still widely used, Units of Study and Fountas & Pinnell Classroom. As an example of an effective curriculum, the CIP will also issue a report on Bookworms.

All of the reviewing teams will include educators who have used the program they’re reviewing, according to the CIP’s founder, Karen Vaites. Vaites said in an online discussion that there’s no indication that’s the case for EdReports reviewers. The CIP teams will also include experts who have seen a range of programs in school districts. “We really think this combination of users and experts is the dream team,” Vaites said during the discussion.

In an email, Vaites told me that the CIP will use its own evaluation rubric, which will draw from existing rubrics such as KMC’s and The Reading League’s. Vaites said the CIP rubric will address usability in more detail than others and will also make “an effort to use plain English” in its reports.

But because EdReports has become so deeply entrenched, it may be hard for alternatives to gain much influence. Several people I spoke with said it would be possible for EdReports to improve its process—perhaps by once again revising their criteria and/or providing more rigorous training to their reviewers—but were dubious that would actually happen.

Walpole, the creator of Bookworms, has ideas for how to make EdReports’ process more consistent and transparent, like limiting its re-reviews to criteria that haven’t been met in a previous round. But more broadly, she feels evaluations of curricula need to “get past the ‘gateways’ idea” and include rigorous studies that could illuminate which curricula work best, in which situations. Such studies would take a significant infusion of funds, she says, most likely requiring the involvement of the federal government. But she notes that the payoff could be substantial improvements in reading achievement.

Unreliable Ratings Could Threaten Progress

In the area of literacy, EdReports was created to encourage alternatives to basals and also to prod basal publishers to improve their offerings. It’s achieved the first of those goals to some extent: there are now far more truly effective literacy curricula than there used to be.

But EdReports’ erratic ratings have led many to overlook at least one effective alternative, Bookworms. And its seal of approval for at least one potentially effective curriculum that is too voluminous to be usable in many districts—EL Education—could lead educators who try it to reject knowledge-building curricula in general. I’ve spoken to one teacher, otherwise a fervent science-of-reading advocate, who had that experience.

As for the second goal, some experts I spoke with say basal publishers are now including more complex texts, which is an improvement. But if basals continue to get undeserved greens for knowledge-building and usability, the progress that has been made could be lost.

“I think this is a real risk for the Science of Reading and high-quality curriculum movements,” Boruff wrote in her blog post. “When teachers are given materials that are supposedly high-quality but are actually garbage, many teachers will tune out, or assume they are living another pendulum swing.”

David Liben says that EdReports is “opening up the door back to the basals. If EdReports didn’t bless the basals, there would be far more districts using knowledge-building curricula.”

Some now see EdReports as a booster of basals, even though it has given at least as many all-greens to curricula that are more carefully curated and build knowledge more coherently. For example, a recent open letter to the governor of Massachusetts signed by hundreds of teachers criticized the state’s curriculum list because it was heavily influenced by EdReports. “The problem,” according to the letter, “is that Ed Reports tends to favor basal readers.”

No method of evaluating curriculum will be perfect—just as no curriculum is perfect. But given the importance of curriculum decisions, in terms of their cost and their importance to the prospects of millions of children, the literacy experts I spoke with all believe that states, districts, and schools need more reliable guidance than EdReports has been providing.

This post originally appeared in a somewhat different form on Forbes.com.