Are Charter Schools Dramatically Outperforming Public Schools? A New Report Says Look More Carefully

Over the past two months, headlines have declared that charter schools are outperforming traditional public schools, based on a new study from the Center for Research on Education Outcomes (CREDO), but a critique from the Network for Public Education suggests that the results are being overblown.

CREDO is housed at Stanford University but appears to be associated primarily with the conservative Hoover Institute (also housed at Stanford), with large chunks of funding coming from the pro-charter Walton Family Foundation and City Fund.

CREDO’s report highlights differences between charters and traditional public schools in days of learning. But “days of learning” doesn’t actually mean days of learning. Instead, it’s a metric that CREDO invented back in a 2012 paper as a way of rendering standard deviations of test scores more accessible to the average reader. By dividing one standard deviation in tests scores by the 720 days between 4th grade and 8th grade tests.

So 0.01 standard deviation translates to 5.78 days of learning.

CREDO finds charters come out ahead by 16 days of learning for reading, and 6 days of learning for math. That translates 0.011 and 0.028 standard deviations over traditional public schools.

But is that a remarkable difference?

To answer that question, NPE turned to another CREDO report.

In reading, charter students, on average, realize a growth in learning that is .01 standard deviations less than their TPS counterparts. This small difference — less than 1 percent of a standard deviation — is significant statistically but is meaningless from a practical standpoint. Differences of the magnitude described here could arise simply from the measurement error in the state achievement tests that make up the growth score, so considerable caution is needed in the use of these results.

In math, the analysis shows that students in charter schools gain significantly less than their virtual twin. Charter students on average have learning gains that are .03 standard deviations smaller than their TPS peers. Unlike reading, the observed difference in average math gains is both significant and large enough to be meaningful. In both cases, however, the absolute size of the effect is small.

In other words, when a study found charter schools behind traditional public schools by that amount, CREDO found the effects “meaningless” and “small.”

NPE also faults the study’s Volume II for being selective in its choice of charter management organizations to include in the study. In particular, NPE notes, CREDO did not include Charter Schools USA, which operates nearly 100 schools, the Michigan-based Leona Group, which operates 58 schools, and Pearson’s Connection Academy, the second-largest national chain of on-line charter schools. Just these three chains of the several left out of the study would potentially have large effects on the results.

The report points out that CREDO methodology has been criticized by scholars in the past, and that CREDO research is generally not peer-reviewed. “CREDO’s report engaged in misleading reporting of its own findings but continues to use a flawed methodology, as scholars have repeatedly shown when reviewing prior CREDO reports,” argues NPE.

I reached out to CREDO for their response to the NPE report. If they reply, that will be added to this post.