KIPP’s success significant but hard to explain, says study co-author

This article was originally published in The Notebook. In August 2020, The Notebook became Chalkbeat Philadelphia.

A new study by Mathematica Policy Research found that students at KIPP charter schools showed greater gains in test scores in core subjects during their middle school years than their peers at traditional public schools. Brian Gill, one of the report’s authors, answered several questions about the implications and particulars of the KIPP study, shedding light on the significance of its conclusions.

The main takeaway from the study, according to Gill, is this: “KIPP’s impacts on student achievement are statistically significant, substantial in size, consistent across four academic subjects, consistent across the great majority of schools included in the study, and consistent even on a nationally normed assessment we administered that had no stakes for students. It’s pretty robust positive findings on achievement impacts. … In 15 years of studying educational interventions, I personally haven’t studied anything that had impacts this consistently positive.”

Impact estimates were done separately for each school, and although most schools had positive effects, Gill said, there was considerable variation in the size of the effects. But the researchers had an agreement that schools wouldn’t be identified in the report. That means how the two Philadelphia KIPP schools did compared to others or to the national sample won’t be made known.

Gill noted that, because the methodology was largely a value-added one — tracking individual student improvement, not overall proficiency rates — it is possible for a school not to meet federal learning goals (Adequate Yearly Progress) and still show significant student growth.

“Given the population," he said, "it wouldn’t surprise me if some schools were producing substantial positive effects and not making AYP.”

So what about the KIPP schools led to more rapid student improvement?

“We didn’t learn as much as we hoped to learn there,” said Gill. The researchers surveyed KIPP principals in an effort to distinguish the best performers from those that didn’t do as well. Gill said he didn’t find anything statistically significant, with a couple of exceptions.

“One was related to having a comprehensive system of high expectations for student behavior. Schools that had that tended to do better,” he said.

The other exception, related to school time, he said, was a little more complicated. Although all KIPP schools have days that are notably longer, the ones that had the longest school days actually did worse, according to Gill.

"This possibly says something about how far you can push the school day," he said. "At the same time, we found that schools that put more time into core academic subjects got better results.” Understanding this interplay completely, he said, “was not possible to tease out in our research design.”

Many in Philadelphia and elsewhere believe that one reason KIPP performs well is that it doesn’t replace students who wash out, for one reason or another. The study found a similar attrition rate for KIPP and comparison schools — about 37 percent. But Gill said that the study included all students who started at KIPP, regardless of where they ended up.

“That is not a problem in our analysis because we keep the kids and continue to count them as KIPP kids even if they left KIPP,” he said. “There is no way the results can be artificially inflated by losing those kids.”

On the other hand, he said there could be a more subtle "peer effect" due to more motivated students staying together. “That’s where the attrition and replacement is potentially relevant — not to the validity of findings, but to their interpretation," he said. “But in that regard, it’s clear, we don’t see evidence that peer effect could explain any substantial part of KIPP’s effect.”

Finally, as part of the study, the researchers interviewed both those students who won the KIPP lottery and those who didn’t to try to identify any effects of KIPP on student attitudes and behaviors. They found, not surprisingly, that KIPP students do more homework. But the other finding here was surprising and the only negative note in the report.

“According to the students’ self-reports, the KIPP kids are more likely to engage in kinds of undesirable behavior that lead to conflicts with teachers or parents.” The researchers are not sure why that might be, but Gill has a couple of guesses: KIPP students possess more self-awareness, or they simply have more time to get into arguments, given the longer days and additional homework.

The study did not follow KIPP students beyond their middle school years to see how well they do in high school or college.

“We don’t have data to follow the kids long enough for that,” Gill said. "We would like to do that at some point in the future.”