This article was originally published in The Notebook. In August 2020, The Notebook became Chalkbeat Philadelphia.
This is a reprint of an article that originally appeared in Education Week.
by Katie Ash
A new report finds that students in KIPP charter schools experience significantly greater learning gains in math, reading, science, and social studies than do their peers in traditional public schools.
The study, which analyzed data from 43 middle schools run by KIPP, officially known as the Knowledge Is Power Program, was conducted by Mathematica Policy Research, a research center based in Princeton, N.J. It concludes that students in the charter program, over a three-year period, gained an additional 11 months of learning in math, eight additional months in reading, 14 additional months of learning in science, and 11 additional months of learning in social studies when compared to students in comparable traditional public schools.
The study is based on an examination of standardized test scores, as well as students’ scores on the TerraNova assessment—a nationally norm-referenced, low-stakes test.
KIPP chief executive officer Richard Barth said he was pleased but not surprised by the results.
"I know how hard our kids are working and how hard the teachers are working. People are on the same page working together, and this reflects that," he said in an interview.
The Mathematica researchers used two methodologies to draw comparisons between KIPP students and their peers in regular public schools. First, they used a matched comparison, linking KIPP students with students in district schools with similar demographics and baseline academic achievement. Forty-one schools were used in the matched sample.
In addition, researchers compared students through a lottery-based analysis, which compared two sets of students—those who applied to a KIPP school through a random, lottery-based system and were not selected, and those who went through the same process and gained admittance. This method is designed to account for differences in the motivation levels of parents and families who ended up in the KIPP schools and those whose children were not selected and therefore attended school in the traditional district, the researchers say. However, this sample size of schools was much smaller than the matched comparison—only 13 schools—because not all KIPP schools have enough demand to warrant a random lottery selection of students.
Overall, 125 KIPP schools are operating in 20 states and the District of Columbia.
The findings were consistent using both methodologies, which reinforces and bolsters the study’s conclusions, said Ron Zimmer, an associate professor of public policy and education in the Peabody College at Vanderbilt University, who has studied the performance of charters and traditional public schools.
"This is hard work to do the different sensitivity analyses," he said. "The results are robust, and I have a lot of confidence in their results that they present here."
The study also delved into the demographics of KIPP students compared to their district counterparts. It found that KIPP schools have a higher proportion of low-income and black students but typically have fewer special-education students and English-language learners than traditional district schools. The study also found that KIPP students typically enter the program with a lower baseline math and reading achievement than students at the traditional elementary schools that feed into KIPP middle schools.
Skeptics have claimed that as KIPP grows, it will attract more high-performing students, said Steve Mancini, a spokesman for KIPP. But the study finds that in fact, "the kids we’re attracting are even less proficient [than district schools’ populations]," he said.
In addition, to address the claim that KIPP expels or encourages low-achieving students to drop out of the program in a way that inflates the network’s performance, in the report, the academic achievement of students who started in the KIPP program is counted as part of the KIPP cohort regardless of whether they later decide to leave KIPP for another school. For example, a student could leave KIPP for another school in 6th grade, but their performance at the new school is counted towards the academic achievement of KIPP students overall regardless. The report also found that KIPP schools have similar attrition rates as traditional district schools (37 percent over three years for both sets of students).
"It’s a credible way to deal with the criticism that [KIPP] is selectively counseling out kids who aren’t doing well," said Jeffrey Henig, a professor of political science and education at Teachers College, Columbia University.
A study published by Gary Miron from Western Michigan University in 2011 found that while KIPP’s attrition rates were comparable to traditional public schools, the network did not replace low-performing students who left KIPP, which could have a positive impact on its schools’ overall academic performance.
A working paper released by Mathematica in September 2012 found that "in 7th and 8th grades, KIPP schools were less likely to replace those students who did leave early. Thus, this could affect the composition of students at KIPP schools during the latter years of middle school, but not in 5th and 6th grade, when the impacts of KIPP on student achievement are strong," said Philip Gleason, a lead author of the working paper as well as the new study, in an email interview. Another researcher on the study, Brian Gill, added, "lower backfill rates in 7th and 8th grades cannot ‘inflate’ the [academic gains shown by students in the new study] anyway, because we are measuring impacts on individual students and including the kids who leave KIPP."
As noted by Education Week in a recent series on charter school discipline policies, KIPP officials say they have circulated data on the attrition of schools within their network in recent years, a step they believe has called attention to the issue and resulted in schools taking steps to keep more students in school.
When it comes to why KIPP schools have seen such gains, said Henig of Columbia University, "the study is a lot stronger about making the case that there is something working than it does about what it is that makes [KIPP schools] stronger."
One factor that researchers pointed to as a possible key to KIPP’s success is the amount of time spent in school. KIPP students spend an average of nine hours per day, for 192 days each year, in school, compared to 6.6 hours per day, for 180 days, for traditional public schools, the report found. In addition, KIPP students spend an extra 35-53 minutes on homework each night than students not enrolled in KIPP. But the study also found that having a longer school day doesn’t always correlate with greater achievement. In fact, the KIPP schools with the longest school days actually had slightly lower achievement than KIPP schools with shorter days. Researchers found that the element most strongly associated with higher achievement was more time spent on core-subject areas.
Whether those results could be applied to traditional public schools is up for debate, said Henig. "Fiscally, and politically, and given union contracts," dramatically extending the school day in district schools would be very difficult, he said.
The report also points to KIPP’s comprehensive behavior system as a contributing factor to its success. In schools where school-wide behavior standards and discipline policies are consistently communicated and enforced, the school rewards students for positive behavior, and the school punishes students who violate the rules, reading and math scores went up, researchers found. No other school climate factors yielded a statistically significant correlation with academic performance.
Using student and parent surveys, the researchers also found that students in KIPP schools were more likely to admit to arguing or lying to their parents, giving teachers a hard time, or losing their temper at school. The researchers speculate that this finding could be based on negative impacts of KIPP on student behavior or a willingness of KIPP students to self-report their conflicts more honestly than other students.
One area of concern could be the amount of teacher turnover in KIPP schools and potentially overworked principals, said Henig. The study found teacher turnover in KIPP schools to occur at slightly higher rates than traditional public schools (21 percent compared to 15 percent). KIPP principals in the study reported working an average of 74 hours per week on work-related activities—an average of 12 hours per day, six days per week.
Those findings raise concerns about "how replicable this model is in terms of going to scale given the high demands [on teachers and principals,]" said Henig. "When the economy [is] tough, it might be easier to get young college grads …. but they may find that it’s harder to keep filling the bucket in terms of teachers and principals."
Barth, KIPP’s CEO, agreed that principals at KIPP work long hours, but said that as the network evolves, greater efficiencies could be achieved. "We think that as we have matured we’re finding ways to work smarter as well, and as principals stay with us over time, the amount of hours they work will decline," he said.
The study builds on previous research conducted by Mathematica about KIPP schools. Future research will delve further into the topics of leadership practices in KIPP, said Christina Tuttle, a researcher with Mathematica and a lead author of the study, in a conference call with reporters on Tuesday.