This article was originally published in The Notebook. In August 2020, The Notebook became Chalkbeat Philadelphia.
Sulton Glass is just 7 years old, but he can ride his bike in traffic.
He’s learned to strap on his helmet, check his tires, and follow the rules of the road. That means he can join the other kids from the Neighborhood Bike Works for their weekly ride, a wobbly, giggling excursion through University City to the Woodlands Cemetery. There he’ll listen to a repair lesson, practice his hand signals, and swoop happily up and down the hilly, car-free roads that wind through the headstones. Finally he’ll follow the group down Chester Avenue, past Clark Park, up Locust Street, and safely back to his waiting mother.
She says Sulton has come a long way. “He thought he was the bike king until he came here and learned how to do things the proper way!” laughs Sharon Glass, sitting on a bench outside Bike Works headquarters. Now, she says, he’s having fun, he’s learning, and he’s losing some of the sass he was learning in the neighborhood.
“You’ve got to let him go outside and be a boy. But he started picking up bad habits. I was having little behavior issues with him,” Glass says.
The step-by-step instruction he gets at the Bike Works – he attends two days a week after school, learning to take bikes apart and put them together again – is exactly the “hands-on” work he needs. He’s getting more focused and less disruptive. “I see the difference,” says Glass.
So she’s glad she spotted the Bike Works on a flyer handed out at her son’s school. It’s not always easy for parents to find quality afterschool programs, Glass says.
If the city had a good directory of what was available, how it worked, and how much it cost, “that would be good. They need that.”
City officials know she’s right. But organizing Philadelphia’s sprawling collection of so-called “out-of-school-time” (OST) programs is much easier said than done.
A ‘system of systems’
“It’s been a big mash of programs for a long time,” says Thomas Sheaffer, the policy director in the city’s Office for Health & Opportunity. “There’s not anything quite like it.”
Sheaffer is the point man of the Philadelphia OST Project, the city’s effort to create a “system of systems” to coordinate the vast array of OST programs that collectively serve more than 40,000 children each day. Backed by the Wallace Foundation, the William Penn Foundation, and the United Way, Sheaffer is working with a wide array of OST providers and managers to improve the quality, accessibility, and distribution of OST programming citywide.
These programs come in all shapes and sizes. The city funds programs through the Department of Human Services, the Department of Parks and Recreation, and the Philadelphia Free Library. The state funds programs through its 21st Century Community Learning Center grants. There are citywide efforts like the After School Activities Partnership (ASAP), community-based efforts by nonprofits like Congreso, and countless independent operations like Neighborhood Bike Works.
ASAP’s online OST directory – the most comprehensive available – includes nearly 800 listings. Just keeping it up to date is “an ongoing battle,” says ASAP director Justin Ennis.
Identifying what’s effective
Finding out what works– and sharing that information with parents, funders, and policymakers – is harder still.
“A couple of funders have approached us and said, ‘What’s a good program?’ And I say, ‘Well, I’m not sure,’” Sheaffer says.
“But I think if we can pull off what we’re trying to do with our grant, we’ll be able to tell you. If an agency is working with us, that means they’ve committed to data collection, and something other than throwing a ball out and letting the kids play for an hour.”
Philadelphia’s challenge is replicated nationwide.
“Cities don’t have the kind of information they need to make decisions, especially around funding,” says Nancy Devine, Wallace’s director of learning and enrichment.
Cities struggle to determine how good programs are, how effectively they’re distributed, and who is being served, she says. Are there language-enriching programs available in immigrant communities? Is one neighborhood stronger than another in science or math-based opportunities? Are there enough safe-haven options in high-crime communities? Better answers could help cities make better decisions about where to spend what they’ve got.
Likewise, better data about program quality would help funders and parents alike. But that’s where things get really complicated.
“I wish it were easier!” says Amy Friedlander of the Philadelphia Health Management Corporation (PHMC), which serves as the “intermediary” monitoring city-funded OST programs. “It should be so straightforward, but it’s far from it.”
PHMC collects basic information from DHS-funded providers about attendance, safety, and contract compliance, she says. And in recent years it has taken on the task of boosting the quality of its providers’ offerings by asking them all to implement a “project-based learning” model that provides students with structured activities – anything from recording weather conditions in the local park to producing a play or hosting a cooking competition – organized enough to have demonstrable pedagogical value. Each DHS-funded site gets at least three visits a year from PHMC field workers, who work with site staff to implement project-based learning.
Need for more data
But even with that, when it comes to quality, the most PHMC can say is whether the city’s programs replicate models that have proven successful elsewhere. “Right now, we just kind of say, ‘The national data shows this, and therefore we’re monitoring our programs and pushing towards high utilization,’” Friedlander says.
Thus, a major goal for the Philadelphia OST Project, launched in 2012, is to figure out how to collect and disseminate more useful data about attendance, demographics, results, and more.
It’s a complicated prospect in more ways than one, Sheaffer says. Among the challenges, DHS can only “compel” those programs it directly funds to collect certain data. Getting comparable data from programs run or funded elsewhere can be tricky. Many recreation centers, for example, don’t even have the computers needed to enter data about program participation.
“That’s something we need to address, and we’re intending [to],” Sheaffer says.
What’s more, he says, because OST programs are so diverse, not all share the same priorities about what constitutes a “successful” outcome. Some want their programs to be as little like school as possible.
“Certain recreational programs say, ‘Don’t hold me accountable for academic improvement, that’s not what we do,’” Sheaffer says.
So it’s Sheaffer’s job to help find common ground among the providers and departments that manage them. “We’re working on it,” he says.
No one expects OST programs to start monitoring student test scores to see whether a given program leads directly to better grades. But Sheaffer believes that program providers could feasibly report more about things like student demographics (who’s serving high concentrations of special education or ESOL students?); program profiles and goals (how structured is the program? What’s a typical day like? What’s the student-teacher ratio?); and program outcomes (do attendees skip school less often?).
Christine Caputo at the Free Library of Philadelphia says her organization is “really excited” about the prospect of joining the “system of systems,” in part because better information about participation and outcomes would help the library build support from the city and private donors.
“A kid gives you a hug, and that’s the greatest feeling in the world,” she says. “But it’s hard to report that to the city.”
At the same time, she said, the success of the library’s programs will always depend on their staff’s relationships with students. It would be helpful to have a data-sharing system in place that could tell the library which students are struggling in school, she says, but that can’t replace the value of a stable and effective staff that can get to know students personally.
“More data is always useful, but it can be double-edged,” she says. “Kids can get pigeonholed.”
Right now, Friedlander says, the OST Project has decided on some basic quality measures it hopes all providers can collect. Programs should be able to report whether attendees show improved life skills, better relationships with adults, more clearly focused goals, better engagement with school, and, for older students, college- and work-readiness.
It’s all data that would help clarify which programs are effective and why.
“Part of why we want data is [to] document impact,” she says. But how might that data be reported and collected? That’s work yet to be done, Friedlander says, “We have not yet identified all of the tools we would need.”
‘Where can we send them?’
Damien Ramirez, 11, stacks tires before a weekly bike ride organized by Neighborhood Bike Works. (Photo: Charles Mostoller)
Back at the Neighborhood Bike Works, Liz Pisarczyk is also thinking about impact. The program has typically served about 500 students a year, but a recent strategic planning process has them thinking less about quantity and more about quality.
“We’re recognizing how important it is to reach the students who keep coming back,” says Pisarczyk, the Bike Works’ program director. “Not to say we’re not trying to reach a certain quantity, but maybe we can offer more long-term activities for older youth who’ve stuck around.”
That means focusing on some of the same things that Sheaffer and Friedlander hope other OST programs can report. “We’re really focusing on preparing our youth for career skills and life skills,” Pisarczyk says.
Likewise, she says, the Bike Works could really benefit from the kind of comprehensive directory that the OST Project envisions. “Of course we want kids to come into our programs – but we also want them to discover other programs, especially if they’re in need of tutoring or something like that,” she said. “Right now, that’s where we struggle – a kid needs something very specifically, and we’re kind of, ‘Um, where can we send them?’ I wouldn’t say it happens frequently – but it happens.”
For now, the best resource available remains the ASAP program guide, which has provided program names and basic information by zip code.
An updated version will be online soon, Ennis says. “Now you’ll be able to search for specific programs, or programs that are just for teens, or where meals are provided, where transportation is provided.”
And as the OST Project’s work takes shape, Ennis says, the directory could be even more comprehensive and quality-focused.
But exactly how that might work is still not clear. Under consideration, Sheaffer says, is whether a guide should include some kind of “seal of approval” or rating system – the equivalent of the “Keystone STARS” system used for preschool and child care. Also being considered is a Yelp-style public input process, “where kids and parents could post comments about the program being good or lousy,” Sheaffer says. “That’s a little scary, but might be better than any master quality measurement.”
Also unclear is the impact that the School District’s so-called “doomsday budget,” which promises to strip schools of all but the minimum of legally required staff, will have on next year’s OST programming. The battle for more funding looks set to carry on all summer, and officials say the exact budgets and staffing levels for individual schools won’t be clear until close to September. However, District spokesman Fernando Gallard said that austerity could require the District to close up its buildings at 3 p.m. More than 100 of DHS’s programs are located in school buildings.
In response, ASAP’s Ennis says that his organization is working on two fronts. The first is ensuring that “receiving” schools carry on the programs lost at schools that will close in September. ASAP, which operates chess, drama, Scrabble, and debate clubs for more than 5,000 students, is also preparing for the possibility that it won’t be able to run OST programming in any District schools.
If that’s the case, Ennis said, ASAP will look for other sites: recreation centers, Catholic and charter schools, libraries and so on. “In the worst-case scenario, if they can’t keep the buildings open, we will try to expand our partnerships with out-of-school providers,” he said.
The clock is ticking
Despite all the turmoil and uncertainty, Sheaffer is hopeful that the OST Project can produce not only an agreed-upon set of data points that all OST providers report, but a system for collecting, housing, and managing that data.
He knows that the diversity of OST programming – with its multiple funding streams and its wide array of visions for serving students – won’t change anytime soon. “There’s no way to buck that,” he says. “I think we have to learn how to build on that.”
He also knows that the clock is ticking. When the Wallace funding runs out at the end of 2015, the OST Project will almost surely need to win the support of the next mayor, who’ll likely have to decide whether the city should support it permanently.
That means not only getting a “system of systems” at least partially up and running before Mayor Nutter leaves office, but showing that it can be useful: helping parents choose where to send their children; helping funders decide where to send their dollars; and helping programs evaluate themselves and improve.
Wallace’s Devine says the new system won’t have to be perfect, but it will probably have to show progress in order for a new mayor to support it. Sheaffer agrees, saying that if the “system of systems” doesn’t win the support it needs to be sustained once the Wallace funding and other support dries up, he says, the OST landscape will revert to the fragmented status quo.
For now, parents like Sharon Glass will continue to rely on the more traditional non-system of systems: word of mouth, recommendations from school, and the occasional Google search.
But something better would be welcome, Glass says, because plenty of kids in her West Philadelphia neighborhood aren’t getting what they need. She sees them on the corner every day. “When we leave, they’re out there,” she says. “And when we come back, they’re still out there.”