Become a Catalyst member

Join the conversation

We encourage our readers to leave comments and engage in dialogue about our stories. But before you do, please check out our "rules of the road."

Subscribe to catalyst-chicago.org by e-mail

catalyst-chicago.org feeds

Current Issue

College and careers

An overhaul of the district’s career education programs seeks to make classes more challenging and put career-track students on the path to higher ed, but many schools have lost programs, and fewer students are participating overall.

Turnaround study shows only small gains

UPDATED: A long-awaited study that CPS touted as proof of the success of turnarounds was released Wednesday--and instead showed only a small amount of progress and added fuel to calls for the district to back off.

This morning, a coalition of Local School Council members, backed by, Chicago Teachers Union announced that they plan to file a lawsuit asking for an injunction to stop the district from moving ahead with plans for 10 turnarounds, including six at schools that would be handed over to the non-profit Academy for Urban School Leadership. The crux of the lawsuit is that state law requires local school councils to be given specific action plans before being turned around. The LSC members from the targeted schools say they were never given such directives. 

The CTU has long fought against school closings and pointed out they result in the firing of mostly black teachers. One finding of the study by the Consortium on Chicago School Research supports that claim: At turnaround schools, the number of black teachers plummeted, while the number of white, less experienced teachers increased.

But the study provides only a little evidence that the upheaval and financial investment in turnarounds is worth the substantial cost, estimated to be at least $20 million in the coming school year. That does not include millions more in capital spending on building renovations at these schools.

In fact, the study draws no conclusion about whether turnarounds are the best strategy to improve the lowest-achieving schools, or whether the Academy for Urban School Leadership is the best organization to carry it out.

“That is not what we were studying,” said Elaine Allensworth, a senior director and chief research officer at the Consortium.

The study found that student achievement improved at schools that underwent particular types of intervention. But the study made no distinction between three radically different strategies: schools that were turned around; schools that were closed for low achievement and re-opened as new schools, with a mostly-new group of already higher-performing students; and schools whose principals were sent to a two-year professional development program at the University of Virginia. 

The only school-by-school performance data provided indicates that, compared to similar low-performing schools, AUSL's turnaround schools moved their students ahead at a slightly faster pace. But the improvement was small, and one expert questions its significance.

Allensworth and co-author Marisa de la Torre said the growth of AUSL students is statistically significant, but refused to attach any adjective to the improvement and declined to provide an example of how much actual test scores differed or changed between the turnaround and comparison schools. (Instead of actual test scores, the study relied on standard deviation of test scores.)

“All I can say is that for some people the differences will be big, for some other people not so much,” Torres said.

CPS Chief Education Officer Noemi Donoso said the fact that the turnaround schools did better, regardless of how much better, was enough for her.

“The alternative is the status quo,” she said. “If the choice is between flat-lining and closing the gap, I would rather close the gap.”

But education experts consider the statistical figure of average yearly growth by turnarounds cited in the study as evidence of improvement to be relatively small. 

“0.07 is a pretty small effect,” said Dan McCaffrey, a statistician at the Rand Corporation, a not-for-profit research and analysis organization. “It doesn’t mean that it is not meaningful, but it is small."

McCaffrey also criticized the study's inclusion of schools that underwent drastically different interventions. The schools that were closed and re-opened as charters and magnets got virtually all-new students, who were higher-performing when they enrolled than the students who were in the school previously, according to information in the study.  

“Shifting students and changing labels is not a legitimate way to improve a school,” said McCaffrey. 

A student shift, though much smaller, could be at play in some turnarounds. At Morton and Howe —the two highest-performing AUSL turnaround schools—students at the schools in the fall of the first year of the turnaround had significantly higher reading performance than students from the prior, according to the Consortium. These two schools also experienced enrollment increases, which seems to suggest that better-performing students were attracted to or recruited to attend.

Challenge from the feds

Even before the Consortium released the study, it had already been challenged. The study was funded by the Institute for Education Sciences, the federal agency that oversees education research and reports education statistics. But Rebecca Maynard, a commissioner at the agency, refused to approve the final version. (The Institute is headed by John Easton, the former director of the Consortium.)

Allensworth said Maynard wrote a short explanation that indicated she didn’t agree with the methodology.

But Maynard said she was worried that the study was drawing a conclusion without enough information. “Parts of the report were written in a way that could suggest the study was intended to answer more complex questions than was judged to be possible with the available data,” she said in an e-mail.

The Consortium analysis compared a school that underwent an intervention to a single other school with similar demographics and performance. With such a small sample--44 schools in all--other factors could have affected whether or not schools improved, such as principal turnover and extra resources. 

Donoso acknowledged that schools often wind up making trade-offs because of scarce dollars, choosing between hiring extra security or paying for after-school programs. But, she said, that is the choice of the local school council and the principal.

The study also looked at the progress of two initiatives to improve high schools. One, reconstitution, was a process that mirrored turnaround in that staff were replaced. According to previous research, that strategy failed.  

And the first results from turnaround high schools are not good. Because the first turnaround high school was opened in 2009, no test scores were available for inclusion in the Consortium study. However, the attendance rate increased in the first year--then declined after that.

In addition, there were only small improvements in the freshman on-track rate, which takes into account absences and grades in freshman year and is considered a strong predictor of graduation--or dropping out.

7 comments

Rod Estvan wrote 2 years 10 weeks ago

the Consortium study

I have to assume the Consortium study will be released later today because it was not yet posted early this morning. Ms. Karp I am confident captured the essence of the study. I am sure of this because I have always found her reporting to be excellent. I will none the less read the report myself and draw my own conclusions.

The quote from CPS Chief Education Officer Noemi Donoso that limited progress is better than "flat-lining," was very disturbing to me. The reason I found it to be disturbing was because the turnaround process as Ms. Karp's article points out is not cheap, and the CPS has a responsibility to the tax payers and citizens of Chicago to operate in a cost effective manner. Since we are operating currently in an economic situation of a state which is in deep crisis CPS has to assess the turnaround process not just in terms of small test score gains, but also based on a cost benefit analysis on a district level.

Apparently any cost benefit analysis related to the turn around process is not on the radar screen of CPS. The consequences of CPS failing to examine whether the existing turnaround process is simply worth it will come back to haunt the district because both the citizens of Chicago and the Illinois General Assembly will at some point come after CPS with the pitch forks out unless there is greater accountability.

When I do get to read the Consortium study I will be looking for its analysis of how students with disabilities at these turnaround schools improved or did not improve schools. I will be very pleasantly surprised if I find data on this issue in the study.

Rod Estvan

Lorraine Forte wrote 2 years 10 weeks ago

Responding to your comment

Rod, thanks for your perceptive-as-usual comments. The study does not include any analysis pertaining to students with disabilities.

Anonymous wrote 2 years 10 weeks ago

Junk Man

So, I'd have to say that such a study that does not break out for performance of students with disabilities is junk. Sorry.

lforte wrote 2 years 10 weeks ago

student performance

the report does not break out students with disabilities, nor does it differentiate between the different strategies, as our story pointed out

Rod Estvan wrote 2 years 10 weeks ago

I have now read the report

I have now had a chance to read the Consortium report and have to agree with Ms. Forte that there is no analysis in the report of the academic growth or lack of such growth for students with disabilities for any of the turnaround schools included in the study. But it is very clear that the researchers had this data at their disposal because all measurements of the turnaround schools are adjusted for changes in special education enrollments at the turnaround schools before and after the turn around. However, this adjustment did not appear to take into consideration more complex factors such as the types of disabling conditions or the percentage of disabled students who were given the IAA as opposed to the ISAT.

If we look at the ten AUSL elementary schools (see Consortium report page 7 for a list of these schools) in the study and examine their special education enrollments using their data from FY 2006 and compare that data to FY2012 (all data retrieved from http://research.cps.k12.il.us/cps/accountweb/Reports/allschools.html) we see some very interesting things. In FY2006 these ten elementary schools combined enrolled 554 students with disabilities by FY 2011 these schools enrolled 518 students with disabilities representing a decline of 6.5%.

Of these ten elementary schools three schools experienced an increase in special education enrollments (Bethune had the largest increase of 28.5%) and six schools experienced a decrease in special education enrollments (Morton had the largest decline of 33.3%). One school had no increase or decrease. This decline in the total special education enrollment of these ten AUSL elementary schools is not consistent with the district wide special education enrollment figures. On a citywide basis in FY 2006 there were 49,374 students with disabilities and in FY 2012 there were 49,419 such students, a very minor increase of 45 students on a district wide basis or .09%.

To be fair we need to look at total enrollments at these ten schools. On the 20th day of 2006 these ten schools collectively had 5,802 students, by the 20th day of 2012 these same schools had 4,938 students or a decline in enrollment of 14.9%. We can compare this enrollment decline at these ten schools for the school district as a whole. On the 20th day 2006 CPS enrolled 420,925 students and by the 20th day 2012 the enrollment stood at 404,151 or a district wide decline of 3.9%.

As is obvious the 10 AUSL elementary schools exhibit data trends that are very different than for the school district as a whole, which in part is why the Consortium used what it called a "matched group of schools" to compare these 10 schools to. I look forward to seeing the Consortium's full report so I can see if the schools the Consortium used to compare to these 10 AUSL schools to experienced similar special education enrollment declines or overall enrollment declines.

Rod Estvan

Marc Sims wrote 2 years 9 weeks ago

Turnaround study shows only small gains

Most Chicago Public Schools and Charter Schools in the African American community are up against a culture where education is not a top priority.

The Chicago Public Schools need to institute mandatory parenting classes for all CPS parents.

marcsimschicago@gamil.com

Clara Fitzpatrick wrote 2 years 9 weeks ago

The Culture of African American Communities

Your statement is much like the Consortium report, drawing conclusions that the data don't support.

Add your comment

The content of this field is kept private and will not be shown publicly.
go here for more