The School Improvement Grant (SIG) program, supercharged by a $3 billion windfall under the federal economic-stimulus program in 2009, has jump-started aggressive moves by states and districts. To get their share of the money, they had to quickly identify some of their most academically troubled schools, craft new teacher-evaluation systems, and carve out more time for instruction, among other steps.

Some schools and districts spent millions of dollars on outside experts and consultants. Others went through the politically ticklish process of replacing teachers and principals, while combating community skepticism and meeting the demands of district and state overseers.

It’s not at all clear if the federal prescription can cure the most ailing schools and lead to long-term improvements, but preliminary student achievement data for the program offer some promise. The U.S. Department of Education looked at about 700 of the schools in their second year of the program and found that a quarter of them posted double-digit gains in math during the 2010-11 school year. Another 20 percent showed similar progress in reading.

A collaborative reporting project drawing on the efforts of more than 20 news organizations (including Catalyst Chicago) and affiliated journalists paints a mixed picture of how the SIG program is playing out on the ground. The project was organized by the Education Writers Association, the Hechinger Report at Teachers College- Columbia University and Education Week. 

The major findings show:

• States have pulled SIG money from at least a dozen schools that showed anemic progress on early indicators of success, such as teacher and student attendance, according to the U.S. Department of Education.

• Schools nationwide, especially those in rural areas, are wrestling with staff turnover and the need for new teacher-evaluation systems driven by the program’s requirements, along with the challenge of adding extra instructional time.

• Though millions of dollars in grant money has gone to outside contractors, few states track the details of how that money is being spent—and some contractor-run schools have seen student performance sink.

 At the same time, the program’s supporters can point to encouraging—though early—developments. And some of the best early reviews come from students, who say their schools are calmer and more academically rigorous.

“I feel more safe, and I feel like I’m learning more. They are starting to have challenges for us,” said Jasmine Dukes, a seventh-grader at Friendship Preparatory Academy at Calverton, formerly Calverton Middle School, in Baltimore.

 U.S. Secretary of Education Arne Duncan also sees signs of recovery in schools across the country, even as he cautions that it’s still too early to draw conclusions about the program’s effectiveness.

“Big picture, there’s really significant movement in a very short amount of time, which I think a lot of folks felt wasn’t possible,” Duncan said. But he doesn’t expect overnight success: “This is really, really hard work; there’s a reason the country took a pass on this for a couple of decades.”

All of which argues for caution in assessing the program’s effectiveness so far.

“There’s evidence on both sides of the coin,” said Robert Balfanz, the director of the Everyone Graduates Center at Johns Hopkins University and a leading researcher on school improvement. “The big question is whether those changes are going to lead to improvement.”

In an April 11 report, the Government Accountability Office, the investigative arm of Congress, said the Education Department failed to do enough to ensure that states and districts monitor the performance of contractors hired with SIG funds. GAO also recommended that the department do a better job of spelling out how states should make "evidence-based" decisions about whether to remove schools' grants. The report found that among 23 of the 44 states responding to a GAO question about such decisions, most or all of the schools that had their funding renewed failed to meet annual goals.

In a formal response, Michael Yudin, the acting assistant secretary for elementary and secondary education, called the GAO report “incomplete and somewhat misleading.”  He agreed that the Education Department needs to do a better job of helping districts and states decide when to pull the plug on a grant. But he disagreed with the content