Florida took another meaningful step today with academic oversight of the private schools in the Tax Credit Scholarship program, releasing a report disclosing test scores gains not only statewide but for individual schools as well. The report documents a fifth consecutive year of learning gains for the low-income scholarship students while at the same time revealing the challenges of reporting on schools with as few as 30 students taking the test.
First, let’s review the big picture. Today’s report looks at norm-referenced tests administered during 2010-11, a year in which the total K-12 enrollment was 34,550. The state hires an independent researcher, the respected Northwestern University professor David Figlio, to collect and analyze the test data. He received 16,575 test score gains from students in grades 3-10 who had test scores for two consecutive years.
This is the fifth annual report and it says once again that the students who choose the scholarship are the lowest-performing students from disproportionately low-performing public schools. That’s reassuring because it means the scholarship continues to serve the students in greatest need, and does not skim public schools of higher-performing low-income students. Dr. Figlio was quite clear on that point: “The tendency for the weakest prior performers on standardized tests to choose to participate in the FTC program is becoming stronger over time.”
These same students, he finds, are showing test-score gains in reading and math that almost precisely track the progress made by all students of all income levels nationally. For numbers crunchers, it looks like this: The typical student in the program scored at the 45th percentile in reading and the 46th percentile in math. When compared to rankings the previous year, the change in reading was 0 percentile points and in math was -0.9, which is statistically the same as 0. This is a case when zero is a good score. It means the students are keeping pace with the national cohort, which is the policy equivalent of saying they gained a year’s worth of knowledge in a year.
Given that these are students who previously struggled and whose household income is only 12 percent above poverty, those gains are particularly encouraging.
Unfortunately, Dr. Figlio was unable this year to conduct a regression analysis to get an accurate view of how these scores compare with low-income students who remain in public schools. Step Up For Students, the nonprofit that administers the scholarship, improved its online application system in 2010-11 to include a pre-screen that helps families understand whether they meet the income eligibility prior to starting a full application. That left him with only a small cohort of families who applied but were denied because their income was only slightly above the threshold – the very cohort he used for his regression analysis.
In turn, the report uses a direct comparison with all students on free or reduced-price lunch in Florida public schools and shows that scholarship students had roughly the same gains statistically speaking – 1.1 higher in math and 0.8 lower in reading. (That math gain score represents a 3.5 percentile point increase from 2009-10 and reading increased by 1.7 as well.) But Dr. Figlio termed the direct comparison “problematic” because the scholarship students are considerably poorer and left public school with considerably lower test scores than their public school counterparts.
That’s the backdrop to the new portion of the report – a requirement enacted by the State Legislature in 2010 that individual school gain scores also be disclosed.
The threshold in the law is any school with at least 30 students in grades 3-10 who have current and previous test scores from the same school, which aligns with public school standards even though the test itself is different. Public school students take the Florida Comprehensive Assessment Test (FCAT) and scholarship students take a nationally norm-referenced test approved by the state Department of Education. The large majority take the Stanford Achievement Test. For 2010-11, 70 schools met the threshold. That numbers strikes some people as small, given that 1,114 schools participated that year, but it’s inherent in the Florida program design. The average number of students in each school that year was 31, and a disproportionate share was in untested grades K-2.
The results were so complicated by the small sample size – an average of only 50.3 student scores per school – that Dr. Figlio included a three-year average alongside the one-year score to give greater context. In general terms, the 70 schools tracked the broader statewide distribution of test scores. Dr. Figlio put them in three groupings based on a 95 percent confidence standard. For the one-year learning gains, 50 kept pace with the national sample, eight exceeded and 12 fell short. For the three-year learning gains, 51 kept pace, five exceeded and 14 fell short.
The law provides no rewards or sanctions related to the release of school test gain scores, and we’ll know soon enough how lawmakers, educators and the public react to them. But Step Up administrators reached out to administrators at the schools receiving negative one- or three-year scores and heard some encouraging words. Several principals said they intend to use the scores to dig deeper and to talk with teachers about their approach with scholarship students, and none of them pushed back on the disclosure requirement itself.
These are promising academic signs for Florida and for the estimated 47,000 low-income students who will be receiving scholarships this fall.
