As a young graduate student, I participated in an exercise that should be routinely practiced in social science training.
Our statistical methods professor gave us an article published in a top political science journal and provided the same data used by the authors.
Our assignment was to replicate the results. The authors described the methods they used in the article, we despite having the same data, try as we might, none of us could replicate the results.
Our professor had taught us an important lesson about social “science” without having to say it out loud: caveat emptor.
All kinds of things can go wrong in social science research, some errors more innocent than others, so an informed consumer will be looking for results across multiple studies using higher quality methods and from scholars willing to share their data for others to examine. In situations where data is not shared and methods are complex and opaque, the opportunities for mischief multiply rapidly.
Having a press that tends to breathlessly report on studies that confirm pre-existing political narratives is not helpful, as these invariably travel around the world well before there has been time for replication or scrutiny.
This all came to mind recently when I read the following about research claiming to demonstrate that warm weather negatively impacts the academic achievement of students – but only Black and Hispanic students:
In a paper published Monday in the journal Nature Human Behavior, researchers found that students performed worse on standardized tests for every additional day of 80 degrees Fahrenheit or higher, even after controlling for other factors. Those effects held across 58 countries, suggesting a fundamental link between heat exposure and reduced learning.
But when the researchers looked specifically at the United States, using more granular data to break down the effect on test scores by race, they found something surprising: The detrimental impact of heat seemed to affect only Black and Hispanic students.
So, could there be a correlation between climate and student learning?
There could be, and there should be more research performed. Do we have reasons to be skeptical? I would say we do. Florida, for instance, gets plenty of days of 80 degree plus weather, but here are the academic trends for White, Black and Hispanic students on NAEP:
That looks a lot like across-the-board improvement to these eyes. The percentage of Florida Black students scoring “Basic or Better” more than doubling between 1992 and 2019 in balmy Florida does not mean there is no role for climate, but it does seem it’s possible to overcome whatever that role may be.
I happen to live in a desert in a state (Arizona) which has a large majority of the state’s students living in Maricopa County, the greater Phoenix area, which is a desert – and a hot one to boot. Stanford University recently compiled data that allows us to compare academic growth rates by county. Maricopa County Hispanic students had an academic growth rate 16% above the national average during the period covered by the Stanford data (2009 to 2016) and Maricopa County Black students were 11% above the national average.
Delightfully, Arizona kids did not get the climate memo either.
Currently, approximately half of white students have access to in-person learning compared to only one-quarter of Black and Hispanic students. A recent analysis of district reopening decisions found politics and teacher union strength to be more influential than COVID-19 trends. The same analysis found that districts with more Catholic schools were more likely to reopen for in-person education.
Luckily for them, in my opinion.
This study, of course, deserves the same sort of scrutiny that the previous one discussed. I fear, however, that we have far more immediate concerns regarding growing achievement gaps than climate change.
