First and foremost: The pandemic is not over, and many of these indicators will continue to change in the months and, yes, years to come. This is a first attempt at tallying the pandemic’s impacts at the state level; it’s not the final word.

Second, many of these measures are preliminary. The economic data, in particular, is constantly being updated and retrospectively revised by the government agencies that collect it. States that appear to fare well or poorly now may look different down the road.

Third, some of the indicators, particularly for education, are partial. Again, we can expect more and better data to become available in the future. While imperfect, this preliminary data already reveals emerging patterns that we felt important to include.

We divided the data into four buckets: health, economy, education and social well-being, and — in most cases — compared the metrics to a pre-pandemic baseline.

Once we had the four policy area scores, we faced the decision of whether to average those scores and if so, how to weight those categories. We decided that weighting any category more than the others — deciding, for instance, that vaccination rates should count more than math testing scores — is a political decision, not an editorial one. There are always trade-offs in public policy, and the decision on how to prioritize the relative value of health, economic, education and social outcomes are exactly the decisions that politicians are elected to make. To stay as neutral as possible we’ve weighed those categories equally in our average score, but we encourage readers to closely review the individual categories and the data behind them in reaching their own conclusions.

Importantly, we believe in showing and sharing our work. Readers can download the full data set here. Below, we have a more detailed explanation of our methodology and provide links to the underlying data.

This scorecard should be considered a tool in an evolving policy landscape that shows where states are struggling and where they are doing better. We hope that the patterns that are emerging in this data will be helpful to public officials in evaluating the trade-offs they are still facing as we grapple with this ongoing public health crisis.

— Maura Reynolds, editor, POLITICO’s Recovery Lab and Sean McMinn, data and graphics editor, POLITICO

METHODOLOGY

For each of the four categories — health, economy, education and social well-being — we calculated a score between zero and 100. Those scores represent the average of each of the metrics, outlined below, that go into those categories.

For each of the metrics in the four categories, we compared a state’s performance to other states using a z-score, which is a commonly used statistical measurement to determine how far off of “average” a given data point is. That z-score can then be converted to a percentile from 0-100 by using a cumulative distribution function.

Those percentiles were averaged for each metric across a category to calculate the category’s score.

We averaged those four category scores with each other to come up with an average score for each state. If a state was missing data for any metric, that metric was not included in the average.

The District of Columbia was not included in this analysis because it faced different challenges getting federal aid than states did and its mayor does not exercise the same set of authorities as a governor.

Health

To score states’ performance in health care, we used four metrics:

- Deaths per capita
- Hospitalizations per capita
- Covid tests completed, compared to hospital admissions
- Vaccine doses administered per capita

Deaths per capita

For deaths, we calculated the total number of Covid-19 deaths per capita starting March 26, 2020 and as of Nov. 26. We also checked to see if the age distribution of states made a substantial difference in the rankings using the CDC’s relative risk breakdown, and it did not.

Hospitalizations per capita

For hospitalizations, we compiled the total number of Covid-19-confirmed hospital admissions (adult and pediatric) per capita starting March 26, 2020 and as of Nov. 28.

Covid tests completed, compared to hospital admissions

For testing, we summed the total number of positive, negative and inconclusive tests conducted per capita as of Nov. 25. We then divided that number by the state’s hospitalizations per capita to calculate a ratio of how many tests were conducted per hospitalization. This weighting compensated for the fact that states with surges would likely see an increase in testing, and it benefited states that were testing preventatively — even when there weren’t surges.

For example, if at any point in time State A had a surge and State B had a lull in cases, you would expect there to be more tests taking place in State A. If State B was doing the same number of tests as State A, State B would appear better in our rankings because their public health operation was being more proactive.

Vaccine doses administered per capita

We calculated the total number of vaccines administeredin each state, including booster doses, per capita as of Nov. 24.

Economy

To score state economic performance we used three metrics. For each, we compared seasonally adjusted post-pandemic 2020 and 2021 numbers to those of 2019.

- GDP by state
- Job creation
- Unemployment rate

GDP

Beginning with the second quarter of 2020, we took the state GDP for each quarter and compared it to the same quarter in 2019 to come up with a percent change. We then averaged those percent changes.

Job creation

We calculated the average monthly change in nonfarm employee payrolls compared to the same month in 2019, starting with March 2020.

Unemployment

We calculated the average monthly change in the state unemployment rate compared to the same month in 2019, starting with March 2020.

Social well-being

Measuring social well-being is inherently difficult but we looked for nationwide metrics that were available with state breakdowns. We found three that met our criteria:

- Food insecurity
- Ability to afford household expenses
- Violent crime

We compared these metrics against the earliest available baseline after March 2020, and calculated the change each state experienced as the pandemic progressed. As a result, our calculations don’t compare current levels of each metric, such as food insecurity, just how much they improved or worsened after the baseline. Therefore a state with a drop-off in violent crime will score higher than a state with an increase in violent crime, even if the level of violent crime in the first state is higher.

Food insecurity

For food insecurity, we used data from the U.S. Census Bureau’s weekly Household Pulse Survey, which began in April 2020. We used the first survey as the baseline. We took the average weekly percent of people who reported that they sometimes or often didn’t have enough to eat, then divided that rate by the percent of people who had struggled with food insecurity during the first week the survey was conducted.

Ability to afford household expenses

In mid-August 2020, or week 13 of the Household Pulse survey, the Census Bureau began asking respondents how their ability to pay for usual household expenses had been affected by the pandemic. We took the average weekly percent of people that said it had been somewhat or very difficult to afford household expenses, then divided that by the initial rate reported in week 13.

Violent crime

For violent crimes, we calculated the percent change in total violent crimes from 2019 to 2020, using data from the FBI. (Data for 2021 has not yet been released.) The violent crimes in the FBI data include homicide, rape, robbery, aggravated assault, burglary, larceny-theft and motor vehicle theft. This data reflects the hierarchy rule, which requires only the most serious offense to be counted in an agency’s crime totals when multiple offenses are committed in an incident.

We chose to focus on violent crime because, as POLITICO’s Recovery Lab previously reported, property crime actually decreased during the pandemic.

Education

To score states’ performance in education, we used three metrics:

- Change in reading achievement
- Change in math achievement
- Change in enrollment

When the pandemic broke out in spring 2020, the federal government postponed the tests it administers every year to track student performance. Those tests, known as the National Assessment of Educational Progress, will resume in spring 2022 but the results will not be available until next summer at the earliest. (Some states conducted their own statewide testing during the pandemic, but those results are not comparable across states.)

In the absence of federal test scores, there is currently no educational assessment data available that reflects the performance of all students and all states. But since pandemic learning loss is expected to have very long-term effects on the economy and society, we didn’t want to exclude education from the scorecard.

As a result, we decided to include the next-best educational data available — assessments by private companies — and create as large a dataset as possible with their help. We worked with three private companies — Curriculum Associates, Renaissance and NWEA — that contract with states and districts around the country to help education officials measure student performance. Each of the testing providers crunched their data into a format that could be aggregated. Each had their own threshold for what level of data they were able to share based on agreements with their clients:

- Curriculum Associates: State-aggregated data with a minimum of 10 districts and 1,000 students tested in a given grade/subject. Only students who tested in school were included.
- NWEA: State-aggregated data only from states where multiple districts were included and approximately 20 percent of pre-K through eighth grade public school students in a state were tested.
- Renaissance: State-aggregated data with a minimum of 10 schools and 2,500 students tested. Only students who tested in school were included.

The aggregated dataset that we created is the best education assessment data currently available for comparing student achievement across states. It includes testing results from 48 states and represents tests taken by over 4 million students between grades one and eight in reading and math. (One testing company, Curriculum Associates, provided data for only third graders.)

The data captures a sample of students in each state, but it’s not a randomized sample, and it has important limitations that readers should keep in mind when evaluating the results. For instance, while the companies excluded small sample sizes from the data they provided, we don’t know to what extent the samples in the combined dataset are representative of the populations of their states in terms of demographics, ethnic diversity, income, whether the students attend an urban, suburban or rural school, or other factors. We also don’t know whether the students took the tests at home or in school; students taking the tests at home tended to score higher. One company provided data only for students who took the test at school, as opposed to at home. The other two reported that a majority of tests were taken at school, or in districts where more than 50 percent of schools were not physically closed.

We did not account for differences in demographic groups between tests, in part because testing providers did not provide detailed district- or school-level breakdowns in order to preserve student privacy. There were, in general, differences between testing companies: Renaissance, for example, reported 47 percent of their test-takers who indicated a race were white, while NWEA reported 58 percent were white. The data may also be sampled from private or religious schools at higher or lower rates than a state’s actual enrollments.

In short, the education data in the scorecard is the best available right now, but it’s imperfect, preliminary and partial, and readers should keep those limitations in mind. Recognizing this, we are not publishing specific testing scores for each state, but are instead assigning a score based on a grouping detailed lower in the methodology.

Maryland and Wyoming did not have private testing data that met the threshold criteria, so they do not have a score in the education category. Those states’ average scores were calculated using only the health, social well-being and economic categories.

Reading and math achievement

These two measures were calculated separately and together comprised two-thirds of the education score. For each testing provider that had data for a given state, we took the median percentile of spring 2021 compared to a pre-pandemic nationwide norm and converted it to a normal-curve equivalent using the following formula:

( 21.06 * percent-point-function of (PERCENTILE_RANKING/100) ) + 50

In order to obtain a unique baseline for each state, we took a pre-pandemic testing period for each state (Fall 2019 for NWEA and Renaissance and spring 2019 for Curriculum Associates) and calculated a normal-curve equivalent for that also. We then calculated the change between the state-specific spring 2021 data and the state-specific pre-pandemic data.

If there was only one testing company that provided data for a given state, the change from pre-pandemic to post-pandemic was used to score the state’s reading and math scores. If there were multiple testing companies that provided data for a given state, each of their percent changes were calculated separately and then averaged together, weighted by the number of students in each testing company’s spring 2021 sample.

For states where the weighted average percent change was better than -1 percent, we assigned them into a group labeled “little to no learning loss” and awarded 100 points in the reading or math category. The remaining states were divided into two equally sized groups. The states in the middle group were labeled “some learning loss” and were awarded 67 points, and the states in the bottom group — which showed the most learning loss — earned 33 points and received a “most learning loss” label.

Change in enrollment

We calculated the percent change between 2020-2021 academic year enrollment to 2019-2020 enrollment using public school data available for each state from the National Center for Education Statistics. Because of data limitations at the time this analysis was done, pre-K enrollment data was excluded. Private schools were also not included.

Illinois did not provide enrollment data for the 2020-2021 academic year, so only its testing scores were used to calculate its education score.

Annette Choi contributed to this methodology.