Non-Response Bias
In 2020, the COVID-19 pandemic disrupted the lives of people across the country.
There are large gaps in 2020 data—specifically from March to June—and organizations that collect survey data are unable to use typical standardization techniques to ensure the quality of data and have therefore been unable to release those data to the public.
The COVID-19 pandemic interrupted systems of data collection and reporting in the U.S. The onset of the pandemic made survey collection efforts difficult, especially for communities drastically impacted by the pandemic. Further, many collection efforts were halted as field surveyors were forced to abstain from their usual door-to-door work and mail operations were largely scaled down. Thus, the information that was collected during the initial months of the pandemic was limited to areas with convenient access to internet and mail services.
Evaluating trends over time might render meaningless, or at worst, deceptive predictions without accounting for non-response bias. The unpredictability of the pandemic has made the forecasting of trends and needs increasingly challenging, and many organizations have turned back to descriptive statistics summarizing data from the present or recent past. Though the 2020-2021 datasets listed on this page can be used to compare across years, this should be done with some discretion and the author should caution readers of non-response in 2020 data.
In general, survey-based data sources can be used appropriately to summarize short-term trends or to identify current statistics to compare across age, gender, racial, and/or ethnic groups. Sources on this page would serve best as a reference point for predicting trends in pandemic and post-pandemic years rather than for comparing to previous years’ data—for instance, comparing 2020-2022 trends when more current data are made available would be more reliable than comparing 2019-2021 trends.
Previous status of women reports created by the Wisconsin Women's Council and the Women's Fund Consortium of Northeast Wisconsin relied heavily on the U.S. Census Bureau's American Community Survey (ACS), whose operations and survey results were significantly disrupted in 2020.
The ACS typically collects responses from upwards of 3.5 million addresses through mail operations as well as in-person interviewing. In 2020, the data collection process was modified in light of the pandemic. Mail operations were suspended mid-March to June 2020, and limited mail operations returned in July. The ACS’s usual 5-piece mail strategy was not reinstalled until April 2021. In-person interviews were also suspended from mid-March to June 2020; only phone interviews were conducted during this time. Though there was an internet option for the survey, it was only able to cover a small subset of the full sample due to reduced mailings.
The COVID-19 pandemic largely disrupted methods of data collection typically used by state and federal agencies. Not only were survey response rates down in 2020, but U.S. Census Bureau researchers identified a non-response bias in the collected data. This non-response likely biased 2019 income estimates up and poverty statistics down. Therefore, we cannot arbitrarily draw conclusions about 2020 without considering this non-response bias. Though some data—like the ACS 2016-2020 5-year estimates—has been exhaustively evaluated and standardized for quality for public use, other data sources that have not yet been modified to account for lower responses may still reflect this non-response bias. Fortunately, because the U.S. Census Bureau was able to postpone the 2020 decennial census, the quality of responses was maintained.
Additional Resources
The following resources from the U.S. Census Bureau provide detailed insight on the effects of the COVID-19 pandemic on the collection and quality of data.
Pandemic Impact on 2020 American Community Survey 1-Year Data
Webinar: Impact of Pandemic on the American Community Survey
2020 Census Data Quality
2020 Census Data Quality Results Prerelease Webinar
An Equitable Approach to Data
When determining the necessary distribution of resources, data are only a starting point.
To understand the implications of trends and statistics that might materialize, it is imperative that some context is provided in addition to reporting raw numbers. According to the University of Wisconsin-Madison Population Health Institute, we can address disaggregated data more accurately by:
Including population adjustments,
Acknowledging existing inequities in social and economic conditions, and
Identifying limitations in data and analysis.
Sufficient contextualization of information is especially important when we are dealing with data that are biased themselves as a result of inequitable contexts. It is imperative that this be taken into consideration when reporting results and limitations in the availability and quality of data collected during the COVID-19 pandemic so that analyses reflect the effects of larger systems of oppression like racism, classism, and sexism. Learn more about data equity in the era of COVID-19 here.