What We’ve Learned About So-Called ‘Lockdowns’ and the COVID-19 Pandemic – FactCheck.org

The studies in this group used the Oxford COVID-19 Government Response Tracker, which looked at government responses worldwide to the pandemic and created a stringency index, measuring how strict the measures were over time. The index is from 0 to 100, with 100 being the most stringent restrictions. For instance, the OxCGRT heat map shows that many countries around the world had stringency levels above 70 in April 2020.

The working paper calculates mortality impact estimates for each of the seven studies aiming to show the effect of the average mandated restrictions in Europe and the United States early in the pandemic compared with a policy of only recommendations. The paper then calculates a weighted average, giving more weight to studies that said their findings were more precise. Nearly all of the weight 91.8% goes to one study, even though the working paper rejects the conclusions of that study.

That study coauthored by Carolyn Chisadza, a senior lecturer in economics at the University of Pretoria, and published on March 10, 2021, in the journal Sustainability looked at a sample of countries between March and September 2020 and concluded: Less stringent interventions increase the number of deaths, whereas more severe responses to the pandemic can lower fatalities.

The working paper claims the researchers conclusion is incorrect but it uses the studys estimates, saying the figures show an increase in mortality due to lockdowns.

Chisadza told us in an email that the study showed: Stricter lockdowns will reduce the rate of deaths than would have occurred without lockdown or too lenient of restrictions. But Hanke said the data from Chisadza and her colleagues only show that stricter lockdowns will reduce mortality relative to the worst possible lockdown, meaning a more lenient lockdown that, under the study, was associated with the highest rate of deaths.

We reached out to a third party about this disagreement. Victor Chernozhukov, a professor in the Massachusetts Institute of Technologys Department of Economics and the Statistics and Data Science Center, along with Professor Hiroyuki Kasahara and Associate Professor Paul Schrimpf, both with the Vancouver School of Economics at the University of British Columbia the authors of another study that was included in the working paper looked at the Chisadza study and provided FactCheck.org with a peer review of it. They found the Chisadza study only measured the correlation between current death growth rates and current policies. It did not show the lagged effect of more stringent policies, implemented three weeks prior, on current death growth rates, which is what one would want to look at to evaluate the effectiveness of lockdowns.

In an email and in a phone interview, Chernozhukov told us the Chisadza study made an honest mistake. He said the working paper is deeply flawed partly because it relies heavily on a study that estimates the wrong effect very precisely.

In their review, Chernozhukov, Kasahara and Schrimpf write that the Chisadza et al. study should be interpreted as saying that the countries currently experiencing high death rates (or death growth rates) are more likely to implement more stringent current policy. That is the only conclusion we can draw from [the study], because the current policy can not possibly influence the current deaths, given the several weeks of delay between new infections and deaths.

The effect that should be examined for the meta-analysis is the effect of the previous (e.g., 3 week lagged) policy stringency index on the current death growth rates.

Chernozhukov, Kasahara and Schrimpf conducted a quick reanalysis of similar data to the Chisadza study, finding results that suggest that more stringent policies in the past predict lower death growth rates. Chernozhukov said much more analysis would be needed to further characterize this effect, but that it is actually quite substantial.

If the Chisadza study were removed from the working paper, according to one of the papers footnotes, the result would be a weighted average reduction in mortality of 3.5%, which Hanke said doesnt change the overall conclusions. He said it simply demonstrates the obvious fact that the conclusions contained in our meta-analysis are robust.

But experts have pointed out other issues with the meta-analysis. Chernozhukov also said the paper excluded a whole bunch of studies, including synthetic control method studies, which evaluate treatment effects. He also questioned the utility of looking at a policy index that considers the U.S. as a whole, lumping all the states together. He said the meta-analysis is not credible at all.

Among the other six stringency index studies included in the meta-analysis, only one concluded that its findings suggested lockdowns had zero effect on mortality. In a review of 24 European countries weekly mortality rates for the first six months of 2017-2020, the study, published in CESifo Economic Studies, found no clear association between lockdown policies and mortality development. The author and Herby, one of the authors of the working paper, have written for the American Institute for Economic Research, which facilitated the controversial Great Barrington Declaration, an October 2020 statement advocating those at low risk of dying from COVID-19 live their lives normally to build up immunity to the virus through natural infection, while those at highest risk are protected.

The other studies found lockdown policies helped COVID-19 health outcomes. For instance, a CDC study published in the agencys Morbidity and Mortality Weekly Report in January 2021, on the experience of 37 European countries from Jan. 23 to June 30, 2020, concluded that countries that implemented more stringent mitigation policies earlier in their outbreak response tended to report fewer COVID-19 deaths through the end of June 2020. These countries might have saved several thousand lives relative to countries that implemented similar policies, but later.

A working paper from Harvard Universitys Center for International Development, which looked at 152 countries from the beginning of the pandemic until Dec. 31, 2020, found that lockdowns tend to significantly reduce the spread of the virus and the number of related deaths. But the effect fades over time, so lengthy (after four months) or second-phase lockdowns dont have the same impact.

A study published in World Medical & Health Policy in November 2020 that looked at whether 24 European countries responded quickly enough found that the fluctuating containment measures, from country to country and over time, prohibited a clear association with the mortality rate. But it said the implementation speed of these containment measures in response to the coronavirus had a strong effect on the successful mitigation of fatalities.

Many studies found restrictions worked. Meyerowitz-Katz noted that the working paper authors disagreed with the conclusions of other studies included in the review, pointing to one included in the group of shelter-in-place orders. Meyerowitz-Katz said that study found that significant restrictions were effective, but is included in this review as estimating a 13.1% INCREASE in fatalities.

That study, by Yale School of Management researchers, published by The Review of Financial Studies in June 2021, developed a time-series database on several types of restrictions for every U.S. county from March to December 2020. The authors concluded: We find strong evidence consistent with the idea that employee mask policies, mask mandates for the general population, restaurant and bar closures, gym closures, and high-risk business closures reduce future fatality growth. Other business restrictions, such as second-round closures of low- to medium-risk businesses and personal care/spa services, did not generate consistent evidence of lowered fatality growth and may have been counterproductive. The authors said the studys findings lie somewhere in the middle of the existing results on how NPIs influenced the spread of COVID-19.

In terms of hard figures on fatality reductions, the study said the estimates suggest a county with a mandatory mask policy would see 15.3% fewer new deaths per 10,000 residents on average six weeks later, compared with a county without a mandatory mask policy. The impact for restaurant closures would be a decrease of 36.4%. But the estimates suggest other measures, including limits on gatherings of 100 people or more, appeared to increase deaths. The authors said one possible explanation of such effects could be that the public is substituting other activities that actually increase transmission of the virus such as hosting weddings with 99 people in attendance, just under the 100-person limitation.

Another study in the shelter-in-place group is the study by Chernozhukov, Kasahara and Schrimpf, published in theJournal of Econometrics in January 2021. It looked at the policies in U.S. states and found that nationally mandating face masks for employees early in the pandemic could have led to as much as 19 to 47 percent less deaths nationally by the end of May, which roughly translates into 19 to 47 thousand saved lives. It found cases would have been 6% to 63% higher without stay-at-home orders and found considerable uncertainty over the impact of closing schools. It also found substantial declines in growth rates are attributable to private behavioral response, but policies played an important role as well.

The working paper considered 13 studies that evaluated stay-in-place orders, either alone or in combination with other NPIs. The estimated effect on total fatalities for each study calculated by the authors varied quite widely, from a decrease of 40.8% to an increase of 13.1% (the study above mentioned by Meyerowitz-Katz). The authors then combined the studies into a weighted average showing a 2.9% decrease in mortality from these studies on shelter-in-place orders.

Sizable impact from some NPIs. The working paper actually found a sizable decrease in deaths related to closing nonessential businesses: a 10.6% weighted average reduction in mortality. The authors said this is likely to be related to the closure of bars. It also calculated a 21.2% weighted average reduction in deaths due to mask requirements, but notes this conclusion is based on only two studies.

As with the shelter-in-place group, the calculated effects in the specific NPIs group varied widely from a 50% reduction in mortality due to business closures to a 36% increase due to border closures. The paper said differences in the choice of NPIs and in the number of NPIs make it challenging to create an overview of the results.

The review itself does refer to other papers that reported that the lockdowns had a significant impact in preventing deaths, Dr. Lee Riley, chair of the Division of Infectious Disease and Vaccinology at the University of California, Berkeley School of Public Health, told us when we asked for his thoughts on the working paper. The pandemic has now been occurring long enough that its not surprising to begin to see many more reports that now contradict each other. As we all know, the US and Europe went through several periods when they relaxed their lockdowns, which was followed by a resurgence of the cases.

Riley said that many of the studies that this review included may suffer from the classic chicken-or-egg bias. Whenever there was an increase in cases of deaths, lockdowns got instituted so its not surprising that some of the studies showed no impact of the lockdowns. If there was no surge of cases or deaths, most places in the US did not impose restrictions.

Meyerowitz-Katz noted on Twitter that the impact of lockdowns is very hard to assess, if for no other reason than we have no good definition of lockdown in the first place. In most cases, it seems the authors have taken estimates for stay-at-home orders as their practical definition of lockdown (this is pretty common) And honestly, Id agree that the evidence for marginal benefit from stay-at-home orders once youve already implemented dozens of restrictions is probably quite weak.

But, if we consider lockdown to be any compulsory restriction at all, the reality is that virtually all research shows a (short-term) mortality benefit from at least some restrictions.

Weve already mentioned two studies beyond those in the working paper: the Nature June 2020 study by Imperial College London researchers that estimated interventions in 11 countries in Europe in the first few months of the pandemic reduced transmission and averted 3.1 million deaths; and the Nature May 2020 study that estimated cases in mainland China would have been 67-fold greater without several NPIs by the end of February.

There are many more that aimed to evaluate the effectiveness of various mitigation strategies, not included in the working papers analysis.

Near the end of his lengthy Twitter thread on the working paper, Meyerowitz-Katz said he agrees that a lot of people originally underestimated the impact of voluntary behaviour change on COVID-19 death rates its probably not wrong to argue that lockdowns werent as effective as we initially thought. He pointed to the Nature Communications study mentioned above, showing less of an impact from NPIs in a second wave of COVID-19 and positing individual safety behaviors were playing more of a role in that second wave.

HOWEVER, this runs both ways, Meyerowitz-Katz said. [I]t is also quite likely that lockdowns did not have the NEGATIVE impact most people propose, because some behaviour changes were voluntary!

He and others examined whether lockdowns were more harmful than the pandemic itself in a 2021 commentary piece in BMJ Global Health. They concluded that government interventions, even more restrictive ones such as stay-at-home orders, are beneficial in some circumstances and unlikely to be causing harms more extreme than the pandemic itself. Analyzing excess mortality suggested that lockdowns are not associated with large numbers of deaths in places that avoided large COVID-19 epidemics, such as Australia and New Zealand, they wrote.

Editors note:SciChecks COVID-19/Vaccination Projectis made possible by a grant from the Robert Wood Johnson Foundation. The foundation hasno controlover FactCheck.orgs editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation. The goal of the project is to increase exposure to accurate information about COVID-19 and vaccines, while decreasing the impact of misinformation.

Herby, Jonas et al. A Literature Review and Meta-Analysis of the Effects of Lockdowns on COVID-19 Mortality. Studies in Applied Economics, Institute for Applied Economics, Global Health, and the Study of Business Enterprise, Johns Hopkins University. posted Jan 2022.

World Health Organization. Coronavirus disease (COVID-19): Herd immunity, lockdowns and COVID-19. 31 Dec 2020.

Flaxman, Seth et al. Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature. 584 (2020).

Lai, Shengjie et al. Effect of non-pharmaceutical interventions to contain COVID-19 in China. Nature. 585 (2020).

Sharfstein, Joshua, vice dean of the Johns Hopkins Bloomberg School of Public Health. Statement emailed to FactCheck.org. 8 Feb 2022.

Best, Paul. Lockdowns only reduced COVID-19 death rate by .2%, study finds: Lockdowns should be rejected out of hand.' Fox News. 1 Feb 2022.

Meyerowitz-Katz, Gideon. @GidMK. This paper has been doing the rounds, claiming that lockdown was useless (the source of the 0.2% effect of lockdown claim). Dozens of people have asked my opinion of it, so here we go: In my opinion, it is a very weak review that doesnt really show much, if anything 1/n. Twitter.com. 4 Feb 2022.

Hanke, Steve H., founder and co-director of the Johns Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise. Email interview with FactCheck.org. 18 Feb 2022.

Ferguson, Neil, director of the MRC Centre for Global Infectious Disease Analysis, Jameel Institute, Imperial College London. Statement posted by Science Media Centre. 3 Feb 2022.

Oxford COVID-19 Government Response Tracker. Blavatnik School of Government, University of Oxford. https://covidtracker.bsg.ox.ac.uk/. website accessed 20 Feb 2022.

Chisadza, Carolyn, senior lecturer in economics at the University of Pretoria. Email interview with FactCheck.org. 15 Feb 2022.

Clance, Matthew, associate professor in the Department of Economics at the University of Pretoria. Email interview with FactCheck.org. 16 Feb 2022.

Our World in Data. Cumulative confirmed COVID-19 deaths. website accessed 22 Feb 2022.

Bjornskov, Christian. Did Lockdown Work? An Economists Cross-Country Comparison. CESifo Economic Studies. 67.3 (2021).

Fuller, James A. et al. Mitigation Policies and COVID-19Associated Mortality 37 European Countries, January 23June 30, 2020. Morbidity and Mortality Weekly Report. 70.2 (2021).

Goldstein, P. et al. Lockdown Fatigue: The Diminishing Effects of Quarantines on the Spread of COVID-19. Harvard University Center for International Development. 2021.

Stockenhuber, Reinhold. Did We Respond Quickly Enough? How Policy-Implementation Speed in Response to COVID-19 Affects the Number of Fatal Cases in Europe. World Medical & Health Policy. 12.4 (2020).

Riley, Lee, chair of the Division of Infectious Disease and Vaccinology at the University of California, Berkeley School of Public Health. Email interview with FactCheck.org. 14 Feb 2022.

Spiegel, Matthew and Heather Tookes. Business Restrictions and COVID-19 Fatalities. The Review of Financial Studies. 34.11 (2021).

Chernozhukov, Victor et al. Causal impact of masks, policies, behavior on early covid-19 pandemic in the U.S. Journal of Econometrics. 220. 1 (2021).

Hale, Thomas et al. Global Assessment of the Relationship between Government Response Measures and COVID-19 Deaths. medrxiv.org. 6 Jul 2020.

Hale, Thomas et al. Government responses and COVID-19 deaths: Global evidence across multiple pandemic waves. Plos One. 9 Jul 2021.

Brauner, Jan M. et al. Inferring the effectiveness of government interventions against COVID-19. Science. 371.6531 (2020).

Haug, Mils et al. Ranking the effectiveness of worldwide COVID-19 government interventions. Nature Human Behaviour. 4 (2020).

Sharma, Mrinank et al. Understanding the effectiveness of government interventions against the resurgence of COVID-19 in Europe. Nature Communications. 12 (2021).

Yang, Jiannan et al. The impact of non-pharmaceutical interventions on the prevention and control of COVID-19 in New York City. Chaos: An Interdisciplinary Journal of Nonlinear Science. 31.2 (2021).

Achenbach, Joel and Laura Meckler. Shutdowns prevented 60 million coronavirus infections in the U.S., study finds. Washington Post. 8 Jun 2020.

Hsiang, Solomon et al. The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature. 584 (2020).

Chernozhukov, Victor et al. Comments on the John Hopkins Meta Study (Herby et al., 2022) and Chisadza et al. (2021). Provided to FactCheck.org. 4 Mar 2022.

Chernozhukov, Victor, professor, Massachusetts Institute of Technology Department of Economics and the Statistics and Data Science Center. Phone interview with FactCheck.org. 8 Mar 2022.

View post:

What We've Learned About So-Called 'Lockdowns' and the COVID-19 Pandemic - FactCheck.org