From the Fraser Institute

By Kenneth P. Green

Nearly two and a half years since COVID-19 overturned much of the world, vigorous debate
still exists over whether or not governments “got COVID-19 right.” That is, despite more
than two years for analysis and retrospective studies, it is unclear if governments—any governments—implemented the best policies (or even 2nd best policies) as the COVID pandemic exploded in 2020, and as it unfolded through 2021 and 2022.

This situation is troubling, as the world will undoubtedly face similar, and perhaps more severe pandemic challenges again in the future. Society needs to know what worked and what did not work, not only with regard to vaccines, but with regard to the many other interventions government deployed to fight the COVID epidemic.

One key dispute regarding what went right or what went wrong with regard to COVID policies revolves around whether or not the public policies promulgated by governments and public health authorities were science- or evidence-based. Governments, regulatory bodies, and public health institutions around the world were not only unified, but strident in their proclamations that policies were “following the science.” Skeptics of such pronouncements were given little attention at the time, and were more often than not simply attacked as being peddlers of misinformation, or dis-information, and were squelched in public discourse. This did not, however, settle the dispute.

In fact, the battle lines over this question—whether governments were actually following the science or evidence available at the time, or were merely asserting that they were doing so—have only hardened since the pandemic struck in 2020. Even as this essay is being written, for example, a war is raging across Twitter (now “X”) between a prominent COVID vaccine researcher (Peter Hotez), and people who question whether his representation of his work is honest. The questioners seek a public debate, but the scientist involved (backed by many other prominent colleagues) have declared the very idea of societal debate to be anathema to the idea of science itself. The “Twitter war” over this dispute has drawn in one of the world’s richest men (Elon Musk), one of the world’s loudest populist broadcasters (Joe Rogan), and a host of high-level scientists and heads of scientific agencies (Mikhail, 2023).

This dynamic, of public accusations of obstruction, and demonizing of those making the accusations, bodes poorly for the prospects of improving future policy by learning from experience. It also bodes poorly for future confidence in governmental policy responses to threatening situations. It seems self-evident that the public will be less likely to follow governmental recommendations or guidance in future if the consensus develops that they cannot be trusted when they claim to be telling the truth, or following the evidence, or, as such truth-telling was cast in this case, “following the science.”

So what is the answer? Were governments following the science or the evidence extant at the time of COVID’s emergence and progression through the population, or were they following the science selectively, creating more of a veneer of science than a solid policy foundation of science?

Let’s examine this question by examining the two highly controversial, major non-pharmaceutical interventions, or NPIs, imposed during the COVID pandemic. NPIs implemented throughout the COVID-19 outbreak were policies intended to slow or limit the spread of the virus, and to reduce risk of infection via measures of physical separation including: enhanced hygiene, social distancing (keeping separate from others at a distance of 2 meters), erecting physical air/particle barricades, wearing gloves, donning masks, accepting voluntary and involuntary isolation, and a variety of restrictions on assemblage, including closures of parks, businesses, schools, and houses of worship, etc.

Gathering restrictions/closures (bars, businesses, churches, schools)

Several articles have looked into the question of “what the science said” about gathering restrictions imposed in response to COVID-19: closures of businesses, schools, public gatherings, etc., as that science already existed in 2020, and before.

The Cochrane Library, a respected clearinghouse of scientific and medical information,
published one such retrospective look at “the science” conducted in mid-2020, and entitled Quarantine Alone or in Combination with Other Public Health Measures to Control COVID-19: A Rapid Review (Nussbaumer-Streit, 2020; for simplicity, Nussbaumer).

Nussbaumer searched for studies related to quarantine efficacy on the PubMed, Ovid MEDLINE, WHO Global Index Medicus, Embase, and CINAHL databases on June 23, 2020, only about six months into the pandemic. Nussbaumer specifically searched for “Cohort studies, case-control studies, time series, interrupted time series, case series, and mathematical modelling studies that assessed the effect of any type of quarantine to control COVID-19.” The authors even included studies on SARS (Severe Acute Respiratory Syndrome) and MERS (Middle East Respiratory Syndrome) for indirect evidence that might apply to COVID-19. The net was cast widely. Nussbaumer identified 51 studies that met their criteria for inclusion in their analysis. Ten were modeling studies on COVID19; 15 were modelling studies of SARS and MERS; and the team found four observational studies
on SARS and MERS.

Nussbaumer found that while modeling studies were uniform in predicting that quarantine, in a variety of forms, would effectively reduce transmission of SARS, MERS, or COVID-19, the quality of evidence from these modeling studies was characterized as having low certainty or very low certainty, and the studies were, primarily, as the authors themselves noted, “modeling studies that make parameter assumptions based on the current, fragmented knowledge.”1

1 The four observational studies identified suggested quarantine was effective as a control strategy for SARS and MERS. However, Nussbaumer deemed them unlikely to be reflective of COVID-19 controls.

School closures

As discussed in a separate essay on the issue of school closures and educational performance (McPherson and Green, 2023), the question of whether school closures reduced either children’s risk directly, or society’s risk of COVID-contagion vie exposure to children, remains an important and particularly contentious area of debate. We are still trying to assess the damage done to children’s educational attainment in the nearly three years of intermittent school closures in Canada, and it will be some time before the extent of that damage is well understood. Longer term impacts on children’s mental health and social development will likely take many years to ascertain.

When COVID-19 struck in 2020, understanding how school closures related to the spread of a virus through school populations were based mostly on studies of Influenza pandemics that preceded COVID.

In another Cochrane Library review of studies on school closures published as of December 2020, the team of Krishnaratne et al. surveyed research on the issue of school closures published in “the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, and the Educational Resources Information Center, as well as COVID-19-specific databases, including the Cochrane COVID-19 Study Register and the WHO COVID-19 Global literature on coronavirus disease” (2022). They found 38 unique studies in their analysis, 33 of which were modeling studies, three of which were observational studies, one study that was described as “quasi-experimental,” and one “experimental study with modelling components.”

Krishnaratne et al. concluded that

• a broad range of measures implemented in the school setting can have positive impacts
  on the transmission of SARS-CoV-2, and on healthcare utilisation outcomes related
  to COVID-19. The certainty of the evidence for most intervention-outcome combinations is very low, and the true effects of these measures are likely to be substantially
  different from those reported here. Measures implemented in the school setting may
  limit the number or proportion of cases and deaths, and may delay the progression
  of the pandemic. However, they may also lead to negative unintended consequences,
  such as fewer days spent in school (beyond those intended by the intervention).
  (Krishnaratne et al., 2022)

This aligns with by another review of the literature published in the British Medical Journal in 2023. In that paper, Hume et al. characterized the findings of 26 systemic reviews pertinent to the school-transmission question. They conclude, “We found evidence that both school closures and in-school mitigations may have had a beneficial impact on reducing COVID-19 transmission in the community. However, the GRADE [a measure of research quality] certainty was very low in both outcomes. We also found that school closures may have had negative impacts on children, including reduced learning, increased anxiety and increased rates of obesity. However, GRADE certainties were low or very low in these outcomes. Overall, confidence in the included SRs was generally low or critically low” (Hume, et al., 2023).

Other studies pre-dating COVID-19 also raised questions about the potential utility of school closures. One of the more interesting studies on the question from much farther back—a decade back, in fact—came out of University College London, by a research
team that included Neil M. Ferguson, the same researcher whose initial modeling of COVID-19 shaped the world’s response to the pandemic. That study concluded regarding school closures for Influenza, “Although some health benefits can be expected, there is
still substantial debate about if, when, and how school closure policy should be used. There is no consensus on the scale of the benefits to be expected, and recent reviews highlighted the lack of evidence for social distancing measures such as school closure. Even if benefits are substantial, they must be weighed against the potential high economic and social costs of proactively closing schools, which also can have negative effects on key workers since, for example, many doctors and nurses are also parents” (Cauchemez et al, 2009).

• “School closures may have had negative impacts on children, including reduced learning, increased anxiety and increased rates of obesity.”

Enhanced Hygiene: Hand-washing, mask-wearing, and social distancing

The Cochrane Library has again been a world leader in performing retrospective reviews of the effectiveness (and state of knowledge) regarding interventions intended to manage the COVID-19 pandemic.

In 2020, the research team of Jefferson et al. searched the extant scientific literature and
published its findings in the Cochrane Library’s Physical Interventions to Interrupt or Reduce the Spread of Respiratory Viruses. Searching an array of medical and scientific literature indices and databases, the Jefferson team amassed a total of 67 studies which met the criteria for inclusion in their retrospective analysis of the evidence regarding enhanced hygiene (hand-washing and mask-wearing). All of these studies pre-dated the COVID-19 pandemic and focused on previous disease outbreaks of influenza, including the highly virulent H1:N1 variant that wrought worldwide havoc in 2009.

As there is unusually contentious debate over Cochrane’s research on this topic (a fairly vicious battle between the “masks don’t work” and “masks do work” tribes), it is best to read the team’s conclusions in the authors’ own words:

• The high risk of bias in the trials, variation in outcome measurement, and relatively low compliance with the interventions during the studies hamper drawing firm conclusions and generalising the findings to the current COVID-19 pandemic. There is uncertainty about the effects of face masks. The low-moderate certainty of the evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect. The pooled results of randomized trials did not show a clear reduction in respiratory viral infection with the use of medical/surgical masks during seasonal influenza. There were no clear differences between the use of medical/surgical masks compared with N95/P2 respirators in healthcare workers when used in routine care to reduce respiratory viral infection. Hand hygiene is likely to modestly reduce the burden of respiratory illness. Harms associated with physical interventions were under-investigated. (Jefferson et al., 2020: 2–3).

Team Jefferson would update this research in 2023, amassing an expanded set of 78 randomized clinical trials involving physical interventions that had been conducted prior to the time of search (Jefferson et al., 2023). Most of those studies were also conducted prior to the COVID-19 outbreak, and studied the effectiveness of physical interventions against various types of influenza. However, six of the studies included in the 2023 review were conducted during the COVID-19 pandemic.

I’ll also reproduce this latest summary of findings from Jefferson et al. in the authors’ own
words:

• There is uncertainty about the effects of face masks. The low to moderate certainty
  of evidence means our confidence in the effect estimate is limited, and that the true
  effect may be different from the observed estimate of the effect. The pooled results
  of RCTs did not show a clear reduction in respiratory viral infection with the use of
  medical/surgical masks. There were no clear differences between the use of medical/
  surgical masks compared with N95/P2 respirators in healthcare workers when used in
  routine care to reduce respiratory viral infection. Hand hygiene is likely to modestly
  reduce the burden of respiratory illness, and although this effect was also present
  when ILI [Influenza-Like Illness] and laboratory-confirmed influenza were analysed
  separately, it was not found to be a significant difference for the latter two outcomes.
  Harms associated with physical interventions were under-investigated. (Jefferson et
  al., 2023: 3).

Discussion

Governments and government policies were deeply involved in the development, definition, and deployment of both pharmaceutical and non-pharmaceutical interventions throughout the COVID pandemic, far more directly than was routinely seen in past public health crises or in contagious disease outbreaks. Whereas in the past one might have consulted with one’s doctor to get an idea about what to do to protect one’s health, there was no need for that in 2020: government pronouncements at the highest levels of health care bureaucracy were loud, clear, and insistent from the earliest days of the COVID pandemic.

At the time, governments, regulatory agencies, and a vast panoply of public health authorities asserted that they were, in all cases, diligently “following the science” in the policy guidances and prescriptions they wrote for society. “Following the science” became a constant refrain in explaining everything government did. It was also a bludgeon when used in the negative: “not following the science” was applied to anyone critical of what government did. And despite the fact that science is generally a relatively slow process of theory, experimentation, and publication of findings, somehow, as of January 2020, “the science” took to changing on a daily, even hourly basis.

Despite government assertions that they were following the science, reviews of the literature on the subject extant at the time suggest that governments were selectively interpreting the studies that already existed.

In fact, studies regarding the effectiveness of the measures governments adopted to fight COVID were relatively few in number, were mostly in the past tense as of 2020, were largely characterized as having mixed results, were highly uncertain, and were primarily driven by computer models.

The “consensus” of reviews of the literature extant at the time, conducted both during 2020 and afterward, come to basically the same conclusions. Specifically, they find with regard to the non-pharmaceutical interventions of enhanced hygiene, mask-wearing, and restrictions on gathering that while these NPIs might have helped, they might also have harmed, but that there was no strong evidence one way or the other.

Thus, it’s safe to say that while governments did not actively lie about “following the science,” they most certainly misrepresented “the science” in that they exaggerated certainty (a constant problem of government), they chose to believe models over observational studies, they only acknowledged half the science (about slowing transmission, not about causing harm to people economically and socially), and they spoke in tones of absolute certainty that denied the essence of science. That essence understands that knowledge is provisional, that this was a novel virus, and that there was, in fact, no
solid reason to find prior research outcomes dispositive in terms of giving guidance about what to do.

In acting with such certainty, governments not only did harm to their own future credibility, but they did harm to the credibility of public health institutions, biomedical institutions, pharmaceutical institutions, and others that governments chose to use as fig-leaves to obscure what were, in very large measure, arbitrary choices that governments made based on fragmentary, cherry-picked, model-dominated studies that mostly pre-existed the COVID pandemic. At the same time, governments also suppressed dissent among and elevated the authoritarian voices and managers of those institutions.

In an ideal world, there would be some process by which our public health agencies, at least, could come to recognize and admit how badly they managed to “follow the science” on COVID-19, and how vast was the gulf between their expressions of absolute certainty and what the scientific literature showed at the time was, in fact, a sea of uncertainty. Until such a “truth and reconciliation” process takes place, it is hard to see how public trust in public health institutions might be restored.

About the author

Kenneth P. Green is a Fraser Institute senior fellow and author of over 800 essays and articles on public policy, published by think tanks, major newspapers, and technical and trade journals in North America. Mr. Green holds a doctoral degree in environmental science and engineering from UCLA, a master’s degree in molecular genetics from San Diego State University, and a bachelors degree in general biology from UCLA.