A recent study[1]claimed that the best protection against COVID-19 illness comes from a combination of a full series of COVID-19 vaccination and prior infection with SARS-CoV-2.
Analysing data from 26 previously published studies, the authors claim that this “hybrid immunity” is a clear indication of the need for continued vaccination – even among people who have had COVID-19 and recovered. Those with hybrid immunity were reported to have above 95% protection against severe disease and hospitalization for up to 12 months. The study claims to help public policymakers understand the optimal timing of COVID-19 booster doses.
Despite being credited with “[delivering] decision-grade evidence to the world,”[2]a closer investigation of the study’s methodology in both the main article and its supplementary documentation diminishes the results.
The study, published in The Lancet Infectious Diseases, is called a systematic review and meta-analysis. As is generally the case with study designs, the study by Bobrovitz et al. has its strengths and limitations.
The strengths of the systematic review include the well-defined research question, the comprehensive search for evidence, the criterion-based selection of relevant evidence, the rigorous appraisal of validity, and an objective or quantitative summary of the collected evidence.[3]
However, the strengths of the systematic review by Bobrevitz et al. have also turned into weaknesses. The primary problem is that the specific focus and prescribed methods call into question the comparability of the studies used in their analyses. These included combining studies that could not:
- Appropriately control for age, comorbidities, testing frequency, and risk of exposure to SARS-CoV-2 infection
- Discern between the sequence and timing of when people were vaccinated, and previous infection occurred
- Differentiate between “possible”, “probable”, and “confirmed” reinfections
While these limitations are explicitly mentioned by Bobrovitz et al., transparency about methodological limits does not make them – or their impact on a study’s results – inconsequential. Flaws are flaws.
The picture of hybrid immunity and its pattern in populations overall are extremely complicated. People not only differ in history of infection, and the variants to which they have been exposed, but with the type of vaccine they have received, the number of doses, and how their immune system responded to each of those challenges.[4]This checkered pattern in population immunity makes those with the hybrid version an immunological gobbledygook that is falsely deemed equivalent by Bobrovitz et al.
Wholesale conjecture from limited data
While the authors considered many different limiting factors related to the individual studies in their systematic review, they seem to have forgotten – or do not mention – how such missing information would affect their meta-analysis.
When comparing 12-month percentage-point changes in immune protection against severe disease or hospitalization, Bobrovitz et al. rely on extrapolated estimates – those derived from assuming that existing trends continue beyond available data – in the hybrid immunity groups (see Figure 1, panels B and C).
Such gaffes are more common than many would think in the scientific community, and are usually considered statistical misdemeanours.[5]However, when fabricated levels of protection are compared to empirical ones, it becomes less and less likely that what has been produced by these authors is “decision-grade”. Developing policy on a narrative that compares the “real” with the “hypothetical” is bound to be fraught with limitations, which is quite the omission given that the protection afforded by different types of immunity is the crux of their conclusions.
Figure 1. Protection against hospital admission or severe disease conferred by previous infection (panel A) or hybrid immunity, with primary series (panel B), or with booster doses (panel C) of COVID-19 vaccines. Points of the same colour represent estimates from the same study. The diameter of the circles varies with the sample size of the study. Solid black lines represent the best-fit of a log-odds regression model. Dashed lines represent 95% confidence intervals. Solid red outline highlights model predictions beyond the range of the available data. This figure has been modified from Figure 2 in Bobrovitz et al.
Stricter definitions of “severe disease” reverse the study’s original results
To check the robustness of their results, Bobrovitz et al. performed two sensitivity analyses. These are a critical way to assess the impact, effect, or influence of key assumptions – such as different methods of analysis, definitions of outcomes, protocol deviations, and missing data – on the overall conclusions of a study.
These included:
- Removing datasets that were judged to be at “high” risk of bias – leaving only those studies at “moderate” risk of bias; and
- Redefining “severe disease and hospitalization” to match World Health Organization guidelines, which excludes all-cause hospitalization, and retains hospitalizations directly related to COVID-19.
While the authors claim that these sensitivity analyses generated “similar findings” to the results of the main article, their supplementary documents reveal that results are in the opposite direction of what was originally observed (see Figure 2). For example, when the outcome of severe disease and hospitalization is redefined, the percentage-point change of the “previous infection” group demonstrates a 5.7-percentage-point increase in immune protection versus the 7.8-percentage-point decline in the original analysis. The opposite trend was observed among those with “hybrid immunity”.
Figure 2. Percentage-point changes in protection against hospital admission or severe disease conferred by previous infection or hybrid immunity between 3 and 12 months. Data plotted for the “Original Analysis” are taken from Table 2 in Bobrovotz et al.’s main article, while data plotted for the sensitivity analysis using “WHO-defined hospitalization” are taken from Table 13 in the Supplementary Appendix[6]. Values above 0 (the horizontal dashed line) indicate an increase in protection over the time window of 3-12 months; values below 0 indicate a decrease in protection.
Last, there are notable conflicts of interest among many of the authors, either through previous links to Pfizer, The Bill & Melinda Gates Foundation, The Coalition for Epidemic Preparedness Innovations, or as employees of federal government agencies.
While I do not think conflicts of interest prohibit Bobrovitz et al. from participating in research, they are arguably as important to consider as the study’s limitations. Declaring a conflict of interest does not suddenly make the analysis, or its interpretation, free of bias or agenda.
The widely disseminated[7]conclusions of the paper, which are regarded as further proof of the preeminent role vaccination plays in protecting against severe COVID-19, come as no surprise. Many of the authors, as indicated by their past partnerships with pharmaceutical companies and government agencies could not be reasonably expected to publish radical conclusions.
Attaching the catchy label “hybrid immunity” seems like a last-gasp effort to reclaim some semblance of success from a lacklustre intervention.
Regardless of any potential bias that may have affected the authors’ work and conclusions, the published paper makes a decisive contribution to the problem at hand. It has publicly exposed how superficial and messy the current state of scientific knowledge is regarding the benefits of COVID-19 vaccines.
References
[1] Protective effectiveness of previous SARS-CoV-2 infection and hybrid immunity against the omicron variant and severe disease: a systematic review and meta-regression, Bobrovitz N, et al. Lancet Infect Dis 2023; published online January 18, 2023
[2] UCalgary and WHO researchers find hybrid immunity is the best protection against COVID-19, UCalgary News
[4] Why Hybrid Immunity is so Triggering
[6] Supplement to Bobrovitz et al.
[7] Protective effectiveness of previous SARS-CoV-2 infection and hybrid immunity against the omicron variant and severe disease: a systematic review and meta-regression, PlumX Metrics
