Evaluation of COVID-19 vaccine effectiveness

Overview

Vaccine effectiveness and impact document provides interim best practice guidance on how to assess COVID-19 vaccine effectiveness (VE) using observational study designs. It discusses critical considerations in the design, analysis and interpretation of COVID-19 VE evaluations, as biased results may be produced even in settings where data completeness and quality are high. This guidance is targeted mostly for evaluations undertaken in low- and middle-income countries but most of the concepts apply to VE evaluations in high-income settings as well.

• Due to their methodological complexity and susceptibility to biases, COVID-19 VE evaluations do not need to be conducted by all countries introducing COVID-19 vaccines. A checklist of criteria to have in place when considering such evaluations is provided.
• Objectives of VE evaluations are to evaluate real-world performance of vaccines, to address gaps in evidence from clinical trials (including effectiveness in subgroups, effectiveness against variants of concern and duration of protection), to provide input into impact models, and to provide post-authorization confirmation of effectiveness of conditionally approved products.
• The most feasible outcomes to evaluate in VE evaluations in most settings are symptomatic disease and severe disease. VE studies of death, infection and transmission, while of great public health importance, generally require targeted special studies with more resources.
• We recommend the use of laboratory-confirmed outcomes in VE evaluations. At this time, we recommend the use of real-time reverse-transcription polymerase chain reaction (rRT-PCR) for laboratory testing of participants. Specimens should be taken within 10 days of disease onset.
• We recommend against the use of self-reported COVID-19 vaccination as the sole source indicating whether a person is vaccinated, due to recall bias and lack of product details. Documented vaccination should be used for the primary analysis; self-reported vaccination could be included in a secondary analysis.
• Although not without potential biases, we recommend the test-negative design as the most efficient and logistically feasible method to assess VE in L/MICs, with the advantage of some degree of comparability between cases and controls since they all sought care for a similar illness at the same facilities. Other methods that could be considered are cohort studies, case-control studies, and the screening method (in certain settings with reliable information on coverage at different times during the study period).
• Due to lack of randomization of vaccination in real-world settings, all observational study designs are subject to bias because vaccinated persons often differ from unvaccinated persons in their disease risk, independent of vaccination. Important biases include: confounding by health care seeking, exposure risk, misclassification of outcomes due to diagnostic errors, prior SARS-CoV-2 infection, and spurious inferences of waning. Collection of key covariates to control for confounding bias in the analysis is important.
• For the primary analysis of VE studies, a conservative approach is recommended in considering a person as potentially protected from vaccination only from 14 days after the date of vaccination (the time required to achieve protection for the majority of vaccine recipients for most vaccines), for both first and (if applicable) second doses of vaccine. Secondary analyses with different time intervals post-vaccination can be conducted to help guide future policy.
• The primary analysis should compare persons receiving the recommended number of doses of the same vaccine with unvaccinated individuals. Secondary analyses include partially vaccinated persons, persons receiving doses of two different vaccines, targeted subgroups, viral variants, and history of prior SARS-CoV-2 infection or disease if available. Even though partial vaccination and the use of different vaccines to complete a course are not currently recommended by WHO, these might happen in the real world and findings could inform future policy.
• VE estimates that vary from efficacy in clinical trials could be valid or not valid; thorough investigation into reasons for the difference should be undertaken.
• Existing platforms that can be used for VE evaluations include surveillance systems for severe acute respiratory infections (SARI), influenza-like illness (ILI), other syndromic disease surveillance in sentinel hospitals, health worker surveillance, administrative databases and well-defined outbreaks.
• We recommend standardized reporting of the results of studies based on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidance, as well as suggested additional COVID-19 specific elements described below.

Questions should be directed to covidve@who.int.

Sample size calculator for evaluation of COVID-19 vaccine effectiveness (Excel)