Real-life demonstration of the value of statistical controls

In an email conversation with me, a doctor who's making managerial decisions in a health-care system is concerned about whether we'll be able to see the effect of vaccination in the data on cases. There are at least two sources of variation that could pollute the data:

<1> the emergence of new variants, especially fast-spreading variants of the virus

<2> the natural boom-bust cycle (waves) of the pandemic

If the cases start falling, do we know for sure it is due to the vaccines? Or could it be a natural ebbing of the current wave? If the cases continue to grow, do we know the vaccines are ineffective, or just ineffective for a new variant?

Other combinations are also possible. The vaccines can be effective against the current variant and less effective against the fast-spreading variant, leading to either (a) a net acceleration or (b) a net deceleration of cases, depending on the strength of each factor.

***

Ouch, you say, so we can't know anything.

If we can't, it's because we choose not to. This is why statisticians pleaded that the pharmas be required to continue running the placebo groups in the ongoing vaccine trials. These placebo groups are statistical controls.

If there is a new variant of the virus, it will raise the case counts in the placebo and vaccine groups both. So if we look at the difference between the two groups, the effect of the new variant is cancelled out.

Similarly, if there is a natural ebbing of the pandemic, it will affect both placebo and vaccine groups. When we look at the difference between the groups, this effect will also be self-cancelling.

The placebo control group establishes the baseline used to judge the vaccinated group. Besides, I have mentioned frequently here that we should keep running the placebo groups in order to establish long-term efficacy and safety, and to gain confidence in subgroup analyses.

***

This self-cancellation is valid only if the placebo and vaccine treatments are randomized. This means any participant has equal chance of being in placebo and vaccine groups. If placebo participants are offered the chance to get vaccinated, then the randomized structure of the vaccine trial has been destroyed.

If that happens, we have to analyze the group comparison as an "observational study". We do some form of matching to create quasi-randomized comparison groups. However, there are some risks. For example, the self-selection bias may be so severe that we can't find comparable people. Everyone above 75 may decide to take the vaccine, which leaves us with nobody in that age group who's unvaccinated.

Statistical controls are really valuable, and solve many problems all at once! Unfortunately, in this case, I believe they are destroying the placebo groups, which means we will not obtain those insights, which sadly makes us wander forward somewhat blinded.

(It's not all or none either. Given what we learned about overall efficacy, I believe the placebo group can be reduced in size.)

 

P.S. If the placebo groups are intact, then I hope the pharmas are still passing data to the FDA. Assuming we have a way to detect the new variants effectively, we have direct evidence on whether the vaccines work with the new variants. The scientific value of the placebo groups goes way beyond the pre-planned analyses!