Adolescent trial showcases pre-registration and the Law of Small Numbers

The scientific report on Pfizer's adolescent trial did not bring closure on the two key issues I flagged in my previous post (link). Here are my notes on this trial. 

The Key Efficacy Finding Does Not Require a Randomized Controlled Trial

The researchers explained that based on the low expected Covid-19 case rates among 2,000 teenagers (split into two groups), they were not going to see enough cases to offer the same kind of analysis as in the adult trial. They have convinced the FDA (or the other way round) to accept immunogenicity findings as a substitute. That is to say, Pfizer will show only that injecting teenagers with the vaccine generates antibodies intended to fight the SARS-CoV-2 virus. 

The sample size issue is real. The infection rate in teenagers is well known to be much lower than in adults. It's hard to find national numbers split by age group, though. I found out that in California, roughly 4% of children below 17 have tested positive for Covid-19 during the whole pandemic while the number was close to 10% for people above 18. These numbers are cumulative for the 12-15 months of the pandemic whereas a typical vaccine trial spans 3-4 months at the time of interim analysis. Detecting a difference in tiny signals (rare events) is challenging and requires a large sample size. Two thousand is like 20 times fewer participants than the adult trial. 

Typically, immunogenicity results advance a treatment to Phase 3. These lab tests prove that the vaccine produces a good amount of antibodies as intended. However, having antibodies does not guarantee the vaccinated people will be protected from infection. Nor do lab results suffice to decide what level of antibodies is sufficient to reduce infection (and transmission). Further, Phase 2 trials involve few people. Thus, a Phase 3 trial is required for FDA approval. Phase 3 trials involve thousands of people, and measure clinically relevant outcomes, which for a vaccine usually means reduction of infection.

With immunogencity as the key efficacy metric, the adolescent trial is in spirit an expanded Phase 2 trial. You don't even need to have a placebo group, nor randomize treatment. Injecting someone with saline will not produce antibodies against the coronavirus. They also measured antibodies in only 200 kids, a 20%-ish subset of those enrolled in the adolescent trial. They could have obtained the same outcome by recruiting 200 kids, giving them two shots, and measuring their antibody levels afterwards. No RCT is necessary.

The average antibody level of the 12-to-15-year-olds is compared to a number they previously obtained from the adult trial, involving those in the 16-to-25-year-old age group. If similar levels of antibodies are found, they presume that the vaccine will work in teenagers.

There are several assumptions required to justify such a claim. First, they assume the vaccine works with the same efficacy for 16-to-25-year-olds in the adult trial as the overall VE. This is not a given since the trial did not contain enough people in that (or any other) age group to produce a reliable estimate of subgroup efficacy. Second, they assume there are no material differences between the two age groups that affect immune response. Third, they assume that a direct link between antibody levels and reduction in case rate has been established in the 16-to-25-year-old age group. If that is true, I haven't seen the evidence.

In the protocol, the scientists were supposed to provide an "immunobridging" analysis, which sounds like a mathematical model that justifies the third assumption. They ultimately did not come through with that model. 

 

The Trial Provides Minimal Data on Safety

While the placebo arm is redundant for the pre-specified efficacy analysis, the placebo participants are useful to establish a background level of side effects, to help the vaccine developer make the case that the vaccine does not cause more adverse incidents than expected.

Nevertheless, the conclusion one can draw from a study of 1,000 vaccinated participants is highly limited. If a deadly side effect has a 1 in 5,000 chance of happening, there is over 80% chance that there will be zero cases of this side effect during this trial. With probably ~15 million Americans aged 12-15, such a side effect would affect 3,000 kids. Fewer than 200 American teens have died from Covid-19.

The trial did not identify any major concerns but what we don't know, we don't know.

 

What about that 100% VE talking point?

On Twitter and cable TV, the talking heads have mostly not presented the trial results as I have above. Below is a typical example:

Notice that they ignored the immunogenicity results and trumpeted a vaccine efficacy (VE) number. In the study, Pfizer reported a "surprise," saying that they found 16 cases of Covid-19 among the placebo group, and zero cases in the vaccine arm. 

In the NEJM paper, on the other hand, the immunogenicity outcome was unmistakably featured as the primary endpoint. This raises a critical scientific issue that is perhaps hard for an outsider to grasp.

All clinical trials are required to submit protocols and analysis plans well ahead of enrollment. This process is called pre-registration (investigators are not supposed to make edits, especially major edits, while the trials are running although this practice has become a casualty of the pandemic). Pre-specifying your analysis prevents data mining for results after the data have materialized. Such data mining frequently results in spurious findings - mistaking one-off phenomenon as generalizable outcome. Spurious findings will not replicate if the trials were to be repeated.

In the adolescent trial, pre-registration made Pfizer think about how to demonstrate efficacy. They feared that using the usual measurement based on symptomatic, PCR-positive Covid-19 cases is likely to saddle the study without a conclusion because they were only willing to run a trial with 2,000 participants. Therefore, they chose the immunogenicity outcome as the primary endpoint.

Now, after the trial finished, the Pfizer press release - and its readers - trumpeted the 100% VE metric as if that was the pre-specified analysis. This behavior usurps the principle of pre-registration. 

So let me now explain why there is a good chance the observed VE is an outlier. If the whole trial were repeated, I'd be surprised to see the same result repeated.

According to the FDA's press release on approving Pfizer's vaccine for teenagers, the case rate in the placebo arm was 16/978 = 1.6%. In Pfizer's adult trial, the corresponding rate was 162/18325 = 0.9%. So, the teenagers were getting sick at a rate 80 percent higher than that of adults. If I assume that the kids were equally likely to get infected as adults at 0.9%, the chance that we would find 16 or more cases in 978 kids is less than 2 percent. If we were to run this trial 100 times, 98 or more of them would observe fewer than 16 cases in the placebo group.

But teenagers have less than half the case rate of adults as mentioned above. If I assume that the case rate for kids is at 0.45%, the chance that we would find 16 or more cases in 978 kids is 0.001%. Don't forget, also, that the case rate only counts adjudicated, lab-confirmed, symptomatic Covid-19 cases occurring at least 7 days after the 2nd dose so the real-life infection rate is even higher than the 1.6%.

This is very strong evidence that what was observed during this trial was an outlier event. 

Nevertheless, this outlier event has been painted as a "pleasant surprise". We didn't think we'd have enough cases to show anything but how could we refuse the unexpected gift of 100% efficacy? The real surprise was not the VE but the inexplicable, highly unlikely incidence of Covid-19 among teenagers enrolled in this specific trial.  

In fact, this is exactly the reason why pre-registration is good practice. If we allowed this type of post-hoc data mining, spurious findings will bloom. 

***

See also my previous rant about the way the paper was written up as if the adolescent and 16-to-25-year-old age group were originally planned as two arms of a trial.

See also my post on the Law of Small Numbers, one of the seminal contributions by Tversky and Kahneman. The Law of Small Numbers is the false belief that the Law of Large Numbers apply to one's small sample.