“Reduce likelihood of a tick bite by 73.6 times”? Forking paths on the Appalachian Trail.

Shira writes:

As an Appalachian Trail hiker, I always treat my clothes with permethrin. I’m a big fan of Sawyer products, but this claim caught my eye:

Reduce likelihood of a tick bite by 73.6 times by treating shoes and socks with Permethrin (University of Rhode Island study – 2011).

So I [Shira] found the paper

I thought I’d try your challenge to come up with forking path possibilities from Active Statistics chapter corresponding to ROS 16 “2. Lucky golf balls and implausible effect sizes.”

2 outcomes: bite/attached or not, dead or not. Their main results focus on the first. Maybe they looked at others ?

3 levels of treatment (at-home treatment, commercial treatment, untreated) , but they combine the first 2 in their main results. They could have kept all levels separate. Doing the sneaker/sock logistic regression comparing at-home to untreated, the odds ratio point estimate drops from 73 to 36, both statistically significant.

3 locations of tick placement: near sneakers/socks, near shorts, near t-shirts. The result highlighted by Sawyer is for the sneakers/socks location only. I think ticks were applied directly to the shoes and so received an immediate dose of permethrin, but I’m confused about whether ticks applied to the leg and arm were applied to skin or clothing. They describe “ticks applied to the leg and arm were temporarily shielded by a cotton pad.” So it isn’t unreasonable that the effect for shoes would be much higher than for other locations.

15 human subjects, each gets 10 ticks per location per day. The 15 humans are divided into 5 per treatment group. Across 2 days, the humans serve as their own controls, in a (partial) crossover/within-subject design with logical assignment to new treatment groups on day 2, though 5 humans will never get control.

Unlike the Lucky Golf Ball example, I think most of these forking paths would still result in statistical significance. The logistic coefficient for shoes is -4 with standard error 1. Suppose its effect was almost as low as for shorts, only a logistic coefficient of -2, then the power is around 50%. But the forking paths could still have gotten them to larger estimates?

I replied that forking paths are there for sure. My quick reaction here is that, as in our Millennium Village project analysis, the right thing to do is to look at all possible comparisons of interest and fit a multilevel model. That is, the point of thinking about forking paths is not to correct the p-value or worry about statistical significance but rather to be aware of the many potential things you might want to study. The original sin is not the paths or even the p-value, it’s the implicit or explicit decision to look only at the biggest differences and not put them in context.

Shira responded:

Yes, that makes sense.

I think in most of your stories about this issue, looking at the biggest differences is what got them the statistical significance. And the smaller differences were not statistically significant. So the punchline is like “ah if they had only looked at the full context, we would see all these null results.”

In this tick paper, I think even the smaller differences are statistically significant. But the point of the original sin still stands. Maybe that’s why I like this example, because it is still a problem to only present the biggest difference, even when the smaller differences are statistically significant too.

She adds:

Oh and to be clear: I really do think Appalachian Traile hikers should permethrin-treat their clothes/shoes/gear! Lyme disease is terrible.