What the DST researchers actually found

To add to my prior post, having now read the published paper on the effect of DST on heart attacks, I can confirm that I disagree with the way the publicist hired by the journal messaged the research conclusion. And some of the fault lies with the researchers themselves who appear to have encouraged the exaggerated claim.

Here is the summary of the research as written up by the researchers themselves. First I note the following conclusion:

and right before, they write this explanation of the "timing" effect:

So indeed, if I were to believe the research, someone may have a heart attack on Monday instead of Tuesday "as a result of" daylight savings time in the spring. And wait a minute, by reversing this change in the fall, we seemingly postpone some heart attacks by two days. Hence my assertion that even if true, the phenomenon is not interesting.

In fact, I think this study provides negative evidence toward the idea that DST causes heart attacks. Here is how the authors describe their hypothesis:

The new data show no statistical difference in overall heart attack (admissions) for either period. That is their main result.

***

In this post, I want to discuss the challenges of this type of research. The underlying data is OCCAM (see definition here). It is observational in nature, it has no controls, it is seemingly complete (for "non-federal hospitals in Michigan), it is adapted and merged (as explained in the prior post).

Start with the raw data, in which there is a blip observed the Monday after Spring Forward. This problem is one of reverse causation: we see a blip, now we want to explain it.

Spring Forward is put forward as a hypothetical "cause" of this blip. But, we should realize that there is an infinity of alternative causes.

Seasonality is clearly something that needs to be considered. Is it normal to see an increase in admissions from Sunday (weekend) to Monday? To establish how unusual that blip is, we need to manufacture a "control," because none exists in the data.

In the poster presentation, the researchers use a simple control: what happened the week before? (This is known as a pre-post analysis.) The red line shown on the chart would suggest that a jump on Monday is unusual. This chart is a reproduction of the two charts from the poster but superimposed.

One can complain that the pre-1-week control is too simplistic. What if the week before was anomalous? A natural way forward is to use more weeks of data in the control. In the published paper, the researchers abandon the pre-1-week control, and basically use several years of data to establish a trend.

But this effort is complicated by the substantial variability in the data over time:

(I can't explain why the counts here are so much lower than the counts given in the post-DST week line in the first chart. In the paper, they describe the range of daily counts as 14 to 53.)

So expanding the window of analysis is double-edged. On the one hand, we guard against the one week prior to Spring Forward being an anomaly; on the other hand, we include other weeks of the year that are potentially not representative of the period immediately prior to Spring Forward.

The researchers do not simply average the prior weeks--they actually produce a statistical adjustment on the raw data, and call that the "trend model prediction". This is a very appealing concept. What we really want to know (but can't) is the "counterfactual": the number of cases if there were no DST time change.

In the next chart (reproduced from their paper), the "trend" line is what the authors claim the counterfactual counts would have been. They then compare the red line to the blue line (actual counts) and make claims about excess cases.

***

Of course, the devil is in the details. If you're going to make predictions about the counterfactual, the reader has to gain confidence in the assumptions you use to create those predictions.

One way to understand the statistical adjustment is to plot the raw data and the adjusted data side by side. Unfortunately we don't have the raw data. We do have the one week of pre-DST data from the poster. So I compare that to the "trend".

This chart raises two questions. First, the predicted counts in the paper are about 30% higher than the counts in pre-week of the poster. Second, the pre-week distribution of count by day matches the "trend" poorly.

 

While the pre-count is not expected to match the predicted "trend" perfectly, I'd expect that the post-counts should match since both the poster and the paper address what happens the week after the DST time change.

Strangely enough, the counts in the paper are 35% higher than those in the poster for the post-DST week! I'm not sure what to make of this: maybe they have expanded the definition of what counts as "hospital admissions for AMI requiring PCI".

The attempt to establish a control by predicting the counterfactual is a good idea. Given the subjectivity of such adjustments, researchers should be rigorous in explaining the effect of the adjustments. Stating the methodology or the equations involved is not sufficient. The easiest way to explain the adjustments is to visualize the unadjusted versus the adjusted data. The direction and magnitude of the adjustments should make sense.

 ***

Going back to the problem of reverse causation. Seasonality, trend and DST are only three possible causes for the Monday blip. Analysts must make an effort to rule out all other plausible explanations, such as bad data (e.g. every time the time changes, some people forget to move their clocks).

As I am testing your patience again with the length of this post, I will put my remaining comments in a third post.