Evaluating the impacts of welfare reform?

John Pugliese writes:

I was recently in a conversation with some colleagues regarding the evaluation of recent welfare reform in California. The discussion centered around what types of design might allow us to understand the impact the changes. Experimental designs were out, as random assignment is not feasible. Our data is pre/post, and some of my colleagues believed that the best we can do under these circumstance was a descriptive study; i.e. no causal inference. All of us were concerned with changes in economic and population changes over the pre-to-post period; i.e. over-estimating the effects in an improving economy.

I was thought a quasi-experimental design was possible using MLM. Briefly, my suggestion was the following:

Match our post-participants to a set of pre-participants on relevant person level factors, and treat the pre/post differences as a random effect at the county level. Next, we would adjust the pre/post differences by changes in economic and population factors at level 2 (county level) in order to produce an estimate of the change.

I was wondering if you might share some thoughts on this approach?

In reply to your colleagues who believe all that can be done is description, recall Jennifer’s dictum that the goal of inference is always causal. A good description is fine—I spend much of my time as an applied researcher doing descriptive inference—but, implicitly or explicitly, it will usually be used for causal purposes, so it’s worth making that link and understanding the assumptions required for any given causal interpretation.

Regarding your idea, it seems to me that you’re considering variation of the treatment at the county level. From the perspective of data analysis, multilevel modeling is definitely the way to go. But I don’t know enough about the substantive context to say more than this. Maybe my commenters will have more to offer.