Emily Oster with another reason not to trust observational studies. (Hello, most nutritional research.)
Consider a case in which a new research finding links a health behavior with good health outcomes. A possible consequence is additional take-up of this behavior among individuals who engage in other positive health behaviors. If this occurs, then later analysis of observational data may be biased by the change in selection. Even sampling-driven false positive results may be confirmed in subsequent work. This paper asks whether these dynamic biases occur, whether they are quantitatively important in empirical settings, and whether standard selection-on-observables adjustments are sufficient to address them. Using data from vitamin supplementation and diet behaviors I show that selection responds endogenously to health recommendations. When behaviors are more recommended, they are more frequently adopted by individuals who engage in other positive health behaviors (not smoking, exercising) and who are better educated and richer. Further, the relationship between these behaviors and health outcomes changes over time. When behaviors are more recommended they are more strongly associated with positive health outcomes, including survival, weight and heart health. The effects are large and adjustment for selection on observables is insufficient to address the bias. This suggests research findings themselves may endogenously bias observational studies.