Program evaluation: estimation vs. simulation

Simulations are a tool that is more and more used to evaluate policies. This is in particular important when these policies have never been implemented before and thus there is no historical data to draw on. How good are such simulations? One way to check this is to do an ex-ante simulation of a policy change that has actually been implemented thereafter, and then estimate ex-post its effect.

Fabian Bornhorst does this with the PROGRESA program in Mexico. There, some families receive transfers if their children go to school. Bornhorst first draws a model of occupational choice and then estimates it using data that was available before PROGRESA was implemented. He then looks at outcomes if PROGRESA is applied in this model economy. This is then compared to actual outcomes. How well does the simulation fare? Not bad at all, but not perfectly either. It tends to indicate somewhat stronger outcomes than those actually observed. Also, it does not capture some differences across groups, but this can mainly be attributed to the level of heterogeneity within the simulated model, so this is fixable. Overall, very encouraging.