Assess external validity: to what populations, programs and places can the results be safely extrapolated?

Program Evaluation CritiqueDue: May 7th, 2020 No matter where you find yourself a year from now, learning how to critically evaluate research is an essential part of informing and making policy. This exercise is designed to help you hone those skills. Select one of the four articles listed below to read and offer a summary of the findings (including a description of the research question, what econometric techniques the authors used to answer the research question, who is “treated” and who is not, what assumptions the authors make about the treatment and comparison group, and the main findings), the strengths and weaknesses of the analysis, and the policy implications of the findings. Students should work independently and write a report on their findings.To help guide your assessment of the article you choose, I have enclosed a list of questions that you should make sure you have answered in your assessment. These questions serve as a guide—please do NOT turn in a sheet with just a list of your answers to these questions. Write in complete sentences as if you were writing a policy brief for someone.Note: You are welcome to discuss the articles with your classmates, but each student must independently write up their own report.Hoynes, H. W., & Schanzenbach, D. W. (2009). Consumption responses to in-kind transfers: Evidence from the introduction of the food stamp program. American Economic Journal: Applied Economics, 1(4), 109-139.Cohodes, Sarah R., & Goodman, Joshua S. 2014. Merit aid, college quality, and college completion: Massachusetts’ Adams scholarship as an in-kind subsidy. American Economic Journal: Applied Economics, 6(4), 251-285.Deming, D., 2009. Early childhood intervention and life-cycle skill development: Evidence from Head Start. American Economic Journal: Applied Economics, 1(3), pp.111-34.Rossin-Slater, Maya; Christopher J. Ruhm., and Jane Waldfogel. 2013. The effects of California’s paid family leave program on mothers’ leave-taking and subsequent labor market outcomes. Journal of Policy Analysis and Management. 32(2): 224-245.General Advice on Reading Empirical Studies(courtesy of Prof. Sue Dynarski)Expect jargon and poor writing (enjoy the rare exceptions!). Plowing through nonsense in order to grasp the gist of a study is a critical task for policy analysts. You have an excellent grounding in empirical methods and are quickly gaining the skills needed to assess just about any empirical evaluation. You may not be able to interpret every equation or sentence, but if the authors have done their job then you will be able to evaluate their results and judge the quality of their analysis.Questions to Answer1)FAQs (Angrist and Pischke, MHE)a)What is the causal question of interest?b)Ideal experiment: What would be the ideal experiment you would conduct to answer this question?c)Identification strategy: How does the study use observational data to approximate an ideal experiment?2)Important Detailsa)What is the estimated effect of X on Y?i)What is its statistical significance?ii)What is its practical significance (magnitude)?b)Datai)What are some key characteristics of the sample (you could say a lot here – try to focus on what is relevant for assessing internal and external validity).3)Assess internal validity: is the identification strategy a good one?a)To answer this, first think about (and discuss) why treatment status varies in general. Do people choose treatment? Is it chosen for them? By what process?b)Then describe the identifying variation in this study. This is the variation left after we control for the other variables.c)What are the key threats to the internal validity of the study? As you list them, make sure you tell us how these threats could affect the results (e.g., sign the bias).4)Assess external validity: to what populations, programs and places can the results be safely extrapolated?