Comparing Job Training Impact Estimates Using Survey and Administrative Data

Publisher: Washington, DC: Mathematica Policy Research
Sep 28, 2018
Annalisa Mastri, Dana Rotz, and Elias S. Hanno
The data source used for analysis can affect the accuracy and precision of a study’s estimates. In this paper, we examine three sources used to produce impact estimates for the WIA Adult and Dislocated Worker Programs Gold Standard Evaluation: (1) survey data, (2) administrative data from the National Directory of New Hires (NDNH), and (3) Administrative Tax Data. We further explore the strengths and drawbacks of administrative and survey data by directly comparing estimates produced using the survey data and NDNH. We found that both sources were consistent in their main conclusions about program effectiveness but that impacts calculated using the NDNH were smaller than those calculated using survey data, consistent with previous studies of similar sources. Three factors likely explain most of the observed differences: (1) many survey respondents reported jobs not captured by the NDNH; (2) survey respondents typically overreported earnings in any given job, especially early in the follow-up period; and (3) survey respondents typically underreported the number of jobs they held early in the follow-up period.