Bounding Interviewer Effects: Data Quality when Objective Measures are Missing
Conference
10th International Conference on Agricultural Statistics
Format: CPS Paper - ICAS 2026
Keywords: data quality, interviewers
Abstract
For most survey-based research, interviewers identify respondents, pose survey questions, record answers, and often navigate complex social interactions to collect data. Using a "backcheck” quality-control process from nine large household surveys, we find strong evidence of interviewer variance in face-to-face and telephone interviews. We reject the null hypothesis that interviewer effects are consistent with classical measurement error. We then extend the Abowd (2013) econometric strategy to estimate reliability ratios using prior beliefs on mode accuracy to bound interviewer measurement error. Reliability ratios differ by variable type and confidence in the accuracy of resurvey data with specific applications to agricultural variables. Pooling across all outcomes, we find reliability ratios are 18 percent lower in telephone surveys than face-to-face surveys. We discuss the implications of these reliability ratios as data quality measures.