Abstract

In recent years, Machine Learning (ML) is increasingly being used for official statistics. However, as its usage for statistical estimation is relatively new, the effect of using ML on the total error of the estimate needs to be better understood. Existing quality frameworks such as the Total Error Framework (TEF) and the Total Survey Error Framework (TSE) help identifying some errors, but do not address other errors specific to the process of applying ML.

To fill this gap, Puts, Daas & Salgado developed the Total Machine Learning Error framework (TMLE), which is inspired by TSE. Its key contribution is the observation that data is collected, processed, and applied twice when using ML: (1) for the training data to estimate model parameters, and (2) for the inference data to estimate the desired statistic. Both steps may introduce measurement and representational errors, e.g. specification and sampling errors, to the respective datasets. This results in a biased model and accumulates in the estimate’s total error.

The TMLE will be applied on a smart survey project conducted by Statistics Netherlands: the household budget. A theoretical analysis will be performed to study the errors and their effects on the ML model and the estimate. We then demonstrate the usefulness of the TMLE in improving the data quality. Lastly, we discuss the challenges and criteria for the proper application of ML on smart survey data.