Abstract

Automatic coding based on machine learning is increasingly adopted by National Statistical Institutes (NSIs) to support large-scale classification tasks involving hierarchical classifications. While deep learning models achieve high predictive accuracy, their operational deployment in official statistics requires stronger guarantees in terms of uncertainty management, explainability, and trustworthiness. In practice, the key challenge is not only to predict accurately, but to decide when a prediction can be trusted and safely automated.

This paper presents a practitioner-oriented yet technically grounded framework for uncertainty-aware and trustworthy automatic coding with machine learning, combining architectural design, statistical uncertainty quantification, and explainability methods. The approach is illustrated in a real-world text classification setting with hierarchical coding schemes.

We first introduce a custom deep learning architecture with multiple prediction heads aligned with the levels of the classification. At inference time, a confidence-based fallback mechanism allows the system to revert to a parent-level code when confidence at a finer level is insufficient. This design supports partial automation while limiting high-risk errors and facilitating human-in-the-loop validation.

Second, we address uncertainty estimation and calibration in deep text classifiers. We propose a custom loss, leveraging the hierarchy of classifications, designed to enhance calibration. We then discuss practical calibration metrics and post hoc calibration techniques, and show how calibrated probabilities enable meaningful confidence thresholding. To support operational decision-making, we rely on risk–coverage curves, which explicitly characterize the trade-off between automation coverage and prediction risk as confidence thresholds vary. These curves provide an intuitive tool to compare models, assess calibration quality, and select operating points consistent with institutional risk constraints.

Third, we present the use of conformal prediction to complement calibrated probabilities with formal error guarantees. Conformal methods enable the construction of prediction sets and abstention strategies with controlled risk, and can be naturally combined with hierarchical fallback mechanisms.

Finally, we explore explainable AI (xAI) mechanisms for deep learning–based automatic coding, leveraging cross-attention between text and label embeddings and attribution methods from Captum, including Layer Integrated Gradients at the embedding level. These explanations support model validation, error analysis, and user trust in production environments.

We conclude with practical recommendations for post-deployment monitoring, including calibration drift, confidence distribution shifts, and risk–coverage stability, illustrating how these tools contribute to the sustainable deployment of trustworthy machine learning systems for automatic coding in NSIs.