Abstract

In statistical production, classifying free-text descriptions into official statistical classifications (e.g., NACE, ISCO, ...etc) is a critical yet labor-intensive task. While Machine Learning (ML) can automate and scale this process, its adoption requires addressing quality assurance, reproducibility, and auditability. The proposed framework establishes a common approach to data handling, model training, validation, deployment, and documentation, facilitating governance and reproducibility while enabling reuse across use cases within Luxembourg National Statistical Institute (STATEC). To ensure data sovereignty and security, we leverage on-premise, open-source tools, eliminating cloud-based risks while maintaining full control over the ML lifecycle.

In complement to this framework, a human-in-the-loop (HITL) system, where experts validate ML generated classification proposals, enhances higher accuracy and trust. To operationalize this approach, we created a tool, which provides business experts with a modern, intuitive interface for efficient interaction with the ML system.