66th ISI World Statistics Congress

66th ISI World Statistics Congress

Smart use of Large Language Models for production-ready statistical classification coding (National Statistical Institute of Spain, INE)

Author

AP
Adrián Pérez Bote

Co-author

Conference

66th ISI World Statistics Congress

Format: WSC 2027: IPS Abstract

Keywords: classification of economic activities, coding, large language models (llms), multiclass classification, synthetic_data

Abstract

Automatic coding of free-text descriptions based on statistical classifications is of great importance for statistical organizations. However, it remains an unsolved challenge. The landscape of solutions has evolved from rule-based strategies to supervised machine learning (ML), and currently Large Language Models (LLM)-based methods such as Retrieval-Augmented Generation (RAG) are gaining traction. Each of these approaches presents its own drawbacks.
The most widespread approach is training a supervised ML model. This approach requires hundreds of thousands or even millions of labelled samples to achieve the desired performance. Obtaining these labelled samples may be infeasible in some contexts. In addition, supervised ML techniques perform poorly on out-of-distribution (OOD) samples, that is, samples drawn from a distribution different from that of the training data.
LLMs offer outstanding capabilities in Natural Language Processing, solving many tasks using zero-shot or few-shot strategies. RAG is among the most popular LLM-based approaches, as it reduces the latency and cost compared with simple prompting techniques. Due to its design, RAG is expected to perform well across a wide variety of scenarios, since the absence of training dataset removes the distinction between in-distribution (ID) and OOD samples. However, RAG presents high latency and computational costs, making it infeasible for many realistic scenarios.
We propose a framework that combines ideas from supervised ML and LLM-based strategies and achieves a balance between performance, development effort and production feasibility. It consists of gathering organic training data if available, generating labelled texts using the ZeroGen technique and constructing a dataset from pre-existing resources such as ontologies and indexes. We then filter incorrect labels using embedding-based similarity measures. Finally, we fine-tune pre-trained BERT-based models on an aggregation of these three datasets.
The proposed methodology was tested over the NACE Rev. 2.1. classification, the European version of ISIC Rev. 5. We compared it against RAG and an ML model trained solely on organic data. Our methodology achieved the best accuracy and macro-F1 score in almost all test sets. Compared with the baseline supervised ML approach, it improves the macro-F1 score by up to 20pp.
BERT-based models trained only on synthetic data (both generated using ZeroGen and extracted from pre-existing resources) achieve competitive performance compared with the same models trained on organic data and even surpass them on OOD samples. This suggests that competitive coders can be developed without using organic data, enabling the rapid and flexible development of automatic coders.

Figures/Tables

perf_nace_rev21_zoomout