Abstract

The study reports early results from work carried out within the international project “One-stop-shop for Artificial Intelligence and Machine Learning for Official Statistics (AIML4OS)”. The initial aim of this proof-of-concept is the statement whether machine learning (ML) and large language models (LLMs) can assign Polish Classification of Activities (PKD, the national adaptation of NACE) codes from website text.
The first objective was to build dataset of a quality that would allow to use it as a training set. Company websites served as the source of the training data, since they offer detailed descriptions of activities, products and services and are frequently more up to date than business register records. Information collected from websites were long and noisy. Given the unstructured nature of the data, we had to clean and normalise the text, remove HTML markup, and anonymise personal details using regular expressions and NLP methods. The data were strongly imbalanced across classes, from about 800 observations in the largest class to none in others. To address the problem, the training set was extended with LLM-generated synthetic company descriptions.
After vectorisation with TF-IDF, the text was used to train three classical models (logistic regression, decision tree and random forest) and to fine-tune a multilingual transformer, XLM-RoBERTa. Hyperparameters were tuned by grid search and validated with three-fold cross-validation. Logistic regression with TF-IDF achieved the highest accuracy, outperforming the transformer model. Performance was assessed with accuracy and with top-5 and top-10 accuracy, the latter being relevant where a classifier is meant to propose a ranked list of candidate codes to a human coder rather than assign a code automatically.
The next stage of the work extends the pipeline with a Retrieval-Augmented Generation (RAG) architecture. The embedding of a company description serves as a query to a vector database (ChromaDB) containing NACE definitions, standardised descriptions of business activities and labelled example texts. The most similar entries are retrieved, optionally reordered by a reranking model, and supplied to the LLM as context, so that the model chooses among a short list of candidate codes instead of generating a code from memory. Grounding predictions in official definitions is expected to improve their quality and to reduce hallucinations. It also allows the knowledge base to be updated, for example after a change of classification revision, without retraining the model. If successful, the methodology could be adapted to other classifications, such as COICOP and ISCO.