Abstract

Monitoring labor markets, especially in countries with fragile economies and vulnerable to shocks, requires high quality and up-to-date statistics as the basis for making informed decisions in policy-making. Since AI has shown better performance than statistical models in predicting labor market indicators and its use in government statistical agencies is still relatively low due to issues of transparency, replicability, and institutional credibility, this paper presents an explainable AI framework for monitoring unemployment levels in both micro- and macro-levels, which combines technical developments and statistical agency operations.
This paper uses micro-data from the Palestinian Labour Force Surveys (LFS) for the years 2015-2024, as provided by the Palestinian Central Bureau of Statistics (PCBS). It provides a dual approach for analyzing unemployment trends at both the micro and macro levels. On the micro-level, it identifies the probability of each person being unemployed based on a set of socio-demographic, educational and geographical variables; and, for this purpose, it trains CatBoost classification models. For ensuring the transparency of the model, it utilizes SHAP (SHapley Additive exPlanations), and hence allows identifying the most significant factors of unemployment risks for different subpopulations. On the macro level, it uses LSTM and Gradient Boosting Models to predict and now-cast quarterly unemployment rate based on the historical data available. As part of a rigorous evaluation process, it also implements an expanding window protocol to prevent information leakage and to obtain the best possible performance in terms of out-of-sample predictions.
As a final step, the paper proposes a complete, reproducible and deployable pipeline, which combines the micro and macro levels into a single pipeline. In particular, the micro-level risk scores will be used for monitoring the population and regions separately, and the macro-level forecasts will allow identifying the turning points and providing early warnings for short-term future changes. In addition, the empirical results show that the AI-based models are capable of better predictions than the statistical models, especially in times of high volatility and structural disruptions. Moreover, the application of the explainability mechanism will make the predictions more interpretable and useful for policymakers.
In addition to the improvement of the prediction accuracy, the paper highlights the importance of governance, transparency and operationality. The framework is developed according to the international guidelines on AI in official statistics, such as the OECD's recommendations on explainable AI and the UNSD’s principles for data innovation. The case study on Palestine demonstrates how advanced AI techniques may be deployed within official labor market statistics without losing credibility or respect for statistical standards.
Finally, the paper highlights the experience gained by national statistical agencies that would like to adopt AI-based forecasting and monitoring tools as well as their potential for scalable application in other developing and fragile economic environments.