Abstract

Today, National Statistical Offices (NSOs) face a critical challenge: the amount of available data has increased dramatically, but valuable insights often remain trapped in fragmented silos. Traditional statistical systems that treat census and survey data as separate sources are not enough to provide timely, integrated, and high-quality information. This paper introduces the Integrated Processing and Analysis System (IPAS), a transformative ecosystem developed in-house by Statistics Indonesia (BPS) to address this challenge.

IPAS moves beyond simple digitization to establish a truly interoperable data environment. The system leverages distributed computing technologies such as Spark on a Data Lake. It shifts from rigid, disparate data formats to a unified modern architecture. Additionally, IPAS establishes a "Data Lakehouse" paradigm that harmonizes the ingestion, processing, and analysis of diverse datasets into a single source of truth. This ecosystem allows for the seamless integration of large-scale text census data, images, and big data with high-frequency surveys. Consequently, statisticians can uncover cross-domain insights that were previously computationally prohibitive or methodologically complex.

This paper will demonstrate how IPAS serves as a blueprint for the future of statistical infrastructure. We will explore how the system fosters data interoperability through implementation of GSBPM, standardized metadata governance and unified workflows, effectively bridging the gap between data engineering and statistical methodology through the strategic utilization of Artificial Intelligence (AI). Ultimately, IPAS illustrates that navigating the data revolution requires more than just new tools; it demands a fundamental restructuring of the statistical ecosystem to ensure that data is not just stored, but is accessible, connected, and ready to deliver impact for policy and society.