Innovations and improvements in collecting and processing household spending diary data
Conference
65th ISI World Statistics Congress
Format: IPS Abstract - WSC 2025
Thursday 9 October 2 p.m. - 3:40 p.m. (Europe/Amsterdam)
Abstract
The Living Costs and Food Survey (LCF) collects information on spending patterns and the cost of living that reflects household budgets across the UK.
Currently, detailed expenditure data is collected by interviewers using Excel spreadsheets, including embedding images of receipts within them. The process was developed at pace when the pandemic hit in March 2020. No user testing was carried out prior to roll out, due to the pace of implementation needed to minimise the pause in data collection for the LCF.
Prior to the pandemic diary data for the LCF was collected using person level paper diaries. Each eligible household member recorded their irregular spending for 14 days. As such, when the Excel process was implemented, the burden of diary completion switched from the respondent to the interviewer, significantly increasing interviewer administration time.
Approval to develop a more user-friendly interviewer facing collection tool in Blaise 5 was granted in November 2022. The aims of the project were to reduce interviewer and coder burden, develop a more user-friendly tool, and improve data quality.
Development of the tool involved users from the start of the project to ensure an improved user journey. Regular focus groups were held to gather feedback, and users were involved in iterative testing. User feedback gathered informed the ongoing development of the tool. Automated transfer of data from the interviewer tool to the coder system was introduced to realise efficiencies in processing.
Alongside development of the interviewer facing tool, approval was sought to explore development of a longer-term respondent facing digital collection tool. The discovery was completed in spring/summer 2023 and confirmed the current excel diary is not fit for purpose. The recommendation was to progress to alpha to explore whether a browser based digital solution is fit for purpose for respondents. The Alpha project is being progressed in autumn 2024, with the aim of answering 3 key research questions:
● Collecting data at source: Does a digital diary enable a reduction in missing/incomplete data and a reduction in interviewer/coder queries?
● Ensuring that people can use the technology: Does the technology have sufficient usability to enable users to easily enter expenditure details in real life settings?
● Determine whether a browser solution provides an acceptable user experience and complete data: How do respondents manage over multiple days without offline data capture?
In addition to development of the interviewer and respondent facing tools, work has continued to explore methods for receipt scanning and automated classification. The aim is to implement efficiencies in processing of LCF data. The work is currently being led by the ONS Data Science Campus.
Experiences in the development of both the interviewer and respondent facing tool will be shared during the session, including progress with development of the automated methods to enable further efficiencies in data processing to be realised.