Abstract

Regulators increasingly require fairness audits of automated decisions, and the Royal Statistical Society asks that the evidence submitted about AI systems be statistically grounded and tested under distribution shift. To our knowledge, no audit framework specifies how an audit should proceed when decisions are made about people in places where outcomes cluster geographically for reasons that may or may not be legitimate and where location itself carries information about protected status. We propose a six-step protocol for such audits, built on B-Fair, a Bayesian measure of disparity among merit-comparable individuals that is aligned with protected context. Two steps are declarations that should be made before the outcomes are examined: who counts as comparable, and which spatial variation in outcomes a fair system may show. Three are computations: nuisance spatial models fitted without the outcomes, a posterior distribution of the disparity, and a calibration of the decision against the audited population itself. The last is a report that a regulator can read as evidence rather than as a score. A simulation study shows why each step is needed. A fair world that ignores legitimate spatial clustering flags 15.9% of fair systems. A decision threshold carried over from another population flags up to 21.0%, because what a fair system looks like depends on the audited geography; calibrating each audit to its own population reduces this to 3.4%, at the cost of 18% of detections. The failure of a borrowed threshold persists across effect sizes, spatial clustering ranges, and sample sizes; in larger samples, it becomes relatively worse, not better. How much spatial variation is legitimate cannot be learned from the data, so it should be declared in advance. A worked audit report shows that the same transported threshold both condemns a fair city and clears an unfair one, while the protocol decides both correctly. A fairness threshold is not a portable standard; an audit procedure can be.