Uncovering large inconsistencies between machine learning derived gridded settlement datasets

High-resolution human settlement maps provide detailed delineations of where people live and are vital for scientific and practical purposes, such as rapid disaster response, allocation of humanitarian resources, and international development. The increased availability of high-resolution satellite imagery, combined with powerful techniques from machine learning and artificial intelligence (AI), has spurred the creation of a wealth of settlement datasets. The agreement and alignment between these datasets has not been studied in detail. We compare three settlement maps developed by Google (Open Buildings), Meta (High Resolution Population Density Maps) and Microsoft (Global Building Footprints), and uncover which factors drive mismatch. Our study focuses on 44 African countries. We build a global machine learning model to predict where datasets agree, and find that geographic and socio-economic factors considerably impact overlap. However, we also find there is great variability across countries, suggesting complex interactions between country morphology and dataset overlap. It is vital to understand the shortcomings of AI-derived settlement layers as international organizations, governments, and NGOs are already experimenting with incorporating these into programmatic work. We anticipate our work to be a starting point for more critical and detailed analyses of AI derived datasets for humanitarian, policy, and scientific purposes.