Making Metadata Count

The goal of this project is to improve the role of demographic meta-data in machine learning benchmarks, focused on computer-aided diagnosis. In clinical settings, meta-data such as age of sex of patient are crucial to the diagnosis of that patients. However, when developing machine learning algorithms for diagnosis, such meta-data is often not taken into account. In fact, our preliminary study shows that 90% of diagnostic algorithms at a recent conference, do not mention such information, in part because medical image data for machine learning do not include meta-data. This can lead to (i) poor and/or biased algorithm predictions, and (ii) underexplored research questions. Addressing the missing meta-data problem is therefore crucial for responsible translation of diagnostic algorithms to real-life settings. This project will consist of a systematic review of the use of meta-data in computer-aided diagnosis benchmark, and investigate how to best include such meta-data when training algorithms.