Claims Reloaded
The mission of Claims Reloaded is to provide an open-source toolkit for assessing the validity of claims of superior performance in biomedical segmentation and classification tasks, enabling researchers and reviewers to detect unsupported claims.
Performance comparisons are fundamental in medical imaging AI research, often driving claims of superiority based on relative improvements in common performance metrics. However, such claims frequently rely on empirical mean performance. Claims Reloaded quantifies the probability of false claims based on a Bayesian approach that leverages reported results alongside empirically estimated model congruence to estimate whether the relative ranking of methods is likely to have occurred by chance.
Publications
False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims
Christodoulou E, Reinke A, Andrè P, Godau P, Kalinowski P, Houhou R, Erkan S, Sudre C, Burgos N, Boutaj S, Loizillon S, Solal M, Cheplygina V, Heitz C, Kozubek M, Antonelli M, Rieke N, Gilson A, Mayer L, Tizabi M, Cardoso M, Simpson A, Kopp-Schneider A, Varoquaux G, Colliot O, Maier-Hein L - arXiv - 2025