Metrics Reloaded
Keywords: Challenges, Competitions, Validation, Model Validation, Evaluation, Metrics, Metric Selection, Good Scientific Practice, Image Processing, Image Analysis, Computer Vision, Classification, Segmentation, Instance Segmentation, Semantic Segmentation, Detection, Localization, Medical Imaging, Biological Imaging
The mission of Metrics Reloaded is to guide researchers in the selection of appropriate performance metrics for biomedical image analysis problems, as well as provide a comprehensive online resource for metric-related information and pitfalls.
The reliable validation of algorithms is key for the translation of biomedical image analysis research into practice. However, an increasing body of work shows that this is often not the case, with the metrics used frequently not reflecting the underlying biomedical problem, which significantly diminishes the validity of the research. To resolve this issue, we have compiled the Metrics Reloaded Framework which enables users to make educated decisions on which metrics to use for a given biomedical problem. The framework currently supports image-level classification, semantic segmentation, object detection and instance segmentation problems.
Publications
Understanding metric-related pitfalls in image analysis validation
Reinke A, Tizabi M, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Kavur A, Rädsch T, Sudre C, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Buettner F, Cardoso M, Cheplygina V, Chen J, Christodoulou E, Cimini B, Farahani K, Ferrer L, Galdran A, van Ginneken B, Glocker B, Godau P, Hashimoto D, Hoffman M, Huisman M, Isensee F, Jannin P, Kahn C, Kainmueller D, Kainz B, Karargyris A, Kleesiek J, Kofler F, Kooi T, Kopp-Schneider A, Kozubek M, Kreshuk A, Kurc T, Landman B, Litjens G, Madani A, Maier-Hein K, Martel A, Meijering E, Menze B, Moons K, Müller H, Nichyporuk B, Nickel F, Petersen J, Rafelski S, Rajpoot N, Reyes M, Riegler M, Rieke N, Saez-Rodriguez J, Sánchez C, Shetty S, Summers R, Taha A, Tiulpin A, Tsaftaris S, Van Calster B, Varoquaux G, Yaniv Z, Jäger P, Maier-Hein L - Nature Methods - 2024
Metrics reloaded: recommendations for image analysis validation
Maier-Hein L, Reinke A, Godau P, Tizabi M, Buettner F, Christodoulou E, Glocker B, Isensee F, Kleesiek J, Kozubek M, Reyes M, Riegler M, Wiesenfarth M, Kavur A, Sudre C, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Rädsch T, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Blaschko M, Cardoso M, Cheplygina V, Cimini B, Collins G, Farahani K, Ferrer L, Galdran A, van Ginneken B, Haase R, Hashimoto D, Hoffman M, Huisman M, Jannin P, Kahn C, Kainmueller D, Kainz B, Karargyris A, Karthikesalingam A, Kofler F, Kopp-Schneider A, Kreshuk A, Kurc T, Landman B, Litjens G, Madani A, Maier-Hein K, Martel A, Mattson P, Meijering E, Menze B, Moons K, Müller H, Nichyporuk B, Nickel F, Petersen J, Rajpoot N, Rieke N, Saez-Rodriguez J, Sánchez C, Shetty S, van Smeden M, Summers R, Taha A, Tiulpin A, Tsaftaris S, Van Calster B, Varoquaux G, Jäger P - Nature Methods - 2024
