How to Exploit Weaknesses in Biomedical Challenge Design and Organization. - Annika Reinke; Matthias Eisenmann; Sinan Onogur; Marko Stankovic 002; Patrick Scholz 001; Peter M. Full; Hrvoje Bogunovic; Bennett A. Landman; Oskar Maier; Bjoern H. Menze; Gregory C. Sharp; Korsuk Sirinukunwattana; et al. (MICCAI (4), 2018)

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions. - Lena Maier-Hein; Matthias Eisenmann; Annika Reinke; Sinan Onogur; Marko Stankovic 002; Patrick Scholz 001; Tal Arbel; Hrvoje Bogunovic; Andrew P. Bradley; Aaron Carass; Carolin Feldmann; Alejandro F. Frangi; et al. (CoRR, 2018)

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions. - Lena Maier-Hein; Matthias Eisenmann; Annika Reinke; Sinan Onogur; Marko Stankovic 002; Patrick Scholz 001; Peter M. Full (Zenodo, 2018)

Methods and open-source toolkit for analyzing and visualizing challenge results. - Manuel Wiesenfarth; Annika Reinke; Bennett A. Landman; Manuel Jorge Cardoso; Lena Maier-Hein; Annette Kopp-Schneider (CoRR, 2019)

Robust Medical Instrument Segmentation Challenge 2019. - Tobias Roß; Annika Reinke; Peter M. Full; Martin Wagner 001; Hannes Kenngott; Martin Apitz; Hellena Hempe; Diana Mîndroc-Filimon; Patrick Scholz 001; Thuy Nuong Tran; Pierangela Bruno; Pablo Arbeláez; et al. (CoRR, 2020)

Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room. - Lena Maier-Hein; Martin Wagner 001; Tobias Roß; Annika Reinke; Sebastian Bodenstedt; Peter M. Full; Hellena Hempe; Diana Mîndroc-Filimon; Patrick Scholz 001; Thuy Nuong Tran; Pierangela Bruno; Anna Kisilenko; et al. (CoRR, 2020)

BIAS: Transparent reporting of biomedical image analysis challenges. - Lena Maier-Hein; Annika Reinke; Michal Kozubek 001; Anne L. Martel; Tal Arbel; Matthias Eisenmann; Allan Hanbury; Pierre Jannin; Henning Müller; Sinan Onogur; Julio Saez-Rodriguez; Bram van Ginneken; et al. (Medical Image Anal., 2020)

Common Limitations of Image Processing Metrics: A Picture Story. - Annika Reinke; Matthias Eisenmann; Minu Dietlinde Tizabi; Carole H. Sudre; Tim Rädsch; Michela Antonelli; Tal Arbel; Spyridon Bakas; M. Jorge Cardoso; Veronika Cheplygina; Keyvan Farahani; Ben Glocker; et al. (CoRR, 2021)

The Federated Tumor Segmentation (FeTS) Challenge. - Sarthak Pati; Ujjwal Baid; Maximilian Zenk; Brandon Edwards; Micah J. Sheller; G. Anthony Reina; Patrick Foley; Alexey Gruzdev; Jason Martin; Shadi Albarqouni; Yong Chen 016; Russell Taki Shinohara; et al. (CoRR, 2021)

The Medical Segmentation Decathlon. - Michela Antonelli; Annika Reinke; Spyridon Bakas; Keyvan Farahani; Annette Kopp-Schneider; Bennett A. Landman; Geert Litjens 001; Bjoern H. Menze; Olaf Ronneberger; Ronald M. Summers; Bram van Ginneken; Michel Bilello; et al. (CoRR, 2021)

How can we learn (more) from challenges? A statistical approach to driving future algorithm development. - Tobias Roß; Pierangela Bruno; Annika Reinke; Manuel Wiesenfarth; Lisa Koeppel; Peter M. Full; Bünyamin Pekdemir; Patrick Godau; Darya Trofimova; Fabian Isensee; Sara Moccia; Francesco Calimeri; et al. (CoRR, 2021)

Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark. - Martin Wagner 001; Beat Peter Müller-Stich; Anna Kisilenko; Duc Tran; Patrick Heger; Lars Mündermann; David M. Lubotsky; Benjamin Müller; Tornike Davitashvili; Manuela Capek; Annika Reinke; Tong Yu 009; et al. (CoRR, 2021)

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge. - Tobias Roß; Annika Reinke; Peter M. Full; Martin Wagner 001; Hannes Kenngott; Martin Apitz; Hellena Hempe; Diana Mîndroc-Filimon; Patrick Scholz 001; Thuy Nuong Tran; Pierangela Bruno; Pablo Arbeláez; et al. (Medical Image Anal., 2021)

Metrics reloaded: Pitfalls and recommendations for image analysis validation. - Lena Maier-Hein; Annika Reinke; Evangelia Christodoulou; Ben Glocker; Patrick Godau; Fabian Isensee; Jens Kleesiek; Michal Kozubek 001; Mauricio Reyes 001; Michael A. Riegler; Manuel Wiesenfarth; Michael Baumgartner 001; et al. (CoRR, 2022)

Labeling instructions matter in biomedical image analysis. - Tim Rädsch; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Nicholas Schreck; A. Emre Kavur; Bünyamin Pekdemir; Tobias Roß; Annette Kopp-Schneider; Lena Maier-Hein (CoRR, 2022)

Biomedical image analysis competitions: The state of current participation practice. - Matthias Eisenmann; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Fabian Isensee; Tim J. Adler; Patrick Godau; Veronika Cheplygina; Michal Kozubek 001; Sharib Ali; Anubha Gupta; Jan Kybic; et al. (CoRR, 2022)

Ten years of image analysis and machine learning competitions in dementia. - Esther E. Bron; Stefan Klein 001; Annika Reinke; Janne M. Papma; Lena Maier-Hein; Daniel C. Alexander; Neil P. Oxtoby (NeuroImage, 2022)

MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images. - David Zimmerer; Peter M. Full; Fabian Isensee; Paul Jäger 002; Tim Adler; Jens Petersen; Gregor Köhler; Tobias Roß; Annika Reinke; Antanas Kascenas; Bjørn Sand Jensen; Alison Q. O'Neil; et al. (IEEE Trans. Medical Imaging, 2022)

Challenge Results are not Reproducible. - Annika Reinke; Georg Grab; Lena Maier-Hein (Bildverarbeitung für die Medizin, 2023)

Abstract: MOOD 2020 - A Public Benchmark for Out-of-distribution Detection and Localization on Medical Images. - David Zimmerer; Peter M. Full; Fabian Isensee; Paul Jäger 002; Tim Adler; Jens Petersen; Gregor Köhler; Tobias Roß; Annika Reinke; Antanas Kascenas; Bjørn Sand Jensen; Alison Q. O'Neil; et al. (Bildverarbeitung für die Medizin, 2023)

Why is the Winner the Best? - Matthias Eisenmann; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Fabian Isensee; Tim J. Adler; Sharib Ali; Vincent Andrearczyk; Marc Aubreville; Ujjwal Baid; Spyridon Bakas; Niranjan Balu; et al. (CVPR, 2023)

Deployment of Image Analysis Algorithms Under Prevalence Shifts. - Patrick Godau; Piotr Kalinowski; Evangelia Christodoulou; Annika Reinke; Minu Tizabi; Luciana Ferrer; Paul F. Jäger; Lena Maier-Hein (MICCAI (3), 2023)

Understanding metric-related pitfalls in image analysis validation. - Annika Reinke; Minu Tizabi; Michael Baumgartner 001; Matthias Eisenmann; Doreen Heckmann-Nötzel; A. Emre Kavur; Tim Rädsch; Carole H. Sudre; Laura Ación; Michela Antonelli; Tal Arbel; Spyridon Bakas; et al. (CoRR, 2023)

Why is the winner the best? - Matthias Eisenmann; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Fabian Isensee; Tim J. Adler; Sharib Ali; Vincent Andrearczyk; Marc Aubreville; Ujjwal Baid; Spyridon Bakas; Niranjan Balu; et al. (CoRR, 2023)

Panoptica - instance-wise evaluation of 3D semantic and instance segmentation maps. - Florian Kofler; Hendrik Kristian Möller; Josef A. Buchner; Ezequiel de la Rosa; Ivan Ezhov; Marcel Rosier; Isra Mekki; Suprosanna Shit; Moritz Negwer; Rami Al-Maskari; Ali Ertürk; Shankeeth Vinayahalingam; et al. (CoRR, 2023)

Eliminating flaws in biomedical image analysis validation - Annika Reinke (2023)

Labelling instructions matter in biomedical image analysis. - Tim Rädsch; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Nicholas Schreck; A. Emre Kavur; Bünyamin Pekdemir; Tobias Roß; Annette Kopp-Schneider; Lena Maier-Hein (Nat. Mac. Intell., 2023)

Beyond rankings: Learning (more) from algorithm validation. - Tobias Roß; Pierangela Bruno; Annika Reinke; Manuel Wiesenfarth; Lisa Koeppel; Peter M. Full; Bünyamin Pekdemir; Patrick Godau; Darya Trofimova; Fabian Isensee; Tim J. Adler; Thuy Nuong Tran; et al. (Medical Image Anal., 2023)

Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark. - Martin Wagner 001; Beat P. Müller-Stich; Anna Kisilenko; Duc Tran; Patrick Heger; Lars Mündermann; David M. Lubotsky; Benjamin Müller; Tornike Davitashvili; Manuela Capek; Annika Reinke; Carissa Reid; et al. (Medical Image Anal., 2023)

Sources of performance variability in deep learning-based polyp detection. - Thuy Nuong Tran; Tim J. Adler; Amine Yamlahi; Evangelia Christodoulou; Patrick Godau; Annika Reinke; Minu Dietlinde Tizabi; Peter Sauer; Tillmann Persicke; Jörg Gerhard Albert; Lena Maier-Hein (Int. J. Comput. Assist. Radiol. Surg., 2023)

FISBe: A Real-World Benchmark Dataset for Instance Segmentation of Long-Range thin Filamentous Structures. - Lisa Mais; Peter Hirsch 001; Claire Managan; Ramya Kandarpa; Josef Lorenz Rumberger; Annika Reinke; Lena Maier-Hein; Gudrun Ihrke; Dagmar Kainmueller (CVPR, 2024)

Quality Assured: Rethinking Annotation Strategies in Imaging AI. - Tim Rädsch; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Nicholas Heller; Fabian Isensee; Annette Kopp-Schneider; Lena Maier-Hein (ECCV (78), 2024)

Confidence Intervals Uncovered: Are We Ready for Real-World Medical Imaging AI? - Evangelia Christodoulou; Annika Reinke; Rola Houhou; Piotr Kalinowski; Selen Erkan; Carole H. Sudre; Ninon Burgos; Sofiène Boutaj; Sophie Loizillon; Maëlys Solal; Nicola Rieke; Veronika Cheplygina; et al. (MICCAI (10), 2024)

Metrics Reloaded: A framework for trustworthy image analysis validation. - Ali Emre Kavur; Annika Reinke; Paul F. Jäger; Fabian Isensee; Tim Rädsch; Minu Dietlinde Tizabi; Lena Maier-Hein (Zenodo, 2024)

FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures. - Lisa Mais; Peter Hirsch 001; Claire Managan; Ramya Kandarpa; Josef Lorenz Rumberger; Annika Reinke; Lena Maier-Hein; Gudrun Ihrke; Dagmar Kainmueller (Zenodo, 2024)

Abstract: Metrics Reloaded - Recommendations and Online Toolkit for Image Analysis Validation. - A. Emre Kavur; Lena Maier-Hein; Annika Reinke (Bildverarbeitung für die Medizin, 2025)

Some Hidden Traps of Confidence Intervals in Medical Image Segmentation: Coverage Issues. - Pascaline Andrè; Charles Heitz; Evangelia Christodoulou; Annika Reinke; Carole H. Sudre; Michela Antonelli; M. Jorge Cardoso; Antoine Gilson; Sophie Tezenas du Montcel; Gaël Varoquaux; Lena Maier-Hein; Olivier Colliot (BRIDGE/DeCaF@MICCAI, 2025)

Application-driven validation of posteriors in inverse problems. - Tim J. Adler; Jan-Hinrich Nölke; Annika Reinke; Minu Dietlinde Tizabi; Sebastian Gruber 001; Dasha Trofimova; Lynton Ardizzone; Paul F. Jaeger; Florian Buettner 001; Ullrich Köthe; Lena Maier-Hein (Medical Image Anal., 2025)

Navigating prevalence shifts in image analysis algorithm deployment. - Patrick Godau; Piotr Kalinowski; Evangelia Christodoulou; Annika Reinke; Minu Tizabi; Luciana Ferrer; Paul Jäger 002; Lena Maier-Hein (Medical Image Anal., 2025)

Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation. - Tim Rädsch; Leon D. Mayer; Simon Pavicic; A. Emre Kavur; Marcel Knopp; Baris Öztürk; Klaus H. Maier-Hein; Paul F. Jaeger; Fabian Isensee; Annika Reinke; Lena Maier-Hein (CoRR, 2025)

False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims. - Evangelia Christodoulou; Annika Reinke; Pascaline Andrè; Patrick Godau; Piotr Kalinowski; Rola Houhou; Selen Erkan; Carole H. Sudre; Ninon Burgos; Sofiène Boutaj; Sophie Loizillon; Maëlys Solal; et al. (CoRR, 2025)

Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study. - Leon D. Mayer; Tim Rädsch; Dominik Michael; Lucas Luttner; Amine Yamlahi; Evangelia Christodoulou; Patrick Godau; Marcel Knopp; Annika Reinke; Fiona R. Kolbinger; Lena Maier-Hein (CoRR, 2025)

Federated Learning for Surgical Vision in Appendicitis Classification: Results of the FedSurg EndoVis 2024 Challenge. - Max Kirchner; Hanna Hoffmann; Alexander C. Jenke; Oliver Lester Saldanha; Kevin Pfeiffer; Weam Kanjo; Julia Alekseenko; Claas de Boer; Santhi Raj Kolamuri; Lorenzo Mazza; Nicolas Padoy; Sophia Bano; et al. (CoRR, 2025)