Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation

Authors

Katharina V Hoebel
Christopher P Bridge
Sara Ahmed
Oluwatosin Akintola
Caroline Chung
Raymond Y Huang
Jason M Johnson
Albert Kim
K Ina Ly
Ken Chang
Jay Patel
Marco Pinho
Tracy T Batchelor
Bruce R Rosen
Elizabeth R Gerstner
Jayashree Kalpathy-Cramer

Publication Date

1-1-2024

Journal

Radiology Artificial Intelligence

Abstract

Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality.

Keywords

Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning

Comments

Supplemental material is available

PMID: 38197800