External Validation of an Algorithm To Detect Vertebral Level Mislabeling and Autocontouring Errors

Authors

Tucker J Netherton
Didier Duprez
Tina Patel
Gizem Cifter
Laurence E Court
Christoph Trauernicht
Ajay Aggarwal

Publication Date

4-1-2025

Journal

Physics and Imaging in Radiation Oncology

Abstract

Background and purpose: This work performs external validation of a previously developed vertebral body autocontouring tool and investigates a post-processing method to increase performance to clinically acceptable levels.

Materials and methods: Vertebral bodies within CT scans from two separate institutions (40 from institution A and 41 from institution B) were automatically 1) localized and enumerated, 2) contoured, and 3) screened as a means of quality assurance (QA) for errors. Identification rate, contour acceptability rate, and QA accuracy were calculated to assess the tool's performance. These metrics were compared to those calculated on CTs from the model's original training dataset, and a post-processing technique was developed to increase the tool's accuracy.

Results: When testing the model without post-processing on external datasets A and B, accurate identification rates of 83 % and 92 % were achieved for vertebral bodies (C1-L5). Identification rate, contour acceptability rate and QA accuracy were reduced on both datasets compared to accuracies and rates measured on the model's orginal testing dataset. After algorithm adjustment, identification rate across all vertebrae increased on average by 4 % (p < 0.01) for dataset A and also 4 % on the dataset B (p = 0.01).

Conclusions: A post-processing adjustment within the machine learning pipeline increased performance of vertebral body localization accuracy to acceptable levels for clinical use. External validation of machine learning and deep learning tools is essential to perform before deployment to different insitutions.

Keywords

External validation of machine learning tools, Computed tomography, Vertebral level labeling, Image segmentation, Vertebral body segmentation

DOI

10.1016/j.phro.2025.100738

PMID

40129727

PMCID

PMC11932639

PubMedCentral® Posted Date

2-25-2025

PubMedCentral® Full Text Version

Post-print

Published Open-Access

yes