Training Like a Medical Resident: Universal Medical Image Segmentation via Context Prior Learning

Yunhe Gao1  Zhuowei Li1  Di Liu1  Mu Zhou1  Shaoting Zhang2  Dimitris N. Metaxas1 
1Rutgers University
2Shanghai Artificial Intelligence Laboratory 

PDF ArXiv Code
Clinical diagnostic workflows typically focus on specific specialties and diseases, leading to the curation of image datasets that are partially annotated, multi-modal, and multi-regional. Traditional training paradigms involve training separate models for each segmentation task (or dataset). In this study, we emphasize a universal medical image segmentation paradigm aiming at one model for all, leading to a robust and generalizable universal model for diverse tasks.

Abstract

A major enduring focus of clinical workflows is disease analytics and diagnosis, leading to medical imaging datasets where the modalities and annotations are strongly tied to specific clinical objectives. To date, building task-specific segmentation models is intuitive yet a restrictive approach, lacking insights gained from widespread imaging cohorts. Inspired by the training of medical residents, we explore universal medical image segmentation, whose goal is to learn from diverse medical imaging sources covering a range of clinical targets, body regions, and image modalities. Following this paradigm, we propose Hermes, a context prior learning approach that addresses the challenges related to the heterogeneity on data, modality, and annotations in the proposed universal paradigm. In a collection of seven diverse datasets, we demonstrate the appealing merits of the universal paradigm over the traditional task-specific training paradigm. By leveraging the synergy among various tasks, Hermes shows superior performance and model scalability. Our in-depth investigation on two additional datasets reveals Hermes' strong capabilities for transfer learning, incremental learning, and generalization to different downstream tasks.

Illustration of Hermes. A context prior knowledge pool, including task and modality priors, is learned with the segmentation backbone. Through oracle-guided selection and combination of these priors, Hermes can address a variety of segmentation tasks and image modalities.

BibTeX

@article{gao2023training,
        title={Training Like a Medical Resident: Universal Medical Image Segmentation via Context Prior Learning},
        author={Gao, Yunhe and Li, Zhuowei and Liu, Di and Zhou, Mu and Zhang, Shaoting and Meta, Dimitris N},
        journal={arXiv preprint arXiv:2306.02416},
        year={2023}
}