Semi-supervised 3D universal lesion segmentation in CT thorax-abdomen scans

PURPOSE: Extracting volumetric information using automated lesion segmentation could allow for more accurate quantification of disease response in heterogeneous lesions. Using a single model for Universal Lesion Segmentation has the potential for faster inference times compared to multi-model approaches and allows for internal representation of lesion type features. METHODS: We compiled eight public datasets with segmentation masks for various lesion types in CT thorax-abdomen scans. Scans were resampled to 1mm isotropic voxel spacing and regions of interest were cropped centered on each lesion. A nnUnet was trained with 3213 lesions from 1481 studies and used to predict 3D segmentation masks for the circa 32,000 partially annotated lesions from the DeepLesion dataset. Masks were further refined by applying the GrabCut algorithm in three orthogonal directions based on the provided long and short-axis diameter measurements. We fine-tuned the nnUnet using the resulting masks and evaluated on a test set with full annotations. We experimented with epoch numbers and learning rate decay. All models were trained using 5-fold cross validation. RESULTS: Fine-tuning the model using the DeepLesion masks improved segmentation performance from 0.71 to 0.73 Dice compared to the baseline nnUnet. Segmentation performance ranged from 0.53, 0.61, 0.66, 0.77, 0.79 to 0.9 Dice for colon, pancreas, lymph node, lung, liver and kidney lesions. CONCLUSIONS: 3D universal lesion segmentation using large, aggregated datasets shows promise as an alternative to lesion specific models. By incorporating partially annotated data in a semi-supervised manner we can further increase data volume and model performance with minimal annotation effort. LIMITATIONS: This study used a small number of scans in the test set and did not evaluate model performance on out-of-distribution lesion types. FUNDING: This research was supported by the Eurostars PIANO project E113829.