Benchmarking computer-aided detection of pulmonary nodules on the recently completed publicly available LIDC/IDRI database

PURPOSE The recently completed LIDC/IDRI database provides by far the largest public resource to assess the performance of algorithms for the detection of pulmonary nodules in thoracic CT scans. We report the performance of two detection systems, and address the issue of completeness of the reference standard. METHOD AND MATERIALS The LIDC/IDRI database contains 890 thoracic CT scans with section thickness of 2.5mm or lower, one per patient, from 7 centers acquired with 17 different scanner models from 4 manufacturers. Cases have been annotated in an extensive reading process comprising a blinded and an unblinded review by four radiologists who indicated all nodules <3mm and >3mm effective diameter. We define nodules >3mm indicated by all four observers as positive findings. We applied two pulmonary nodule detection systems: Herakles, an industry research prototype (MeVis Medical Solutions, Bremen, Germany) and ISICAD (Image Sciences Institute, Utrecht, The Netherlands), a system trained with data from the Dutch-Belgian NELSON lung cancer screening trial. We report sensitivity at 1, 2, and 4 false positive (FP) detections per scan and analyze the FPs. RESULTS The 890 scans contained 775 positive findings. At 1, 2, and 4 FP/scan, Herakles had a sensitivity of 69%, 75%, and 79%, respectively. For ISICAD this was 51%, 63%, 72%. We analyzed the FPs of Herakles at an operating point of 2 FP/scan. Of these, 31% were annotated by at least one radiologist as a nodule >3mm. An additional 17% were indicated by at least one radiologist as a nodule <3mm. A human expert visually inspected the remaining FPs using multiple slices of all three orthogonal views. A substantial part of these marks (41%) were located on nodular lesions that had not been indicated by any of the four radiologists involved in the annotation of the LIDC/IDRI data set . CONCLUSION The LIDC/IDRI data set is an excellent benchmarking tool for nodule detection algorithms. Automated detection can identify pulmonary nodules that have not been annotated in an extensive reading process with blinded and unblinded review by four human observers. CLINICAL RELEVANCE/APPLICATION Algorithms for automatic detection of pulmonary nodules can be compared and improved through the availability of a common database for benchmarking.