Malignancy estimation of Pulmonary Nodules using Multi-View Multi-Time Point Convolutional Neural Networks

Lung Cancer is one of the leading causes of cancer-related deaths for both men and women in the United States. The aim of lung cancer screening is to detect lung cancer at an early stage. Majority of the time, after the lung nodule detection phase, only a small portion out of all the nodules that get detected turns out to be cancerous. Compared to traditional techniques that use handcrafted features and furthermore relies on tedious \& time-consuming prior lung nodule segmentation, the proposed method uses deep learning techniques in an end-to-end arrangement that performs both the feature extraction and classification directly from raw nodule patches. In this study, we focus on improving the pulmonary nodule malignancy estimation part by introducing a novel multi-view multi-timepoint convolutional neural network (MVMT-CNN) architecture that uses low dose CT images as its input. The dataset used in this study was taken from the National Lung Cancer Screening Trial (NLST)- which is the largest lung cancer screening trial known to date. We investigate the influence of whether adding temporal information of the same patient can help to improve the diagnosis. The proposed convolutional neural network architecture requires nine 2D patches- each of which represents a certain plane from the extracted 3D nodule patches. The nine planes are analyzed separately in parallel CNN streams and the output features coming from the nine different pathways are fused into one layer before passing it to the classification stage. Additionally, batch normalization and drop out layers are also incorporated in order to decrease the training time and reduce the chances of over-fitting. The average Area Under the ROC curve obtained after 5 fold cross validation along with bootstrapping were used to compare \& select the final best performing architecture. The robustness of the final selected model was examined and verified by swapping the time points to see if the network did actually learn to identify the growth of the nodule between timepoints. The proposed method confirms that using the proposed multi-view multi-timepoint CNN architecture improves the prediction ability of pulmonary nodules significantly.