This folder contains an example to train and validate a 3D detection RetinaNet model. The workflow of MONAI detection module is shown in the following figure.
MONAI detection implementation is based on the following papers:
RetinaNet: Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017.
The dataset we are experimenting with in this example is LUNA16, which is based on LIDC/IDRI database [1,2,3].
LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules and to assign a probability for being a nodule to each location.
Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset! We acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health, and their critical role in the creation of the free publicly available LIDC/IDRI Database used in this study.
We follow the official 10-fold data splitting from the LUNA16 challenge and generate data-split JSON files using the script from nnDetection. The resulting JSON files can be downloaded from this link. In these files, the values of "box" are the ground truth boxes in world coordinates.
- For questions relating to the use of MONAI, please use our Discussions tab on the main repository of MONAI.
- For bugs relating to MONAI functionality, please create an issue on the main repository.
- For bugs relating to the running of a tutorial, please create an issue in this repository.
pip install -r requirements.txtWe provide resampled Nifti images for users to download (recommended). If you do this, please open luna16_prepare_env_files.py, change the value of "raw_data_base_dir" and "resampled_data_base_dir" to the directory where you store the downloaded images, and the value of "downloaded_datasplit_dir" to where you downloaded the data split json files.
Then run the following command and go directly to Sec. 3.2.
python3 luna16_prepare_env_files.pyAlternatively, you can download the original data and resample them by yourself with the following steps. Users can either download 1) mhd/raw data from LUNA16 or its copy, or 2) DICOM data from LIDC-IDRI with NBIA Data Retriever.
The raw CT images in LUNA16 have various voxel sizes. The first step is to resample them to the same voxel size, which is defined in the value of "spacing" in ./config/config_train_luna16_16g.json.
Then, please open luna16_prepare_env_files.py, change the value of "raw_data_base_dir" to the directory where you store the downloaded images, the value of "downloaded_datasplit_dir" to where you downloaded the data-split JSON files, and the value of "resampled_data_base_dir" to the target directory where you will save the resampled images. If you are using DICOM data, please also provide path to "dicom_meta_data_csv" which can be found in the downloaded folder from LIDC-IDRI. Otherwise, please make sure that the "dicom_meta_data_csv" variable in the script equals to None.
If you downloaded mhd/raw data, please resample the images by running
python3 luna16_prepare_env_files.py
python3 luna16_prepare_images.py -c ./config/config_train_luna16_16g.jsonIf you downloaded DICOM data, please resample the images by running
python3 luna16_prepare_env_files.py
python3 luna16_prepare_images_dicom.py -c ./config/config_train_luna16_16g.jsonNote that for DICOM data, the data split json files are based on data downloaded with NBIA Data Retriever.
The resampled images will be with Nifti format. The number of resampled images will be less than the original images, this is because LUNA16 annotation includes only nodules >= 3 mm accepted by at least 3 out of 4 radiologists, and excluded the images without such nodules.
The LUNA16 dataset was split into 10-fold to run cross-fold training and inference.
Taking fold 0 as an example, run:
python3 luna16_training.py \
-e ./config/environment_luna16_fold0.json \
-c ./config/config_train_luna16_16g.jsonThis python script uses batch size and patch size defined in ./config/config_train_luna16_16g.json, which works for a 16G GPU. If you have a different GPU memory size, please change "batch_size", "patch_size", and "val_patch_size" to fit the GPU you use.
For fold i, please run
python3 luna16_training.py \
-e ./config/environment_luna16_fold${i}.json \
-c ./config/config_train_luna16_16g.jsonIf you are tuning hyper-parameters, please also add --verbose flag.
Details about matched anchors during training will be printed out.
For each fold, 95% of the training data is used for training, while the rest 5% is used for validation and model selection. The training and validation curves for 300 epochs of 10 folds are shown below. The upper row shows the training losses for box regression and classification. The bottom row shows the validation mAP and mAR for IoU ranging from 0.1 to 0.5.
With a single DGX1V 16G GPU, it took around 55 hours to train 300 epochs for each data fold.
If you have a different GPU memory size than 16G, please maximize "val_patch_size" in ./config/config_train_luna16_16g.json to fit the GPU you use.
For fold i, please run
python3 luna16_testing.py \
-e ./config/environment_luna16_fold${i}.json \
-c ./config/config_train_luna16_16g.jsonPlease download the official LUNA16 evaluation scripts from https://luna16.grand-challenge.org/Evaluation/, and save it as ./evaluation_luna16. Note that the official LUNA16 evaluation scripts are based on python2.
To run it with python3, please 1) copy the files in ./evaluationScript_py3_update to replace the files in downloaded scripts in ./evaluation_luna16; 2) then run the following command to convert python2 code to python3 code: python -m pip install future; futurize --stage1 -w evaluation_luna16/noduleCADEvaluationLUNA16.py; futurize --stage2 -w evaluation_luna16/noduleCADEvaluationLUNA16.py;
3) In ./evaluation_luna16/noduleCADEvaluationLUNA16.py, replace plt.xscale('log', basex=2) with plt.xscale('log', base=2) and plt.grid(b=True, which='both') with plt.grid(visible=True, which='both').
./evaluation_luna16/noduleCADEvaluationLUNA16.py will be the main python script to generate evaluation scores.
To run evaluation, please first make sure the 10 resulted json files result_luna16_fold{i}.json are in ./result folder. Then run:
./run_luna16_offical_eval.shThis bash script first combines the 10 result JSON files from 10 folds into one CSV file, then runs the official LUNA16 evaluation scripts saved in ./evaluation_luna16.
The evaluation scores will be stored in ./result/eval_luna16_scores
We got FROC result as shown in the table below. It is comparable with the result in nnDetection Table 2.
| Methods | 1/8 | 1/4 | 1/2 | 1 | 2 | 4 | 8 |
|---|---|---|---|---|---|---|---|
| Liu et al. (2019) | 0.848 | 0.876 | 0.905 | 0.933 | 0.943 | 0.957 | 0.970 |
| nnDetection (2021) | 0.812 | 0.885 | 0.927 | 0.950 | 0.969 | 0.979 | 0.985 |
| MONAI detection | 0.835 | 0.885 | 0.931 | 0.957 | 0.974 | 0.983 | 0.988 |
This MONAI example uses similar training and inference workflows as nnDetection LUNA16, with the same hyper-parameters and data augmentation pipeline.
The major differences are as follows:
- We use a different learning rate scheduler than nnDetection.
- We train 300 epochs (around 150k iterations), while nnDetection trains 125k iterations.
- During training, we run validation with 5% of the training data and save the best model, while nnDetection directly saves the model from the last iteration for inference.
- When the input image is too large to fit in the GPU memory, the inference is performed on patches. We do sliding window aggregation on the predicted class logits and box regression, while nnDetection uses a different aggregation strategy from Jaeger et al..
Other differences that may have a minor impact on the performance:
- We use different network architectures. We use RetinaNet with ResNet as a backbone, while nnDetection uses RetinaUnet with U-net as backbone.
- We directly apply the trained model to the images/patches during inference, while nnDetection uses mirror aggregation by applying the trained model to the images/patches flipped in three axes and averaging the flipped-back results.
- We read in boxes from data loader and do data augmentation on boxes. nnDetection, on the other hand, reads in image masks with ellipsoid as foreground and does data augmentation on masks, then convert the masks to boxes to train the network.
We greatly appreciate Michael Baumgartner, one of the main contributors of nnDetection project, for his vital cooperation and help in ensuring the successful completion of this MONAI detection module.