Deep Learning Papers on Medical Image Analysis
To the best of our knowledge, this is the first list of deep learning papers on medical applications. There are couple of lists for deep learning papers in general, or computer vision, for example Awesome Deep Learning Papers. In this list, I try to classify the papers based on their deep learning techniques and learning methodology. I believe this list could be a good starting point for DL researchers on Medical Applications.
- A list of top deep learning papers published since 2015.
- Papers are collected from peer-reviewed journals and high reputed conferences. However, it may have recent papers on arXiv.
- A meta-data is required along with the paper, i.e. Deep Learning technique, Imaging Modality, Area of Interest, Clinical Database (DB).
List of Journals / Conferences (J/C):
- Medical Image Analysis (MedIA)
- IEEE Transaction on Medical Imaging (IEEE-TMI)
- IEEE Transaction on Biomedical Engineering (IEEE-TBME)
- IEEE Journal of Biomedical and Health Informatics (IEEE-JBHI)
- International Journal on Computer Assisted Radiology and Surgery (IJCARS)
- International Conference on Information Processing in Medical Imaging (IPMI)
- International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)
- International Conference on Information Processing in Computer-Assisted Interventions (IPCAI)
- IEEE International Symposium on Biomedical Imaging (ISBI)
Deep Learning Techniques:
- NN: Neural Networks
- MLP: Multilayer Perceptron
- RBM: Restricted Boltzmann Machine
- SAE: Stacked Auto-Encoders
- CAE: Convolutional Auto-Encoders
- CNN: Convolutional Neural Networks
- RNN: Recurrent Neural Networks
- LSTM: Long Short Term Memory
- M-CNN: Multi-Scale/View/Stream CNN
- MIL-CNN: Multi-instance Learning CNN
- FCN: Fully Convolutional Networks
- US: Ultrasound
- MR/MRI: Magnetic Resonance Imaging
- PET: Positron Emission Tomography
- MG: Mammography
- CT: Computed Tompgraphy
- H&E: Hematoxylin & Eosin Histology Images
- RGB: Optical Images
- AutoEncoders/ Stacked AutoEncoders
- Convolutional Neural Networks
- Recurrent Neural Networks
- Generative Adversarial Networks
- Detection/ Localization
- Image Reconstruction and Post-Processing
- Other tasks
- AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
- Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images
|NN||H&E||N/A||Deep learning of feature representation with multiple instance learning for medical image analysis [pdf]||ICASSP||2014|
|M-CNN||H&E||Breast||AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images [pdf]||AMIDA||IEEE-TMI||2016|
|FCN||H&E||N/A||Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation pdf||MICCAI||2017|
|M-CNN||CT||Lung||Multi-scale Convolutional Neural Networks for Lung Nodule Classification [pdf]||LIDC-IDRI||IPMI||2015|
|3D-CNN||MRI||Brain||Predicting Alzheimer’s disease: a neuroimaging study with 3D convolutional neural networks [pdf]||ADNI||arXiv||2015|
|CNN+RNN||RGB||Eye||Automatic Feature Learning to Grade Nuclear Cataracts Based on Deep Learning [pdf]||IEEE-TBME||2015|
|CNN||X-ray||Knee||Quantifying Radiographic Knee Osteoarthritis Severity using Deep Convolutional Neural Networks [pdf]||O.E.1||arXiv||2016|
|CNN||H&E||Thyroid||A Deep Semantic Mobile Application for Thyroid Cytopathology [pdf]||SPIE||2016|
|3D-CNN, 3D-CAE||MRI||Brain||Alzheimer’s Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network [pdf]||ADNI||arXiv||2016|
|M-CNN||RGB||Skin||Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers [pdf]||Dermofit||MLMI||2016|
|CNN||RGB||Skin, Eye||Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes [pdf]||EDRA, DRD||arXiv||2016|
|M-CNN||CT||Lung||Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks [pdf]||LIDC-IDRI, ANODE09, DLCST||IEEE-TMI||2016|
|3D-CNN||CT||Lung||DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf]||LIDC-IDRI, LUNA16||IEEE-WACV||2018|
|3D-CNN||MRI||Brain||3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients [pdf]||MICCAI||2016|
|SAE||US, CT||Breast, Lung||Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans [pdf]||LIDC-IDRI||Nature||2016|
|CAE||MG||Breast||Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring [pdf]||IEEE-TMI||2016|
|MIL-CNN||MG||Breast||Deep multi-instance networks with sparse label assignment for whole mammogram classification [pdf]||INbreast||MICCAI||2017|
|GCN||MRI||Brain||Spectral Graph Convolutions for Population-based Disease Prediction [pdf]||ADNI, ABIDE||arXiv||2017|
|CNN||RGB||Skin||Dermatologist-level classification of skin cancer with deep neural networks||Nature||2017|
|FCN + CNN||MRI||Liver-Liver Tumor||SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf]||ISBI||2017|
|MLP||CT||Head-Neck||3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data [pdf]||MICCAI||2015|
|CNN||US||Fetal||Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks [pdf]||IEEE-JBHI||2015|
|2.5D-CNN||MRI||Femur||Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf]||OAI||ISBI||2015|
|LSTM||US||Fetal||Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks [pdf]||MICCAI||2015|
|CNN||X-ray, MRI||Hand||Regressing Heatmaps for Multiple Landmark Localization using CNNs [pdf]||DHADS||MICCAI||2016|
|CNN||MRI, US, CT||-||An artificial agent for anatomical landmark detection in medical images [pdf]||SATCOM||MICCAI||2016|
|FCN||US||Fetal||Real-time Standard Scan Plane Detection and Localisation in Fetal Ultrasound using Fully Convolutional Neural Networks [pdf]||MICCAI||2016|
|CNN+LSTM||MRI||Heart||Recognizing end-diastole and end-systole frames via deep temporal regression network [pdf]||MICCAI||2016|
|M-CNN||MRI||Heart||Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation Neural Networks [pdf]||IEEE-TMI||2016|
|CNN||PET/CT||Heart||Automated detection of pulmonary nodules in PET/CT images: Ensemble false-positive reduction using a convolutional neural network technique Neural Networks [pdf]||MP||2016|
|3D-CNN||MRI||Brain||Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks [pdf]||IEEE-TMI||2016|
|CNN||X-ray, MG||-||Self-Transfer Learning for Fully Weakly Supervised Lesion Localization [pdf]||NIH,China, DDSM,MIAS||MICCAI||2016|
|CNN||RGB||Eye||Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images [pdf]||DRD, MESSIDOR||MICCAI||2016|
|GAN||-||-||Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery||IPMI||2017|
|FCN||X-ray||Cardiac||CathNets: Detection and Single-View Depth Prediction of Catheter Electrodes||MIAR||2016|
|3D-CNN||CT||Lung||DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf]||LIDC-IDRI, LUNA16||IEEE-WACV||2018|
|U-Net||-||-||U-net: Convolutional networks for biomedical image segmentation||MICCAI||2015|
|FCN||MRI||Head-Neck||Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation [pdf]||arXiv||2016|
|FCN||CT||Liver-Liver Tumor||Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields [pdf]||MICCAI||2016|
|3D-CNN||MRI||Spine||Model-Based Segmentation of Vertebral Bodies from MR Images with 3D CNNs||MICCAI||2016|
|FCN||CT||Liver-Liver Tumor||Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks [pdf]||arXiv||2017|
|FCN||MRI||Liver-Liver Tumor||SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf]||ISBI||2017|
|3D-CNN||Diffusion MRI||Brain||q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.2)||IEEE-TMI||2016|
|GAN||MG||Breast Mass||Adversarial Deep Structured Nets for Mass Segmentation from Mammograms [pdf]||INbreast, DDSM-BCRP||ISBI||2018|
|3D-CNN||CT||Liver||3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes pdf||MICCAI||2017|
|3D-CNN||MRI||Brain||Unsupervised domain adaptation in brain lesion segmentation with adversarial networks pdf||IPMI||2017|
|3D-CNN||CT||Spine||An Artificial Agent for Robust Image Registration [pdf]||2016|
|2.5D-CNN||MRI||Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf]||OAI||ISBI||2015|
|3D-CNN||Diffusion MRI||Brain||q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.1)||[HCP]and other||IEEE-TMI||2016|
|CNN||CS-MRI||A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction pdf||IEEE-TMI||2017|
|GAN||CS-MRI||Deep Generative Adversarial Networks for Compressed Sensing Automates MRI pdf||NIPS||2017|
Deep Learning for Medical Image Processing: Overview, Challenges and Future
Muhammad Imran Razzak, Saeeda Naz and Ahmad Zaib
Medical Image Analysis with Deep Learning
In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data.
By Taposh Roy, Kaiser Permanente.
Analyzing images and videos, and using them in various applications such as self driven cars, drones etc. with underlying deep learning techniques has been the new research frontier. The recent research papers such as “A Neural Algorithm of Artistic Style”, show how a styles can be transferred from an artist and applied to an image, to create a new image. Other papers such as “Generative Adversarial Networks” (GAN) and “Wasserstein GAN” have paved the path to develop models that can learn to create data that is similar to data that we give them. Thus opening up the world to semi-supervised learning and paving the path to a future of unsupervised learning.
While these research areas are still on the generic images, our goal is to use these research into medical images to help healthcare. We need to start with some basics. In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data. In the next article I will deep dive into some convolutional neural nets and use them with Keras for predicting lung cancer.
Basic Image Processing (using python)
There are a variety of image processing libraries, however OpenCV (open computer vision) has become mainstream due to its large community support and availability in C++, java and python. I prefer using opencv using jupyter notebook.
Install OpenCV using: pip install opencv-python or install directly from the source from opencv.org
Now, lets check if you can open an image and view it on your notebook using the code below.
Example image load through OpenCV.
Lets, do something fun such as detecting a face. To detect face we will use an open source xml stump-based 20×20 gentle adaboost frontal face detector originally created by Rainer Lienhart. A good post with details on Haar-cascade detection is here.
Face detection using OpenCV.
Medical Image Data Format
Medical images follow Digital Imaging and Communications (DICOM) as a standard solution for storing and exchanging medical image-data. The first version of this standard was released in 1985. Since then there are several changes made. This standard uses a file format and a communications protocol.
- File Format — All patient medical images are saved in the DICOM file format. This format has PHI (protected health information) about the patient such as — name, sex, age in addition to other image related data such as equipment used to capture the image and some context to the medical treatment. Medical Imaging Equipments create DICOM files. Doctors use DICOM Viewers, computer software applications that can display DICOM images, read and to diagnose the findings in the images.
- Communications Protocol — The DICOM communication protocol is used to search for imaging studies in the archive and restore imaging studies to the workstation in order to display it. All medical imaging applications that are connected to the hospital network use the DICOM protocol to exchange information, mainly DICOM images but also patient and procedure information. There are also more advanced network commands that are used to control and follow the treatment, schedule procedures, report statuses and share the workload between doctors and imaging devices.
A very good blog that goes into details of the DICOM standard is here
Analyze DICOM Images
A very good python package used for analyzing DICOM images is pydicom. In this section, we will see how to render a DICOM image on a Jupyter notebook.
Install OpenCV using: pip install pydicom
After you install pydicom package, go back to the jupyter notebook. In the notebook, import the dicom package and other packages as shown below.
We also use other packages such as pandas, scipy, skimage, mpl_toolkit for data processing and analysis.
There’s a wealth of freely available DICOM datasets online but here’s a few that should help you get started:
- Kaggle Competitions and Datasets: This is my personal favorite. Check out the data for lung cancer competition and diabetes retinopathy.
- Dicom Library : DICOM Library is a free online medical DICOM image or video file sharing service for educational and scientific purposes.
- Osirix Datasets: Provides a large range of human datasets acquired through a variety of imaging modalities.
- Visible Human Datasets: Parts of the Visible Human project are somehow freely distributed here which is weird cause getting that data is neither free nor hassle-free.
- The Zubal Phantom: This website offers multiple datasets of two human males in CT and MRI which are freely distributed.
Download the dicom files and load them on your jupyter notebook.
Now, load the DICOM images into a list.
Step 1 : Basic Viewing of DICOM Image in Jupyter
In the first line we load the 1st DICOM file, which we’re gonna use as a reference named
RefDs, to extract metadata and whose filename is first in the
We then calculate the total dimensions of the 3D NumPy array which are equal to (Number of pixel rows in a slice) x (Number of pixel columns in a slice) x (Number of slices) along the x, y, and z cartesian axes. Lastly, we use the
SliceThickness attributes to calculate the spacing between pixels in the three axes. We store the array dimensions in
ConstPixelDims and the spacing in
Step 2: Looking into details of DICOM format
The unit of measurement in CT scans is the Hounsfield Unit (HU), which is a measure of radiodensity. CT scanners are carefully calibrated to accurately measure this. A detailed understanding on this can be found here.
Each pixel is assigned a numerical value (CT number), which is the average of all the attenuation values contained within the corresponding voxel. This number is compared to the attenuation value of water and displayed on a scale of arbitrary units named Hounsfield units (HU) after Sir Godfrey Hounsfield.
This scale assigns water as an attenuation value (HU) of zero. The range of CT numbers is 2000 HU wide although some modern scanners have a greater range of HU up to 4000. Each number represents a shade of grey with +1000 (white) and –1000 (black) at either end of the spectrum.
Hounsfield Scale [credits: “Introduction to CT physics” (PDF). elsevierhealth.com.]
CT Scanner Image [credits : “Introduction to CT physics” (PDF). elsevierhealth.com.]
In the next part, we will use Kaggle’s lung cancer data-set and Convolution Neural Nets using Keras. We will build upon the information provided by this article to go to the next one.
- Kaggle community for all the different scripts and support
Bio: Taposh Roy leads innovation team in Kaiser Permanente’s Decision Support group. He works with research, technology and business leaders to derive insights from data.
Original. Reposted with permission.