The news in our blog

Deep Learning Papers on Medical Image Analysis

Deep Learning Papers on Medical Image Analysis


To the best of our knowledge, this is the first list of deep learning papers on medical applications. There are couple of lists for deep learning papers in general, or computer vision, for example Awesome Deep Learning Papers. In this list, I try to classify the papers based on their deep learning techniques and learning methodology. I believe this list could be a good starting point for DL researchers on Medical Applications.


  1. A list of top deep learning papers published since 2015.
  2. Papers are collected from peer-reviewed journals and high reputed conferences. However, it may have recent papers on arXiv.
  3. A meta-data is required along with the paper, i.e. Deep Learning technique, Imaging Modality, Area of Interest, Clinical Database (DB).

List of Journals / Conferences (J/C):


Deep Learning Techniques:

  • NN: Neural Networks
  • MLP: Multilayer Perceptron
  • RBM: Restricted Boltzmann Machine
  • SAE: Stacked Auto-Encoders
  • CAE: Convolutional Auto-Encoders
  • CNN: Convolutional Neural Networks
  • RNN: Recurrent Neural Networks
  • LSTM: Long Short Term Memory
  • M-CNN: Multi-Scale/View/Stream CNN
  • MIL-CNN: Multi-instance Learning CNN
  • FCN: Fully Convolutional Networks

Imaging Modality:

  • US: Ultrasound
  • MR/MRI: Magnetic Resonance Imaging
  • PET: Positron Emission Tomography
  • MG: Mammography
  • CT: Computed Tompgraphy
  • H&E: Hematoxylin & Eosin Histology Images
  • RGB: Optical Images

Table of Contents

Deep Learning Techniques

Medical Applications

Deep Learning Techniques

Auto-Encoders/ Stacked Auto-Encoders

Convolutional Neural Networks

Recurrent Neural Networks

Generative Adversarial Networks

Medical Applications


Technique Modality Area Paper Title DB J/C Year
NN H&E N/A Deep learning of feature representation with multiple instance learning for medical image analysis [pdf] ICASSP 2014
M-CNN H&E Breast AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images [pdf] AMIDA IEEE-TMI 2016
FCN H&E N/A Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation pdf MICCAI 2017


Technique Modality Area Paper Title DB J/C Year
M-CNN CT Lung Multi-scale Convolutional Neural Networks for Lung Nodule Classification [pdf] LIDC-IDRI IPMI 2015
3D-CNN MRI Brain Predicting Alzheimer’s disease: a neuroimaging study with 3D convolutional neural networks [pdf] ADNI arXiv 2015
CNN+RNN RGB Eye Automatic Feature Learning to Grade Nuclear Cataracts Based on Deep Learning [pdf] IEEE-TBME 2015
CNN X-ray Knee Quantifying Radiographic Knee Osteoarthritis Severity using Deep Convolutional Neural Networks [pdf] O.E.1 arXiv 2016
CNN H&E Thyroid A Deep Semantic Mobile Application for Thyroid Cytopathology [pdf] SPIE 2016
3D-CNN, 3D-CAE MRI Brain Alzheimer’s Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network [pdf] ADNI arXiv 2016
M-CNN RGB Skin Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers [pdf] Dermofit MLMI 2016
CNN RGB Skin, Eye Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes [pdf] EDRADRD arXiv 2016
M-CNN CT Lung Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks [pdf] LIDC-IDRIANODE09DLCST IEEE-TMI 2016
3D-CNN CT Lung DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf] LIDC-IDRILUNA16 IEEE-WACV 2018
3D-CNN MRI Brain 3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients [pdf] MICCAI 2016
SAE US, CT Breast, Lung Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans [pdf] LIDC-IDRI Nature 2016
CAE MG Breast Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring [pdf] IEEE-TMI 2016
MIL-CNN MG Breast Deep multi-instance networks with sparse label assignment for whole mammogram classification [pdf] INbreast MICCAI 2017
GCN MRI Brain Spectral Graph Convolutions for Population-based Disease Prediction [pdf] ADNIABIDE arXiv 2017
CNN RGB Skin Dermatologist-level classification of skin cancer with deep neural networks Nature 2017
FCN + CNN MRI Liver-Liver Tumor SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf] ISBI 2017

Detection / Localization

Technique Modality Area Paper Title DB J/C Year
MLP CT Head-Neck 3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data [pdf] MICCAI 2015
CNN US Fetal Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks [pdf] IEEE-JBHI 2015
2.5D-CNN MRI Femur Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf] OAI ISBI 2015
LSTM US Fetal Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks [pdf] MICCAI 2015
CNN X-ray, MRI Hand Regressing Heatmaps for Multiple Landmark Localization using CNNs [pdf] DHADS MICCAI 2016
CNN MRI, US, CT - An artificial agent for anatomical landmark detection in medical images [pdf] SATCOM MICCAI 2016
FCN US Fetal Real-time Standard Scan Plane Detection and Localisation in Fetal Ultrasound using Fully Convolutional Neural Networks [pdf] MICCAI 2016
CNN+LSTM MRI Heart Recognizing end-diastole and end-systole frames via deep temporal regression network [pdf] MICCAI 2016
M-CNN MRI Heart Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation Neural Networks [pdf] IEEE-TMI 2016
CNN PET/CT Heart Automated detection of pulmonary nodules in PET/CT images: Ensemble false-positive reduction using a convolutional neural network technique Neural Networks [pdf] MP 2016
3D-CNN MRI Brain Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks [pdf] IEEE-TMI 2016
CNN X-ray, MG - Self-Transfer Learning for Fully Weakly Supervised Lesion Localization [pdf] NIH,ChinaDDSM,MIAS MICCAI 2016
CNN RGB Eye Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images [pdf] DRDMESSIDOR MICCAI 2016
GAN - - Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery IPMI 2017
FCN X-ray Cardiac CathNets: Detection and Single-View Depth Prediction of Catheter Electrodes MIAR 2016
3D-CNN CT Lung DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification [pdf] LIDC-IDRILUNA16 IEEE-WACV 2018


Technique Modality Area Paper Title DB J/C Year
U-Net - - U-net: Convolutional networks for biomedical image segmentation MICCAI 2015
FCN MRI Head-Neck Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation [pdf] arXiv 2016
FCN CT Liver-Liver Tumor Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields [pdf] MICCAI 2016
3D-CNN MRI Spine Model-Based Segmentation of Vertebral Bodies from MR Images with 3D CNNs MICCAI 2016
FCN CT Liver-Liver Tumor Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks [pdf] arXiv 2017
FCN MRI Liver-Liver Tumor SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D convolutional neural networks [pdf] ISBI 2017
3D-CNN Diffusion MRI Brain q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.2) IEEE-TMI 2016
GAN MG Breast Mass Adversarial Deep Structured Nets for Mass Segmentation from Mammograms [pdf] INbreastDDSM-BCRP ISBI 2018
3D-CNN CT Liver 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes pdf MICCAI 2017
3D-CNN MRI Brain Unsupervised domain adaptation in brain lesion segmentation with adversarial networks pdf IPMI 2017


Technique Modality Area Paper Title DB J/C Year
3D-CNN CT Spine An Artificial Agent for Robust Image Registration [pdf] 2016


Technique Modality Area Paper Title DB J/C Year
2.5D-CNN MRI Automated anatomical landmark detection ondistal femur surface using convolutional neural network [pdf] OAI ISBI 2015
3D-CNN Diffusion MRI Brain q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI [pdf] (Section II.B.1) [HCP]and other IEEE-TMI 2016

Image Reconstruction and Post Processing

Technique Modality Area Paper Title DB J/C Year
CNN CS-MRI A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction pdf IEEE-TMI 2017
GAN CS-MRI Deep Generative Adversarial Networks for Compressed Sensing Automates MRI pdf NIPS 2017

Other tasks


A survey on deep learning in medical image analysis

Author links open overlay panelGeertLitjensThijsKooiBabak EhteshamiBejnordiArnaud Arindra AdiyosoSetioFrancescoCiompiMohsenGhafoorianJeroen A.W.M.van der LaakBramvan GinnekenClara I.Sánchez

Medical Image Analysis with Deep Learning 


Medical Image Analysis with Deep Learning

In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data.

By Taposh Roy, Kaiser Permanente.

Analyzing images and videos, and using them in various applications such as self driven cars, drones etc. with underlying deep learning techniques has been the new research frontier. The recent research papers such as “A Neural Algorithm of Artistic Style”, show how a styles can be transferred from an artist and applied to an image, to create a new image. Other papers such as “Generative Adversarial Networks” (GAN) and “Wasserstein GAN” have paved the path to develop models that can learn to create data that is similar to data that we give them. Thus opening up the world to semi-supervised learning and paving the path to a future of unsupervised learning.

While these research areas are still on the generic images, our goal is to use these research into medical images to help healthcare. We need to start with some basics. In this article, I start with basics of image processing, basics of medical image format data and visualize some medical data. In the next article I will deep dive into some convolutional neural nets and use them with Keras for predicting lung cancer.

Basic Image Processing (using python)

There are a variety of image processing libraries, however OpenCV (open computer vision) has become mainstream due to its large community support and availability in C++, java and python. I prefer using opencv using jupyter notebook.

Install OpenCV using: pip install opencv-python or install directly from the source from

Installing opencv.
Now open your Jupyter notebook and confirm you can import cv2. You will also need numpy and matplotlib to view your plots inside the notebook.

Now, lets check if you can open an image and view it on your notebook using the code below.

Example image load through OpenCV.
Basic Face Detection

Lets, do something fun such as detecting a face. To detect face we will use an open source xml stump-based 20×20 gentle adaboost frontal face detector originally created by Rainer Lienhart. A good post with details on Haar-cascade detection is here.

Face detection using OpenCV.
There are a lot of examples for image processing using opencv in the docs section. I leave it up to the reader to play with more examples. Now that we know the basics of image processing, lets move to the next level of understanding medical image format.

Medical Image Data Format

Medical images follow Digital Imaging and Communications (DICOM) as a standard solution for storing and exchanging medical image-data. The first version of this standard was released in 1985. Since then there are several changes made. This standard uses a file format and a communications protocol.

  • File Format — All patient medical images are saved in the DICOM file format. This format has PHI (protected health information) about the patient such as — name, sex, age in addition to other image related data such as equipment used to capture the image and some context to the medical treatment. Medical Imaging Equipments create DICOM files. Doctors use DICOM Viewers, computer software applications that can display DICOM images, read and to diagnose the findings in the images.
  • Communications Protocol — The DICOM communication protocol is used to search for imaging studies in the archive and restore imaging studies to the workstation in order to display it. All medical imaging applications that are connected to the hospital network use the DICOM protocol to exchange information, mainly DICOM images but also patient and procedure information. There are also more advanced network commands that are used to control and follow the treatment, schedule procedures, report statuses and share the workload between doctors and imaging devices.

A very good blog that goes into details of the DICOM standard is here

Analyze DICOM Images

A very good python package used for analyzing DICOM images is pydicom. In this section, we will see how to render a DICOM image on a Jupyter notebook.

Install OpenCV using: pip install pydicom

After you install pydicom package, go back to the jupyter notebook. In the notebook, import the dicom package and other packages as shown below.

We also use other packages such as pandas, scipy, skimage, mpl_toolkit for data processing and analysis.

There’s a wealth of freely available DICOM datasets online but here’s a few that should help you get started:

Download the dicom files and load them on your jupyter notebook.

Now, load the DICOM images into a list.

Step 1 : Basic Viewing of DICOM Image in Jupyter

In the first line we load the 1st DICOM file, which we’re gonna use as a reference named RefDs, to extract metadata and whose filename is first in the lstFilesDCM list.

We then calculate the total dimensions of the 3D NumPy array which are equal to (Number of pixel rows in a slice) x (Number of pixel columns in a slice) x (Number of slices) along the x, y, and z cartesian axes. Lastly, we use the PixelSpacing and SliceThickness attributes to calculate the spacing between pixels in the three axes. We store the array dimensions in ConstPixelDims and the spacing in ConstPixelSpacing [1].

Step 2: Looking into details of DICOM format

The unit of measurement in CT scans is the Hounsfield Unit (HU), which is a measure of radiodensity. CT scanners are carefully calibrated to accurately measure this. A detailed understanding on this can be found here.

Each pixel is assigned a numerical value (CT number), which is the average of all the attenuation values contained within the corresponding voxel. This number is compared to the attenuation value of water and displayed on a scale of arbitrary units named Hounsfield units (HU) after Sir Godfrey Hounsfield.

This scale assigns water as an attenuation value (HU) of zero. The range of CT numbers is 2000 HU wide although some modern scanners have a greater range of HU up to 4000. Each number represents a shade of grey with +1000 (white) and –1000 (black) at either end of the spectrum.

Hounsfield Scale [credits: “Introduction to CT physics” (PDF).]
Some scanners have cylindrical scanning bounds, but the output image is square. The pixels that fall outside of these bounds get the fixed value -2000.

CT Scanner Image [credits : “Introduction to CT physics” (PDF).]
The first step usually is setting these values to 0. Next, let’s go back to HU units, by multiplying with the rescale slope and adding the intercept (which are conveniently stored in the metadata of the scans!).

In the next part, we will use Kaggle’s lung cancer data-set and Convolution Neural Nets using Keras. We will build upon the information provided by this article to go to the next one.


  8. Kaggle community for all the different scripts and support

Bio: Taposh Roy leads innovation team in Kaiser Permanente’s Decision Support group. He works with research, technology and business leaders to derive insights from data.

Original. Reposted with permission.


Deep Learning Applications in Medical Imaging

Deep Learning Applications in Medical Imaging