Deep Reinforcement Learning for Detection of Abnormal Anatomies


  • Paula López Diez Technical University of Denmark
  • Kristine Aavild Juhl Technical University of Denmark
  • Josefine Vilsbøll Sundgaard Technical University of Denmark
  • Hassan Diab The National Medical Research Center for Otorhinolaryngology of the Federal Medico-Biological Agency of Russia
  • Jan Margeta KardioMe
  • François Patou Oticon Medical
  • Rasmus R. Paulsen Technical University of Denmark



deep reinforcement learning, landmarks, anomaly detection, medical image, inner ear, PCA, Procrustes, C-MARL, CT


Automatic detection of abnormal anatomies or malformations of different structures of the human body is a challenging task that could provide support for clinicians in their daily practice. Compared to normative anatomies, there is a low presence of anatomical abnormalities in patients, and the great variation within malformations make it challenging to design deep learning frameworks for automatic detection. We propose a framework for anatomical abnormality detection, which benefits from using a deep reinforcement learning model for landmark detection trained in normative data. We detect the abnormalities using the variability between the predicted landmarks configurations in a subspace based on a point distribution model of landmarks using Procrustes shape alignment and principal component analysis projection from normative data. We demonstrate the performance of this implementation on clinical CT scans of the inner ear, and show how synthetically created abnormal cochlea anatomy can be detected using the prediction of five landmarks around the cochlea. Our approach shows a Receiver Operating Characteristics (ROC) Area Under The Curve (AUC) of 0.97, and 96% accuracy for the detection of abnormal anatomy on synthetic data.


J. C. Gower. Generalized procrustes analysis. Psychometrika, 40(1):33–51, 1975. doi: 10.1007/bf02291478.

C. Baur, B. Wiestler, M. Muehlau, C. Zimmer, N. Navab, and S. Albarqouni. Modeling healthy anatomy with artificial intelligence for unsupervised anomaly detection in brain mri. Radiology: Artificial Intelligence, 3(3):e190169, 2021. doi: 10.1148/ ryai.2021190169. URL

L. Ben Amor, I. Lahyani, and M. Jmaiel. PCA-based multivariate anomaly detection in mobile healthcare applications. In Proc. International Symposium on Distributed Simulation and Real Time Applications (DS-RT), pages 1–8, 2017. doi: 10.1109/DISTRA.2017. 8167682.

R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. 1 2019. URL

T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active shape models-their training and application. Computer vision and image understanding, 61(1):38–59, 1995. doi:

F. C. Ghesu, B. Georgescu, S. Grbic, A. K. Maier, J. Hornegger, and D. Comaniciu. Robust multi-scale anatomical landmark detection in incomplete 3d-CT data. In Proc. MICCAI, pages 194–202, 2017. ISBN 978-3-31966182-7. doi: 10.1007/978-3-319-66182-7 23.

R. S. Gill, S.-J. Hong, F. Fadaie, B. Caldairou, B. C. Bernhardt, C. Barba, A. Brandt, V. C. Coelho, L. d’Incerti, M. Lenge, M. Semmelroch, F. Bartolomei, F. Cendes, F. Deleo, R. Guerrini, M. Guye, G. Jackson, A. SchulzeBonhage, T. Mansi, N. Bernasconi, and A. Bernasconi. Deep convolutional networks for automated detection of epileptogenic brain malformations. In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-L´opez, and G. Fichtinger, editors, Proc. MICCAI, pages 490–497. Springer, 2018. ISBN 978-3-030-00931-1. doi: 10.1007/978-3-030-00931-1 56.

V. A. Krenn, C. Fornai, N. M. Webb, M. A. Woodert, H. Prosch, and M. Haeusler. The morphological consequences of segmentation anomalies in the human sacrum. American Journal of Biological Anthropology, 12 2021. ISSN 2692-7691. doi: 10.1002/ ajpa.24466. URL

G. Leroy, D. Rueckert, and A. Alansary. Communicative reinforcement learning agents for landmark detection in brain images. In Machine Learning in Clinical Neuroimaging and Radiogenomics in Neuro-oncology, pages 177– 186. Springer, 2020. ISBN 978-3-030-66843-3. doi: 10.1007/978-3-030-66843-3 18.

P. Lopez Diez, J. V. Sundgaard, F. Patou, J. Margeta, and R. R. Paulsen. Facial and cochlear nerves characterization using deep reinforcement learning for landmark detection. In Proc. MICCAI, pages 519–528. Springer, 2021. ISBN 978-3-030-87202-1. doi: 10.1007/ 978-3-030-87202-1 50.

P. Seebock, J. I. Orlando, T. Schlegl, S. M. Waldstein, H. Bogunovic, S. Klimscha, G. Langs, and U. Schmidt-Erfurth. Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct. IEEE Transactions on Medical Imaging, 39(1): 87–98, 2020. doi: 10.1109/TMI.2019.2919951.

L. Sennaroglu and M. D. Bajin. Classification and current management of inner ear malformations. Balkan Medical Journal, 34, 08 2017. doi: 10.4274/balkanmedj.2017.0367.

A. Vlontzos, A. Alansary, K. Kamnitsas, D. Rueckert, and B. Kainz. Multiple Landmark Detection Using Multi-agent Reinforcement Learning. In Proc. MICCAI. Springer, 2019. doi: 10.1007/978-3-030-32251-9 29.

P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho, J. C. Gee, and G. Gerig. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage, 31(3):1116–1128, 2006. doi: 10.1016/j.neuroimage.2006.01.015.