LayeredCNN: Segmenting Layers with Autoregressive Models




Segmentation, Deep Learning, Autoregressive, Convolutional Neural Network


We address a subclass of segmentation problems where the labels of the image are structured in layers. We propose applying autoregressive CNNs which, when given an image and a partial segmentation of layers, complete the segmentation. Initializing the model with a user-provided partial segmentation allows for choosing which layers the model should segment. Alternatively, the model can produce an automatic initialization, albeit with some performance loss. The model is trained exclusively on synthetic data from our data generation algorithm. It yields impressive performance on the synthetic data and generalizes to real data it has never seen.

Author Biographies

Jakob L. Christensen, Technical University of Denmark (DTU)

Undergraduate student

Patrick Møller Jensen, Technical University of Denmark (DTU)

PhD Student at DTU Compute

Morten Rieger Hannemose, Technical University of Denmark (DTU)

Postdoc at DTU, Assistant Professor, Department of Applied Mathematics and Computer Science Visual Computing

Anders Bjorholm Dahl, Technical University of Denmark (DTU)

Associate professor, head of section Professor MSO, Head of Section, Department of Applied Mathematics and Computer Science Visual Computing

Vedrana Andersen Dahl, Technical University of Denmark (DTU)

Associate professor at DTU Compute


J. Beutel, H. L. Kundel, and R. L. Van Metter. Handbook of medical imaging, volume 1. Spie Press, 2000.

Dong-Chen He and Abdelmounaime Safia. Brodatz’s texture database.

T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.

L. Hubert and P. Arabie. Comparing partitions. Journal of classification, 2(1):193–218, 1985.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.

N. Jeppesen, A. N. Christensen, V. A. Dahl, and A. B. Dahl. Sparse layered graphs for multi-object segmentation. CVPR, 2020.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.

H. A. Leopold, J. Orchard, J. S. Zelek, and V. Lakshminarayanan. Pixelbnn: Augmenting the pixelcnn with batch normalization and the presentation of a fast architecture for retinal vessel segmentation. Journal of Imaging, 5, 2019.

K. Li, X. Wu, D. Chen, and M. Sonka. Optimal surface segmentation in volumetric images - a graph-theoretic approach. PAMI, 28(1):119–134, 2006.

Paul Mooney. Retinal OCT Images (optical coherence tomography). Kaggle dataset, 2018.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.

T. Salimans, A. Karpathy, X. Chen, and D. P.Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. ICLR, 2017.

A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural net-works. ICML, 2016.

A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelcnn decoders. NIPS, 2016.

X. Wu and D. Z. Chen. Optimal net surface problems with applications. ICALP, 2002.