max planck institut informatik
mpii logo Minerva of the Max Planck Society

Perceptual real-time 2D-to-3D conversion using cue fusion, Graphics Interface 2016


Perceptual real-time 2D-to-3D conversion using cue fusion

Thomas Leimk├╝hler1     Petr Kellnhofer1     Tobias Ritschel2     Karol Myszkowski1     Hans-Peter Seidel1    

1 MPI Informatik     2 University College London    


Stereo images produced from mono images by our automatic real-time 2D-to-3D conversion.

Abstract

We propose a system to infer binocular disparity from a monocular video stream in real-time. Different from classic reconstruction of physical depth in computer vision, we compute perceptually plausible disparity, that is numerically inaccurate, but results in a very similar overall depth impression with plausible overall layout, sharp edges, fine details and agreement between luminance and disparity. We use several simple monocular cues to estimate disparity maps and confidence maps of low spatial and temporal resolution in real-time. These are complemented by spatially-varying, appearance-dependent and class-specific disparity prior maps, learned from example stereo images. Scene classification selects this prior at runtime. Fusion of prior and cues is done by means of robust MAP inference on a dense spatio-temporal conditional random field with high spatial and temporal resolution. Using normal distributions allows this in constant-time, parallel per-pixel work. We compare our approach to previous 2D-to-3D conversion systems in terms of different metrics, as well as a user study.

Materials

Paper (Full Author's Copy) (16 MB)
Video (67 MB)
Slides (10 MB)
Supplemental Data (1 MB)
Results & Stimuli Gallery
Prior Gallery

© 2016 The Authors. This is the authors' version of the work. It is posted here for personal use, not for redistribution.

Citation

Thomas Leimk├╝hler, Petr Kellnhofer, Tobias Ritschel, Karol Myszkowski, Hans-Peter Seidel
Perceptual real-time 2D-to-3D conversion using cue fusion
Proc. Graphics Interface, Victoria/Canada, June 2016

@inproceedings{Leimkuehler2016,
  author = {Thomas Leimk\"uhler and Petr Kellnhofer and Tobias Ritschel and Karol Myszkowski and Hans-Peter Seidel},
  title = {Perceptual real-time 2{D}-to-3{D} conversion using cue fusion},
  booktitle = {Proc. Graphics Interface},
  year = {2016},
}

Acknowledgements

We would like to thank Adam Laskowski, Dushyant Mehta, Elena Arabadzhiyska, Krzysztof Templin, and Waqar Khan.