Perceptual real-time 2D-to-3D conversion using cue fusion
Graphics Interface 2016 / IEEE Transactions on Visualization and Computer Graphics

Perceptual real-time 2D-to-3D conversion using cue fusion

Thomas Leimkühler¹     Petr Kellnhofer²     Tobias Ritschel³     Karol Myszkowski¹     Hans-Peter Seidel<sup>1    

1</sup> MPI Informatik     ² MIT CSAIL     ³ University College London    

Stereo images produced from mono images by our automatic real-time 2D-to-3D conversion.

Abstract

We propose a system to infer binocular disparity from a monocular video stream in real-time. Different from classic reconstruction of physical depth in computer vision, we compute perceptually plausible disparity, that is numerically inaccurate, but results in a very similar overall depth impression with plausible overall layout, sharp edges, fine details and agreement between luminance and disparity. We use several simple monocular cues to estimate disparity maps and confidence maps of low spatial and temporal resolution in real-time. These are complemented by spatially-varying, appearance-dependent and class-specific disparity prior maps, learned from example stereo images. Scene classification selects this prior at runtime. Fusion of prior and cues is done by means of robust MAP inference on a dense spatio-temporal conditional random field with high spatial and temporal resolution. Using normal distributions allows this in constant-time, parallel per-pixel work. We compare our approach to previous 2D-to-3D conversion systems in terms of different metrics, as well as a user study.

Video

Materials

© 2016/2017 The Authors. This is the authors' version of the work. It is posted here for personal use, not for redistribution.

Citation

Thomas Leimkühler, Petr Kellnhofer, Tobias Ritschel, Karol Myszkowski, Hans-Peter Seidel
Perceptual real-time 2D-to-3D conversion using cue fusion
IEEE Transactions on Visualization and Computer Graphics

@article{Leimkuehler2018TVCG,
  author = {Thomas Leimk\"uhler and Petr Kellnhofer and Tobias Ritschel and Karol Myszkowski and Hans-Peter Seidel},
  title = {Perceptual real-time 2{D}-to-3{D} conversion using cue fusion},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year={2018},
  month={June},
  volume={24},
  number={6},
  pages={2037-2050},
  doi = {10.1109/TVCG.2017.2703612}
}

Thomas Leimkühler, Petr Kellnhofer, Tobias Ritschel, Karol Myszkowski, Hans-Peter Seidel
Perceptual real-time 2D-to-3D conversion using cue fusion
Proc. Graphics Interface, Victoria/Canada, June 2016

@inproceedings{Leimkuehler2016GI,
  author = {Thomas Leimk\"uhler and Petr Kellnhofer and Tobias Ritschel and Karol Myszkowski and Hans-Peter Seidel},
  title = {Perceptual real-time 2{D}-to-3{D} conversion using cue fusion},
  booktitle = {Proc. Graphics Interface},
  year = {2016},
}

Acknowledgements

We would like to thank Adam Laskowski, Dushyant Mehta, Elena Arabadzhiyska, Krzysztof Templin, and Waqar Khan.