max planck institut informatik
mpii logo Minerva of the Max Planck Society

Proxy-guided Image-based Rendering for Mobile Devices, Pacific Graphics 2016


Proxy-guided Image-based Rendering for Mobile Devices

Bernhard Reinert1     Johannes Kopf2, 4     Tobias Ritschel3     Edurado Cuervo2     David Chu2, 5     Hans-Peter Seidel1

MPI Informatik1, Microsoft Research2, University College London3, now at Facebook4, now at Google5


Our new image-based rendering algorithm warps original views (with depth, not shown) to produce novel views. It performs as fast as the fastest competitors but maintains much higher visual quality for larger displacements. The insets compare against other common approaches and show the RGB-DSSIM difference. Frame rates are computed on an an Intel Compute Stick at a resolution of 1280x720.


Abstract

VR headsets and hand-held devices are not powerful enough to render complex scenes in real-time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image-based rendering (IBR) architecture for masking the latency. It runs in real-time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server-side dual-view representation that leverages an optimally-placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide-angle projection with favorable directional parameterization. A new client-side IBR algorithm uses a pre-transmitted level-of-detail proxy with an encaging simplification and depth-carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR/mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.

Video

Downloads

Paper (49 MB), Compressed (4.3 MB)
Supplemental (12.1 MB), Compressed (0.1 MB)
Video (432 MB)
Supplemental material website (videos, statistics, etc.)
Pacific Graphics 2016 presentation slides (412 MB).

Citation

Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, Hans-Peter Seidel
Proxy-guided Image-based Rendering for Mobile Devices
Computer Graphics Forum 2016 (Proceedings of Pacific Graphics 2016).

@article{Reinert:2016:ProxyIBR,
  author = { 
	Bernhard Reinert and
	Johannes Kopf and 
	Tobias Ritschel and
	Eduardo Cuervo and
	David Chu and
	Hans-Peter Seidel},
  title = {{Proxy-guided Image-based Rendering for Mobile Devices}},
  journal = {Computer Graphics Forum},
  year = {2016},
  publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
  ISSN = {1467-8659}
}

Acknowledgements

We would like to thank the anonymous reviewers for helpful comments.