Proxy-guided Image-based Rendering for Mobile Devices, Pacific Graphics 2016

Proxy-guided Image-based Rendering for Mobile Devices

Bernhard Reinert¹     Johannes Kopf^{2, 4}     Tobias Ritschel³     Edurado Cuervo²     David Chu^{2, 5}     Hans-Peter Seidel¹

MPI Informatik¹, Microsoft Research², University College London³, now at Facebook⁴, now at Google⁵

Our new image-based rendering algorithm warps original views (with depth, not shown) to produce novel views. It performs as fast as the fastest competitors but maintains much higher visual quality for larger displacements. The insets compare against other common approaches and show the RGB-DSSIM difference. Frame rates are computed on an an Intel Compute Stick at a resolution of 1280x720.

Abstract

VR headsets and hand-held devices are not powerful enough to render complex scenes in real-time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image-based rendering (IBR) architecture for masking the latency. It runs in real-time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server-side dual-view representation that leverages an optimally-placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide-angle projection with favorable directional parameterization. A new client-side IBR algorithm uses a pre-transmitted level-of-detail proxy with an encaging simplification and depth-carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR/mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.

Video

Downloads

Citation

Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, Hans-Peter Seidel
Proxy-guided Image-based Rendering for Mobile Devices
Computer Graphics Forum 2016 (Proceedings of Pacific Graphics 2016).

@article{Reinert:2016:ProxyIBR,
  author = { 
	Bernhard Reinert and
	Johannes Kopf and 
	Tobias Ritschel and
	Eduardo Cuervo and
	David Chu and
	Hans-Peter Seidel},
  title = {{Proxy-guided Image-based Rendering for Mobile Devices}},
  journal = {Computer Graphics Forum},
  year = {2016},
  publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
  ISSN = {1467-8659}
}

Acknowledgements

We would like to thank the anonymous reviewers for helpful comments.