max planck institut informatik
mpii logo Minerva of the Max Planck Society

Efficient Multi-image Correspondences for On-line Light Field Video Processing, Pacific Graphics 2016

Efficient Multi-image Correspondences for On-line Light Field Video Processing

Łukasz Dąbała1,2     Matthias Ziegler3     Piotr Didyk1,4     Frederik Zilly3     Joachim Keinert3    
Karol Myszkowski1     Hans-Peter Seidel1     Przemysław Rokita2     Tobias Ritschel5    

1 MPI Informatik     2 Warsaw University of Technology     3 Fraunhofer IIS     4 Saarland University     5 University College London    

Our real-time multi-view correspondence algorithm extracts multi-view depth maps from sparse, wide-baseline light field video (here 3 x 3 cameras), in order to produce high-quality novel views for applications such as virtual apertures or virtual camera positions.


Light field videos express the entire visual information of an animated scene, but their shear size typically makes capture, processing and display an off-line process, i. e., time between initial capture and final display is far from real-time. In this paper we propose a solution for one of the key bottlenecks in such a processing pipeline, which is a reliable depth reconstruction possibly for many views. This is enabled by a novel correspondence algorithm converting the video streams from a sparse array of off-the-shelf cameras into an array of animated depth maps. The algorithm is based on a generalization of the classic multi-resolution Lucas-Kanade correspondence algorithm from a pair of images to an entire array. Special inter-image confidence consolidation allows recovery from unreliable matching in some locations and some views. It can be implemented efficiently in massively parallel hardware, allowing for interactive computations. The resulting depth quality as well as the computation performance compares favorably to other state-of-the art light field-to-depth approaches, as well as stereo matching techniques. Another outcome of this work is a data set of light field videos that are captured with multiple variants of sparse camera arrays.



Paper (Full Author's Copy) (77 MB) Paper (Reduced size Author's Copy) (1,1 MB) Official online version: Computer Graphics Forum
Video with results (122 MB)
Pacific Graphics 2016 presentation slides (125 MB)
Dataset description


Łukasz Dąbała, Matthias Ziegler, Piotr Didyk, Frederik Zilly, Joachim Keinert, Karol Myszkowski, Hans-Peter Seidel, Przemysław Rokita, Tobias Ritschel
Efficient Multi-image Correspondences for On-line Light Field Video Processing
Computer Graphics Forum 35(7) (Proc. Pacific Graphics 2016, Okinawa, Japan)

@article {10.1111:cgf.13037,
journal = {Computer Graphics Forum},
title = {{Efficient Multi-image Correspondences for On-line Light Field Video Processing}},
author = {Dąbała, Łukasz and Ziegler, Matthias and Didyk, Piotr and Zilly, Frederik and Keinert, Joachim and Myszkowski, Karol
and Seidel, Hans-Peter and Rokita, Przemysław and Ritschel, Tobias},
year = {2016},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
ISSN = {1467-8659},
DOI = {10.1111/cgf.13037}

© 2016 The Authors. This is the author's version of the work. It is posted here for your personal use. Not for redistribution.


This work was partially supported by the Fraunhofer and Max Planck cooperation program within the framework of the German pact for research and innovation (PFI).

MPI Informatik Warsaw University of Technology Fraunhofer IIS Saarland University University College London