max planck institut
mpii logo Minerva of the Max Planck Society

Interactive System for Dynamic Scene Lighting using Captured Video Environment Maps

Vlastimil Havran, Miloslaw Smyk(#), Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel
MPI Informatik, Germany
(#)Szczecin University of Technology, Poland

System overview
Acquired real world illumination -> approximation with light sources -> interactive rendering.


Our interactive system for dynamic scene lighting consists of three parts:

  1. We acquire an environment illumination on a hemisphere at arbitrary location and arbitrary conditions, using high dynamic range camera (HDR).
  2. We preprocess the environment map to generate a sequence of directional light sources in real-time that can be used to transfer real-world illumination to computer generated scenes. For computation of light sources we use importance sampling algorithm that was presented during conference Eurographics 2003. This method achieves real-time performance while keeping many desirable properties of sampling over hemisphere.
  3. We use the computed light sources for rendering of a virtual object as being illuminated by the acquired environment illumination. We achieve interactive rates for on-line rendering with possibility of display on the high dynamic range display. The computed set of light sources can also be used for off-line rendering used in production. The rendering phase offers a few quality and speed improvements, including culling off invisible light sources and enhanced rendering of glossy surfaces. Since the direct illumination is computed, the geometry, surfaces properties, camera settings can be altered from frame to frame.

Our project can be generally applied in any system that uses the real world illumination in computer graphics and visualization. For instance it can be used in virtual studios for compositing computer generated objects into live video stream.

Paper Abstract

We present an interactive system for fully dynamic scene lighting using captured high dynamic range (HDR) video environment maps. The key component of our system is an algorithm for efficient decomposition of HDR video environment map captured over hemisphere into a set of representative directional light sources, which can be used for the direct lighting computation with shadows using graphics hardware. The resulting lights exhibit good temporal coherence and their number can be adaptively changed to keep a constant frame rate while good spatial distribution (stratification) properties are maintained. We can handle a large number of light sources with shadows using a novel technique which reduces the cost of BRDF-based shading and visibility computations. We demonstrate the use of our system in a mixed reality application in which real and synthetic objects are illuminated by consistent lighting at interactive frame rates.

Technical Details


The environment illumination is captured using a HDR camera with fish-eye lens which allows to acquire hemispherical image of the surrounding. The camera is photometrically calibrated and has the following parameters:

Light Source Computation

The environment map is approximated with a finite number of light sources using an importance sampling algorithm. The design goals for the light source computations are as follows:

All the properties are achieved by our algorithm. The description can be found in our technical paper EGSR'05 conference (see references).

Interactive Rendering

The interactive rendering is achieved using a standard workstation with NVidia 6800GT graphics card. The system is implemented in OpenGL and the key parts of the algorithm are coded in the pixel shader. The implementation allows for rendering with arbitrary number of light sources using shadow maps. The quality of rendering of glossy surfaces is improved via various sampling enhancements (see paper for details). The performance have been optimized through introduction of several techniques including the clustering of light sources for uneven illumination and elimination of invisible light sources. All the principles we suggest can also be used for off-line (CPU-based) renderers.

Sample Results

The following snapshots were taken from our system as real-time screen captures. The top left corner image shows fish-eye view as taken by camera. The top right corner image shows longitude-latitude projection of the environment map. The green dots correspond to the directional light sources which represent the approximation of the acquired illumination. The right image shows the result from rendering using the computed directional light sources.

Proof of Concept

Proof of concept

As a proof of concept, we conducted an experiment in the photo studio with controlled illumination conditions. The model of two angels visible on the table is interactively rendered on the screen to the left. The HDR camera is located in front of the white table. Notice that the shadows and surface illumination visible in 'reality' is finely reproduced on the screen using our rendering system.

Outdoor Driving Scenerio

Outdoor driving scenerio Outdoor driving scenerio

Interactive illumination of a synthetic object using the illumination acquired while driving through the university campus. See the video material for the full sequence.

Indoor Illumination

Indoor illumination Indoor illumination

Illumination of synthetic objects with indoor environment. We consider two scenarios: artificial illumination and daylight.

Video Material

The following video material demonstrates the performance of our system. The video clips have been captured in real-time using a digital frame grabber.


Interactive System for Dynamic Scene Lighting using Captured Video Environment Maps
V. Havran, M. Smyk. G. Krawczyk, K. Myskowski, H.-P. Seidel
In: Proc. of Eurographics Symposium on Rendering 2005.
[bibtex] [paper]

Importance Sampling for Video Environment Maps
V. Havran, M. Smyk. G. Krawczyk, K. Myskowski, H.-P. Seidel
Technical Sketch at SIGGRAPH 2005.
[bibtex] [slides]

Goniometric Diagram Mapping for Hemisphere
V. Havran, K. Dmitriev, H.-P. Seidel
In: Proc. of Eurographics 2003, Short Presentations Section.
[bibtex] [paper] [slides]

Photometric Calibration of High Dynamic Range Cameras
G. Krawczyk, M. Goesele, H.-P. Seidel
In: MPI Research Report MPI-I-2005-4-005.
[bibtex] [report] [poster]



Comments and suggestions related to the project are very welcome. For further information please contact one of the authors: Vlastimil Havran, Grzegorz Krawczyk, or Karol Myszkowski.

Search MPII (type ? for help)