max planck institut
mpii logoMinerva of the Max Planck Society

"Capturing Reality" - 3D Reconstruction of Real World Scenes

Winter Semester 2011

Christian Theobalt

Co-lecturers: Kiran Varanasi, Kwang In Kim

Images courtesy of respective authors.


Since the advent of photography, scientists and engineers have been studying ways to capture the real world in a richer manner that permits new applications. This endeavour has been revolutionized by the development of computers. The twin fields of computer vision and computer graphics have discovered various techniques to capture the real world into rich mathematical models, that allow the user to not only re-visualize the scene but also to interact with it. This has enabled a variety of groundbreaking applications, in particular in computer graphics or animation, where photo-realistic virtual versions of static and dynamic real-world scenes can be displayed. To reach this goal, several aspects of the real world scene have to be captured, including geometry, reflectance and motion. In this course we will review fundamental methods from computer vision and graphics to capture such aspects of real world scenes. We will enrich the disucssion of basic methods by providing examples of the research in our own group on the topics of static and dynamic scene capture.

Course structure

The course is given as a class room lecture. There will be 90 miuntes of lecture every week. The following is needed to pass the course:

Background & Requirements

This is an advaned specialized lecture in visual computing that is mainly geared towards senior undrgraduate students and graduate students. The medium of instruction and communication is English. We expect that the students have taken courses in graphcis and / or vision. Basic knowledge in linear algebra, computer graphics and image processing are required. We briefly revise the fundamentals in these areas in the course, but a more thorough introduction is not possible. An aptitude for programming and good skills in the C++ programming language are absolutely necessary. We shall provide the students with the ennvironment for project developement : Kinect cameras, powerful computer workstations with Microsoft Kinect SDK, and tutorials on how to use the system. A few tutorials are also provided on key ideas of the course. But the students are expected to be proactive and make up for the shortcomings of their backgrounds by themselves.

Course Registration & Evaluation

A more detailed introduction to the course will be given in a preparatory meeting on October 18, 6 pm in room MPI 024.

In case you want to take the course, please register by sending an email to Kiran Varanasi - varanasi (at) This should be done at the latest by October 27.

Attendance is required in all the lectures and tutorials.


Course Instructors : Christian Theobalt, Kiran Varanasi, Kwang-In Kim

Teaching Assistants : Miguel Granados, Chenglei Wu, Ahmed Elhayek

Meeting Hours : Kiran (Tuesdays 17h) Office 223, MPI Building

In case you have further questions about the course, please contact Kiran Varanasi - varanasi (at) .

Mailing Lists

Once the course becomes operational, all the students should subscribe to the following mailing list, where all major announcements regarding the course are made :

An additional mailing list is provided for more informal discussions between the students about the programming projects. All the students are suggested to register as well to :

Time and Place

Lectures : Thursdays 14:00-16:00
Tutorials : Tuesdays 18:00-20:00
First meeting : October 18, 2011   at   18:00 hrs
Room: Building E1.4 (Max-Planck-Institut), Room 024
Registration: open


Date Topic Presenter Description Slides

October 18

Introductory Meeting

Lab tour & Tutorial : Microsoft Kinect

Christian Theobalt

Michal Richter

The contents and objectives of the course are outlined.

Formalities about using the lab are given. A tutorial on using the Microsoft Kinect SDK is presented.

October 20

Maths Intro 1

Kwang-In Kim

Fundamental topics of linear matrix algebra are revised.

Lecture Slides

October 27

Physics of the camera

Kiran Varanasi

The physical process of imaging by a digital camera is explained. The fundamental aesthetics of photography are explained from a mathematical point of view.

Assignment 1 on "mathematical background" is handed out.

Lecture Slides

Download : Assignment1

November 3

Camera geometry & calibration

Kiran Varanasi

The mathematical model of a pin-hole camera is explained via projective geometry. Image projection from a perspective camera is described as a matrix operation. The process of calibrating internal and external parameters of a camera is described.

Lecture Slides

November 10

Maths Intro 2

Kwang-In Kim

Basic numerical methods for optimization and regression are revised.

Lecture Slides

Matlab Code : Demos

November 15

Tutorial : Project Proposals

Each group gives a short presentation about their project proposal. Questions related to each project are discussed.

November 17

Structure from motion

Kiran Varanasi

The geometrical relationships between multiple cameras imaging the same scene are described. The method behind "match-moving", i.e, automatic estimation of camera pose for visual effects in films, is described.

Assignment 1 is handed in. Assignment 2 on "camera geometry" is handed out.

Lecture Slides

Assignment 2 : Problem Sheet

Assignment 2 :

November 22

Tutorial : Mathematical background

Miguel Granados

Solutions to assignment 1 are presented. Questions on mathematical background are answered.

November 24

** Shifted to Room 019 **

Light transport

Christian Theobalt

Various models for surface reflectance of light are described. The plenoptic function for light transport in a medium, and its use in computer graphics, is explained.

Lecture Slides

Lecture Slides : Extended Version

December 1

2D image features & matching

Kwang-In Kim

Fundamentals of digital image processing - edge & corner detection are revised. 2D image features based on photometric gradients are described.

Modeling spatial relationship between pixels in an image is described. Methods for matching features consistently across images are reviewed.

Assignment 2 is handed in. Assignment 3 on "feature correspondence" is handed out.

Lecture Slides

Assignment 3 : Problem Sheet

December 6

Tutorial : Camera geometry

Chenglei Wu

Solutions to assignment 2 are presented. Questions on projective geometry & camera models are answered.

Solution Sheet : Assignment 2

December 8

Optic flow & scene flow

Kwang-In Kim

The notion of optical flow is introduced for the analysis of a video-stream of data. Variational methods for the computation of the flow are described.

Lecture Slides

December 15

3D scanning

Christian Theobalt

Alternative means of capture through 3D scanners are introduced. Techniques for the analysis and registration of 3D point data are described.

Assignment 3 is handed in.

Lecture Slides

December 20

Tutorial : Feature correspondence

Ahmed Elhayek

Solutions to assignment 3 are presented. Questions on using OpenCV, matching features and computing optic flow are answered.

December 22

Shape from silhouette

Kiran Varanasi

3D reconstruction from image silhouettes is described through the notion of the visual hull. The practical aspects governing a multi-camera studio, that facilitate such capture, are described.

Lecture Slides

January 10

Tutorial : Interim Project Status Report

Each group gives a short presentation about the status of their project. Questions related to how certain problems faced within can be solved are discussed.

January 12

Shape from stereo

Kiran Varanasi

Practical algorithms that reconstruct dense 3D shapes via multi-camera stereo are described.

Lecture Slides

January 19

Skeletal motion capture

Christian Theobalt

The field of motion capture for animation is reviewed. Motion parametrization through a skeletal hierarchy is described. The various phases in marker-based motion capture are studied.

Lecture Slides

January 26

Shape from Shading

Kiran Varanasi

Shading is a fundamental cue for reconstructing 3D shape from an image. The concepts of shape from shading and photometric stereo are described. The construction of a light stage for imaging a dynamic subject under different lighting conditions is described. The application of relighting the shape under new lighting conditions is studied.

Lecture Slides

February 2

Markerless Motion Capture

Christian Theobalt

Vision-based techniques for motion capture (mocap) without the use of markers are described. The statistical modeling of human shape and pose variations, to be used as priors for mocap, is reviewed. Real-time methods for efficient and accurate mocap are described.

Lecture Slides

February 9

Performance Capture

Christian Theobalt

Performance capture means a much denser description of structure and motion than is possible by skeletal pose. For example, subtle muscle deformations and dynamic folds in cloth can be accurately captured as 3D meshes. The underlying theory of mesh deformation and visual analysis for performance capture is described. The challenges in modeling arbitrary 3D scenes are reviewed and addressed.

Lecture Slides

February 16

Demo sessions


Each group gives a live demo of their project.

February 23

Final Exam

Each student will be tested over the material covered in the lectures by an oral examination.

Search MPII (type ? for help)