"Capturing Reality" - 3D Reconstruction of Real World Scenes

Winter Semester 2011

Christian Theobalt

Co-lecturers: Kiran Varanasi, Kwang In Kim

Images courtesy of respective authors.

Overview

Since the advent of photography, scientists and engineers have been studying ways to capture the real world in a richer manner that permits new applications. This endeavour has been revolutionized by the development of computers. The twin fields of computer vision and computer graphics have discovered various techniques to capture the real world into rich mathematical models, that allow the user to not only re-visualize the scene but also to interact with it. This has enabled a variety of groundbreaking applications, in particular in computer graphics or animation, where photo-realistic virtual versions of static and dynamic real-world scenes can be displayed. To reach this goal, several aspects of the real world scene have to be captured, including geometry, reflectance and motion. In this course we will review fundamental methods from computer vision and graphics to capture such aspects of real world scenes. We will enrich the disucssion of basic methods by providing examples of the research in our own group on the topics of static and dynamic scene capture.

Course structure

The course is given as a class room lecture. There will be 90 miuntes of lecture every week. The following is needed to pass the course:

There will be 3 assignments that the students have to pass to continue participating in the course. Each assignment sheet is accompanied by a corresponding tutorial session designed to help the students and answer related practical questions. The threshold for passing the assignments is to answer at least half the questions correctly (50% of points). Only those students who pass the assignments will be admitted to the final exam. The assignment sheets do not contribute to the final grade of the course.
The students have to finish a programming project. The grade of the project counts for 40% of the overall grade.
There will be a final exam, accounting for 60% of the grade for the course. The date of the exam will be announced in the lecture.

Background & Requirements

This is an advaned specialized lecture in visual computing that is mainly geared towards senior undrgraduate students and graduate students. The medium of instruction and communication is English. We expect that the students have taken courses in graphcis and / or vision. Basic knowledge in linear algebra, computer graphics and image processing are required. We briefly revise the fundamentals in these areas in the course, but a more thorough introduction is not possible. An aptitude for programming and good skills in the C++ programming language are absolutely necessary. We shall provide the students with the ennvironment for project developement : Kinect cameras, powerful computer workstations with Microsoft Kinect SDK, and tutorials on how to use the system. A few tutorials are also provided on key ideas of the course. But the students are expected to be proactive and make up for the shortcomings of their backgrounds by themselves.

Course Registration & Evaluation

A more detailed introduction to the course will be given in a preparatory meeting on October 18, 6 pm in room MPI 024.

In case you want to take the course, please register by sending an email to Kiran Varanasi - varanasi (at) mpii.de. This should be done at the latest by October 27.

Attendance is required in all the lectures and tutorials.

Contact

Course Instructors : Christian Theobalt, Kiran Varanasi, Kwang-In Kim

Teaching Assistants : Miguel Granados, Chenglei Wu, Ahmed Elhayek

Meeting Hours : Kiran (Tuesdays 17h) Office 223, MPI Building

In case you have further questions about the course, please contact Kiran Varanasi - varanasi (at) mpi-inf.mpg.de .

Mailing Lists

Once the course becomes operational, all the students should subscribe to the following mailing list, where all major announcements regarding the course are made : https://lists.mpi-inf.mpg.de/listinfo/crcourse

An additional mailing list is provided for more informal discussions between the students about the programming projects. All the students are suggested to register as well to : https://lists.mpi-inf.mpg.de/listinfo/crcourseproject

Time and Place

Lectures :	Thursdays 14:00-16:00
Tutorials :	Tuesdays 18:00-20:00
First meeting :	October 18, 2011 at 18:00 hrs
Room:	Building E1.4 (Max-Planck-Institut), Room 024
Registration:	open

Schedule

Date	Topic	Presenter	Description	Slides
October 18	Introductory Meeting Lab tour & Tutorial : Microsoft Kinect	Christian Theobalt Michal Richter	The contents and objectives of the course are outlined. Formalities about using the lab are given. A tutorial on using the Microsoft Kinect SDK is presented.
October 20	Maths Intro 1	Kwang-In Kim	Fundamental topics of linear matrix algebra are revised.	Lecture Slides
October 27	Physics of the camera	Kiran Varanasi	The physical process of imaging by a digital camera is explained. The fundamental aesthetics of photography are explained from a mathematical point of view. Assignment 1 on "mathematical background" is handed out.	Lecture Slides Download : Assignment1
November 3	Camera geometry & calibration	Kiran Varanasi	The mathematical model of a pin-hole camera is explained via projective geometry. Image projection from a perspective camera is described as a matrix operation. The process of calibrating internal and external parameters of a camera is described.	Lecture Slides
November 10	Maths Intro 2	Kwang-In Kim	Basic numerical methods for optimization and regression are revised.	Lecture Slides Matlab Code : Demos
November 15	Tutorial : Project Proposals		Each group gives a short presentation about their project proposal. Questions related to each project are discussed.
November 17	Structure from motion	Kiran Varanasi	The geometrical relationships between multiple cameras imaging the same scene are described. The method behind "match-moving", i.e, automatic estimation of camera pose for visual effects in films, is described. Assignment 1 is handed in. Assignment 2 on "camera geometry" is handed out.	Lecture Slides Assignment 2 : Problem Sheet Assignment 2 : Data.zip
November 22	Tutorial : Mathematical background	Miguel Granados	Solutions to assignment 1 are presented. Questions on mathematical background are answered.
November 24 Shifted to Room 019	Light transport	Christian Theobalt	Various models for surface reflectance of light are described. The plenoptic function for light transport in a medium, and its use in computer graphics, is explained.	Lecture Slides Lecture Slides : Extended Version
December 1	2D image features & matching	Kwang-In Kim	Fundamentals of digital image processing - edge & corner detection are revised. 2D image features based on photometric gradients are described. Modeling spatial relationship between pixels in an image is described. Methods for matching features consistently across images are reviewed. Assignment 2 is handed in. Assignment 3 on "feature correspondence" is handed out.	Lecture Slides Assignment 3 : Problem Sheet
December 6	Tutorial : Camera geometry	Chenglei Wu	Solutions to assignment 2 are presented. Questions on projective geometry & camera models are answered.	Solution Sheet : Assignment 2
December 8	Optic flow & scene flow	Kwang-In Kim	The notion of optical flow is introduced for the analysis of a video-stream of data. Variational methods for the computation of the flow are described.	Lecture Slides
December 15	3D scanning	Christian Theobalt	Alternative means of capture through 3D scanners are introduced. Techniques for the analysis and registration of 3D point data are described. Assignment 3 is handed in.	Lecture Slides
December 20	Tutorial : Feature correspondence	Ahmed Elhayek	Solutions to assignment 3 are presented. Questions on using OpenCV, matching features and computing optic flow are answered.
December 22	Shape from silhouette	Kiran Varanasi	3D reconstruction from image silhouettes is described through the notion of the visual hull. The practical aspects governing a multi-camera studio, that facilitate such capture, are described.	Lecture Slides
January 10	Tutorial : Interim Project Status Report		Each group gives a short presentation about the status of their project. Questions related to how certain problems faced within can be solved are discussed.
January 12	Shape from stereo	Kiran Varanasi	Practical algorithms that reconstruct dense 3D shapes via multi-camera stereo are described.	Lecture Slides
January 19	Skeletal motion capture	Christian Theobalt	The field of motion capture for animation is reviewed. Motion parametrization through a skeletal hierarchy is described. The various phases in marker-based motion capture are studied.	Lecture Slides
January 26	Shape from Shading	Kiran Varanasi	Shading is a fundamental cue for reconstructing 3D shape from an image. The concepts of shape from shading and photometric stereo are described. The construction of a light stage for imaging a dynamic subject under different lighting conditions is described. The application of relighting the shape under new lighting conditions is studied.	Lecture Slides
February 2	Markerless Motion Capture	Christian Theobalt	Vision-based techniques for motion capture (mocap) without the use of markers are described. The statistical modeling of human shape and pose variations, to be used as priors for mocap, is reviewed. Real-time methods for efficient and accurate mocap are described.	Lecture Slides
February 9	Performance Capture	Christian Theobalt	Performance capture means a much denser description of structure and motion than is possible by skeletal pose. For example, subtle muscle deformations and dynamic folds in cloth can be accurately captured as 3D meshes. The underlying theory of mesh deformation and visual analysis for performance capture is described. The challenges in modeling arbitrary 3D scenes are reviewed and addressed.	Lecture Slides
February 16	Demo sessions	-	Each group gives a live demo of their project.
February 23	Final Exam		Each student will be tested over the material covered in the lectures by an oral examination.