"Capturing Reality" - 3D Reconstruction of Real World Scenes

Information about Student Projects

One requirement to pass the course will be to finish a programming project. For the duration of the course, teams of 2-3 students will be formed who work together on developing a little application for the the Microsoft Kinect camera, a new 3D depth sensor. The Kinect camera does not only capture video frames, like a standard video camera, but it also captures depth maps, i.e. 2.5 D geometry. The camera was originally developed as a game controller, but Microsoft makes available an API that gives access to the data captured by the cameras, as well as some data that are usually extracted from the camera in the context of a gaming, such a joint positions. Having access to the data, developers can now build their own 3D applications that go beyond gaming, e.g, for 3D modeling, interaction. telepresence and so on. In this lecture, we want to give you the chance to develop such an application on your own. The programming projects will be graded and contribute to 40% of the overall grade for the course.

Required Skills

This is an advanced lecture and we will not teach you programming, or how to use certain libraries that are commonly used in vision and/or graphics. We expect that you have programming experience in C++, and we expect that you have experience with respective libraries, e.g., from previous classes in graphics and/or vision. If you lack these skills you should be prepared to acquire the knowledge about using these tools.

Essential prior knowledge

Experience in developing for Windows, in particular with C++ (Visual Studio);
Experience with OpenGL
Experience with tools for developing basic GUIs, e.g. QT, or MFC

Useful additional prior knowledge

Experience with MATLAB / MATLAB Image Processing Toolbox
Experience with Computer Vision Toolboxes, such as OpenCV

Kinect Lab

In MPII room 210, there is our Kinect Lab in which you can work on your projects. The lab features three computers, each equipped with a Kinect. Each computer runs Windows7 and has the following software / libraries installed that may be relevant to your project:

MS Visual Studio
MS Kinect SDK
OpenGL
MATLAB
OpenCV
QT

There will be an introductory tutorial in which you will be explained the use of the lab and in which you will be shown the basic functionality of the Kinect SDK.

Topic

There are two project options. The first option is to complete the default task described below. In this option, the default task is the minimial requirement and there is an optional advanced task which meant to be built upon the default task. The results of optional advanced tasks are properly evaluated and reflected to the final grade. The second option is to design your own project.

Option 1: default task-3D scanning with a Kinect camera

Write a software that scans the shape of a static 3D object with a Kinect camera, and then visualizes the model on screen. The Kinect can only capture the geometry of the object from one side. So, to get the entire 3D shape, you will have to fuse the depth scans from different sides. You will have to think about how to align the scans in 3D and how to visualize the aligned point cloud. You may also have to think on how the best arrangement of scanner and object is: For instance, you could move the object by hand in front of the Kinect, or you could move the Kinect by hand around the object that stands on a table. Each of these operation modes will have its own challenges. Algorithm-wise you will have to think about how to align the individual scans, i.e. how to find the rigid body transforms forms that align the scans. The lecture will cover this. In the end, you should be able to show the 3D object on screen using a simple graphical interface in which you can rotate/translate the scanned shape.

Candidates for advanced task (not limited to)

A 3D space mouse (intract with your model)- Write a software which interprets body and / or hand gestures to rotate / translate and/or scale a 3D object on screen
Surface completion- Fill the holes in the surface obtained as the result of the basic task. In general, surfaces may contain holes.
Texturing- Wrap textures comining from the video camera onto the 3D surface.

Option 2: build your own project

Maybe you have an idea of your own on what project you want to do with the Kinect - a simple game, a telepresence application, teleconferencing system etc. In such a case, we encourage you to follow your own ideas and turn this into your course project. Your own idea will have to be of a similar scope and on a similar level of complexity as the essential task (3D scanning). Decisions on special projects are taken on a case-by-case basis after a thorough discussion between the students and the instructors.

References

Build Your Own 3D Scanner: Optical Triangulation for Beginners. A comprehensive tutorial on building 3D surface given at SIGGRAPH'09. Chapter 6 of the course notes will be our main reference.
Kinect Hacking : Tips and Software by Oliver Kreylos.
PTAM (Parallel Tracking and Mapping): A state-of-the-art camera tracking system. Implementation available.
KinectFusion: A reference project on 3D tracking, reconstruction, and interaction for Kinect.
SIFT (Scale-invariant feature transform): An implementation of SIFT keypoint detector.
Mesh lab A system for the processing and editing 3D triangular meshes.
Marching cubes: An implementation of Marching cubes algorithm which computes and displays a triangle mesh from discrete data.
Poisson surface reconstruction: An implementation of Poisson surface reconstruction.

Project time line

October 18 2011: Tutorial on the lab hardware / lab machines
November 7 2011: By this date project teams need to be formed and announced to us
November 15 2011: Project proposals due. We will have a special session in the usual tutorial slot time in which the groups are supposed to give a presentation on what project they will be working on. We expect a roughly 10 minute long presentation where each group explains the topic/goal of their project, what problems they will have to solve in reaching the goal, and the time line within which they would solve these problems
January 10, 2011: Interim status-report due. We will have a special session in the tututorial slot time, where each team gives a short presentation detailing the status of their project
February 15 2012: Project due date. The project will have to be submitted to us. This includes all the well-documented source code in a manner that can be compiled run by us on one of the lab machines (see Equipment). Also, we expect written reports on each project in which the groups describe their project goal, the algorithmic problems faced and how they were solved, and also in which the groups describe what the share of each group member was on the project development, and specifically what each group member was working on.
February 16 2012: Project presentations Each team will give a 5-10 minute demo of their project in the Kinect Lab at MPI.
February 29, 2012: Written report due Each team should submit a written report for the evaluation of the project. This should explain clearly the problem setting, the methods for how the various sub-problems were solved, and the reasons why such methods were chosen. Each report should be concluded with ideas towards future work and applications.

Submission / Evaluation Criteria for Projects

The following are the criteria for successful projects. Each submitted project contains:

An archive containing the entire well-documented C++ source code of the project, as well as additional libraries needed. There should be a project file such that the entire code can be compiled in Visual Studio on one of the lab machines.
An executable that starts on one of the Kinect Lab machines and that demos the project
A basic instruction file on how to use of the software, for instance as a README.txt file
A written report on the project, which can be written in a similar way as a seminar report. The report should roughly follow the structure of a scientific paper ( 1) Introduction and Motivation, 2) Technical Exposition - problems one had to solve and how the were solved, 3) Results and Discussion: What has been achieved ? What could not be achieved ? Why did some things not work out and why ?). In addition, the reports must contain a section in which it is described to what part of the project each group member contributed and what share (in per cent) of the project was worked on by him/her. Please use the following Latex template to prepare the report.
.tar.gz (Linux) .zip (Windows)
As a rule of thumbs we expect that reports will be around 6-8 pages long in that two-column scientific paper style.
Use of third party libraries: We do not generally discourage to use of third party libraries in your projects. However, we will of course not accept a final project which does nothing more than plugging together existing code from the web. As a guideline, we expect that at least 50% of the essential algorithmic components of your application were developed by yourself. Also, please keep in mind that you will not have full administrator access to th lab machines. We will not be able and to and will not accommodate for installation requests of all possible libraries on the lab machines that you may want to use through administrators, so keep that in mind.

The projects will be evaluated on the basis of:

1) Fulfillment of the above criteria,
2) successful reaching of the project goal,
3) quality of the code and documentation,
4) successful demonstration,
5) the form and content of the written report.

By fulfilling these criteria for the essential task, you will already be able to reach a very good mark. We will give additional credit for achievements in the optional advanced task.