08w5070 Multi-View Image and Geometry Processing for 3D Cinematography

Arriving Sunday, July 13 and departing Friday, July 18, 2008

Organizers: Remi Ronfard (Xtranormal), Gabriel Taubin (Brown University).

Confirmed Participants

Press Release: Multi-View Image and Geometry Processing for 3D Cinematography

Information for Participants

Schedule and Abstracts (PDF file)

Mailing List

Workshop Files

Final Report (PDF file)

Objectives

Several early 3D cinematography systems have already demonstrated partial 3D cinematography capability in various forms and under different names -virtualized reality, free- viewpoint video, 3D video, etc. So far, most of these projects have focused on the computer systems and engineering aspects involved in the implementation. There is a need for a forum where researchers active in the field could focus on the mathematical foundations, and determine what new mathematical results are needed to advance the state-of-the-art in 3D cinematography systems to the next generation.

Recent advances have now clearly shown the promises of 3D cinematography stystems allowing multiple-camera capture, processing, transmission and rendering of 3D models of real scenes in real time. Yet, many problems remain to be solved before such systems can be transposed from blue screen studios to the real world. Problems are both theoretical and practical:

- Scaling issues. How many cameras should be used to fully sample a complex, dynamic scene with multiple actors? How accurately should the cameras be synchronized and calibrated?

- Representation issues. How can the geometry and texture of a scene be separately extracted and represented independently of the original viewpoints, and at which levels of details?

- Modeling issues. Which information can be realiably extracted from video streams to allow temporally consistent 3D reconstruction? How precisely can the geometry and texture of a scene be recovered in real world situations? How can prior knowledge about the scene geometry and appearance be used?

- Compression of multi-view video streams. How can the massive amounts of multiple video and geometry streams generated by 3D cinematography be repfresented in compact form for efficient storage, transmission, and remote visualization?

- Implementation issues. How can the massive amounts of multiple video and geometry streams generated by 3D cinematography be processed and rendered efficiently? How can such processing be performed on-line at video frame rate, without sacrificing quality?

- Human factors and aesthetic issues. How will end users navigate within 3D cinematographic scenes? Will 3D cinematography produce exact copies of the real world Or will it evolve into a more elaborate, yet to be discovered, art form?

Those are difficult and fascinating questions which will no doubt generate new research directions in computer vision, in graphics, and in the underlaying mathematics, for many years to come. One goal of this workshop is to identify and predict these advances.

On one hand this workshop will bring together researchers who have built three-dimen-sional cinematography systems, to present their work, show their latest results, share insights on current issues and new research directions, and discuss real world applications of their work. Another important goal for this workshop is to produce a state-of-the art report on the mathematical foundations of the field, to be published in a major journal, including recomendations for funding, and a comprehensive list of applications areas with measurable benefits to society.

Some topics to be discuss in this workshop:

- Camera and recording technology for 3D video, including the synchronization, optical modeling and calibration of camera arrays

- Projection and display technology for 3D cinema and 3D television, including auto-stereoscopic displays - Geometric modeling of dynamic scenes in four dimensions - dynamic surfels, voxels, mesh, depth images, NURBS surfaces, subdivision surfaces, level sets, etc.

- Bayesian modeling and priors for geometry, texture and motion recovery from multiple video cameras

- Multi-resolution 3D modeling from videos captured at different image resolutions

- Motion tracking and temporal coherence of 3D reconstruction

- Video-based rendering, view morphing and video interpolation in dynamic scenes

- Distributed and embedded algorithms for real-time geometry and video processing

- Multiple-view video compression

- Applications of 3D cinematography in games, virtual reality, medicine, television, education, etc.