Free-Viewpoint Video from Depth Cameras

Depth cameras, which provide color and depth in-formation per pixel at video rates, offer exciting new opportunities in computer graphics. We address the challenge of supporting free-viewpoint video of dynamic 3D scenes using live data captured and streamed from widely-spaced viewpoints by a hand-ful of synchronized depth cameras. We introduce the concept of the depth hull, which is a generaliza-tion of the well-known visual hull. The depth hull reflects all the dense depth information as observed from several centers of projection around the scene. It is the best approximation of the scene geometry that can be obtained from a given set of depth cam-era recordings. We first present a general improvement to the best existing visual hull rendering algorithm, which is of independent interest. We then use this to con-tribute a hardware-accelerated method for rendering novel views from depth hulls in real-time. This method is based on a combination of techniques from projective shadow mapping and constructive solid geometry (CSG). Our rendering method achieves high-quality results even when only a modest number of depth cameras are deployed. They are applicable to any set of images with ac-companying dense depth maps that correspond to arbitrary viewing positions around the scene. We provide experimental results using a system incorporating two depth cameras recording a dy-namic scene. To the best of our knowledge, these are the first results on free-viewpoint image synthe-sis using commercially-available depth cameras.