International Conference on Computer Vision and Pattern Recognition (CVPR 2018 oral)

DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

Tao Yu1,2, Zherong Zheng1, Kaiwen Guo1,3, Jianhui Zhao2, Qionghai Dai1, Hao Li4, Gerard Pons-Moll5, Yebin Liu1,6 

1Tsinghua University, Beijing, China 2Beihang University, Beijing, China 3Google Inc 4University of Southern California / USC Institute for Creative Technologies 5Max-Planck-Institute for Informatics, Saarland Informatics Campus 6Beijing National Research Center for Information Science and Technology (BNRist)
Figure 1: Our system and real-time reconstruction results.
We propose DoubleFusion, a new real-time system that combines volumetric dynamic reconstruction with datadriven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside and a gradually fused outer surface layer. A predefined node graph on the body surface parameterizes the non-rigid deformations near the body and a free-form dynamically changing graph parameterizes the outer surface layer far from the body allowing more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.
Abstract

Figure. 2: The pipeline of our system.

Results
Figure. 3: Example results reconstructed by our system


Video Results

Technical Paper
Citation
@InProceedings{DoubleFusion,
  author = {Yu, Tao and Zheng, Zherong and Guo, Kaiwen and Zhao, Jianhui and Dai, Qionghai and Li, Hao and Pons-Moll, Gerard and Liu, Yebin},
  title = {DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor},
  booktitle = {The IEEE International Conference on Computer Vision and Pattern Recognition(CVPR)},
  month={June},
  year={2018},
  publisher={IEEE}
}
Tao Yu, Zherong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu. "DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor". IEEE CVPR 2018 (oral).
DoubleFusion Homepage