International Conference on Computer Vision and Pattern Recognition (CVPR 2018 oral)
DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor
Tao Yu1,2, Zerong Zheng1, Kaiwen Guo1,3, Jianhui Zhao2, Qionghai Dai1, Hao Li4, Gerard Pons-Moll5, Yebin Liu1,6
1Tsinghua University, Beijing, China 2Beihang University, Beijing, China 3Google Inc4University of Southern California / USC Institute for Creative Technologies
5Max-Planck-Institute for Informatics, Saarland Informatics Campus
6Beijing National Research Center for Information Science and Technology (BNRist)
Software available at [DoubleFusion v1.0]
Figure 1: Our system and real-time reconstruction results.
We propose DoubleFusion, a
new real-time system that combines volumetric dynamic reconstruction
with datadriven template fitting to simultaneously reconstruct detailed
geometry, non-rigid motion and the inner human body shape from a single
depth camera. One of the key contributions of this method is a double
layer representation consisting of a complete parametric body shape
inside and a gradually fused outer surface layer. A predefined node
graph on the body surface parameterizes the non-rigid deformations near
the body and a free-form dynamically changing graph parameterizes the
outer surface layer far from the body allowing more general
reconstruction. We further propose a joint motion tracking method based
on the double layer representation to enable robust and fast motion
tracking performance. Moreover, the inner body shape is optimized online
and forced to fit inside the outer surface layer. Overall, our method
enables increasingly denoised, detailed and complete surface
reconstructions, fast motion tracking performance and plausible inner
body shape reconstruction in real-time. In particular, experiments show
improved fast motion tracking and loop closure performance on more
challenging scenarios.
Abstract
Figure.
2: The pipeline of our system.
Results
Figure. 3: Example results reconstructed by our system
Video Results
Technical Paper
Citation
@InProceedings{DoubleFusion,
author = {Yu, Tao and Zheng, Zerong and Guo, Kaiwen and Zhao, Jianhui and Dai, Qionghai and Li, Hao and Pons-Moll, Gerard and Liu, Yebin},
title = {DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor},
booktitle = {The IEEE International Conference on Computer Vision and Pattern Recognition(CVPR)},
month={June},
year={2018},
publisher={IEEE}
}
author = {Yu, Tao and Zheng, Zerong and Guo, Kaiwen and Zhao, Jianhui and Dai, Qionghai and Li, Hao and Pons-Moll, Gerard and Liu, Yebin},
title = {DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor},
booktitle = {The IEEE International Conference on Computer Vision and Pattern Recognition(CVPR)},
month={June},
year={2018},
publisher={IEEE}
}
Tao
Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li,
Gerard Pons-Moll, Yebin Liu. "DoubleFusion: Real-time Capture of Human
Performances with Inner Body Shapes from a Single Depth Sensor". IEEE
CVPR 2018 (oral).
DoubleFusion Homepage