IEEE TVCG
MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera
Zhaoqi Su1, Weilin Wan2, Tao Yu1,3, Lingjie Liu2, Lu Fang1, Wenping Wang2, Yebin Liu2
1Tsinghua University, Beijing, China
2the University of Hong Kong, China
3Beihang University, Beijing, China
Figure 1: Results generated by our MulayCap system from a monocular RGB video. From left to right: one of input images, four generated results (one
in the reference view and three in different viewing directions), a cloth editing result, and a relighting result rendered under a novel lighting condition.
We introduce MulayCap, a novel human performance capture method using a monocular video camera without the need for
pre-scanning. The method uses “multi-layer” representations for geometry reconstruction and texture rendering, respectively. For
geometry reconstruction, we decompose the clothed human into multiple geometry layers, namely a body mesh layer and a garment
piece layer. The key technique behind is a Garment-from-Video (GfV) method for optimizing the garment shape and reconstructing the
dynamic cloth to fit the input video sequence, based on a cloth simulation model which is effectively solved with gradient descent. For
texture rendering, we decompose each input image frame into a shading layer and an albedo layer, and propose a method for fusing a
fixed albedo map and solving for detailed garment geometry using the shading layer. Compared with existing single view human
performance capture systems, our “multi-layer” approach bypasses the tedious and time consuming scanning step for obtaining a
human specific mesh template. Experimental results demonstrate that MulayCap produces realistic rendering of dynamically changing
details that has not been achieved in any previous monocular video camera systems. Benefiting from its fully semantic modeling,
MulayCap can be applied to various important editing applications, such as cloth editing, re-targeting, relighting, and AR applications.
Abstract
Figure.
2: The pipeline of our system.
Results
Figure. 3: Selected results reconstructed by our system.
Video Results
Technical Paper
Citation
@article{su2022mulaycap,
author={Su, Zhaoqi and Wan, Weilin and Yu, Tao and Liu, Lingjie and Fang, Lu and Wang, Wenping and Liu, Yebin},
journal={IEEE Transactions on Visualization and Computer Graphics},
title={MulayCap: Multi-Layer Human Performance Capture Using a Monocular Video Camera},
year={2022},
volume={28},
number={4},
pages={1862-1879},
doi={10.1109/TVCG.2020.3027763}
}
author={Su, Zhaoqi and Wan, Weilin and Yu, Tao and Liu, Lingjie and Fang, Lu and Wang, Wenping and Liu, Yebin},
journal={IEEE Transactions on Visualization and Computer Graphics},
title={MulayCap: Multi-Layer Human Performance Capture Using a Monocular Video Camera},
year={2022},
volume={28},
number={4},
pages={1862-1879},
doi={10.1109/TVCG.2020.3027763}
}
Zhaoqi Su, Weilin Wan, Tao Yu, Lingjie Liu, Lu Fang, Wenping Wang, and Yebin Liu. "MulayCap: Multi-Layer Human Performance Capture Using a Monocular Video Camera," in IEEE Transactions on Visualization and Computer Graphics, vol. 28, no. 4, pp. 1862-1879, 1 April 2022, doi: 10.1109/TVCG.2020.3027763.
MulayCap Homepage