iHuman3D: Intelligent Human Body 3D Reconstruction using a Single Flying Camera

 Wei Cheng, Lan Xu, Lei Han, Yuanfang Guo, Lu Fang


Fig. 1 Realtime 3D human body reconstruction by a flying camera.

Abstract

Aiming at autonomous, adaptive and real-time human body reconstruction technique, this paper presents iHuman3D: an intelligent human body 3D reconstruction system using a single aerial robot integrated with an RGB-D camera. Specifically, we propose a real-time and active view planning strategy based on a highly efficient ray casting algorithm in GPU and a novel information gain formulation directly in TSDF. We also propose the human body reconstruction module by revising the traditional volumetric fusion pipeline with a compactly-designed non-rigid deformation for slight motion of the human target. We unify both the active view planning and human body reconstruction in the same TSDF volume-based representation. Quantitative and qualitative experiments are conducted to validate that the proposed iHuman3D: system effectively removes the constraints of fixed capture volume and extra manual labor, enabling real-time and intelligent reconstruction of human body.

System Overview

The system architecture of is illustrated in Fig. 2, which relates to: (i) human body reconstruction module, (ii) active view planning module, and (iii) flying camera module. The reconstruction module fuses the live depth input into the TSDF volumes, and meanwhile provides a real-time mesh visualization result. Based on the real-time live TSDF volume, the active view planning module examines the NBV from all the view candidates in parallel on the modern GPU hardware. For the stability of the whole system, the NBV results are transmitted back to the aerial robot in another fixed frame rate (10fps). In the flying camera module, we use the same robot interface to the hardware platform of the aerial robot as FlyCap, which provides a depth stream with the corresponding camera location information in 30fps. Note that the whole system is synchronized with a common NTP server.

Fig. 2 The architecture of iHuman3D.

Results

Our proposed view planning algorithm achieved superior results compared with state-of-art NBV systems Isler and APORA on computation efficiency. Proposed iHuman3D also outperforms FlyCap in terms of reconstruction accuracy in Fig. 3 and efficiency in Fig. 4.

Fig. 3 Quality comparison between (a) iHuman3D and (b) FlyCap.

Fig. 4 (a) Robot motion according to reconstruction frames. (b) Mesh vertices increase according to frames.

Table 1 Computational speed for evaluating candidate viewpoints.

Video: Live demo and experiments on sythetic data.


Code (Simulation)

Code (Reconstruction) coming soon!

Video

Paper


Citation:

@inproceedings{cheng2018ihuman3d,
    title={iHuman3D: Intelligent Human Body 3D Reconstruction using a Single Flying Camera},
    author={Cheng, Wei and Xu, Lan and Han, Lei and Guo, Yuanfang and Fang, Lu},
    booktitle={2018 ACM Multimedia Conference on Multimedia Conference},
    pages={1733--1741},
    year={2018},
    organization={ACM}
  }

Supplementary:

Supplementary Material