FlyFusion: Realtime Dynamic Scene Reconstruction Using a Flying Depth Camera

 Lan Xu, Wei Cheng, Kaiwen Guo, Lei Han, Yebin Liu, Lu Fang


Fig. 1 Realtime 3D human motion capture by a flying camera.

Abstract

While dynamic scene reconstruction has made revolutionary progress from the earliest setup using a mass of static cameras in studio environment to the latest egocentric or hand-held moving camera based schemes, it is still restricted by the recording volume, user comfortability, human labor and expertise. In this paper, a novel solution is proposed through a real-time and robust dynamic fusion scheme using a single flying depth camera, denoted as FlyFusion. By proposing a novel topology compactness strategy for effectively regularizing the complex topology changes, and the Geometry And Motion Energy (GAME) metric for guiding the viewpoint optimization in the volumetric space, FlyFusion succeeds to enable intelligent viewpoint selection based on the immediate dynamic reconstruction result. The merit of FlyFusion lies in its concurrent robustness, efficiency, and adaptation in producing fused and denoised 3D geometries and motions of a moving target interacting with different non-rigid objects in a large space.

System Overview

Our new FlyFusion system captures a moving target's motion autonomously in real-time using a flying camera (UAV equipped with a RGBD camera), with fewer constraints like fixed capture volume or tedious manual labor. Fig. 2 gives a sketch of the working pipeline of our FlyCap system, which can be decomposed into three modules: flying camera, robust dynamic fusion module, and active view planning module, respectively.

Fig. 2 The architecture of iHuman3D.

Results

We first test our robust dynamic fusion scheme on the public RGB-D datasets of several representative single-view non-rigid reconstruction methods, i.e., VolumeDeform, MonoFVV and KillingFusion in Fig. 3. Throughly evaluation on each term of proposed GAME metric is committed in Fig. 4, Fig. 5 and Fig. 6.

Fig. 3 Evaluation on available single view RGB-D datasets. (a) Our results on the sequences boxing, hoodie, minion, roll shirt, sun flower and umbrella from VolumeDeform. (b) Our results on the sequences bag open, boxing and fast loop from MonoFVV. (c) Our results on the sequences frog, duck, snoopy, hat and Alex from KillingFusion.

Fig. 4 Evaluation of the depth term. (a, b) The results with and without the depth term respectively, including the color frame, the geometry result and the input depth image. (c) The quantitative curves of the average depth value of current depth input.

Fig. 5 Evaluation of the center term. (a, b) The results with and without the center term respectively, including the color frame, the geometry result and the input depth image. (c) The quantitative curves of the average horizontal index of current depth input.

Fig. 6 Quantitative evaluation of the motion term in the panda bag sequence. (a) and (b) are the motion maps of the 219th, 309th, 499th and 799th frames for our method with and without the motion term respectively. (c) The numerical motion curve.

Video: Live demo and experiments on sythetic data.


Code (Simulation)

Code (Motion capture) coming soon!

Video

Paper


Citation

FlyFusion: Realtime Dynamic Scene Reconstruction Using a Flying Depth Camera

@Misc{xuFlyFusion,
  title={FlyFusion: Realtime Dynamic Scene Reconstruction Using a Flying Depth Camera},
  author={Lan Xu and Wei Cheng and Kaiwen Guo and Lei Han and Yebin Liu and Lu Fang},
  howpublished={\url{http://www.luvision.net/FlyFusion_tvcg/}},
  year={2018}
}