Code Tutorial

In this tutorial, we will show how to make a novel view synthesis with fantasy effects from sparse multi-view videos.

Capture

Capture the videos with multiple cameras, rounghly synchronize them. Here we show the example data.

Motion Capture

# Reconstruct the human
python3 apps/demo/mocap.py ${data} --work mvmp-wild --ranges 0 300 1 --subs_vis 1 3 5 7 --pids 0 1 2 3 4 5
# Reconstruct the object
python3 apps/demo/mocap.py ${data} --work object --ranges 0 300 1 --subs_vis 1 3 5 7

Skeleton
Mesh
Soccer

Training

python3 apps/neuralbody/demo.py --mode soccer1_6 --gpus 0,1,2,3 ${data}

Rendering

python3 apps/neuralbody/demo.py--mode soccer1_6 --gpus 0,1,2,3 --demo ${data} 

The config for this scene can be found in config/neuralbody/dataset/demo_soccer1_6.yml. We explain this code and show how to perform scene editing step by step.

Clip1

We play the frame from 0 to 40. We set the view from 0 to 80. In this clip, we render the object as the object_keys. Then, we freeze the time and view. We zoom the scene by scale the intrinsinc parameters.

start:
  frame: [0, 40, 1]
  view: [0, 80, 2]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
zoomin:
  frame: [40, 41, 1]
  view: [80, 81, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
  steps: 60
  effect: zoom
  effect_args:
    scale: [1., 2.]
    cx: [1., 1.]
    cy: [1., 1.]

clip1

Clip2

Then we render the bullet time after zooming in and zoom out by change the scale from 2 to 1.

rotate:
  frame: [40, 41, 1]
  view: [80, 200, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
  effect: none
  effect_args:
    use_previous_K: True
zoomout:
  frame: [40, 41, 1]
  view: [200, 201, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
  steps: 60
  effect: zoom
  effect_args:
    scale: [2., 1.]
    cx: [1., 1.]
    cy: [1., 1.]

clip2

Clip3

Besides traditional bullet time effect, our method can control the motion of each person. Here we keep all people static at frame 40 and make a duplicate of human_3. The parameter scale_occ makes the duplicate transparent.

frozen1:
  frame: [40, 200, 2]
  view: [200, 201, 1]
  object_keys: ["ground_@{'frame': 40}", 'ball_0', "human_3_@{'scale_occ': 0.2}", "human_0_@{'frame': 40}", "human_1_@{'frame': 40}", "human_2_@{'frame': 40}", "human_3_@{'frame': 40}", "human_4_@{'frame': 40}", "human_5_@{'frame': 40}"]
frozen2:
  frame: [199, 39, -2]
  view: [200, 201, 1]
  object_keys: ["ground_@{'frame': 40}", 'ball_0', "human_3_@{'scale_occ': 0.2}", "human_0_@{'frame': 40}", "human_1_@{'frame': 40}", "human_2_@{'frame': 40}", "human_3_@{'frame': 40}", "human_4_@{'frame': 40}", "human_5_@{'frame': 40}"]

clip3

Clip4

Then we can play the motion and focus on the logo SIGGRAPH by zooming in.

normal:
  frame: [40, 65, 1]
  view: [200, 250, 2]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
bullet:
  frame: [65, 66, 1]
  view: [250, 490, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
# 5. zoom in , vanish
zoomin2:
  frame: [65, 66, 1]
  view: [490, 491, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
  steps: 60
  effect: zoom
  effect_args:
    scale: [1., 1.5]
    cx: [1., 0.7]
    cy: [1., 1.]

clip4

Clip5

Finally, we show the disappear effect by setting the key of selected people.

disappear3:
  frame: [65, 200, 1]
  view: [490, 491, 1]
  object_keys: [ground, human_0, human_1, human_2, human_3, human_4, human_5, ball_0]
  effect: disappear
  effect_args:
    key: ['human_1', 'human_2', 'human_5']
    use_previous_K: True

clip5