# Recording 173 · Wipe region · footage-faithful twin

A simulation rebuilt from one real first-person recording: Eidon tracker-pov recording 173 (CC BY 4.0), excerpt
8.17–13.41 s. A person wipes crumbs from a wooden dining table with a napkin.

## What is in the videos

- `twin.mp4`: the twin as it would look to the head camera.
  - **Reconstructed from the recording:** the camera path, the table and the background.
  - **Simulated:** the hand and the napkin.
  - **Video-model output:** the simulated hand and napkin are repainted for realism by Wan2.1-VACE-14B.
- `physics.mp4`: the same twin without the video-model repaint. It shows the simulated hand and napkin as they are, over a Gaussian-splat reconstruction of the scene.
- `compare.mp4`: the real footage, the earlier authored gym and the twin, side by side.

## How it was built

1. **Masks:** Grounding DINO with SAM 2.1 segmented the hand, napkin, table and objects in every frame.
2. **Camera:** Depth Anything 3 estimated a camera for each frame, then a planar bundle adjustment refined them (median reprojection error 3.8 px at 1920 px). MoGe-2 metric depth set the scale; the head was 33–60 cm above the table.
3. **Appearance:**
   - The table texture was projected from the footage.
   - The wall, curtain, chair, glass, tumbler and rack come from a 3D Gaussian splat trained on the frames, with the hand and napkin masked out.
4. **Physics:** a human-like hand in MuJoCo follows the hand path tracked in the clip. Every joint is actuated; there is no mocap or welding. It presses a cloth napkin textured from the footage, and the napkin moves only through contact.

## How faithful it is

The scores compare each version with the real frames. They use the same 20 held-out frames, which the reconstruction never used, and the values are medians.

| Version | SSIM | PSNR |
|---|---|---|
| Twin (photoreal) | 0.88 | 20.7 dB |
| Earlier authored Panda gym | 0.56 | 8.5 dB |

Most of this similarity comes from the camera path, the table and the background. The hand and napkin match the footage poorly; see Limits.

Two negative controls check the physics. With an idle hand, or a hand that never presses, the napkin moves 0.0 cm. With the tracked hand it moves 6.9 cm.

## Limits

- **Hand and napkin:** they only roughly follow the real ones. Their mask overlap with the footage is 0.08 IoU for the hand and about 0 for the napkin. The real person grips the napkin, while the simulated hand only presses it. Through much of the clip, the simulated hand and napkin are in a different place from the real ones, sometimes over bare table.
- **Hand look:** the raw simulated hand is a simple capsule hand. Its realistic look in `twin.mp4` comes from the video model.
- **Scope:** this is one example. Every other gallery gym task is still an authored scene.

All models are Apache-2.0 or MIT. `metrics.json` has the full scores and provenance.