Structure from the proxy, appearance from the first frame
A proxy sequence carries the layout, object classes and motion of a scene, at any level of detail. A reference first frame supplies its appearance. Each chapter below changes one input and holds the others fixed. Every clip on this page is model output, shown with the proxy that drove it.
01
Camera control
Each pair opens on one first frame of one scene, then the proxy flies two different routes: a dive down a glass tower, a dolly zoom on a bridge, a chase out of a tunnel, a drone beside a rocket launch. The video takes whichever route its proxy flies.
Fixed: first frame, scene Varies: camera route
02
World dynamics
The camera holds its path and the proxy changes what happens in the world. At one intersection, the cross traffic moves on one signal plan and the pedestrians cross on the other. The video follows each plan car by car and person by person.
Fixed: first frame, camera Varies: motion in the scene
03
Proxy correspondence
Every element in the proxy lands at the same place in the video: buildings in a street grid, pillars and a figure, the rings of a machine, the parts of a motorcycle, the lamps of a car as the camera circles behind it. Fade the proxy over the output to compare them in place.
Fixed: one proxy and first frame per scene Varies: scene
04
Any granularity
One first frame and one camera route, with four proxy styles: a depth map, fine geometry, a subject silhouette over a banded background, and a wireframe. Each level of detail carries a different amount of scene structure into the video.
Fixed: first frame, camera route Varies: proxy granularity
05
Play it live
Infra
Real-time streaming
The time until the first 47 frames are out dropped from 4.6 s to 2.0 s. The video is generated chunk by chunk. Each chunk takes four denoising passes of the DiT on 8 GPUs (Ulysses8). The first pass is fused with the commit of clean history and proxy condition into the KV cache, which keeps a sink and a recent window and evicts the rest. Between DiT passes, the codec encodes the next proxy chunk and decodes the current RGB chunk at the same time.
8 GPUs · Ulysses8 Fused pass 1 + KV commit Concurrent encode / decode