Space station. Model output, first frame.

Structure from the proxy, appearance from the first frame

A proxy sequence carries the layout, object classes and motion of a scene, at any level of detail. A reference first frame supplies its appearance. Each chapter below changes one input and holds the others fixed. Every clip on this page is model output, shown with the proxy that drove it.

The inputs and the output. The chapters below show them side by side.

01

Camera control

Each pair opens on one first frame of one scene, then the proxy flies two different routes: a dive down a glass tower, a dolly zoom on a bridge, a chase out of a tunnel, a drone beside a rocket launch. The video takes whichever route its proxy flies.

Fixed: first frame, scene Varies: camera route

Tower dive. Proxies for routes A and B above, model outputs for each route below, first frame.
Tower dive · Training proxy Both routes open on one first frame, then split. Each proxy is inset at top left.

02

World dynamics

The camera holds its path and the proxy changes what happens in the world. At one intersection, the cross traffic moves on one signal plan and the pedestrians cross on the other. The video follows each plan car by car and person by person.

Fixed: first frame, camera Varies: motion in the scene

Intersection. Proxies for versions A and B above, model outputs for each version below, first frame.
Intersection · Training proxy Both versions open on one first frame and share the camera. Only the motion in the proxy differs. Each proxy is inset at top left.

03

Proxy correspondence

Every element in the proxy lands at the same place in the video: buildings in a street grid, pillars and a figure, the rings of a machine, the parts of a motorcycle, the lamps of a car as the camera circles behind it. Fade the proxy over the output to compare them in place.

Fixed: one proxy and first frame per scene Varies: scene

Cloud city with a semantic L3 proxy. Proxy and model output, first frame.
Cloud city · Semantic L3 Proxy drawn over the output at the chosen opacity, frame-synced.

04

Any granularity

One first frame and one camera route, with four proxy styles: a depth map, fine geometry, a subject silhouette over a banded background, and a wireframe. Each level of detail carries a different amount of scene structure into the video.

Fixed: first frame, camera route Varies: proxy granularity

Rainy chase with a depth proxy. Proxy and model output, first frame.
Rainy chase · Depth Proxy inset at top left, frame-synced with the output. Switching style keeps the playback position.

05

Play it live

Racing session. Model output full frame, proxy at bottom left, the player's controller at bottom right.

Infra

Real-time streaming

The time until the first 47 frames are out dropped from 4.6 s to 2.0 s. The video is generated chunk by chunk. Each chunk takes four denoising passes of the DiT on 8 GPUs (Ulysses8). The first pass is fused with the commit of clean history and proxy condition into the KV cache, which keeps a sink and a recent window and evicts the rest. Between DiT passes, the codec encodes the next proxy chunk and decodes the current RGB chunk at the same time.

8 GPUs · Ulysses8 Fused pass 1 + KV commit Concurrent encode / decode

Infra overview. A KV cache holds a sink, evicted slots, earlier chunks of clean history H and proxy condition C, and the newly committed KV. Two DiT blocks, each four passes on 8 GPUs with Ulysses8, read the cache; pass 1 is fused with the commit of H and C. Between them, a codec interval runs the encoder (×2) and decoder (×4) concurrently, taking proxy chunk n+1 in and putting RGB chunk n out.
Infra overview. Concept diagram, not drawn to time scale. H clean history · C proxy condition · D noisy latent.

Stage timeline, before and after optimization

Measured stage intervals for three consecutive chunks, aligned to the start of the first chunk and drawn on one time axis. Dark marks show when each chunk's frames are out, labelled with the frame count.