ModelsVideos

Five models × six videos

Five models on six hard videos (man_airport, man_eyeline, man_park, man_winter, man_standing_rotate, breakdance (frames 590-1180)). Each video (for breakdance: frames 590-1180 of the video) is run in a single forward pass (centre crop to 16:9, 672×378); for every model the page shows depth and depth-only (oblique 30°) videos, an interactive 3D point cloud and temporal-consistency metrics.

Models

All videos
How to read the videos, the 3D point cloud and the metrics

How to read this page

  1. Input: every frame of each video at its native fps, centre-cropped to 16:9 and resized to 672×378; the whole video in one forward pass (long videos use a chunked but mathematically equivalent forward).
  2. depth: inverse depth, turbo colour map; the colour range is the 2–98 % range of the whole video, fixed over the video, so colour flicker = flicker of the predicted depth.
  3. confidence (hidden by default; tick "confidence" above): sigmoid(conf), viridis 0–1. The point clouds draw only points with conf > 0.1 that are not on depth edges (the rule of the Pi3 example script).
  4. Dynamic objects: SegFormer-B0 (ADE20K, the weights shipped with Pi3) marks people, animals and vehicles as dynamic objects (red); everything else is static background, and the per-pixel jitter is measured separately for the two groups. Segmentation errors (e.g. a missed person) leak into "static": compare with the mask video.
  5. Depth only (30° oblique) (a collapsed section on every card): the points of the current frame stay in their own camera coordinates (no pose) and are seen from 30° to the left of the subject; the virtual camera turns about the median subject depth of the video and stays fixed. Pose errors do not appear at all; what still moves is depth jumping back and forth along the line of sight (seen from the side as a sideways shift of about sin 30° = half the depth error). Real forward / backward motion of the subject also becomes a sideways shift, and the subject may leave the view when it is much nearer or farther than the median; use the 3D viewer then.
  6. Interactive 3D point cloud (bottom of each card, one open at a time): about 16,000 points per frame (one per 4×4 pixels), by default only points with conf ≥ 0.1 that are not on depth edges, in a fixed world view: every model's points are placed in its own first-frame camera coordinates (that camera at the origin, looking along +z), and the models are stacked top to bottom with a shared time. With "link views" (on by default) all panels share one virtual camera: the same viewing direction and field of view (the median of the five predicted focal lengths), and each model's cloud scaled by the median depth of the static background pixels (outside the SegFormer objects) that all five models keep in that first frame, so that the static scenes of the five models coincide; what still differs between the panels is the models' own geometry and camera motion. The optional camera frusta and image planes use each model's own predicted focal length. Accumulating 4–8 frames with the time colour shows whether the static background stays one thin layer; "accumulate static background only" keeps the moving objects to the current frame. The model boxes inside the viewer (also in full screen) and "Models" above show or hide each model.
  7. Metrics (proxies, no ground truth): mean second difference (jitter) of translation, rotation and the per-frame mean log depth; speed = mean first difference; jitter / speed is near 0 for smooth motion and about 1.7 for pure noise. Per-pixel jitter: WAFT optical flow (WAFT-DAv2-a2) chains pixels into three-frame tracks t−1, t, t+1, keeping only pixels that are forward-backward consistent (≤ 1 px), textured (a Shi-Tomasi threshold: WAFT underestimates motion on textureless white walls) and valid points in all three frames; depth jitter = |Δ² log z| (camera coordinates, no pose), world jitter = |Δ² X| ÷ z (with pose); per-frame median, then mean over the video, in %. Region depth jitter = the absolute value of the median of the signed Δ² log z over the same tracks, i.e. the part where a whole region jumps back and forth together (the object as a whole moving nearer / farther); it is never larger than the per-pixel depth jitter. The two close together → the error is global and a per-frame scale / shift correction (two parameters) can handle it; region much smaller than per-pixel → the error is a local change of shape (e.g. a deforming hand) that needs per-point constraints. Computed along tracks, so it is not affected by the frame-to-frame flicker of the mask edges. Focal jitter = |Δ² log f| of the focal length implied by each frame's predicted rays (least squares of (u, v) = f·(x/z, y/z), principal point at the image centre): Pi3X predicts the rays anew in every frame, and focal jumps make the points zoom in and out relative to the image. The card also lists the median focal length. At 24–30 fps real motion is smooth, so the second difference of the static background is almost all prediction jitter (its true value in world coordinates is 0). The dynamic-object values include real acceleration: fast parts close to the camera have a real Δ² that is not small, so compare the dynamic group only between models on the same video, not across videos or with the static group. A frame's group is skipped with fewer than 100 valid tracks (of a 95×168 grid); "–" when fewer than 25 % of the frames have a value. SegFormer works by class, so parked cars or people in murals count as dynamic too; those pixels only make the static group smaller, they do not leak into it. The values are for comparing models, not an absolute quality.