video_editing_poc/docs/cinematic-quality-rules.md

# Cinematic Highlight Quality Rules (source-adaptive)

Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle:
**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant
tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next.

Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x).

## R1 — Exposure: normalize to a target, never crush ✅
- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma.
  `HighlightFfmpegRenderer.probeSourceLuma`.
- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band
  (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1,
  brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098.

## R2 — Orientation: respect the source, never distort ✅
- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H.
- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39
  letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.

## R3 — Audio balance: the story audio must lead ✅
- **Rule:** the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with
  narration); with narration, voice leads via side-chain ducking. No element is silently buried.
- **Accurate mastering:** single-pass `loudnorm` is only ~±2 LUFS accurate, so the finished file is measured
  (`probeIntegratedLoudness`) and a corrective gain (`loudnessGainDb` → `masterLoudness`) is applied to hit the
  target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at
  −15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
- **Verify:** integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.

## R4 — Duration: length follows the story ✅
- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive
  duration + sane playback speed + candidate containment).

## R5 — No dead air: motion in held shots ✅
- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source
  range. `HighlightFfmpegRenderer.probeSegmentMotion`.
- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a
  gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives.
- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active
  celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).

## R6 — Transitions: ease, don't jerk ✅
- **Crossfades:** video xfade + audio acrossfade dissolve montage beats instead of hard-cutting. Run as
  SEPARATE passes then muxed (`xfadeVideoCommand` + `acrossfadeAudioCommand` + `muxCommand`) — one combined
  filtergraph starves/truncates the audio (that cut the music off early). Timeline compresses by (n-1)*xf;
  `shiftOverlayForCrossfade` re-times overlays and the reported duration is adjusted so overlays/loudness/QA
  stay aligned. Opt-in via `editing.crossfade-seconds` (0 = hard cuts default; localpoc 0.25).
- **Speed-ramp into slow-mo:** `speedRampSetpts` gives a slow-mo shot a log-integrated `setpts` that eases the
  playback speed from normal (1.0) down to below the target across the shot, so the payoff decelerates
  smoothly instead of snapping. Stays a SINGLE segment (R5 push-in preserved); the `-t` pin keeps the planned
  duration. Verified on bowling (payoff segment carries the ramp expression; output valid).

## R7 — Overlays: bold, animated, synced ✅
- **Rule:** captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background,
  with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. `drawTextFilter`.
- **Ties to Tier 2 (R9):** the vision director produces the caption *text* ("STRIKE"); R7 makes it *land*. The
  overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.

## R8 — Music dynamics: build to the payoff ✅
- **Rule:** because MusicGen's internal structure is uncontrolled, the mix applies a deterministic swell
  envelope to the score (`volume='min(1,0.5+0.5*t/peak)':eval=frame`) so the music amplitude rises from 0.5x to
  full over the run-up to the payoff, then holds. Driven only by the payoff beat's (crossfade-compressed)
  timeline position — generic for any source. In `audioMixCommand`; verified on bowling (peak at 6.83s).
- **Still open:** aligning MusicGen's *own* melodic climax (vs a volume swell) needs a controllable music
  model or a produced track (P5.3).

## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳
- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve
  (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup →
  continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax →
  resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled`
  (on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut.
- **Tier 2 (semantic VLM director, DONE 2026-07-23/24):** `HighlightVisionDirector` + `tools/vision_caption.py`
  run a local vision-language model (moondream2, offline). It now captions *several* beat frames (two questions
  each in one call: a discriminative description + a punchy label) and:
  1. **guides selection** — `semanticScore`/`semanticCurve` turn the descriptions into a per-window
     highlight-worthiness signal that the montage director blends with audio to place the payoff on the
     semantically-strongest moment (verified: on bowling the payoff moved onto moondream's detected
     celebration);
  2. **decorates** — the payoff label becomes the bold overlay and the description flavors the music.
  All generic: the scoring uses generic emotion/action/idle keywords (no content-specific terms), and it
  fails soft to the measured cut. ~25s/frame CPU; enabled by `vision-director-enabled` (localpoc on).
  **Honest limit:** a small VLM on distant subjects is only weakly discriminative — a terse question collapses
  to a constant answer (use descriptive questions); a stronger VLM or clearer framing would help.
  moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can.

## R10 — Cinematic cadence: 24 fps ✅
- **Why:** 24 fps (with a ~180° shutter) is the film-standard motion cadence used by the overwhelming majority
  of theatrical productions; 30 fps reads as "video". Output frame rate default moved **30 → 24**
  (`editing.output-frame-rate`, `VideoClippingProperties`). Generic, content-agnostic — one global knob.
- Verified: a styled segment renders `r_frame_rate=24/1`, `nb_frames = 24 × seconds`.

## R11 — Filmic grade: S-curve tone + optional licensed LUT ✅
- **Built-in grade (no external asset):** each beat now applies a **filmic tone curve** — a lifted toe
  (shadows never crushed) plus a rolled-off highlight **shoulder** (highlights compressed, not clipped) — which
  is the core of the "film look", on top of the teal-shadow / warm-highlight ("teal-orange") colour language.
  Replaces the old linear `curves=all='0/x 1/1'` ramp. Still **exposure-preserving** (mids ≈ 0.5, gamma ≥ 1):
  measured on bowling, source mean luma 112 → graded 121 (toward the 120 target), never crushed.
- **Optional film-emulation LUT (`editing.film-lut-path`):** when it points to a readable `.cube` that has an
  adjacent, non-blank `<name>.license.txt` sidecar (same provenance discipline as generated assets), the
  renderer applies it via `lut3d` and **suppresses** the built-in colour moves (no double-grading), keeping only
  sharpen + vignette. Empty by default → built-in grade; **fails closed** on a missing/unlicensed LUT, so no
  unlicensed asset is ever applied. This is the upgrade path to real film color science (e.g. Kodak 2383) once a
  commercially-licensed LUT is provisioned.

## R12 — Motion blur: shutter-angle emulation ✅
- **Why:** film's ~180° shutter produces natural motion blur; sharp digital frames strobe at 24 fps. Styled
  shots now blend each frame with its predecessor (`tmix=frames=2`, normalised so static frames are unchanged;
  only moving pixels gain a short trail). Toggle `editing.cinematic-motion-blur` (default on).
- **Honest limit:** on already-soft / low-resolution source (e.g. 576p phone footage) motion blur can further
  reduce perceived sharpness — it most helps crisp, high-shutter footage. Disable it for soft source.

## R13 — Beat-synced cuts: cut to the music ✅
- **Why:** cinematic edits cut on the beat; consumer AI editors (CapCut, DJI LightCut) beat-sync automatically.
  After the score is generated, `tools/beat_detect.py` (librosa) extracts its beat grid and `HighlightBeatSync`
  snaps each internal cut boundary onto the nearest beat (default ±0.18 s), never reordering shots, never
  changing the total length (the score still fits), and never shortening a shot below 0.5 s. Toggle
  `editing.beat-sync-enabled`. Runs in `HighlightDirectorFlowService` after asset generation, before render.
- **Fails soft:** no beats / unreadable track / missing interpreter → original cuts kept. Verified on bowling:
  194 BPM detected, boundaries {1.7, 5.6, 6.4, 9.26 s} all snapped onto beats.
- **Honest limit:** MusicGen tempo is loose, so beats can be irregular; the tolerance + fail-soft keep it safe.

## R14 — Subject-tracking reframe: follow the subject ✅
- **Why:** DJI/Insta360 "AI reframe" keeps the moving subject framed; a static centre crop lets them drift off.
  `tools/subject_track.py` (YOLO, CV venv) samples the dominant subject's normalised centre across each shot;
  `HighlightSubjectTracker` smooths it and encodes a `pan=` path into the shot treatment; the renderer
  (`subjectFollowFilter`) drives a `zoompan` crop that follows the path (interpolated, clamped inside frame),
  co-existing with the R5 push-in. Toggle `editing.subject-reframe-enabled`.
- **Fails soft:** sparse/no detections → centred crop. Verified on bowling: the bowler's centre tracked
  cx 0.54→0.44→0.68 across the frame.
- **Honest limits:** **YOLOv8 is AGPL-3.0 → non-commercial** (matches the repo's CV stance); per-frame detection
  is CPU-slow; on tiny/distant subjects detection is unreliable.

---
Rules R1–R14 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightBeatSync` /
`HighlightSubjectTracker` and apply to **every** project automatically (R13/R14 behind opt-in flags). Each is
driven by a source measurement or a global cinematic standard, never a per-video constant.

## Still missing for "cinematic" (researched gap — not yet implemented)
Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work:
- **Real film-emulation LUT** — R11 ships the `lut3d` hook but no licensed `.cube` is provisioned; the built-in
  filmic curve is the fallback. Dropping in a commercial LUT (e.g. Kodak 2383) is the biggest colour upgrade.
- **Shallow depth of field** — subject/background separation; hard to fake post-hoc without segmentation/depth.
- **Non-code ceilings:** higher-resolution source (576p upscales soft) and a produced/licensed music track are
  the two largest quality levers and are outside the renderer.