video_editing_poc/docs/cinematic-quality-rules.md

# Cinematic Highlight Quality Rules (source-adaptive)

Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle:
**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant
tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next.

Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x).

## R1 — Exposure: normalize to a target, never crush ✅
- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma.
  `HighlightFfmpegRenderer.probeSourceLuma`.
- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band
  (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1,
  brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098.

## R2 — Orientation: respect the source, never distort ✅
- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H.
- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39
  letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.

## R3 — Audio balance: the story audio must lead ✅
- **Rule:** the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with
  narration); with narration, voice leads via side-chain ducking. No element is silently buried.
- **Accurate mastering:** single-pass `loudnorm` is only ~±2 LUFS accurate, so the finished file is measured
  (`probeIntegratedLoudness`) and a corrective gain (`loudnessGainDb` → `masterLoudness`) is applied to hit the
  target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at
  −15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
- **Verify:** integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.

## R4 — Duration: length follows the story ✅
- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive
  duration + sane playback speed + candidate containment).

## R5 — No dead air: motion in held shots ✅
- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source
  range. `HighlightFfmpegRenderer.probeSegmentMotion`.
- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a
  gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives.
- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active
  celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).

## R6 — Transitions: ease, don't jerk — crossfades ✅ / speed-ramp ⏳
- **Crossfades (DONE):** `xfadeTimelineCommand` dissolves montage beats (video xfade + audio acrossfade)
  instead of hard-cutting, incl. the cut into the slow-mo payoff. Timeline compresses by (n-1)*xf;
  `shiftOverlayForCrossfade` re-times overlays and the reported duration is adjusted so overlays/loudness/QA
  stay aligned. Opt-in via `editing.crossfade-seconds` (0 = hard cuts default; localpoc 0.25), clamped to
  half the shortest beat. Verified on bowling (visible dissolve, overlay stayed on the payoff).
- **Speed-ramp into slow-mo (deferred):** easing the playback-speed change itself conflicts with the R5
  per-shot push-in (sub-segment splitting would reset the zoom); needs a time-varying `setpts` or a push-in
  that spans sub-segments. The crossfade already softens the *cut* into the payoff.

## R7 — Overlays: bold, animated, synced ✅
- **Rule:** captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background,
  with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. `drawTextFilter`.
- **Ties to Tier 2 (R9):** the vision director produces the caption *text* ("STRIKE"); R7 makes it *land*. The
  overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.

## R8 — Music dynamics: build to the payoff ✅
- **Rule:** because MusicGen's internal structure is uncontrolled, the mix applies a deterministic swell
  envelope to the score (`volume='min(1,0.5+0.5*t/peak)':eval=frame`) so the music amplitude rises from 0.5x to
  full over the run-up to the payoff, then holds. Driven only by the payoff beat's (crossfade-compressed)
  timeline position — generic for any source. In `audioMixCommand`; verified on bowling (peak at 6.83s).
- **Still open:** aligning MusicGen's *own* melodic climax (vs a volume swell) needs a controllable music
  model or a produced track (P5.3).

## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳
- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve
  (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup →
  continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax →
  resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled`
  (on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut.
- **Tier 2 (semantic VLM director, DONE 2026-07-23/24):** `HighlightVisionDirector` + `tools/vision_caption.py`
  run a local vision-language model (moondream2, offline). It now captions *several* beat frames (two questions
  each in one call: a discriminative description + a punchy label) and:
  1. **guides selection** — `semanticScore`/`semanticCurve` turn the descriptions into a per-window
     highlight-worthiness signal that the montage director blends with audio to place the payoff on the
     semantically-strongest moment (verified: on bowling the payoff moved onto moondream's detected
     celebration);
  2. **decorates** — the payoff label becomes the bold overlay and the description flavors the music.
  All generic: the scoring uses generic emotion/action/idle keywords (no content-specific terms), and it
  fails soft to the measured cut. ~25s/frame CPU; enabled by `vision-director-enabled` (localpoc on).
  **Honest limit:** a small VLM on distant subjects is only weakly discriminative — a terse question collapses
  to a constant answer (use descriptive questions); a stronger VLM or clearer framing would help.
  moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can.

---
Rules R1–R5 are live in `HighlightFfmpegRenderer` / `FfmpegClipInspector` / `HighlightDirectorPlanValidator`
and apply to **every** project automatically. R6–R9 are the next implementation targets; each must likewise be
driven by a source measurement, never a per-video constant.