82 lines
6.3 KiB
Markdown
82 lines
6.3 KiB
Markdown
# Cinematic Highlight Quality Rules (source-adaptive)
|
||
|
||
Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle:
|
||
**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant
|
||
tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next.
|
||
|
||
Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x).
|
||
|
||
## R1 — Exposure: normalize to a target, never crush ✅
|
||
- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma.
|
||
`HighlightFfmpegRenderer.probeSourceLuma`.
|
||
- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band
|
||
(~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
|
||
- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1,
|
||
brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
|
||
- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098.
|
||
|
||
## R2 — Orientation: respect the source, never distort ✅
|
||
- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H.
|
||
- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39
|
||
letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.
|
||
|
||
## R3 — Audio balance: the story audio must lead ✅
|
||
- **Rule:** the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with
|
||
narration); with narration, voice leads via side-chain ducking. No element is silently buried.
|
||
- **Accurate mastering:** single-pass `loudnorm` is only ~±2 LUFS accurate, so the finished file is measured
|
||
(`probeIntegratedLoudness`) and a corrective gain (`loudnessGainDb` → `masterLoudness`) is applied to hit the
|
||
target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at
|
||
−15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
|
||
- **Verify:** integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.
|
||
|
||
## R4 — Duration: length follows the story ✅
|
||
- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive
|
||
duration + sane playback speed + candidate containment).
|
||
|
||
## R5 — No dead air: motion in held shots ✅
|
||
- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source
|
||
range. `HighlightFfmpegRenderer.probeSegmentMotion`.
|
||
- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a
|
||
gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives.
|
||
- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active
|
||
celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).
|
||
|
||
## R6 — Transitions: ease, don't jerk — crossfades ✅ / speed-ramp ⏳
|
||
- **Crossfades (DONE):** `xfadeTimelineCommand` dissolves montage beats (video xfade + audio acrossfade)
|
||
instead of hard-cutting, incl. the cut into the slow-mo payoff. Timeline compresses by (n-1)*xf;
|
||
`shiftOverlayForCrossfade` re-times overlays and the reported duration is adjusted so overlays/loudness/QA
|
||
stay aligned. Opt-in via `editing.crossfade-seconds` (0 = hard cuts default; localpoc 0.25), clamped to
|
||
half the shortest beat. Verified on bowling (visible dissolve, overlay stayed on the payoff).
|
||
- **Speed-ramp into slow-mo (deferred):** easing the playback-speed change itself conflicts with the R5
|
||
per-shot push-in (sub-segment splitting would reset the zoom); needs a time-varying `setpts` or a push-in
|
||
that spans sub-segments. The crossfade already softens the *cut* into the payoff.
|
||
|
||
## R7 — Overlays: bold, animated, synced ✅
|
||
- **Rule:** captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background,
|
||
with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. `drawTextFilter`.
|
||
- **Ties to Tier 2 (R9):** the vision director produces the caption *text* ("STRIKE"); R7 makes it *land*. The
|
||
overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.
|
||
|
||
## R8 — Music sync: climax on the payoff ⏳ (P5.13)
|
||
- **Measure:** the payoff beat's timeline position.
|
||
- **Adapt:** prompt/trim the score so its peak lands on the payoff, not wherever the generator happened to put it.
|
||
|
||
## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳
|
||
- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve
|
||
(YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup →
|
||
continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax →
|
||
resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled`
|
||
(on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut.
|
||
- **Tier 2 (semantic VLM director, DONE 2026-07-23):** `HighlightVisionDirector` + `tools/vision_caption.py`
|
||
run a local vision-language model (moondream2, offline) to caption the payoff frame, then AUGMENT the Tier-1
|
||
montage with a semantic overlay and a scene-informed music direction. On bowling it read the celebration and
|
||
produced the overlay "STRIKE" and a scene-accurate music prompt — automatically. Enabled by
|
||
`highlight-scheduler.vision-director-enabled` (localpoc on); ~25s/frame on CPU; fails soft (Tier-1 stands).
|
||
moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
|
||
- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can.
|
||
|
||
---
|
||
Rules R1–R5 are live in `HighlightFfmpegRenderer` / `FfmpegClipInspector` / `HighlightDirectorPlanValidator`
|
||
and apply to **every** project automatically. R6–R9 are the next implementation targets; each must likewise be
|
||
driven by a source measurement, never a per-video constant.
|