video_editing_poc/docs/cinematic-quality-rules.md

Cinematic Highlight Quality Rules (source-adaptive)

Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle: every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant tuned to one clip. A fixed grade/geometry/level that looks right on one video is wrong on the next.

Legend: ✅ implemented · ⏳ planned (see cinematic-highlight-poc-plan.md P5.x).

R1 — Exposure: normalize to a target, never crush ✅

• Measure: sample source frames (ffmpeg signalstats YAVG, 2 fps, bounded) → mean luma. HighlightFfmpegRenderer.probeSourceLuma.
• Adapt: exposureNormalizationFilter picks a gamma that maps the measured mean → a target band (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
• Invariant: the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1, brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
• Evidence: bowling final went 63 → ~100 with the source-measured gamma 1.098.

R2 — Orientation: respect the source, never distort ✅

• Measure: read display rotation (FfmpegClipInspector side-data / rotate tag) → effective W×H.
• Adapt: outputGeometry renders portrait sources to a portrait frame, landscape to widescreen; the 2.39 letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.

R3 — Audio balance: the story audio must lead ✅

• Rule: the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with narration); with narration, voice leads via side-chain ducking. No element is silently buried.
• Accurate mastering: single-pass loudnorm is only ~±2 LUFS accurate, so the finished file is measured (probeIntegratedLoudness) and a corrective gain (loudnessGainDb → masterLoudness) is applied to hit the target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at −15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
• Verify: integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.

R4 — Duration: length follows the story ✅

• No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive duration + sane playback speed + candidate containment).

R5 — No dead air: motion in held shots ✅

• Measure: per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source range. HighlightFfmpegRenderer.probeSegmentMotion.
• Adapt: pushInAmount turns low motion into a stronger in-shot zoompan push-in and high motion into a gentle one (with a floor so every shot has a little life); applied before setpts so slow-motion survives.
• Evidence (bowling): static opening shot measured lowest motion → strongest push (0.117); active celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).

R6 — Transitions: ease, don't jerk ✅

• Crossfades: video xfade + audio acrossfade dissolve montage beats instead of hard-cutting. Run as SEPARATE passes then muxed (xfadeVideoCommand + acrossfadeAudioCommand + muxCommand) — one combined filtergraph starves/truncates the audio (that cut the music off early). Timeline compresses by (n-1)*xf; shiftOverlayForCrossfade re-times overlays and the reported duration is adjusted so overlays/loudness/QA stay aligned. Opt-in via editing.crossfade-seconds (0 = hard cuts default; localpoc 0.25).
• Speed-ramp into slow-mo: speedRampSetpts gives a slow-mo shot a log-integrated setpts that eases the playback speed from normal (1.0) down to below the target across the shot, so the payoff decelerates smoothly instead of snapping. Stays a SINGLE segment (R5 push-in preserved); the -t pin keeps the planned duration. Verified on bowling (payoff segment carries the ramp expression; output valid).

R7 — Overlays: bold, animated, synced ✅

• Rule: captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background, with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. drawTextFilter.
• Ties to Tier 2 (R9): the vision director produces the caption text ("STRIKE"); R7 makes it land. The overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.

R8 — Music dynamics: build to the payoff ✅

• Rule: because MusicGen's internal structure is uncontrolled, the mix applies a deterministic swell envelope to the score (volume='min(1,0.5+0.5*t/peak)':eval=frame) so the music amplitude rises from 0.5x to full over the run-up to the payoff, then holds. Driven only by the payoff beat's (crossfade-compressed) timeline position — generic for any source. In audioMixCommand; verified on bowling (peak at 6.83s).
• Still open: aligning MusicGen's own melodic climax (vs a volume swell) needs a controllable music model or a produced track (P5.3).

R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳

• Tier 1 (measurement director, DONE): HighlightMontageDirector measures a per-window motion curve (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup → continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax → resolution button, trimming a high-motion camera-whip tail. Enabled by highlight-scheduler.auto-director-enabled (on in localpoc); writes director/montage.json after analysis. Verified on bowling: auto cut ≈ the hand cut.
• Tier 2 (semantic VLM director, DONE 2026-07-23/24): HighlightVisionDirector + tools/vision_caption.py run a local vision-language model (moondream2, offline). It now captions several beat frames (two questions each in one call: a discriminative description + a punchy label) and:
  1. guides selection — semanticScore/semanticCurve turn the descriptions into a per-window highlight-worthiness signal that the montage director blends with audio to place the payoff on the semantically-strongest moment (verified: on bowling the payoff moved onto moondream's detected celebration);
  2. decorates — the payoff label becomes the bold overlay and the description flavors the music. All generic: the scoring uses generic emotion/action/idle keywords (no content-specific terms), and it fails soft to the measured cut. ~25s/frame CPU; enabled by vision-director-enabled (localpoc on). Honest limit: a small VLM on distant subjects is only weakly discriminative — a terse question collapses to a constant answer (use descriptive questions); a stronger VLM or clearer framing would help. moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
• Source caveat: a director can only cut what was filmed. If the camera never shows the pins, no tier can.

R10 — Cinematic cadence: 24 fps ✅

• Why: 24 fps (with a ~180° shutter) is the film-standard motion cadence used by the overwhelming majority of theatrical productions; 30 fps reads as "video". Output frame rate default moved 30 → 24 (editing.output-frame-rate, VideoClippingProperties). Generic, content-agnostic — one global knob.
• Verified: a styled segment renders r_frame_rate=24/1, nb_frames = 24 × seconds.

R11 — Filmic grade: S-curve tone + optional licensed LUT ✅

• Built-in grade (no external asset): each beat now applies a filmic tone curve — a lifted toe (shadows never crushed) plus a rolled-off highlight shoulder (highlights compressed, not clipped) — which is the core of the "film look", on top of the teal-shadow / warm-highlight ("teal-orange") colour language. Replaces the old linear curves=all='0/x 1/1' ramp. Still exposure-preserving (mids ≈ 0.5, gamma ≥ 1): measured on bowling, source mean luma 112 → graded 121 (toward the 120 target), never crushed.
• Optional film-emulation LUT (editing.film-lut-path): when it points to a readable .cube that has an adjacent, non-blank <name>.license.txt sidecar (same provenance discipline as generated assets), the renderer applies it via lut3d and suppresses the built-in colour moves (no double-grading), keeping only sharpen + vignette. Empty by default → built-in grade; fails closed on a missing/unlicensed LUT, so no unlicensed asset is ever applied. This is the upgrade path to real film color science (e.g. Kodak 2383) once a commercially-licensed LUT is provisioned.

R12 — Motion blur: shutter-angle emulation ✅

• Why: film's ~180° shutter produces natural motion blur; sharp digital frames strobe at 24 fps. Styled shots now blend each frame with its predecessor (tmix=frames=2, normalised so static frames are unchanged; only moving pixels gain a short trail). Toggle editing.cinematic-motion-blur (default on).
• Honest limit: on already-soft / low-resolution source (e.g. 576p phone footage) motion blur can further reduce perceived sharpness — it most helps crisp, high-shutter footage. Disable it for soft source.

R13 — Beat-synced cuts: cut to the music ✅

• Why: cinematic edits cut on the beat; consumer AI editors (CapCut, DJI LightCut) beat-sync automatically. After the score is generated, tools/beat_detect.py (librosa) extracts its beat grid and HighlightBeatSync snaps each internal cut boundary onto the nearest beat (default ±0.18 s), never reordering shots, never changing the total length (the score still fits), and never shortening a shot below 0.5 s. Toggle editing.beat-sync-enabled. Runs in HighlightDirectorFlowService after asset generation, before render.
• Fails soft: no beats / unreadable track / missing interpreter → original cuts kept. Verified on bowling: 194 BPM detected, boundaries {1.7, 5.6, 6.4, 9.26 s} all snapped onto beats.
• Honest limit: MusicGen tempo is loose, so beats can be irregular; the tolerance + fail-soft keep it safe.

R14 — Subject-tracking reframe: follow the subject ✅

• Why: DJI/Insta360 "AI reframe" keeps the moving subject framed; a static centre crop lets them drift off. tools/subject_track.py (YOLO, CV venv) samples the dominant subject's normalised centre across each shot; HighlightSubjectTracker smooths it and encodes a pan= path into the shot treatment; the renderer (subjectFollowFilter) drives a zoompan crop that follows the path (interpolated, clamped inside frame), co-existing with the R5 push-in. Toggle editing.subject-reframe-enabled.
• Fails soft: sparse/no detections → centred crop. Verified on bowling: the bowler's centre tracked cx 0.54→0.44→0.68 across the frame.
• Honest limits: YOLOv8 is AGPL-3.0 → non-commercial (matches the repo's CV stance); per-frame detection is CPU-slow; on tiny/distant subjects detection is unreliable.

R15 — Decisive moment: measurement proposes, the vision model judges ✅ (with a hard VLM ceiling)

A highlight is an action unit — a start, a decisive peak, and an outcome/reaction — and choosing which moment is the highlight is a question of meaning, not of motion or loudness. There is no generic rule in measurement alone: a camera turn-away or a loud aftermath routinely out-scores a quiet celebration on both motion and audio (measured on the real bowling clip: the turn-around has the clip's highest motion and is louder than the celebration). So the design separates the two responsibilities — no positional bands, no per-video thresholds, ever:

• Measurement PROPOSES (HighlightMontageDirector.candidatePeaks): the local maxima of intensity (normalised motion + audio), strongest first, min-separated. Every real event becomes a candidate — a strike, a celebration, a turn-away, a goal, an anticipation — with no opinion about which is the highlight.
• The vision model JUDGES (HighlightVisionDirector.rankDecisiveMoment): it captions each candidate and scores highlight-worthiness by meaning — a celebration or a scored goal outranks a loud turn-away or an "about to…" build (anticipation is explicitly not the payoff). The highest-scoring candidate wins.
• Fallback: if the model declines/fails, the strongest-intensity peak is used (a measured last resort).
• Segment (composeMontageAt): around the chosen peak, the onset (measured motion rising into it, build-capped) and the resolution (measured motion settling after it, sweeping in the outcome + reaction — the pins falling and the celebration; the ball crossing the line and settling in the net) are built.
• Honest labelling (R9/C): a dense read of the shown segment names it and never asserts an action the segment doesn't contain — an anticipatory cut becomes a teaser question, not a false "KICK".

Honest ceiling (verified, not theoretical): the judge is only as good as its eyes. The local moondream2 model reliably perceives some actions (soccer: "kicking a soccer ball" → the goal is chosen correctly) but cannot perceive others — on the distant, portrait bowling clip it describes every frame as "standing" / "walking" / "a bowling alley" and never sees the arms-raised celebration, regardless of prompt (a posture-focused prompt collapsed to a constant "Standing still"). When the model can't discriminate, all candidates tie and it falls back to the loudest peak (the turn-away). This is a model-capability limit, not a design flaw — the fix is a stronger local VLM, which is a drop-in: the judge is a clean interface (MomentChooser / rankDecisiveMoment) with no director changes required. Forcing the weak case with more heuristics is prohibited — that is the hack this rule exists to avoid.

---

Rules R1–R15 are live in HighlightFfmpegRenderer / HighlightDirectorFlowService / HighlightMontageDirector / HighlightVisionDirector / HighlightBeatSync / HighlightSubjectTracker and apply to every project automatically (R13/R14 behind opt-in flags). Each is driven by a source measurement or a global cinematic standard, never a per-video constant.

Still missing for "cinematic" (researched gap — not yet implemented)

Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work:

• Real film-emulation LUT — R11 ships the lut3d hook but no licensed .cube is provisioned; the built-in filmic curve is the fallback. Dropping in a commercial LUT (e.g. Kodak 2383) is the biggest colour upgrade.
• Shallow depth of field — subject/background separation; hard to fake post-hoc without segmentation/depth.
• Non-code ceilings: higher-resolution source (576p upscales soft) and a produced/licensed music track are the two largest quality levers and are outside the renderer.