JSL_Digital_Solution
/
video_editing_poc
forked from jsl/video_editing_poc
2
0
Fork 0
Code Pull Requests Activity
video_editing_poc/docs/cinematic-quality-rules.md

5.3 KiB
Raw Blame History

Cinematic Highlight Quality Rules (source-adaptive)

Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle: every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant tuned to one clip. A fixed grade/geometry/level that looks right on one video is wrong on the next.

Legend: ✅ implemented · ⏳ planned (see cinematic-highlight-poc-plan.md P5.x).

R1 — Exposure: normalize to a target, never crush ✅

  • Measure: sample source frames (ffmpeg signalstats YAVG, 2 fps, bounded) → mean luma. HighlightFfmpegRenderer.probeSourceLuma.
  • Adapt: exposureNormalizationFilter picks a gamma that maps the measured mean → a target band (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
  • Invariant: the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1, brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
  • Evidence: bowling final went 63 → ~100 with the source-measured gamma 1.098.

R2 — Orientation: respect the source, never distort ✅

  • Measure: read display rotation (FfmpegClipInspector side-data / rotate tag) → effective W×H.
  • Adapt: outputGeometry renders portrait sources to a portrait frame, landscape to widescreen; the 2.39 letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.

R3 — Audio balance: the story audio must lead ✅

  • Rule: the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with narration); with narration, voice leads via side-chain ducking. No element is silently buried.
  • Accurate mastering: single-pass loudnorm is only ~±2 LUFS accurate, so the finished file is measured (probeIntegratedLoudness) and a corrective gain (loudnessGainDb → masterLoudness) is applied to hit the target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at −15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
  • Verify: integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.

R4 — Duration: length follows the story ✅

  • No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive duration + sane playback speed + candidate containment).

R5 — No dead air: motion in held shots ✅

  • Measure: per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source range. HighlightFfmpegRenderer.probeSegmentMotion.
  • Adapt: pushInAmount turns low motion into a stronger in-shot zoompan push-in and high motion into a gentle one (with a floor so every shot has a little life); applied before setpts so slow-motion survives.
  • Evidence (bowling): static opening shot measured lowest motion → strongest push (0.117); active celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).

R6 — Transitions: ease, don't jerk ⏳ (P5.11)

  • Ramp speed into a slow-motion beat (don't hard-switch playback speed); crossfade on beat boundaries where the cut isn't meant to be a hard cut.

R7 — Overlays: bold, animated, synced ⏳ (P5.12)

  • Title/label text scaled to the frame, with an entrance (scale/fade) timed to the musical/edit accent — not small static text dropped on screen.

R8 — Music sync: climax on the payoff ⏳ (P5.13)

  • Measure: the payoff beat's timeline position.
  • Adapt: prompt/trim the score so its peak lands on the payoff, not wherever the generator happened to put it.

R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳

  • Tier 1 (measurement director, DONE): HighlightMontageDirector measures a per-window motion curve (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup → continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax → resolution button, trimming a high-motion camera-whip tail. Enabled by highlight-scheduler.auto-director-enabled (on in localpoc); writes director/montage.json after analysis. Verified on bowling: auto cut ≈ the hand cut.
  • Tier 2 (semantic VLM director, DONE 2026-07-23): HighlightVisionDirector + tools/vision_caption.py run a local vision-language model (moondream2, offline) to caption the payoff frame, then AUGMENT the Tier-1 montage with a semantic overlay and a scene-informed music direction. On bowling it read the celebration and produced the overlay "STRIKE" and a scene-accurate music prompt — automatically. Enabled by highlight-scheduler.vision-director-enabled (localpoc on); ~25s/frame on CPU; fails soft (Tier-1 stands). moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
  • Source caveat: a director can only cut what was filmed. If the camera never shows the pins, no tier can.

Rules R1–R5 are live in HighlightFfmpegRenderer / FfmpegClipInspector / HighlightDirectorPlanValidator and apply to every project automatically. R6–R9 are the next implementation targets; each must likewise be driven by a source measurement, never a per-video constant.