jsl
/
video_editing_poc
1
0
Fork 1
Code Issues Pull Requests Packages Projects Releases Wiki Activity
video_editing_poc/docs/cinematic-quality-rules.md

12 KiB
Raw Blame History

Cinematic Highlight Quality Rules (source-adaptive)

Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle: every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant tuned to one clip. A fixed grade/geometry/level that looks right on one video is wrong on the next.

Legend: ✅ implemented · ⏳ planned (see cinematic-highlight-poc-plan.md P5.x).

R1 — Exposure: normalize to a target, never crush ✅

  • Measure: sample source frames (ffmpeg signalstats YAVG, 2 fps, bounded) → mean luma. HighlightFfmpegRenderer.probeSourceLuma.
  • Adapt: exposureNormalizationFilter picks a gamma that maps the measured mean → a target band (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
  • Invariant: the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1, brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
  • Evidence: bowling final went 63 → ~100 with the source-measured gamma 1.098.

R2 — Orientation: respect the source, never distort ✅

  • Measure: read display rotation (FfmpegClipInspector side-data / rotate tag) → effective W×H.
  • Adapt: outputGeometry renders portrait sources to a portrait frame, landscape to widescreen; the 2.39 letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.

R3 — Audio balance: the story audio must lead ✅

  • Rule: the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with narration); with narration, voice leads via side-chain ducking. No element is silently buried.
  • Accurate mastering: single-pass loudnorm is only ~±2 LUFS accurate, so the finished file is measured (probeIntegratedLoudness) and a corrective gain (loudnessGainDb → masterLoudness) is applied to hit the target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at −15.5 needs no fix; one at −18.7 is boosted). Handles MusicGen loudness variance.
  • Verify: integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.

R4 — Duration: length follows the story ✅

  • No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive duration + sane playback speed + candidate containment).

R5 — No dead air: motion in held shots ✅

  • Measure: per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source range. HighlightFfmpegRenderer.probeSegmentMotion.
  • Adapt: pushInAmount turns low motion into a stronger in-shot zoompan push-in and high motion into a gentle one (with a floor so every shot has a little life); applied before setpts so slow-motion survives.
  • Evidence (bowling): static opening shot measured lowest motion → strongest push (0.117); active celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).

R6 — Transitions: ease, don't jerk ✅

  • Crossfades: video xfade + audio acrossfade dissolve montage beats instead of hard-cutting. Run as SEPARATE passes then muxed (xfadeVideoCommand + acrossfadeAudioCommand + muxCommand) — one combined filtergraph starves/truncates the audio (that cut the music off early). Timeline compresses by (n-1)*xf; shiftOverlayForCrossfade re-times overlays and the reported duration is adjusted so overlays/loudness/QA stay aligned. Opt-in via editing.crossfade-seconds (0 = hard cuts default; localpoc 0.25).
  • Speed-ramp into slow-mo: speedRampSetpts gives a slow-mo shot a log-integrated setpts that eases the playback speed from normal (1.0) down to below the target across the shot, so the payoff decelerates smoothly instead of snapping. Stays a SINGLE segment (R5 push-in preserved); the -t pin keeps the planned duration. Verified on bowling (payoff segment carries the ramp expression; output valid).

R7 — Overlays: bold, animated, synced ✅

  • Rule: captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background, with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. drawTextFilter.
  • Ties to Tier 2 (R9): the vision director produces the caption text ("STRIKE"); R7 makes it land. The overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.

R8 — Music dynamics: build to the payoff ✅

  • Rule: because MusicGen's internal structure is uncontrolled, the mix applies a deterministic swell envelope to the score (volume='min(1,0.5+0.5*t/peak)':eval=frame) so the music amplitude rises from 0.5x to full over the run-up to the payoff, then holds. Driven only by the payoff beat's (crossfade-compressed) timeline position — generic for any source. In audioMixCommand; verified on bowling (peak at 6.83s).
  • Still open: aligning MusicGen's own melodic climax (vs a volume swell) needs a controllable music model or a produced track (P5.3).

R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳

  • Tier 1 (measurement director, DONE): HighlightMontageDirector measures a per-window motion curve (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup → continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax → resolution button, trimming a high-motion camera-whip tail. Enabled by highlight-scheduler.auto-director-enabled (on in localpoc); writes director/montage.json after analysis. Verified on bowling: auto cut ≈ the hand cut.
  • Tier 2 (semantic VLM director, DONE 2026-07-23/24): HighlightVisionDirector + tools/vision_caption.py run a local vision-language model (moondream2, offline). It now captions several beat frames (two questions each in one call: a discriminative description + a punchy label) and:
    1. guides selection — semanticScore/semanticCurve turn the descriptions into a per-window highlight-worthiness signal that the montage director blends with audio to place the payoff on the semantically-strongest moment (verified: on bowling the payoff moved onto moondream's detected celebration);
    2. decorates — the payoff label becomes the bold overlay and the description flavors the music. All generic: the scoring uses generic emotion/action/idle keywords (no content-specific terms), and it fails soft to the measured cut. ~25s/frame CPU; enabled by vision-director-enabled (localpoc on). Honest limit: a small VLM on distant subjects is only weakly discriminative — a terse question collapses to a constant answer (use descriptive questions); a stronger VLM or clearer framing would help. moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
  • Source caveat: a director can only cut what was filmed. If the camera never shows the pins, no tier can.

R10 — Cinematic cadence: 24 fps ✅

  • Why: 24 fps (with a ~180° shutter) is the film-standard motion cadence used by the overwhelming majority of theatrical productions; 30 fps reads as "video". Output frame rate default moved 30 → 24 (editing.output-frame-rate, VideoClippingProperties). Generic, content-agnostic — one global knob.
  • Verified: a styled segment renders r_frame_rate=24/1, nb_frames = 24 × seconds.

R11 — Filmic grade: S-curve tone + optional licensed LUT ✅

  • Built-in grade (no external asset): each beat now applies a filmic tone curve — a lifted toe (shadows never crushed) plus a rolled-off highlight shoulder (highlights compressed, not clipped) — which is the core of the "film look", on top of the teal-shadow / warm-highlight ("teal-orange") colour language. Replaces the old linear curves=all='0/x 1/1' ramp. Still exposure-preserving (mids ≈ 0.5, gamma ≥ 1): measured on bowling, source mean luma 112 → graded 121 (toward the 120 target), never crushed.
  • Optional film-emulation LUT (editing.film-lut-path): when it points to a readable .cube that has an adjacent, non-blank <name>.license.txt sidecar (same provenance discipline as generated assets), the renderer applies it via lut3d and suppresses the built-in colour moves (no double-grading), keeping only sharpen + vignette. Empty by default → built-in grade; fails closed on a missing/unlicensed LUT, so no unlicensed asset is ever applied. This is the upgrade path to real film color science (e.g. Kodak 2383) once a commercially-licensed LUT is provisioned.

R12 — Motion blur: shutter-angle emulation ✅

  • Why: film's ~180° shutter produces natural motion blur; sharp digital frames strobe at 24 fps. Styled shots now blend each frame with its predecessor (tmix=frames=2, normalised so static frames are unchanged; only moving pixels gain a short trail). Toggle editing.cinematic-motion-blur (default on).
  • Honest limit: on already-soft / low-resolution source (e.g. 576p phone footage) motion blur can further reduce perceived sharpness — it most helps crisp, high-shutter footage. Disable it for soft source.

R13 — Beat-synced cuts: cut to the music ✅

  • Why: cinematic edits cut on the beat; consumer AI editors (CapCut, DJI LightCut) beat-sync automatically. After the score is generated, tools/beat_detect.py (librosa) extracts its beat grid and HighlightBeatSync snaps each internal cut boundary onto the nearest beat (default ±0.18 s), never reordering shots, never changing the total length (the score still fits), and never shortening a shot below 0.5 s. Toggle editing.beat-sync-enabled. Runs in HighlightDirectorFlowService after asset generation, before render.
  • Fails soft: no beats / unreadable track / missing interpreter → original cuts kept. Verified on bowling: 194 BPM detected, boundaries {1.7, 5.6, 6.4, 9.26 s} all snapped onto beats.
  • Honest limit: MusicGen tempo is loose, so beats can be irregular; the tolerance + fail-soft keep it safe.

R14 — Subject-tracking reframe: follow the subject ✅

  • Why: DJI/Insta360 "AI reframe" keeps the moving subject framed; a static centre crop lets them drift off. tools/subject_track.py (YOLO, CV venv) samples the dominant subject's normalised centre across each shot; HighlightSubjectTracker smooths it and encodes a pan= path into the shot treatment; the renderer (subjectFollowFilter) drives a zoompan crop that follows the path (interpolated, clamped inside frame), co-existing with the R5 push-in. Toggle editing.subject-reframe-enabled.
  • Fails soft: sparse/no detections → centred crop. Verified on bowling: the bowler's centre tracked cx 0.54→0.44→0.68 across the frame.
  • Honest limits: YOLOv8 is AGPL-3.0 → non-commercial (matches the repo's CV stance); per-frame detection is CPU-slow; on tiny/distant subjects detection is unreliable.

Rules R1–R14 are live in HighlightFfmpegRenderer / HighlightDirectorFlowService / HighlightBeatSync / HighlightSubjectTracker and apply to every project automatically (R13/R14 behind opt-in flags). Each is driven by a source measurement or a global cinematic standard, never a per-video constant.

Still missing for "cinematic" (researched gap — not yet implemented)

Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work:

  • Real film-emulation LUT — R11 ships the lut3d hook but no licensed .cube is provisioned; the built-in filmic curve is the fallback. Dropping in a commercial LUT (e.g. Kodak 2383) is the biggest colour upgrade.
  • Shallow depth of field — subject/background separation; hard to fake post-hoc without segmentation/depth.
  • Non-code ceilings: higher-resolution source (576p upscales soft) and a produced/licensed music track are the two largest quality levers and are outside the renderer.