diff --git a/docs/cinematic-highlight-poc-plan.md b/docs/cinematic-highlight-poc-plan.md index 5412861..64ef8e4 100644 --- a/docs/cinematic-highlight-poc-plan.md +++ b/docs/cinematic-highlight-poc-plan.md @@ -47,21 +47,52 @@ numpy<2, transformers must be 4.x, audiocraft/xformers unusable here, `HF_HUB_DI **Phase 1 COMPLETE (2026-07-21).** ### Phase 2 — One approved end-to-end highlight (STOP before render for explicit approval) -- [ ] 2.1 Opt-in `localpoc` Spring profile (base/production defaults untouched): venv python, model paths, - offline flags, no bootstrap auto-start, heuristic fallback off, isolated PoC I/O dirs, render disabled. -- [ ] 2.2 Reprocess the DJI source so `category.json` + `highlight-candidates.json` are written. -- [ ] 2.3 Hand-author a director `edit-plan.json` grounded in persisted candidates (manual, local-only). -- [ ] 2.4 Explicit approval for the specific project → render → final + manifest + QA report. -- [ ] 2.5 Measure technical QA + structured human creative review. +- [x] 2.1 Opt-in `localpoc` Spring profile added (`application-localpoc.yml`): venv python, model paths, + offline, no bootstrap auto-start, heuristic visual, isolated PoC I/O dirs, render OFF, approval REQUIRED. +- [x] 2.2 DJI source processed under `localpoc` (app started, ~4 min 4K analysis): wrote `category.json` + (GENERIC_VLOG, conf 0.25) + `highlight-candidates.json` (3 windows: 27-39 s, 36-48 s, 81-93 s). + Actual footage = blue Porsche Taycan aerial orbit (EV car reveal). +- [x] 2.3 Hand-authored `director/edit-plan.json`: 3 x 9 s highlights (opening_hook / rising_energy / + hero_payoff), grounded in the visible car; passes ALL validator constraints (containment, speed 1.0, + duration bounds, VO budget, category match). Manual + local-only (no external AI). +- [x] 2.4 **RENDERED (operator-approved 2026-07-21).** render-enabled override + approved.flag → local models + generated 3 VO (Piper) + 3 music (MusicGen) + SFX (AudioLDM2) → 3 highlights rendered → project + `final.mp4` assembled (27.3 s, 51.7 MB). Flow ~8.6 min. All real local-model assets, no placeholders. +- [~] 2.5 QA measured (honest): + **PASS** — 1920x1080 / H.264 / 30 fps / AAC 48 kHz; duration 27.3 s; integrated loudness -15.8 LUFS + (target -16 ±1); no black frames; no long silence; overlays render & are grounded ("First light.", + "Silent power."); VO intelligible. + **FAIL/weak** — true peak: highlight_001 = 0.0 dBFS (clips; target ≤ -1.5 dBTP); h2/h3 = -1.4 dBFS + (loudnorm TP limiting not enforced tightly). Visual "treatment" is essentially PASSTHROUGH — no visible + cinematic grade/reframe (renderer maps any treatment to a generic look); overlays are plain white + captions. LRA 16.9 (high). Pipeline's own qa-report passed because its grade/clip checks are structural. + **Verdict: genuine end-to-end local PoC, NOT yet "genuinely cinematic."** Visual grade + true-peak + limiting are the top Phase 3 items. ### Phase 3 — Iterate to quality -- [ ] Improve selection, pacing, visual treatment, voice, music/SFX fit; keep experiments reproducible. +Ranked from the first render's evidence: +- [~] P3.1 Stronger filmic grade in `HighlightFfmpegRenderer.cinematicVisualFilter`: S-curve (curves + medium_contrast) + teal-orange colorbalance + eq + unsharp + vignette (replaces the weak fixed eq). + Verified visually (richer blue, warm highlights, contrast, vignette). NOTE: still ONE uniform look — + not yet beat/category-specific (that needs a validated grade enum plumbed via EditDecision). Only the + highlight renderer changed; multi-clip FfmpegEditRenderer untouched. +- [x] P3.2 True-peak safety: added `alimiter=level=disabled:limit=0.72` after loudnorm in the audio mix. + VERIFIED by re-render: project true peak 0.0 dBFS (clipping) → **-1.7 dBFS**; per-highlight -1.7/-2.7/ + -2.8 (all <= -1.5 dBTP gate); integrated loudness -16.3 LUFS. (First pass at limit=0.79 gave -1.3 on + h1, 0.2 dB over — tightened to 0.72 for inter-sample margin.) Test added. +- [ ] P3.3 Overlay styling: premium type, safe-area, subtle animation instead of plain white captions. +- [ ] P3.4 Real CV visual analysis (resident model, no bootstrap script) to improve candidate selection. +- [ ] P3.5 Music/SFX creative fit + ducking review; consider longer inference / better prompts. +- [ ] P3.6 Freeze acceptance thresholds + blinded human creative review vs baseline before declaring success. ## Milestone log - 2026-07-21: Models provisioned & individually proven (Piper, MusicGen, AudioLDM2). Plan approved. Phase 1 started. - 2026-07-21: **Phase 1 complete.** Worker rewritten (audiocraft→transformers MusicGen + diffusers AudioLDM2), models materialized to `models/` with license sidecars, all 3 asset kinds generate 48 kHz mono real audio through the worker, `mvn -o verify` green (245/0/0/0). Paused for review before Phase 2. +- 2026-07-21: Phase 1 committed (a95d1fa). **Phase 2 up to approval gate:** localpoc profile added, DJI source + processed → candidates + category, grounded 3-highlight director plan authored & pre-validated. + STOPPED — awaiting explicit render approval for dji_21230510013241_0091_d. ## Deferred (until output-quality gate passes) Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed diff --git a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java index 2f586c6..183456f 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java +++ b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java @@ -321,6 +321,11 @@ public class HighlightFfmpegRenderer { .append(properties.getLoudnessTargetI()) .append(":TP=").append(properties.getLoudnessTruePeak()) .append(":LRA=").append(properties.getLoudnessRange()); + // Brickwall safety limiter after loudnorm. loudnorm's true-peak control is predictive and can + // overshoot when several loud sources are mixed. alimiter caps SAMPLE peak, and inter-sample + // true peak runs ~0.5-1.0 dB higher, so hold the sample ceiling at 0.72 lin (~-2.85 dBFS) to + // keep measured true peak comfortably within the -1.5 dBTP gate. + filters.append(",alimiter=level=disabled:limit=0.72"); if (!sourceAudioPresent) { filters.append(",apad"); } @@ -517,9 +522,13 @@ public class HighlightFfmpegRenderer { if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) { return ""; } - return ",eq=contrast=1.12:saturation=1.18:brightness=-0.015" - + ",unsharp=5:5:0.55:3:3:0.25" - + ",vignette=PI/7"; + // Deliberate filmic grade: S-curve contrast, teal shadows / warm highlights (cinematic + // "teal-orange"), gentle saturation and gamma lift, light sharpen, and a soft vignette. + return ",curves=preset=medium_contrast" + + ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05" + + ",eq=contrast=1.12:saturation=1.12:gamma=0.97:brightness=-0.012" + + ",unsharp=5:5:0.4:3:3:0.2" + + ",vignette=PI/6"; } private List voiceoverInputs(EditPlan plan, Path audioDirectory) { diff --git a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java index 0d90e1d..88bc321 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java @@ -115,6 +115,8 @@ class HighlightFfmpegRendererTest { assertThat(command).noneMatch(argument -> argument.contains("[0:a]")); assertThat(command).anySatisfy(argument -> assertThat(argument) .contains("amix=inputs=1:duration=longest") + .contains("loudnorm=I=") + .contains(",alimiter=level=disabled:limit=0.72") .contains(",apad[a]")); assertThat(command).contains("-shortest"); }