Improve highlight render: filmic grade + true-peak limiter
Phase 3 quality fixes for the highlight renderer (multi-clip FfmpegEditRenderer left untouched): - cinematicVisualFilter: replace the weak fixed eq with a deliberate filmic grade (curves medium_contrast S-curve + teal-orange colorbalance + eq + unsharp + vignette). Richer blue, warm highlights, tonal contrast. Still one uniform look; beat/category-specific grading is future work (needs a validated grade enum). - audio mix: add a brickwall limiter (alimiter limit=0.72) after loudnorm. The first render clipped at 0.0 dBFS true peak; measured re-render now lands -1.7/-2.7/-2.8 dBFS per highlight (all within the -1.5 dBTP gate), integrated loudness -16.3 LUFS. Test updated; mvn -o verify green (245 tests, 0 failures). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
This commit is contained in:
parent
588c652a2f
commit
0fcfca9283
|
|
@ -47,21 +47,52 @@ numpy<2, transformers must be 4.x, audiocraft/xformers unusable here, `HF_HUB_DI
|
||||||
**Phase 1 COMPLETE (2026-07-21).**
|
**Phase 1 COMPLETE (2026-07-21).**
|
||||||
|
|
||||||
### Phase 2 — One approved end-to-end highlight (STOP before render for explicit approval)
|
### Phase 2 — One approved end-to-end highlight (STOP before render for explicit approval)
|
||||||
- [ ] 2.1 Opt-in `localpoc` Spring profile (base/production defaults untouched): venv python, model paths,
|
- [x] 2.1 Opt-in `localpoc` Spring profile added (`application-localpoc.yml`): venv python, model paths,
|
||||||
offline flags, no bootstrap auto-start, heuristic fallback off, isolated PoC I/O dirs, render disabled.
|
offline, no bootstrap auto-start, heuristic visual, isolated PoC I/O dirs, render OFF, approval REQUIRED.
|
||||||
- [ ] 2.2 Reprocess the DJI source so `category.json` + `highlight-candidates.json` are written.
|
- [x] 2.2 DJI source processed under `localpoc` (app started, ~4 min 4K analysis): wrote `category.json`
|
||||||
- [ ] 2.3 Hand-author a director `edit-plan.json` grounded in persisted candidates (manual, local-only).
|
(GENERIC_VLOG, conf 0.25) + `highlight-candidates.json` (3 windows: 27-39 s, 36-48 s, 81-93 s).
|
||||||
- [ ] 2.4 Explicit approval for the specific project → render → final + manifest + QA report.
|
Actual footage = blue Porsche Taycan aerial orbit (EV car reveal).
|
||||||
- [ ] 2.5 Measure technical QA + structured human creative review.
|
- [x] 2.3 Hand-authored `director/edit-plan.json`: 3 x 9 s highlights (opening_hook / rising_energy /
|
||||||
|
hero_payoff), grounded in the visible car; passes ALL validator constraints (containment, speed 1.0,
|
||||||
|
duration bounds, VO budget, category match). Manual + local-only (no external AI).
|
||||||
|
- [x] 2.4 **RENDERED (operator-approved 2026-07-21).** render-enabled override + approved.flag → local models
|
||||||
|
generated 3 VO (Piper) + 3 music (MusicGen) + SFX (AudioLDM2) → 3 highlights rendered → project
|
||||||
|
`final.mp4` assembled (27.3 s, 51.7 MB). Flow ~8.6 min. All real local-model assets, no placeholders.
|
||||||
|
- [~] 2.5 QA measured (honest):
|
||||||
|
**PASS** — 1920x1080 / H.264 / 30 fps / AAC 48 kHz; duration 27.3 s; integrated loudness -15.8 LUFS
|
||||||
|
(target -16 ±1); no black frames; no long silence; overlays render & are grounded ("First light.",
|
||||||
|
"Silent power."); VO intelligible.
|
||||||
|
**FAIL/weak** — true peak: highlight_001 = 0.0 dBFS (clips; target ≤ -1.5 dBTP); h2/h3 = -1.4 dBFS
|
||||||
|
(loudnorm TP limiting not enforced tightly). Visual "treatment" is essentially PASSTHROUGH — no visible
|
||||||
|
cinematic grade/reframe (renderer maps any treatment to a generic look); overlays are plain white
|
||||||
|
captions. LRA 16.9 (high). Pipeline's own qa-report passed because its grade/clip checks are structural.
|
||||||
|
**Verdict: genuine end-to-end local PoC, NOT yet "genuinely cinematic."** Visual grade + true-peak
|
||||||
|
limiting are the top Phase 3 items.
|
||||||
|
|
||||||
### Phase 3 — Iterate to quality
|
### Phase 3 — Iterate to quality
|
||||||
- [ ] Improve selection, pacing, visual treatment, voice, music/SFX fit; keep experiments reproducible.
|
Ranked from the first render's evidence:
|
||||||
|
- [~] P3.1 Stronger filmic grade in `HighlightFfmpegRenderer.cinematicVisualFilter`: S-curve (curves
|
||||||
|
medium_contrast) + teal-orange colorbalance + eq + unsharp + vignette (replaces the weak fixed eq).
|
||||||
|
Verified visually (richer blue, warm highlights, contrast, vignette). NOTE: still ONE uniform look —
|
||||||
|
not yet beat/category-specific (that needs a validated grade enum plumbed via EditDecision). Only the
|
||||||
|
highlight renderer changed; multi-clip FfmpegEditRenderer untouched.
|
||||||
|
- [x] P3.2 True-peak safety: added `alimiter=level=disabled:limit=0.72` after loudnorm in the audio mix.
|
||||||
|
VERIFIED by re-render: project true peak 0.0 dBFS (clipping) → **-1.7 dBFS**; per-highlight -1.7/-2.7/
|
||||||
|
-2.8 (all <= -1.5 dBTP gate); integrated loudness -16.3 LUFS. (First pass at limit=0.79 gave -1.3 on
|
||||||
|
h1, 0.2 dB over — tightened to 0.72 for inter-sample margin.) Test added.
|
||||||
|
- [ ] P3.3 Overlay styling: premium type, safe-area, subtle animation instead of plain white captions.
|
||||||
|
- [ ] P3.4 Real CV visual analysis (resident model, no bootstrap script) to improve candidate selection.
|
||||||
|
- [ ] P3.5 Music/SFX creative fit + ducking review; consider longer inference / better prompts.
|
||||||
|
- [ ] P3.6 Freeze acceptance thresholds + blinded human creative review vs baseline before declaring success.
|
||||||
|
|
||||||
## Milestone log
|
## Milestone log
|
||||||
- 2026-07-21: Models provisioned & individually proven (Piper, MusicGen, AudioLDM2). Plan approved. Phase 1 started.
|
- 2026-07-21: Models provisioned & individually proven (Piper, MusicGen, AudioLDM2). Plan approved. Phase 1 started.
|
||||||
- 2026-07-21: **Phase 1 complete.** Worker rewritten (audiocraft→transformers MusicGen + diffusers AudioLDM2),
|
- 2026-07-21: **Phase 1 complete.** Worker rewritten (audiocraft→transformers MusicGen + diffusers AudioLDM2),
|
||||||
models materialized to `models/` with license sidecars, all 3 asset kinds generate 48 kHz mono real audio
|
models materialized to `models/` with license sidecars, all 3 asset kinds generate 48 kHz mono real audio
|
||||||
through the worker, `mvn -o verify` green (245/0/0/0). Paused for review before Phase 2.
|
through the worker, `mvn -o verify` green (245/0/0/0). Paused for review before Phase 2.
|
||||||
|
- 2026-07-21: Phase 1 committed (a95d1fa). **Phase 2 up to approval gate:** localpoc profile added, DJI source
|
||||||
|
processed → candidates + category, grounded 3-highlight director plan authored & pre-validated.
|
||||||
|
STOPPED — awaiting explicit render approval for dji_21230510013241_0091_d.
|
||||||
|
|
||||||
## Deferred (until output-quality gate passes)
|
## Deferred (until output-quality gate passes)
|
||||||
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
|
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
|
||||||
|
|
|
||||||
|
|
@ -321,6 +321,11 @@ public class HighlightFfmpegRenderer {
|
||||||
.append(properties.getLoudnessTargetI())
|
.append(properties.getLoudnessTargetI())
|
||||||
.append(":TP=").append(properties.getLoudnessTruePeak())
|
.append(":TP=").append(properties.getLoudnessTruePeak())
|
||||||
.append(":LRA=").append(properties.getLoudnessRange());
|
.append(":LRA=").append(properties.getLoudnessRange());
|
||||||
|
// Brickwall safety limiter after loudnorm. loudnorm's true-peak control is predictive and can
|
||||||
|
// overshoot when several loud sources are mixed. alimiter caps SAMPLE peak, and inter-sample
|
||||||
|
// true peak runs ~0.5-1.0 dB higher, so hold the sample ceiling at 0.72 lin (~-2.85 dBFS) to
|
||||||
|
// keep measured true peak comfortably within the -1.5 dBTP gate.
|
||||||
|
filters.append(",alimiter=level=disabled:limit=0.72");
|
||||||
if (!sourceAudioPresent) {
|
if (!sourceAudioPresent) {
|
||||||
filters.append(",apad");
|
filters.append(",apad");
|
||||||
}
|
}
|
||||||
|
|
@ -517,9 +522,13 @@ public class HighlightFfmpegRenderer {
|
||||||
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||||
return "";
|
return "";
|
||||||
}
|
}
|
||||||
return ",eq=contrast=1.12:saturation=1.18:brightness=-0.015"
|
// Deliberate filmic grade: S-curve contrast, teal shadows / warm highlights (cinematic
|
||||||
+ ",unsharp=5:5:0.55:3:3:0.25"
|
// "teal-orange"), gentle saturation and gamma lift, light sharpen, and a soft vignette.
|
||||||
+ ",vignette=PI/7";
|
return ",curves=preset=medium_contrast"
|
||||||
|
+ ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05"
|
||||||
|
+ ",eq=contrast=1.12:saturation=1.12:gamma=0.97:brightness=-0.012"
|
||||||
|
+ ",unsharp=5:5:0.4:3:3:0.2"
|
||||||
|
+ ",vignette=PI/6";
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<VoiceoverInput> voiceoverInputs(EditPlan plan, Path audioDirectory) {
|
private List<VoiceoverInput> voiceoverInputs(EditPlan plan, Path audioDirectory) {
|
||||||
|
|
|
||||||
|
|
@ -115,6 +115,8 @@ class HighlightFfmpegRendererTest {
|
||||||
assertThat(command).noneMatch(argument -> argument.contains("[0:a]"));
|
assertThat(command).noneMatch(argument -> argument.contains("[0:a]"));
|
||||||
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
||||||
.contains("amix=inputs=1:duration=longest")
|
.contains("amix=inputs=1:duration=longest")
|
||||||
|
.contains("loudnorm=I=")
|
||||||
|
.contains(",alimiter=level=disabled:limit=0.72")
|
||||||
.contains(",apad[a]"));
|
.contains(",apad[a]"));
|
||||||
assertThat(command).contains("-shortest");
|
assertThat(command).contains("-shortest");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue