112 lines
8.2 KiB
Markdown
112 lines
8.2 KiB
Markdown
# Local Cinematic Highlight PoC — Plan & Milestones
|
||
|
||
Owner: (Jaden) · Started: 2026-07-21 · Machine: Intel Mac (x86_64, 16 CPU, 32 GB, no GPU)
|
||
|
||
## Objective
|
||
|
||
Produce ONE genuinely cinematic highlight from a local source video, end-to-end through the existing
|
||
pipeline, using only local models resident in the service runtime:
|
||
Piper (voiceover) + MusicGen (music) + AudioLDM2 (SFX), with deliberate visual treatment and pacing,
|
||
and explicit human approval before any render.
|
||
|
||
A valid MP4 or a passing test is NOT success. Success = representative output passes measured media QA
|
||
and a human creative review.
|
||
|
||
## Non-negotiables (still in force during the PoC)
|
||
|
||
No external AI services in the media path · no placeholder silence/tones/OS `say` as a finished asset ·
|
||
no unlicensed assets · no rendering without explicit approval for the specific project · no inference-time
|
||
network (models load from local dirs, `HF_HUB_OFFLINE=1`). NOTE: the user authorized model DOWNLOADS on
|
||
2026-07-21 (one-time provisioning), reversing the earlier no-download stance; inference stays offline.
|
||
|
||
## Proven capability baseline (2026-07-21)
|
||
|
||
| Model | Runtime | Status | Evidence |
|
||
|---|---|---|---|
|
||
| Piper `en_US-lessac-medium` | piper-tts 1.5.0 | ✅ works | real speech, 22.05 kHz |
|
||
| MusicGen `musicgen-small` | transformers 4.44.2 + torch 2.2.2 | ✅ works | 5 s / 44.7 s CPU, mean −18 dB, 32 kHz |
|
||
| AudioLDM2 `cvssp/audioldm2` | diffusers 0.30.3 | ✅ works | 3 s / 22.3 s CPU, mean −21.8 dB, 16 kHz |
|
||
|
||
Environment traps (see memory `local-model-runtime-intel-mac`): torch capped at 2.2.2 (Intel-Mac),
|
||
numpy<2, transformers must be 4.x, audiocraft/xformers unusable here, `HF_HUB_DISABLE_XET=1` for downloads.
|
||
|
||
## Plan
|
||
|
||
### Phase 1 — Generate all three assets THROUGH the pipeline
|
||
- [x] 1.1 Rewrite `tools/local_asset_worker.py`: music→transformers MusicGen, sfx→diffusers AudioLDM2
|
||
(resample to 48 kHz mono), voiceover→Piper (unchanged). audiocraft path removed. CLI + exit codes preserved.
|
||
- [x] 1.2 Materialize models into stable `models/` dirs via `save_pretrained` (`tools/provision_local_models.py`):
|
||
`models/musicgen-small` (2.2 G), `models/audioldm2` (4.2 G); `models/piper` (60 M) already present.
|
||
- [x] 1.3 License/provenance `.license.txt` sidecars for each model marker file (config.json / model_index.json /
|
||
voice .onnx). BOTH audio models are NON-COMMERCIAL (MusicGen CC-BY-NC-4.0, AudioLDM2 CC-BY-NC-SA-4.0) —
|
||
flagged for production review. Honors the `AssetLicensePolicy` regular-file gate; no Java change.
|
||
- [x] 1.4 Smoke test: voiceover 3.67 s/−15.8 dB, music 4.94 s/−12.0 dB, sfx 3.00 s/−20.6 dB — all 48 kHz mono,
|
||
real signal, exit 0.
|
||
- [x] 1.5 `mvn -o verify` → 245 tests / 62 classes / 0 failures / 0 errors / 0 skips (unchanged from baseline).
|
||
|
||
**Phase 1 COMPLETE (2026-07-21).**
|
||
|
||
### Phase 2 — One approved end-to-end highlight (STOP before render for explicit approval)
|
||
- [x] 2.1 Opt-in `localpoc` Spring profile added (`application-localpoc.yml`): venv python, model paths,
|
||
offline, no bootstrap auto-start, heuristic visual, isolated PoC I/O dirs, render OFF, approval REQUIRED.
|
||
- [x] 2.2 DJI source processed under `localpoc` (app started, ~4 min 4K analysis): wrote `category.json`
|
||
(GENERIC_VLOG, conf 0.25) + `highlight-candidates.json` (3 windows: 27-39 s, 36-48 s, 81-93 s).
|
||
Actual footage = blue Porsche Taycan aerial orbit (EV car reveal).
|
||
- [x] 2.3 Hand-authored `director/edit-plan.json`: 3 x 9 s highlights (opening_hook / rising_energy /
|
||
hero_payoff), grounded in the visible car; passes ALL validator constraints (containment, speed 1.0,
|
||
duration bounds, VO budget, category match). Manual + local-only (no external AI).
|
||
- [x] 2.4 **RENDERED (operator-approved 2026-07-21).** render-enabled override + approved.flag → local models
|
||
generated 3 VO (Piper) + 3 music (MusicGen) + SFX (AudioLDM2) → 3 highlights rendered → project
|
||
`final.mp4` assembled (27.3 s, 51.7 MB). Flow ~8.6 min. All real local-model assets, no placeholders.
|
||
- [~] 2.5 QA measured (honest):
|
||
**PASS** — 1920x1080 / H.264 / 30 fps / AAC 48 kHz; duration 27.3 s; integrated loudness -15.8 LUFS
|
||
(target -16 ±1); no black frames; no long silence; overlays render & are grounded ("First light.",
|
||
"Silent power."); VO intelligible.
|
||
**FAIL/weak** — true peak: highlight_001 = 0.0 dBFS (clips; target ≤ -1.5 dBTP); h2/h3 = -1.4 dBFS
|
||
(loudnorm TP limiting not enforced tightly). Visual "treatment" is essentially PASSTHROUGH — no visible
|
||
cinematic grade/reframe (renderer maps any treatment to a generic look); overlays are plain white
|
||
captions. LRA 16.9 (high). Pipeline's own qa-report passed because its grade/clip checks are structural.
|
||
**Verdict: genuine end-to-end local PoC, NOT yet "genuinely cinematic."** Visual grade + true-peak
|
||
limiting are the top Phase 3 items.
|
||
|
||
### Phase 3 — Iterate to quality
|
||
Ranked from the first render's evidence:
|
||
- [x] P3.1 Filmic grade in `HighlightFfmpegRenderer.cinematicVisualFilter`: S-curve + teal-orange
|
||
colorbalance + eq + unsharp + vignette (replaces the weak fixed eq). Only the highlight renderer
|
||
changed; multi-clip FfmpegEditRenderer untouched.
|
||
- [x] P3.1b **Beat-specific grading** (a0b6023): grade now varies by story beat (read from the EditPlan
|
||
style key, no EditDecision/serialization change) — opening=cool/soft (linear_contrast), rising=balanced
|
||
(medium_contrast), hero=rich/warm (strong_contrast). Verified visually (calm→dramatic arc) + unit test.
|
||
- [x] P3.2 True-peak safety: added `alimiter=level=disabled:limit=0.72` after loudnorm in the audio mix.
|
||
VERIFIED by re-render: project true peak 0.0 dBFS (clipping) → **-1.7 dBFS**; per-highlight -1.7/-2.7/
|
||
-2.8 (all <= -1.5 dBTP gate); integrated loudness -16.3 LUFS. (First pass at limit=0.79 gave -1.3 on
|
||
h1, 0.2 dB over — tightened to 0.72 for inter-sample margin.) Test added.
|
||
- [x] P3.3 Overlay styling: refined 48px caption, soft drop shadow, thin subtle border, smooth alpha
|
||
fade in/out (0.4s ramps). Verified rendering; committed 24bd1d7. (Placement/safe-area unchanged.)
|
||
- [ ] P3.4 Real CV visual analysis (resident model, no bootstrap script) to improve candidate selection.
|
||
- [ ] P3.5 Music/SFX creative fit + ducking review; consider longer inference / better prompts.
|
||
- [ ] P3.6 Freeze acceptance thresholds + blinded human creative review vs baseline before declaring success.
|
||
|
||
## Milestone log
|
||
- 2026-07-21: Models provisioned & individually proven (Piper, MusicGen, AudioLDM2). Plan approved. Phase 1 started.
|
||
- 2026-07-21: **Phase 1 complete.** Worker rewritten (audiocraft→transformers MusicGen + diffusers AudioLDM2),
|
||
models materialized to `models/` with license sidecars, all 3 asset kinds generate 48 kHz mono real audio
|
||
through the worker, `mvn -o verify` green (245/0/0/0). Paused for review before Phase 2.
|
||
- 2026-07-21: Phase 1 committed (a95d1fa). **Phase 2 up to approval gate:** localpoc profile added, DJI source
|
||
processed → candidates + category, grounded 3-highlight director plan authored & pre-validated.
|
||
STOPPED — awaiting explicit render approval for dji_21230510013241_0091_d.
|
||
- 2026-07-21: **Phase 2 rendered** (operator-approved) — first full local source→final highlight reel.
|
||
- 2026-07-22: **Phase 3.1 + 3.2 done & verified.** Filmic grade + true-peak limiter. Re-render measured TP
|
||
-1.7/-2.7/-2.8 dBFS (was 0.0 clipping), I -16.3 LUFS; mvn verify 245/0/0/0. Committed 588c652 (profile),
|
||
0fcfca9 (render quality).
|
||
- 2026-07-22: **Phase 3.3 done & verified.** Premium overlay styling (soft shadow, alpha fade). Committed
|
||
24bd1d7. Combined re-render (grade+limiter+overlays) verified: TP -2.5/-2.6/-2.8 dBFS, I -16.3 LUFS,
|
||
1080p/27.3s; mvn verify 246/0/0/0.
|
||
- 2026-07-22: **Beat-specific grading done & verified** (a0b6023). Grade varies per story beat; final
|
||
re-render TP -2.3/-2.5/-2.8 dBFS, I -16.3 LUFS, 1080p/27.3s; mvn verify 247/0/0/0. Phase 3 visual/audio
|
||
craft items complete. Next: P3.4 real CV selection, P3.5 music/SFX fit, P3.6 frozen thresholds + review.
|
||
|
||
## Deferred (until output-quality gate passes)
|
||
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
|
||
ops, digest-bound authenticated approval. Recorded, not deleted.
|