100 lines
7.3 KiB
Markdown
100 lines
7.3 KiB
Markdown
# AI Handover: Local Cinematic Highlight PoC
|
||
|
||
Last updated: 2026-07-23 (Europe/Berlin). **This file goes stale — verify before trusting it.**
|
||
|
||
## FIRST: check for staleness (prevents the 2026-07-23 stale-session incident)
|
||
|
||
This handover was once resumed from an old commit while the repo had advanced 2 days on another
|
||
branch/agent, wasting a whole session re-deriving already-built work. Do NOT let that happen:
|
||
|
||
```bash
|
||
git rev-parse --short HEAD # compare to "Recorded HEAD" below
|
||
git log -8 --oneline --decorate
|
||
git status --short # uncommitted work may be the real current state
|
||
```
|
||
|
||
- **Recorded HEAD when this was written: `9fb24e8`** ("R6 speed-ramp: ease into slow-motion"). The 2026-07-23/24
|
||
session's quality work: `1da0661` two-tier director + source-adaptive rendering, `30c832e` loudness mastering,
|
||
`f03d547` R6 crossfades, `67bdf68` R7 bold overlays, `407f6b5` Tier-2 caption-driven selection, `a2a7d7d` R8
|
||
music swell, `4d2cea1` fix music-cut-off-early (separate video/audio crossfade passes), `9fb24e8` R6 speed-ramp.
|
||
**The cinematic ruleset is now R1–R14 (docs/cinematic-quality-rules.md).** 2026-07-25 session added, from a
|
||
web review of "what makes video cinematic" + how DJI LightCut/Insta360 auto-edit: R10 24fps cadence, R11
|
||
filmic tone-curve grade + optional licensed `lut3d` hook, R12 motion blur, R13 beat-synced cuts
|
||
(`HighlightBeatSync` + `tools/beat_detect.py`, librosa), R14 subject-tracking reframe (`HighlightSubjectTracker`
|
||
+ `tools/subject_track.py`, YOLO/AGPL). Also fixed a latent 444→420p playback-compat defect (all encode passes
|
||
now pin `-pix_fmt yuv420p`). R13/R14 are opt-in flags, ON in localpoc. `mvn verify` = 288 tests green. Bowling
|
||
re-render verified: 24fps, yuv420p, −16.0 LUFS, 3 cut boundaries beat-snapped, all 5 shots subject-followed.
|
||
NOT yet committed as of this note. **The cinematic ruleset R1–R9 baseline remains COMPLETE.**
|
||
- **Then production-hardening started (recorded HEAD `fd22412`):** `31efe33` CI (.github/workflows/ci.yml) +
|
||
README, `430d71f` render-approval gate on `POST /v1/edit-projects/{id}:render` (config `require-render-approval`,
|
||
default true, 409 without `approved.flag`), `452bab6` Dockerfile + .dockerignore (models mounted at runtime;
|
||
NOT build-validated — no Docker daemon here), `c323523` Maven Wrapper (pins Maven 3.9.9; CI uses `./mvnw`),
|
||
`fd22412` ported the e2e runbook + local-model facts into repo docs (`docs/RUNBOOK-highlight-e2e.md`,
|
||
`docs/LOCAL-MODELS.md`) so any AI/human reading the repo has them. `mvn -o verify` (with the tmpdir workaround)
|
||
= 271 tests, 0 failures, coverage met.
|
||
- **Production-readiness verdict (honest, 2026-07-24):** the output is a technically-clean, much-improved
|
||
cinematic DRAFT, NOT a certified production deliverable. Blockers: no blinded human creative Gate-B review has
|
||
passed; source is 576p (soft when upscaled); music is generic musicgen-small; VLM weak on distant subjects;
|
||
audio models are CC-BY-NC (non-commercial). Do not claim production-ready on technical probes alone.
|
||
- **Still remaining (large / decision-gated):** commercial licensing (P5.7), full authN/Spring Security,
|
||
PostgreSQL/Testcontainers, container build + no-egress certification, and the formal human Gate-B review (P5.6).
|
||
- **Env note (2026-07-24):** the sandbox began denying the default `$TMPDIR` and socket binds mid-session; see
|
||
[[highlight-e2e-render-runbook]] for the java.io.tmpdir + dangerouslyDisableSandbox workarounds.
|
||
- If HEAD is newer than `8b68116`, or there are unfamiliar commits/uncommitted files, **this file is behind
|
||
the repo**. The authoritative live state is: the git log, `git status`, and
|
||
`docs/cinematic-highlight-poc-plan.md` (milestone log at its tail). Trust those over this file.
|
||
- If a `claude agents` background session is still "running" for this repo, only one agent should mutate the
|
||
repo at a time (renders/commits/state). Confirm the others are stopped.
|
||
|
||
## What exists NOW (built, not aspirational)
|
||
|
||
The full local single-source cinematic-highlight pipeline is implemented and working end-to-end:
|
||
analysis → candidates/category → (manual) director plan → local asset generation → cinematic FFmpeg render
|
||
→ QA. Models are provisioned and proven (Piper voice, MusicGen music, AudioLDM2 SFX) — see
|
||
[[local-model-runtime-intel-mac]]. The user authorized model downloads on 2026-07-21 (this **reversed** the
|
||
earlier no-download stance; that reversal is real and current).
|
||
|
||
To run source → final render, follow [[highlight-e2e-render-runbook]] — do not re-derive it.
|
||
|
||
## Delivered 2026-07-23 (whole service, not just PoC) — committed at `1da0661`
|
||
|
||
**Automatic two-tier director** (plans were hand-authored before; now auto-generated):
|
||
- Tier 1 `HighlightMontageDirector`: composes `director/montage.json` from measured motion (YDIF) + audio (RMS)
|
||
— setup, continuous action, slow-mo payoff on the audio climax, resolution button, camera-whip tail trimmed.
|
||
- Tier 2 `HighlightVisionDirector` + `tools/vision_caption.py`: local moondream2 (offline) captions the payoff
|
||
→ semantic overlay ("STRIKE") + scene-informed music; fails soft. See [[local-model-runtime-intel-mac]].
|
||
- Wired into the scheduler behind `auto-director-enabled` / `vision-director-enabled` (localpoc on).
|
||
|
||
**Source-adaptive rendering fixes:**
|
||
|
||
1. **Portrait/orientation**: `FfmpegClipInspector` now reads rotation side-data/`rotate` tag and stores the
|
||
EFFECTIVE (display) width/height. `HighlightFfmpegRenderer.outputGeometry()` renders portrait sources to a
|
||
portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait phone footage being
|
||
stretched into landscape.
|
||
2. **Audio mix**: source audio (`[0:a]`) is now ducked (-20 dB, or -24 with narration) so the generated score
|
||
leads instead of being buried; montage music cue raised to -9 dB. In `HighlightFfmpegRenderer.audioMixCommand`
|
||
+ the montage cue in `HighlightDirectorFlowService`.
|
||
3. **Duration constraint removed**: `HighlightDirectorPlanValidator` no longer enforces min/max highlight
|
||
duration (only positive/finite + sane playback speed + candidate containment). Config defaults relaxed
|
||
(min 2 / max 3600 as soft candidate hints only). A highlight can be as long as the story needs. Montage path
|
||
was already unconstrained.
|
||
|
||
`mvn -o verify`: 254 tests, 0 failures (JaCoCo gate passes). 6 new tests cover the above. NOT yet committed —
|
||
the user has not asked to commit.
|
||
|
||
## Still open / next levers (from docs/cinematic-highlight-poc-plan.md Phase 5)
|
||
|
||
- Source footage quality is the ceiling (e.g. 576p phone clips upscale soft). Biggest lever.
|
||
- No resident local **director** — plans are hand-authored (`montage.json`). P5.2/P5.5.
|
||
- Bigger/continuous music (musicgen-medium is slow on CPU) — P5.3.
|
||
- In-shot `zoompan` push-in, beat-synced cut lengths — P5.4.
|
||
- Licensing: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL → non-commercial. Production needs
|
||
commercially-licensed models — P5.7.
|
||
- Production hardening (security, CI/CD, containers, PostgreSQL, no-egress cert) deferred until quality passes.
|
||
|
||
## Non-negotiables still in force (except downloads, which the user authorized)
|
||
|
||
No external AI services in the media path; no unlicensed assets shipped as production; no placeholder
|
||
silence/tones as success; no rendering without explicit approval; record license/provenance for every model.
|
||
Manual approval is fine for the PoC. Do not commit unless the user asks.
|