jsl
/
video_editing_poc
1
0
Fork 1
Code Issues Pull Requests Packages Projects Releases Wiki Activity
video_editing_poc/.claude/memory/ai-handover.md

7.3 KiB
Raw Blame History

AI Handover: Local Cinematic Highlight PoC

Last updated: 2026-07-23 (Europe/Berlin). This file goes stale — verify before trusting it.

FIRST: check for staleness (prevents the 2026-07-23 stale-session incident)

This handover was once resumed from an old commit while the repo had advanced 2 days on another branch/agent, wasting a whole session re-deriving already-built work. Do NOT let that happen:

git rev-parse --short HEAD          # compare to "Recorded HEAD" below
git log -8 --oneline --decorate
git status --short                  # uncommitted work may be the real current state
  • Recorded HEAD when this was written: 9fb24e8 ("R6 speed-ramp: ease into slow-motion"). The 2026-07-23/24 session's quality work: 1da0661 two-tier director + source-adaptive rendering, 30c832e loudness mastering, f03d547 R6 crossfades, 67bdf68 R7 bold overlays, 407f6b5 Tier-2 caption-driven selection, a2a7d7d R8 music swell, 4d2cea1 fix music-cut-off-early (separate video/audio crossfade passes), 9fb24e8 R6 speed-ramp. The cinematic ruleset is now R1–R14 (docs/cinematic-quality-rules.md). 2026-07-25 session added, from a web review of "what makes video cinematic" + how DJI LightCut/Insta360 auto-edit: R10 24fps cadence, R11 filmic tone-curve grade + optional licensed lut3d hook, R12 motion blur, R13 beat-synced cuts (HighlightBeatSync + tools/beat_detect.py, librosa), R14 subject-tracking reframe (HighlightSubjectTracker
    • tools/subject_track.py, YOLO/AGPL). Also fixed a latent 444→420p playback-compat defect (all encode passes now pin -pix_fmt yuv420p). R13/R14 are opt-in flags, ON in localpoc. mvn verify = 288 tests green. Bowling re-render verified: 24fps, yuv420p, −16.0 LUFS, 3 cut boundaries beat-snapped, all 5 shots subject-followed. NOT yet committed as of this note. The cinematic ruleset R1–R9 baseline remains COMPLETE.
  • Then production-hardening started (recorded HEAD fd22412): 31efe33 CI (.github/workflows/ci.yml) + README, 430d71f render-approval gate on POST /v1/edit-projects/{id}:render (config require-render-approval, default true, 409 without approved.flag), 452bab6 Dockerfile + .dockerignore (models mounted at runtime; NOT build-validated — no Docker daemon here), c323523 Maven Wrapper (pins Maven 3.9.9; CI uses ./mvnw), fd22412 ported the e2e runbook + local-model facts into repo docs (docs/RUNBOOK-highlight-e2e.md, docs/LOCAL-MODELS.md) so any AI/human reading the repo has them. mvn -o verify (with the tmpdir workaround) = 271 tests, 0 failures, coverage met.
  • Production-readiness verdict (honest, 2026-07-24): the output is a technically-clean, much-improved cinematic DRAFT, NOT a certified production deliverable. Blockers: no blinded human creative Gate-B review has passed; source is 576p (soft when upscaled); music is generic musicgen-small; VLM weak on distant subjects; audio models are CC-BY-NC (non-commercial). Do not claim production-ready on technical probes alone.
  • Still remaining (large / decision-gated): commercial licensing (P5.7), full authN/Spring Security, PostgreSQL/Testcontainers, container build + no-egress certification, and the formal human Gate-B review (P5.6).
  • Env note (2026-07-24): the sandbox began denying the default $TMPDIR and socket binds mid-session; see highlight-e2e-render-runbook for the java.io.tmpdir + dangerouslyDisableSandbox workarounds.
  • If HEAD is newer than 8b68116, or there are unfamiliar commits/uncommitted files, this file is behind the repo. The authoritative live state is: the git log, git status, and docs/cinematic-highlight-poc-plan.md (milestone log at its tail). Trust those over this file.
  • If a claude agents background session is still "running" for this repo, only one agent should mutate the repo at a time (renders/commits/state). Confirm the others are stopped.

What exists NOW (built, not aspirational)

The full local single-source cinematic-highlight pipeline is implemented and working end-to-end: analysis → candidates/category → (manual) director plan → local asset generation → cinematic FFmpeg render → QA. Models are provisioned and proven (Piper voice, MusicGen music, AudioLDM2 SFX) — see local-model-runtime-intel-mac. The user authorized model downloads on 2026-07-21 (this reversed the earlier no-download stance; that reversal is real and current).

To run source → final render, follow highlight-e2e-render-runbook — do not re-derive it.

Delivered 2026-07-23 (whole service, not just PoC) — committed at 1da0661

Automatic two-tier director (plans were hand-authored before; now auto-generated):

  • Tier 1 HighlightMontageDirector: composes director/montage.json from measured motion (YDIF) + audio (RMS) — setup, continuous action, slow-mo payoff on the audio climax, resolution button, camera-whip tail trimmed.
  • Tier 2 HighlightVisionDirector + tools/vision_caption.py: local moondream2 (offline) captions the payoff → semantic overlay ("STRIKE") + scene-informed music; fails soft. See local-model-runtime-intel-mac.
  • Wired into the scheduler behind auto-director-enabled / vision-director-enabled (localpoc on).

Source-adaptive rendering fixes:

  1. Portrait/orientation: FfmpegClipInspector now reads rotation side-data/rotate tag and stores the EFFECTIVE (display) width/height. HighlightFfmpegRenderer.outputGeometry() renders portrait sources to a portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait phone footage being stretched into landscape.
  2. Audio mix: source audio ([0:a]) is now ducked (-20 dB, or -24 with narration) so the generated score leads instead of being buried; montage music cue raised to -9 dB. In HighlightFfmpegRenderer.audioMixCommand
    • the montage cue in HighlightDirectorFlowService.
  3. Duration constraint removed: HighlightDirectorPlanValidator no longer enforces min/max highlight duration (only positive/finite + sane playback speed + candidate containment). Config defaults relaxed (min 2 / max 3600 as soft candidate hints only). A highlight can be as long as the story needs. Montage path was already unconstrained.

mvn -o verify: 254 tests, 0 failures (JaCoCo gate passes). 6 new tests cover the above. NOT yet committed — the user has not asked to commit.

Still open / next levers (from docs/cinematic-highlight-poc-plan.md Phase 5)

  • Source footage quality is the ceiling (e.g. 576p phone clips upscale soft). Biggest lever.
  • No resident local director — plans are hand-authored (montage.json). P5.2/P5.5.
  • Bigger/continuous music (musicgen-medium is slow on CPU) — P5.3.
  • In-shot zoompan push-in, beat-synced cut lengths — P5.4.
  • Licensing: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL → non-commercial. Production needs commercially-licensed models — P5.7.
  • Production hardening (security, CI/CD, containers, PostgreSQL, no-egress cert) deferred until quality passes.

Non-negotiables still in force (except downloads, which the user authorized)

No external AI services in the media path; no unlicensed assets shipped as production; no placeholder silence/tones as success; no rendering without explicit approval; record license/provenance for every model. Manual approval is fine for the PoC. Do not commit unless the user asks.