video_editing_poc/README.md

Video Editing Service — local cinematic highlight generator

Turn a single source video into a cinematic highlight entirely with local, offline models: it selects the moment, cuts a story-structured montage, generates the music/SFX/voiceover, applies a cinematic grade, and masters the audio — no external AI services, no runtime downloads.

Status: a working, source-adaptive proof of concept. The cinematic quality ruleset (R1–R9, below) is complete and content-agnostic. It is not yet production-hardened (no auth on the render endpoint, no containers/PostgreSQL/no-egress certification, and the local models are non-commercially licensed — see Limitations). Do not deploy as-is.

What it does

For one source clip the single-source highlight pipeline runs, fully offline:

ingest ─► analyze (ffprobe, scenes, audio, frames)
       ─► candidates + category
       ─► DIRECTOR (auto):
            Tier 1  measure motion (YDIF) + audio (RMS) ─► story-structured shot list
            Tier 2  local VLM (moondream2) captions beats ─► semantic payoff selection + overlay + music mood
       ─► generate assets (Piper voice · MusicGen music · AudioLDM2 SFX)
       ─► render (portrait/landscape-aware, exposure-normalized, push-in, crossfades, slow-mo ramp,
                  bold overlay, ducked source under a swelling score)
       ─► master loudness ─► QA probes ─► final.mp4

Rendering is gated: it stays off by default and requires an explicit approval flag per project.

Cinematic quality rules (R1–R9)

Every rule is source-adaptive — it measures the source and adapts, rather than hard-coding constants. Full detail in docs/cinematic-quality-rules.md.

#	Rule	Measure → adapt
R1	Exposure	frame luma → normalize; grade never crushes the subject
R2	Orientation	source rotation → portrait/landscape output, no distortion
R3	Audio balance + loudness	score leads, source ducked; measured loudness corrected to −16 LUFS
R4	Duration	any length, story-driven
R5	Motion push-in	per-shot motion (YDIF) → adaptive in-shot zoompan
R6	Transitions	cross-dissolves between beats + ease into slow-motion
R7	Overlays	bold, outlined, animated entrance
R8	Music dynamics	volume swell builds into the payoff
R9	Show the action	Tier-1 measured + Tier-2 VLM caption-driven selection

Build & test

Requires JDK 21 and ffmpeg/ffprobe on the PATH.

mvn -B verify          # compile, run all tests, JaCoCo gate

CI (.github/workflows/ci.yml) runs this on every push.

Run the highlight pipeline (local PoC)

The localpoc Spring profile wires the pipeline to pre-provisioned local model paths, isolates its input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is documented step-by-step in the operator runbook. Outputs land under output/localpoc/highlight-projects/<project>/final.mp4.

Local models used (each with a provenance sidecar under models/):

Model	Role	License
Piper (en_US-lessac-medium)	voiceover	MIT / Blizzard dataset
MusicGen small	music	CC-BY-NC
AudioLDM2	SFX	CC-BY-NC-SA
moondream2	Tier-2 vision director	Apache-2.0
YOLOv8n (optional CV)	visual analysis	AGPL-3.0

Non-negotiable constraints

• No automatic dependency/model downloads at runtime; models load offline from models/ + the local cache.
• No external AI services in the media path.
• No unlicensed assets; no placeholder silence/tones passed off as generated audio (the pipeline fails closed).
• No rendering without an explicit approval flag.

Limitations

• Licensing: MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are non-commercial/copyleft. Commercial use requires swapping in commercially-licensed models/assets.
• Not production-hardened: no Spring Security/authN, no container/K8s/deployment manifests, REST persistence defaults to in-memory, POST /v1/edit-projects/{projectId}:render has no approval gate, and no-egress operation is not yet certified.
• VLM quality: on distant/small subjects the small local VLM is only weakly discriminative; a stronger model or closer framing improves Tier-2 selection.
• A director can only cut what was filmed — it cannot show a moment the camera never captured.

Repository map

• src/main/java/org/example/videoclips/editing/ — highlight analysis, two-tier director, renderer, QA.
• tools/ — local model workers (local_asset_worker.py, vision_caption.py).
• src/main/resources/application-localpoc.yml — the opt-in PoC profile.
• docs/ — the PoC plan, the cinematic quality rules, acceptance review.