# Video Editing Service — local cinematic highlight generator

Turn a single source video into a cinematic highlight **entirely with local, offline models**: it selects the
moment, cuts a story-structured montage, generates the music/SFX/voiceover, applies a cinematic grade, and
masters the audio — no external AI services, no runtime downloads.

> **Status:** a working, source-adaptive **proof of concept**. The cinematic quality ruleset (R1–R9, below) is
> complete and content-agnostic. It is **not yet production-hardened** (no auth on the render endpoint,
> no containers/PostgreSQL/no-egress certification, and the local models are non-commercially licensed — see
> [Limitations](#limitations)). Do not deploy as-is.

## What it does

For one source clip the single-source highlight pipeline runs, fully offline:

```
ingest ─► analyze (ffprobe, scenes, audio, frames)
       ─► candidates + category
       ─► DIRECTOR (auto):
            Tier 1  measure motion (YDIF) + audio (RMS) ─► story-structured shot list
            Tier 2  local VLM (moondream2) captions beats ─► semantic payoff selection + overlay + music mood
       ─► generate assets (Piper voice · MusicGen music · AudioLDM2 SFX)
       ─► render (portrait/landscape-aware, exposure-normalized, push-in, crossfades, slow-mo ramp,
                  bold overlay, ducked source under a swelling score)
       ─► master loudness ─► QA probes ─► final.mp4
```

Rendering is gated: it stays off by default and requires an explicit approval flag per project.

## Cinematic quality rules (R1–R9)

Every rule is **source-adaptive** — it measures the source and adapts, rather than hard-coding constants.
Full detail in [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules.md).

| # | Rule | Measure → adapt |
|---|---|---|
| R1 | Exposure | frame luma → normalize; grade never crushes the subject |
| R2 | Orientation | source rotation → portrait/landscape output, no distortion |
| R3 | Audio balance + loudness | score leads, source ducked; measured loudness corrected to −16 LUFS |
| R4 | Duration | any length, story-driven |
| R5 | Motion push-in | per-shot motion (YDIF) → adaptive in-shot `zoompan` |
| R6 | Transitions | cross-dissolves between beats + ease into slow-motion |
| R7 | Overlays | bold, outlined, animated entrance |
| R8 | Music dynamics | volume swell builds into the payoff |
| R9 | Show the action | Tier-1 measured **+ Tier-2 VLM caption-driven** selection |

## Build & test

Requires **JDK 21** and **ffmpeg/ffprobe** on the PATH.

```bash
mvn -B verify          # compile, run all tests, JaCoCo gate
```

CI (`.github/workflows/ci.yml`) runs this on every push.

## Run the highlight pipeline (local PoC)

The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its
input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable
bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is in
**[`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md)**. Outputs land under
`output/localpoc/highlight-projects/<project>/final.mp4`.

Local models used (provisioning + version traps in **[`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md)**; each
needs a provenance sidecar under `models/`):

| Model | Role | License |
|---|---|---|
| Piper (`en_US-lessac-medium`) | voiceover | MIT / Blizzard dataset |
| MusicGen small | music | **CC-BY-NC** |
| AudioLDM2 | SFX | **CC-BY-NC-SA** |
| moondream2 | Tier-2 vision director | Apache-2.0 |
| YOLOv8n (optional CV) | visual analysis | **AGPL-3.0** |

## Non-negotiable constraints

- No automatic dependency/model downloads at runtime; models load offline from `models/` + the local cache.
- No external AI services in the media path.
- No unlicensed assets; no placeholder silence/tones passed off as generated audio (the pipeline fails closed).
- No rendering without an explicit approval flag.

## Limitations

- **Licensing:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are **non-commercial/copyleft**.
  Commercial use requires swapping in commercially-licensed models/assets.
- **Not production-hardened:** no Spring Security/authN, no container/K8s/deployment manifests, REST persistence
  defaults to in-memory, and no-egress operation is not yet certified. `POST /v1/edit-projects/{projectId}:render`
  now requires an `approved.flag` in the project directory, but that is a basic presence gate — not yet an
  authenticated, digest-bound authorization.
- **VLM quality:** on distant/small subjects the small local VLM is only weakly discriminative; a stronger
  model or closer framing improves Tier-2 selection.
- A director can only cut what was filmed — it cannot show a moment the camera never captured.

## Repository map

- `src/main/java/org/example/videoclips/editing/` — highlight analysis, two-tier director, renderer, QA.
- `tools/` — local model workers (`local_asset_worker.py`, `vision_caption.py`).
- `src/main/resources/application-localpoc.yml` — the opt-in PoC profile.
- `docs/` — the PoC plan, the cinematic quality rules, acceptance review.