docs: portable e2e runbook + local-models reference (any-AI usable)

Port the run-to-final flow and the local model runtime/version-trap facts from
Claude-specific auto-memory into repo Markdown so any assistant or human reading
the repo (Codex, Claude, etc.) has them. Runbook updated for the auto-director
(the plan is generated automatically now; manual authoring is an override).
Linked from the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
This commit is contained in:
JSLMPR 2026-07-24 15:47:45 +02:00
parent c32352390b
commit fd224124cf
3 changed files with 109 additions and 3 deletions

View File

@ -58,11 +58,12 @@ CI (`.github/workflows/ci.yml`) runs this on every push.
The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its
input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable
bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is in
documented step-by-step in the operator runbook. Outputs land under **[`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md)**. Outputs land under
`output/localpoc/highlight-projects/<project>/final.mp4`. `output/localpoc/highlight-projects/<project>/final.mp4`.
Local models used (each with a provenance sidecar under `models/`): Local models used (provisioning + version traps in **[`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md)**; each
needs a provenance sidecar under `models/`):
| Model | Role | License | | Model | Role | License |
|---|---|---| |---|---|---|

39
docs/LOCAL-MODELS.md Normal file
View File

@ -0,0 +1,39 @@
# Local model runtime & provisioning
The pipeline runs entirely on local models. This documents what works and the version traps, so the runtime
can be reproduced (reference environment: **x86_64 macOS, no GPU**; a Linux/GPU host is easier). Python venv:
`./.venv-local-asset/bin/python` (py3.12). Nothing here downloads at service runtime — models load offline.
## Models (each needs a provenance sidecar under `models/`)
| Model | Role | Runtime | Notes | License |
|---|---|---|---|---|
| Piper `en_US-lessac-medium` | voiceover | `piper-tts` 1.5.0 + onnxruntime | real speech; CLI takes `--model`/`--output_file` | MIT / Blizzard dataset |
| MusicGen small | music | `transformers` 4.44.2 | `facebook/musicgen-small`, CPU ~9× realtime, 32 kHz | **CC-BY-NC** |
| AudioLDM2 | SFX | `diffusers==0.30.3` | `cvssp/audioldm2`, CPU ~7× realtime, **16 kHz → resample** | **CC-BY-NC-SA** |
| moondream2 | Tier-2 vision director | `transformers` + `torchvision==0.17.2` | `vikhyatk/moondream2` rev `2024-08-26`, `trust_remote_code`, offline; ~25 s/frame CPU | Apache-2.0 |
| YOLOv8n (optional CV) | visual analysis | `.venv-local-cv` (ultralytics) | `./yolov8n.pt` loads offline; runs behind loopback HTTP `:8091` | **AGPL-3.0** |
## Version traps (x86_64 macOS — all real)
- **PyTorch caps at `torch==2.2.2` / `torchaudio==2.2.2`** (last x86_64 macOS wheels). numpy must be **< 2**
(pinned `numpy==1.26.4`, `numba==0.60.0`, `llvmlite==0.43.0`, `scipy==1.13.1`).
- **transformers must be 4.x** (pinned `4.44.2`). transformers 5.x silently disables PyTorch (needs torch ≥ 2.4)
→ models unavailable.
- **audiocraft does NOT work here** (hard top-level `from xformers import ops`; xformers has no cp312 x86_64
wheel/sdist). AudioGen is audiocraft-only → **SFX uses AudioLDM2 (diffusers)** instead. MusicGen runs via
`transformers`, not audiocraft.
- **HF downloads:** set `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks). HF may **429** after
many pulls — retry resumes from cache.
## Wiring notes
- `tools/local_asset_worker.py` synthesizes voice/music/SFX; `tools/vision_caption.py` runs moondream (batch,
load-once) for the Tier-2 director.
- `LocalAssetSynthesizer` requires each model path to be a licensed **regular file** (adjacent `.license.txt`,
non-blank, not `UNTRACKED`) — `AssetLicensePolicy`.
- Do **not** run `tools/run_local_cv_worker.sh` / `tools/run_local_asset_worker.sh` in a certified environment:
their `auto` modes `pip install` and can fetch a named YOLO model.
- **Licensing blocker for commercial use:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are
non-commercial/copyleft. Swap in commercially-licensed models/assets before any commercial release. moondream2
(Apache-2.0) and Piper are fine.

View File

@ -0,0 +1,66 @@
# Runbook — source video → rendered cinematic highlight (local)
Exact steps to run one source clip end-to-end with the `localpoc` profile. Everything is local/offline.
Run from the repo root. Models must be provisioned first — see [`LOCAL-MODELS.md`](LOCAL-MODELS.md).
> The plan is now generated **automatically** by the two-tier director (Tier-1 measured + Tier-2 vision),
> so you no longer author `montage.json` by hand — but you can override it (see "Manual override").
## 1. Stage the source
```bash
cp <video> input/localpoc/highlights/source/<name>.mp4
```
## 2. Analyze + auto-direct (render stays OFF)
```bash
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc > /tmp/app.log 2>&1 &
```
The scheduler ingests the file, analyzes it, and (localpoc has `auto-director-enabled` + `vision-director-enabled`)
writes `director/montage.json` + a minimal `director/edit-plan.json` automatically. Wait for
`event=highlight_auto_director_completed`, then `pkill -f VideoClippingApplication`. The Tier-2 vision pass
(moondream captioning several frames, ~25 s/frame CPU) makes this take a few minutes. Artifacts land in
`output/localpoc/highlight-projects/<projectId>/`.
## 3. Approve (rendering is gated)
```bash
echo approved > output/localpoc/highlight-projects/<projectId>/director/approved.flag
```
## 4. Render — render-enabled MUST be a command-line arg
The `localpoc` profile hard-codes `render-enabled: false`, so the `VIDEO_EDITING_*` env var is ignored;
override with a Spring program arg (highest precedence):
```bash
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc \
-Dspring-boot.run.arguments="--video-clipping.editing.highlight-scheduler.render-enabled=true" \
> /tmp/render.log 2>&1 &
```
MusicGen generates the score (~95 s CPU) → FFmpeg render. Wait for `final.mp4` at the project root, then
`pkill -f VideoClippingApplication`. Output: `output/localpoc/highlight-projects/<projectId>/final.mp4`.
## 5. Re-render after editing the plan/code
The render scanner skips projects with a root `final.mp4` / status `RENDERED`. Reset:
```bash
BP=output/localpoc/highlight-projects/<projectId>
rm -f $BP/final.mp4 $BP/final-preview.mp4 $BP/render-manifest.json; rm -rf $BP/highlights/* $BP/project-render-work
python3 -c "import json;p='$BP/project.json';d=json.load(open(p));d['status']='WAITING_FOR_DIRECTOR';json.dump(d,open(p,'w'),indent=2)"
```
Then repeat step 4.
## Manual override (optional)
To hand-author the cut instead of the auto-director, write `director/montage.json` yourself (it takes
precedence over `edit-plan.json` at render — `HighlightDirectorFlowService`; the render scanner still needs
`edit-plan.json` to exist — `HighlightDirectorPlanScanner`). Schema:
`{projectId, sourceVideoFileName, grade("hero"), musicDirection, voiceover[],
overlays[{text,timelineStartSeconds,timelineEndSeconds,placement}],
shots[{sourceStartSeconds, durationSeconds, zoom, speed}]}`. Shot `durationSeconds` = TIMELINE seconds;
source consumed = `durationSeconds * speed` (speed < 1 = slow-mo).
## Gotchas
- **HF downloads:** export `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks; classic HTTPS works).
- **Sandbox / temp dir:** if the app dies with `Operation not permitted` on a socket bind or `/var/folders/.../T`
(Tomcat/`@TempDir`), the environment restricted the default `$TMPDIR`/network. Run the app JVM with
`-Dspring-boot.run.jvmArguments="-Djava.io.tmpdir=<writable dir>"` and Maven tests with
`-DargLine="-Djava.io.tmpdir=<writable dir>"` (JaCoCo still attaches).
- **Measure output:** `ffprobe` geometry + `ffmpeg -i final.mp4 -filter_complex ebur128=peak=true -f null -`
(target ≈ 16 LUFS, TP ≤ 1.5) + `blackdetect`/`silencedetect`. A valid MP4 is **not** proof of cinematic
quality — that needs a human creative review (see [`cinematic-highlight-acceptance-review.md`](cinematic-highlight-acceptance-review.md)).