forked from jsl/video_editing_poc
docs: portable e2e runbook + local-models reference (any-AI usable)
Port the run-to-final flow and the local model runtime/version-trap facts from Claude-specific auto-memory into repo Markdown so any assistant or human reading the repo (Codex, Claude, etc.) has them. Runbook updated for the auto-director (the plan is generated automatically now; manual authoring is an override). Linked from the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
This commit is contained in:
parent
c32352390b
commit
fd224124cf
|
|
@ -58,11 +58,12 @@ CI (`.github/workflows/ci.yml`) runs this on every push.
|
||||||
|
|
||||||
The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its
|
The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its
|
||||||
input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable
|
input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable
|
||||||
bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is
|
bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is in
|
||||||
documented step-by-step in the operator runbook. Outputs land under
|
**[`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md)**. Outputs land under
|
||||||
`output/localpoc/highlight-projects/<project>/final.mp4`.
|
`output/localpoc/highlight-projects/<project>/final.mp4`.
|
||||||
|
|
||||||
Local models used (each with a provenance sidecar under `models/`):
|
Local models used (provisioning + version traps in **[`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md)**; each
|
||||||
|
needs a provenance sidecar under `models/`):
|
||||||
|
|
||||||
| Model | Role | License |
|
| Model | Role | License |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
# Local model runtime & provisioning
|
||||||
|
|
||||||
|
The pipeline runs entirely on local models. This documents what works and the version traps, so the runtime
|
||||||
|
can be reproduced (reference environment: **x86_64 macOS, no GPU**; a Linux/GPU host is easier). Python venv:
|
||||||
|
`./.venv-local-asset/bin/python` (py3.12). Nothing here downloads at service runtime — models load offline.
|
||||||
|
|
||||||
|
## Models (each needs a provenance sidecar under `models/`)
|
||||||
|
|
||||||
|
| Model | Role | Runtime | Notes | License |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Piper `en_US-lessac-medium` | voiceover | `piper-tts` 1.5.0 + onnxruntime | real speech; CLI takes `--model`/`--output_file` | MIT / Blizzard dataset |
|
||||||
|
| MusicGen small | music | `transformers` 4.44.2 | `facebook/musicgen-small`, CPU ~9× realtime, 32 kHz | **CC-BY-NC** |
|
||||||
|
| AudioLDM2 | SFX | `diffusers==0.30.3` | `cvssp/audioldm2`, CPU ~7× realtime, **16 kHz → resample** | **CC-BY-NC-SA** |
|
||||||
|
| moondream2 | Tier-2 vision director | `transformers` + `torchvision==0.17.2` | `vikhyatk/moondream2` rev `2024-08-26`, `trust_remote_code`, offline; ~25 s/frame CPU | Apache-2.0 |
|
||||||
|
| YOLOv8n (optional CV) | visual analysis | `.venv-local-cv` (ultralytics) | `./yolov8n.pt` loads offline; runs behind loopback HTTP `:8091` | **AGPL-3.0** |
|
||||||
|
|
||||||
|
## Version traps (x86_64 macOS — all real)
|
||||||
|
|
||||||
|
- **PyTorch caps at `torch==2.2.2` / `torchaudio==2.2.2`** (last x86_64 macOS wheels). numpy must be **< 2**
|
||||||
|
(pinned `numpy==1.26.4`, `numba==0.60.0`, `llvmlite==0.43.0`, `scipy==1.13.1`).
|
||||||
|
- **transformers must be 4.x** (pinned `4.44.2`). transformers 5.x silently disables PyTorch (needs torch ≥ 2.4)
|
||||||
|
→ models unavailable.
|
||||||
|
- **audiocraft does NOT work here** (hard top-level `from xformers import ops`; xformers has no cp312 x86_64
|
||||||
|
wheel/sdist). AudioGen is audiocraft-only → **SFX uses AudioLDM2 (diffusers)** instead. MusicGen runs via
|
||||||
|
`transformers`, not audiocraft.
|
||||||
|
- **HF downloads:** set `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks). HF may **429** after
|
||||||
|
many pulls — retry resumes from cache.
|
||||||
|
|
||||||
|
## Wiring notes
|
||||||
|
|
||||||
|
- `tools/local_asset_worker.py` synthesizes voice/music/SFX; `tools/vision_caption.py` runs moondream (batch,
|
||||||
|
load-once) for the Tier-2 director.
|
||||||
|
- `LocalAssetSynthesizer` requires each model path to be a licensed **regular file** (adjacent `.license.txt`,
|
||||||
|
non-blank, not `UNTRACKED`) — `AssetLicensePolicy`.
|
||||||
|
- Do **not** run `tools/run_local_cv_worker.sh` / `tools/run_local_asset_worker.sh` in a certified environment:
|
||||||
|
their `auto` modes `pip install` and can fetch a named YOLO model.
|
||||||
|
- **Licensing blocker for commercial use:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are
|
||||||
|
non-commercial/copyleft. Swap in commercially-licensed models/assets before any commercial release. moondream2
|
||||||
|
(Apache-2.0) and Piper are fine.
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
# Runbook — source video → rendered cinematic highlight (local)
|
||||||
|
|
||||||
|
Exact steps to run one source clip end-to-end with the `localpoc` profile. Everything is local/offline.
|
||||||
|
Run from the repo root. Models must be provisioned first — see [`LOCAL-MODELS.md`](LOCAL-MODELS.md).
|
||||||
|
|
||||||
|
> The plan is now generated **automatically** by the two-tier director (Tier-1 measured + Tier-2 vision),
|
||||||
|
> so you no longer author `montage.json` by hand — but you can override it (see "Manual override").
|
||||||
|
|
||||||
|
## 1. Stage the source
|
||||||
|
```bash
|
||||||
|
cp <video> input/localpoc/highlights/source/<name>.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Analyze + auto-direct (render stays OFF)
|
||||||
|
```bash
|
||||||
|
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc > /tmp/app.log 2>&1 &
|
||||||
|
```
|
||||||
|
The scheduler ingests the file, analyzes it, and (localpoc has `auto-director-enabled` + `vision-director-enabled`)
|
||||||
|
writes `director/montage.json` + a minimal `director/edit-plan.json` automatically. Wait for
|
||||||
|
`event=highlight_auto_director_completed`, then `pkill -f VideoClippingApplication`. The Tier-2 vision pass
|
||||||
|
(moondream captioning several frames, ~25 s/frame CPU) makes this take a few minutes. Artifacts land in
|
||||||
|
`output/localpoc/highlight-projects/<projectId>/`.
|
||||||
|
|
||||||
|
## 3. Approve (rendering is gated)
|
||||||
|
```bash
|
||||||
|
echo approved > output/localpoc/highlight-projects/<projectId>/director/approved.flag
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Render — render-enabled MUST be a command-line arg
|
||||||
|
The `localpoc` profile hard-codes `render-enabled: false`, so the `VIDEO_EDITING_*` env var is ignored;
|
||||||
|
override with a Spring program arg (highest precedence):
|
||||||
|
```bash
|
||||||
|
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc \
|
||||||
|
-Dspring-boot.run.arguments="--video-clipping.editing.highlight-scheduler.render-enabled=true" \
|
||||||
|
> /tmp/render.log 2>&1 &
|
||||||
|
```
|
||||||
|
MusicGen generates the score (~95 s CPU) → FFmpeg render. Wait for `final.mp4` at the project root, then
|
||||||
|
`pkill -f VideoClippingApplication`. Output: `output/localpoc/highlight-projects/<projectId>/final.mp4`.
|
||||||
|
|
||||||
|
## 5. Re-render after editing the plan/code
|
||||||
|
The render scanner skips projects with a root `final.mp4` / status `RENDERED`. Reset:
|
||||||
|
```bash
|
||||||
|
BP=output/localpoc/highlight-projects/<projectId>
|
||||||
|
rm -f $BP/final.mp4 $BP/final-preview.mp4 $BP/render-manifest.json; rm -rf $BP/highlights/* $BP/project-render-work
|
||||||
|
python3 -c "import json;p='$BP/project.json';d=json.load(open(p));d['status']='WAITING_FOR_DIRECTOR';json.dump(d,open(p,'w'),indent=2)"
|
||||||
|
```
|
||||||
|
Then repeat step 4.
|
||||||
|
|
||||||
|
## Manual override (optional)
|
||||||
|
To hand-author the cut instead of the auto-director, write `director/montage.json` yourself (it takes
|
||||||
|
precedence over `edit-plan.json` at render — `HighlightDirectorFlowService`; the render scanner still needs
|
||||||
|
`edit-plan.json` to exist — `HighlightDirectorPlanScanner`). Schema:
|
||||||
|
`{projectId, sourceVideoFileName, grade("hero"), musicDirection, voiceover[],
|
||||||
|
overlays[{text,timelineStartSeconds,timelineEndSeconds,placement}],
|
||||||
|
shots[{sourceStartSeconds, durationSeconds, zoom, speed}]}`. Shot `durationSeconds` = TIMELINE seconds;
|
||||||
|
source consumed = `durationSeconds * speed` (speed < 1 = slow-mo).
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
- **HF downloads:** export `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks; classic HTTPS works).
|
||||||
|
- **Sandbox / temp dir:** if the app dies with `Operation not permitted` on a socket bind or `/var/folders/.../T`
|
||||||
|
(Tomcat/`@TempDir`), the environment restricted the default `$TMPDIR`/network. Run the app JVM with
|
||||||
|
`-Dspring-boot.run.jvmArguments="-Djava.io.tmpdir=<writable dir>"` and Maven tests with
|
||||||
|
`-DargLine="-Djava.io.tmpdir=<writable dir>"` (JaCoCo still attaches).
|
||||||
|
- **Measure output:** `ffprobe` geometry + `ffmpeg -i final.mp4 -filter_complex ebur128=peak=true -f null -`
|
||||||
|
(target ≈ −16 LUFS, TP ≤ −1.5) + `blackdetect`/`silencedetect`. A valid MP4 is **not** proof of cinematic
|
||||||
|
quality — that needs a human creative review (see [`cinematic-highlight-acceptance-review.md`](cinematic-highlight-acceptance-review.md)).
|
||||||
Loading…
Reference in New Issue