From fd224124cfba7c2d6e2b20613b4a1c269a5b2101 Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Fri, 24 Jul 2026 15:47:45 +0200 Subject: [PATCH] docs: portable e2e runbook + local-models reference (any-AI usable) Port the run-to-final flow and the local model runtime/version-trap facts from Claude-specific auto-memory into repo Markdown so any assistant or human reading the repo (Codex, Claude, etc.) has them. Runbook updated for the auto-director (the plan is generated automatically now; manual authoring is an override). Linked from the README. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN --- README.md | 7 ++-- docs/LOCAL-MODELS.md | 39 +++++++++++++++++++++ docs/RUNBOOK-highlight-e2e.md | 66 +++++++++++++++++++++++++++++++++++ 3 files changed, 109 insertions(+), 3 deletions(-) create mode 100644 docs/LOCAL-MODELS.md create mode 100644 docs/RUNBOOK-highlight-e2e.md diff --git a/README.md b/README.md index 246060d..ac8dcad 100644 --- a/README.md +++ b/README.md @@ -58,11 +58,12 @@ CI (`.github/workflows/ci.yml`) runs this on every push. The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable -bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is -documented step-by-step in the operator runbook. Outputs land under +bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is in +**[`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md)**. Outputs land under `output/localpoc/highlight-projects//final.mp4`. -Local models used (each with a provenance sidecar under `models/`): +Local models used (provisioning + version traps in **[`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md)**; each +needs a provenance sidecar under `models/`): | Model | Role | License | |---|---|---| diff --git a/docs/LOCAL-MODELS.md b/docs/LOCAL-MODELS.md new file mode 100644 index 0000000..fe71965 --- /dev/null +++ b/docs/LOCAL-MODELS.md @@ -0,0 +1,39 @@ +# Local model runtime & provisioning + +The pipeline runs entirely on local models. This documents what works and the version traps, so the runtime +can be reproduced (reference environment: **x86_64 macOS, no GPU**; a Linux/GPU host is easier). Python venv: +`./.venv-local-asset/bin/python` (py3.12). Nothing here downloads at service runtime — models load offline. + +## Models (each needs a provenance sidecar under `models/`) + +| Model | Role | Runtime | Notes | License | +|---|---|---|---|---| +| Piper `en_US-lessac-medium` | voiceover | `piper-tts` 1.5.0 + onnxruntime | real speech; CLI takes `--model`/`--output_file` | MIT / Blizzard dataset | +| MusicGen small | music | `transformers` 4.44.2 | `facebook/musicgen-small`, CPU ~9× realtime, 32 kHz | **CC-BY-NC** | +| AudioLDM2 | SFX | `diffusers==0.30.3` | `cvssp/audioldm2`, CPU ~7× realtime, **16 kHz → resample** | **CC-BY-NC-SA** | +| moondream2 | Tier-2 vision director | `transformers` + `torchvision==0.17.2` | `vikhyatk/moondream2` rev `2024-08-26`, `trust_remote_code`, offline; ~25 s/frame CPU | Apache-2.0 | +| YOLOv8n (optional CV) | visual analysis | `.venv-local-cv` (ultralytics) | `./yolov8n.pt` loads offline; runs behind loopback HTTP `:8091` | **AGPL-3.0** | + +## Version traps (x86_64 macOS — all real) + +- **PyTorch caps at `torch==2.2.2` / `torchaudio==2.2.2`** (last x86_64 macOS wheels). numpy must be **< 2** + (pinned `numpy==1.26.4`, `numba==0.60.0`, `llvmlite==0.43.0`, `scipy==1.13.1`). +- **transformers must be 4.x** (pinned `4.44.2`). transformers 5.x silently disables PyTorch (needs torch ≥ 2.4) + → models unavailable. +- **audiocraft does NOT work here** (hard top-level `from xformers import ops`; xformers has no cp312 x86_64 + wheel/sdist). AudioGen is audiocraft-only → **SFX uses AudioLDM2 (diffusers)** instead. MusicGen runs via + `transformers`, not audiocraft. +- **HF downloads:** set `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks). HF may **429** after + many pulls — retry resumes from cache. + +## Wiring notes + +- `tools/local_asset_worker.py` synthesizes voice/music/SFX; `tools/vision_caption.py` runs moondream (batch, + load-once) for the Tier-2 director. +- `LocalAssetSynthesizer` requires each model path to be a licensed **regular file** (adjacent `.license.txt`, + non-blank, not `UNTRACKED`) — `AssetLicensePolicy`. +- Do **not** run `tools/run_local_cv_worker.sh` / `tools/run_local_asset_worker.sh` in a certified environment: + their `auto` modes `pip install` and can fetch a named YOLO model. +- **Licensing blocker for commercial use:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are + non-commercial/copyleft. Swap in commercially-licensed models/assets before any commercial release. moondream2 + (Apache-2.0) and Piper are fine. diff --git a/docs/RUNBOOK-highlight-e2e.md b/docs/RUNBOOK-highlight-e2e.md new file mode 100644 index 0000000..960b7a5 --- /dev/null +++ b/docs/RUNBOOK-highlight-e2e.md @@ -0,0 +1,66 @@ +# Runbook — source video → rendered cinematic highlight (local) + +Exact steps to run one source clip end-to-end with the `localpoc` profile. Everything is local/offline. +Run from the repo root. Models must be provisioned first — see [`LOCAL-MODELS.md`](LOCAL-MODELS.md). + +> The plan is now generated **automatically** by the two-tier director (Tier-1 measured + Tier-2 vision), +> so you no longer author `montage.json` by hand — but you can override it (see "Manual override"). + +## 1. Stage the source +```bash +cp