The director's vision JUDGE is only as good as its model, and moondream2 can't
perceive some actions on hard footage (R15 ceiling). Make the captioner backend a
config choice so a stronger local VLM drops in with no code change:
- tools/vision_caption_llamacpp.py: same manifest->JSON contract as
tools/vision_caption.py, but backed by llama.cpp `llama-mtmd-cli` (GGUF). Runs on
this x86 CPU via AVX and bypasses the torch==2.2.2 / transformers 4.x trap
entirely (no PyTorch). Model/mmproj/binary paths come from env vars; fully
offline, serverless (per-frame CLI, mmap stays warm).
- editing.vision-caption-script selects the worker (default: moondream). The Java
HighlightVisionDirector now reads the configured script instead of a hardcoded
path -- nothing else changes.
- docs/LOCAL-MODELS.md: provisioning + enablement for Qwen2.5-VL-3B (Apache-2.0)
via llama.cpp; alternatives (Qwen3-VL, Gemma 3 4B). Honest note: likely improves
the bowling case but unverified until tested with real weights.
mvn verify: 293 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rendering (whole service, driven by source measurements, not constants):
- Orientation-aware geometry: FfmpegClipInspector reads display rotation and
stores effective dims; HighlightFfmpegRenderer.outputGeometry renders portrait
sources portrait and skips the 2.39 letterbox on portrait (landscape unchanged).
- Dynamic exposure: probeSourceLuma measures the frames; exposureNormalizationFilter
maps the mean toward a target; the grade is now exposure-preserving (no crushed
subjects: bowling final went ~63 -> ~100 mean luma).
- Motion-adaptive in-shot push-in (zoompan), amount from per-shot YDIF.
- Audio mix ducks source audio under the generated score so it leads.
- Highlight duration is no longer capped (validator + config).
Automatic director (plans were hand-authored before):
- Tier 1 HighlightMontageDirector: composes the montage from measured motion (YDIF)
and audio-energy (RMS) curves -- setup, continuous action/tension, slow-mo payoff
on the audio climax, resolution button, camera-whip tail trimmed.
- Tier 2 HighlightVisionDirector + tools/vision_caption.py: a local, offline
vision-language model (moondream2) captions the payoff frame and augments the
montage with a semantic overlay ("STRIKE") and scene-informed music; fails soft.
- Wired into the scheduler behind auto-director-enabled / vision-director-enabled
(on in the localpoc profile).
Docs: cinematic-quality-rules.md (R1-R5, R9 both tiers), poc-plan milestones.
Tests: mvn -o verify -> 262 passing, 0 failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
Replace the unusable audiocraft path (requires xformers, which has no Intel-Mac
build) with runtimes proven to work offline on this machine:
- music: transformers MusicGen (facebook/musicgen-small)
- sfx: diffusers AudioLDM2 (cvssp/audioldm2), resampled 16k -> 48k
- voiceover: Piper (unchanged), normalized to 48 kHz mono
The worker CLI contract and exit codes are preserved, so the Java
LocalAssetSynthesizer license gate and fail-closed behavior are unchanged.
Add tools/provision_local_models.py to materialize models into models/ from the
local HF cache with no network. Models and their license sidecars live under the
git-ignored models/ dir; both audio models are non-commercial (CC-BY-NC-4.0 /
CC-BY-NC-SA-4.0), recorded for later production review.
Add docs/cinematic-highlight-poc-plan.md tracking the PoC plan and milestones.
mvn -o verify: 245 tests, 0 failures/errors/skips (unchanged).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
JSLMPR