Compare commits

..

48 Commits

Author SHA1 Message Date
JSLMPR e6a7aecd97 fix(render): loop a bounded music bed so long reels render (MusicGen CPU cap)
A multi-segment reel needs music as long as the reel, but MusicGen-small stalls on
CPU past ~45s (confirmed on the 49s downhill/soccer reels). Decouple music-generation
length from reel length: generate a bounded bed (music-gen-max-seconds, default 15s)
and loop it across the timeline (-stream_loop -1 on the music input; the existing
atrim bounds it to the cue). Reel length stays uncapped; music generation stays
feasible. Verified: 7-segment 45s downhill reel now renders (14.9s bed looped,
1080x1920/24fps/yuv420p/-15.6 LUFS).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 03:08:15 +02:00
JSLMPR 5b98c0538f feat(director): multi-segment highlight reel with intensity-aware selection
The director produced ONE segment around a single chosen moment, so a video with
several highlight-worthy actions got only one. Now it understands the whole clip and
builds a REEL:

- candidatePeaks proposes ALL intensity peaks across the clip (no cap), time-ordered.
- The vision judge (judgeMoments) rates EVERY candidate in one batched pass and
  supplies an honest overlay per candidate (replaces the single-pick rankDecisiveMoment
  + the separate groundOverlay pass).
- composeReel selects every candidate whose blended score clears the bar
  (0.6*judge-worthiness + 0.4*measured-intensity; the single best is always kept),
  builds an action segment (entry -> slow-mo peak -> exit) per selection, merges
  overlapping ones, and concatenates them into one video with one overlay per segment.
- Intensity in the blend means continuous-motion clips (a downhill ride, where the
  model rates everything 'riding') still pick the most dynamic sections, not all of them.
- Config: highlight-select-threshold (default 0.5). No cap on count or length.

mvn verify: 295 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 19:07:33 +02:00
JSLMPR 70dffaea6c feat(vision): default localpoc to Qwen2.5-VL with loud moondream fallback
Make the stronger Tier-2 judge the localpoc default (vision-caption-script ->
vision_caption_llamacpp.py). Guard it so it never silently degrades: resolveCaptionScript
checks the llama.cpp backend is provisioned (binary env + weights) and, if not, falls
back to moondream with a WARN (event=vision_backend_not_ready). Worker defaults the
model/mmproj to the repo's ./models/qwen2.5-vl-3b paths, so only the machine-specific
binary env is mandatory. mvn verify: 294 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 17:55:46 +02:00
JSLMPR 8c0ddc1796 perf(vision): add resident llama-server backend (model loads once)
The CLI backend reloads the ~3GB model per frame (~2min/frame). Add a SERVER mode:
when LLAMACPP_SERVER_BIN is set, start llama-server once, POST base64 frames to its
OpenAI /v1/chat/completions endpoint on loopback, stop it at the end. Same answers,
~3x faster on a full clip (measured: 3 frames 181s incl. one-time load vs ~6-8min).
CLI mode (LLAMACPP_MTMD_BIN) remains the simple fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 17:49:36 +02:00
JSLMPR 1ea6640da0 fix(vision): overlay describes the chosen peak, not the trailing outcome
groundOverlay sampled the payoff shot THROUGH the final (button/outcome) shot, so a
post-payoff reaction frame could supply the overlay (bowling: 'SMILING' from the
bowler lowering his arms after the celebration). Confine the overlay caption to the
payoff (slow-mo) shot — the decisive moment the judge chose.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 16:49:22 +02:00
JSLMPR cf8ec76430 fix(vision): Qwen2.5-VL llama.cpp backend — verified to break the moondream ceiling
Provisioned and tested the stronger Tier-2 judge on this Intel Mac (CPU-only):

- vision_caption_llamacpp.py: force CPU (-ngl 0 --no-mmproj-offload) because the
  integrated GPU times out on the vision encoder (Metal command-buffer timeout);
  extract the final assistant turn from the chat-templated output.
- docs/LOCAL-MODELS.md: the VERIFIED build+run recipe, incl. two real gotchas ->
  Command Line Tools libc++ mismatch (add -isystem <SDK>/usr/include/c++/v1 to the
  cmake flags, else ggml-base fails on <array>), and the Intel-GPU Metal timeout.

Verified result: where moondream described the bowling celebration as "standing in
a bowling alley", Qwen2.5-VL-3B says "raising their arms in a celebratory gesture"
(highlight-worthiness 1.0) and rates turn-around/anticipation low (0.2 / 0.15). End
to end, the director's judge now chooses time=10.5s score=1.0 (the celebration) vs
moondream's 13.0s score=0.15 (the turn-away). Model weights are gitignored under
models/ (Apache-2.0, provisioned offline).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:50:12 +02:00
JSLMPR 3652cefacc feat(vision): pluggable llama.cpp/GGUF Tier-2 VLM backend (stronger judge)
The director's vision JUDGE is only as good as its model, and moondream2 can't
perceive some actions on hard footage (R15 ceiling). Make the captioner backend a
config choice so a stronger local VLM drops in with no code change:

- tools/vision_caption_llamacpp.py: same manifest->JSON contract as
  tools/vision_caption.py, but backed by llama.cpp `llama-mtmd-cli` (GGUF). Runs on
  this x86 CPU via AVX and bypasses the torch==2.2.2 / transformers 4.x trap
  entirely (no PyTorch). Model/mmproj/binary paths come from env vars; fully
  offline, serverless (per-frame CLI, mmap stays warm).
- editing.vision-caption-script selects the worker (default: moondream). The Java
  HighlightVisionDirector now reads the configured script instead of a hardcoded
  path -- nothing else changes.
- docs/LOCAL-MODELS.md: provisioning + enablement for Qwen2.5-VL-3B (Apache-2.0)
  via llama.cpp; alternatives (Qwen3-VL, Gemma 3 4B). Honest note: likely improves
  the bowling case but unverified until tested with real weights.

mvn verify: 293 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 19:11:54 +02:00
JSLMPR d89a28e084 refactor(director): decisive moment = measurement proposes, vision model judges
Choosing WHICH moment is the highlight is a question of meaning, not motion or
loudness, and there is no generic rule in measurement alone: on the real bowling
clip a camera turn-away has the highest motion AND is louder than the celebration.
Positional bands / thresholds only move which video breaks. So responsibilities
are now split, with zero per-video constants:

- HighlightMontageDirector.candidatePeaks: measurement PROPOSES the intensity
  (motion+audio) local maxima, strongest-first, min-separated. No opinion on which
  is the highlight; no band, no threshold.
- HighlightVisionDirector.rankDecisiveMoment: the vision model JUDGES each
  candidate by highlight-worthiness (a celebration/goal outranks a loud turn-away
  or an "about to..." build; anticipation is not the payoff). Highest score wins;
  falls back to the strongest peak only if the model declines.
- composeMontageAt: builds the action segment (measured onset -> chosen peak ->
  measured resolution that sweeps in the outcome+reaction) around the choice.
- MomentChooser interface makes the judge a drop-in: a stronger local VLM plugs in
  with no director changes.

Removes the previous positional-band / semantic-weight heuristics.

Honest, verified ceiling (documented in R15): the judge is only as good as its
eyes. moondream2 perceives some actions (soccer: "kicking a soccer ball" -> the
goal is chosen correctly) but not others -- on distant portrait bowling footage it
describes every frame as "standing"/"walking" and never sees the celebration
(a posture prompt collapsed to a constant "Standing still"). When it can't
discriminate, candidates tie and it falls back to the loudest peak. This is a
model-capability limit, not a design flaw; the fix is a stronger VLM (drop-in).

mvn verify: 293 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:37:38 +02:00
JSLMPR 8b00b41d7c fix(director): select the decisive moment, include its outcome, honest overlay
The highlight could cut off before the actual action and then assert an action
that was never shown (a "KICK" overlay over a stop-before-the-kick cut). Root
causes, all generic and fixed here:

- Localization (A): the climax search only scanned the central 25-80% band on
  audio(+semantic), structurally excluding a late payoff and often landing on the
  anticipation. Now it searches nearly the whole clip and localizes primarily by
  MOTION (the decisive action is a motion event), audio equal, semantic a weak
  tie-breaker, with a robust percentile-based camera-whip guard.
- Outcome (B): the window ended at a fixed climax+offset. It now extends past the
  climax until motion settles (the action AND its result), whip-guarded and capped.
- Honest overlay (C): a DENSE pass over the shown payoff window (not 7 sparse
  whole-clip frames) picks the best action frame; an honesty rule never asserts an
  action the window does not show -- anticipation gets a grounded teaser question
  instead (HighlightVisionDirector.groundOverlay/honestOverlayText/isAnticipatory).
  The VLM no longer picks the moment (it mislocalized onto anticipation), only
  grounds the overlay.
- Pacing cap: montage-max-build-seconds bounds the single pre-climax build shot so
  a distant action spike on a long source can't create one runaway shot (dead air
  plus an impractically long generated score).

Also: re-ingesting a source whose name already exists in processed/ no longer
fails -- moveToDirectory picks a unique "<name>-<n>.<ext>" instead of refusing.

Verified end-to-end on a new 63s landscape soccer clip: the director now finds the
scoring kick (ball in net), keeps the outcome, honestly labels "KICK", 24fps/420p/
-16 LUFS, beat-synced, subject-followed. mvn verify: 292 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:50:57 +02:00
JSLMPR 9e401870a1 feat(cinematic): R10–R14 grade/cadence/edit rules + 420p compat fix
Add five source-adaptive cinematic rules, all content-agnostic and
  fail-soft, derived from a web review of the film look and how consumer
  AI editors (DJI LightCut, Insta360) auto-edit:

  - R10: 24fps cinematic cadence (output frame rate 30 -> 24)
  - R11: filmic tone-curve grade (lifted toe + highlight roll-off);
         optional licensed film LUT via lut3d, gated on a .license.txt
         sidecar (fails closed, no unlicensed asset applied)
  - R12: shutter-angle motion blur (tmix), toggle
  - R13: beat-synced cuts — snap cut boundaries onto the score's beat
         grid (HighlightBeatSync + tools/beat_detect.py, librosa)
  - R14: subject-tracking reframe — follow the detected subject instead
         of a static crop (HighlightSubjectTracker + tools/subject_track.py,
         YOLO/AGPL, non-commercial); renderer subjectFollowFilter
  - Fix: pin -pix_fmt yuv420p on every encode pass (was yuv444p, which
         browsers/QuickTime reject)

  R13/R14 are opt-in flags, on in localpoc. mvn verify: 288 tests green.
  E2e bowling re-render verified: 24fps, yuv420p, -16.0 LUFS, 3 cut
  boundaries beat-snapped, all 5 shots subject-followed.
  Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:36:18 +02:00
JSLMPR b455a11d82 Add AGENTS.md (any-AI entry point) + Gate-B review scorecard
- AGENTS.md: portable onboarding for any assistant (Codex/Claude/etc.) at the repo
  root -- what the project is, read-order, build/test/run, hard constraints, honest
  status, env gotchas. Makes the whole repo usable by any AI, not just Claude.
- docs/gate-b-review-bowling.md: the production-readiness scorecard. Gate A
  (technical) measured = PASS; Gate B (human creative rubric) = PENDING. Output is
  production-ready only when BOTH pass; this is the artifact a human fills to decide.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 21:21:43 +02:00
JSLMPR 7012daa6b6 docs(memory): save session state at HEAD fd22412 (R1-R9 done; hardening started; honest not-production-ready verdict)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 16:19:17 +02:00
JSLMPR fd224124cf docs: portable e2e runbook + local-models reference (any-AI usable)
Port the run-to-final flow and the local model runtime/version-trap facts from
Claude-specific auto-memory into repo Markdown so any assistant or human reading
the repo (Codex, Claude, etc.) has them. Runbook updated for the auto-director
(the plan is generated automatically now; manual authoring is an override).
Linked from the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 15:47:45 +02:00
JSLMPR c32352390b Add Maven Wrapper (pin Maven 3.9.9); CI uses ./mvnw
Deterministic build across environments. Closes the wrapper follow-up noted when
CI was added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 15:41:17 +02:00
JSLMPR 846cb05aa5 docs(memory): handover HEAD 452bab6; production hardening started
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 15:15:07 +02:00
JSLMPR 452bab6a1b Add Dockerfile + .dockerignore (app image; models mounted at runtime)
Multi-stage build (maven:3.9-temurin-21 -> eclipse-temurin:21-jre): builds the jar,
installs ffmpeg, runs as a non-root user. The large, non-commercially-licensed AI
models and Python venv are deliberately NOT baked in -- mount them read-only at
runtime; REST/folder workflows need no models. .dockerignore keeps generated media,
models, and local state out of the build context.

Not build-validated in this environment (no running Docker daemon); the file is a
standard reviewable artifact and a starting point -- no-egress operation, scanning,
and further hardening remain to be validated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 15:13:29 +02:00
JSLMPR 430d71fae8 Gate the REST render endpoint behind an approval flag
POST /v1/edit-projects/{projectId}:render was unauthenticated -- anyone could
trigger a render. It now requires an approved.flag in the project directory and
returns 409 otherwise (config video-clipping.editing.require-render-approval,
default true). This closes the "no check at all" hole; it is a basic presence gate,
not yet authenticated/digest-bound authorization (a remaining hardening item).
New SpringBootTest asserts 409 without approval; the delegation unit test disables
the gate. mvn verify 271/0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 12:38:12 +02:00
JSLMPR 31efe33081 Add CI workflow and a production README
- .github/workflows/ci.yml: build + test on every push/PR (JDK 21 + ffmpeg,
  mvn -B verify, uploads surefire reports). Closes the "no CI" gap.
- README.md: honest overview of the local offline highlight pipeline, the R1-R9
  cinematic quality rules, how to build/test/run, the local models and their
  (non-commercial) licenses, constraints, and limitations. Explicitly states it is
  a PoC, not production-hardened. Closes the "no README" gap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 12:17:08 +02:00
JSLMPR 2552a7cf06 docs(memory): update handover HEAD to 9fb24e8; R1-R9 complete 2026-07-24 12:06:11 +02:00
JSLMPR 9fb24e8f19 R6 speed-ramp: ease into slow-motion
A slow-motion shot now decelerates smoothly instead of snapping to slow-mo:
speedRampSetpts builds a log-integrated setpts that ramps playback speed from
normal (1.0) down to below the target across the shot. The shot stays a SINGLE
segment (so the R5 per-shot push-in is preserved) and the existing -t pin keeps
the planned output duration. Normal-speed shots keep a plain constant setpts.
Unit-tested; verified in a real render (payoff carries the ramp, output valid).
Completes R6 (crossfades + speed-ramp).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 12:04:32 +02:00
JSLMPR 4d2cea15a7 Fix: music/audio cut off early with crossfades (separate video+audio passes)
R6 ran the video xfade and audio acrossfade in one filtergraph on the same inputs,
which starved the audio path and truncated it (final audio 4.2s vs 9.7s video) --
so the music appeared to end early. Split into two passes (xfadeVideoCommand video
only, acrossfadeAudioCommand audio only) and mux them. Verified: final audio now
9.8s matching the 9.67s video, music present through the end. Isolation and unit
tests confirm the two chains each produce the full compressed length.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 11:52:43 +02:00
JSLMPR 99ec273020 docs(memory): record current HEAD (a2a7d7d) + tmpdir/sandbox workaround
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 09:42:57 +02:00
JSLMPR a2a7d7d736 R8: swell the music into the payoff
MusicGen's internal structure is uncontrolled, so the mix applies a deterministic
swell envelope to the score (volume='min(1,0.5+0.5*t/peak)':eval=frame): the music
amplitude rises from 0.5x to full over the run-up to the payoff, then holds. The
peak is the payoff (slow-motion) beat's timeline midpoint on the crossfade-
compressed timeline -- generic, driven only by the plan, no content assumptions.
No-ops when there is no slow-mo beat. Filter string unit-tested; verified in a real
render (swell peaks at 6.83s, output -16.1 LUFS).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 09:37:43 +02:00
JSLMPR 407f6b5e04 Tier-2 caption-driven shot selection (semantic editor, not just decorator)
The vision director now captions several beat frames (two questions per frame in
one worker call: a discriminative description + a punchy label) and turns the
descriptions into a per-window "highlight-worthiness" curve via generic
emotion/action/idle keyword scoring (semanticScore/semanticCurve). The montage
director blends that curve with audio to place the payoff on the semantically
strongest moment; the payoff label becomes the bold overlay and the description
flavors the music. Everything is content-agnostic and fails soft to the measured
cut. Verified on bowling: the payoff moved onto moondream's detected celebration.

Honest limit: a small VLM on distant subjects is only weakly discriminative;
descriptive questions beat terse ones (which collapse to a constant answer).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 01:31:48 +02:00
JSLMPR 67bdf683ca R7: bold, animated overlay captions
Overlays are now large (fontsize 84) with a thick outline + strong drop shadow so
they read on any background, and animate in: a snappy 0.18s alpha punch plus a
34px rise-up over 0.22s, with a soft ease-out. Placed on the payoff beat so the
entrance lands on the musical/edit accent. This presents the Tier-2 vision
director semantic caption ("STRIKE") boldly. Overlay style/animation asserted in
the existing overlay test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-24 00:05:39 +02:00
JSLMPR f03d5472c5 R6: cross-dissolve transitions between montage beats
Beats now blend instead of hard-cutting (incl. the cut into the slow-mo payoff)
via an xfade + acrossfade chain (xfadeTimelineCommand). The timeline compresses
by (n-1)*xf, so shiftOverlayForCrossfade re-times overlays and the reported
duration is reduced to keep overlays, loudness mastering, and QA aligned. Opt-in
via editing.crossfade-seconds (0 = hard cuts default; localpoc 0.25), clamped to
half the shortest beat. Offset math + overlay shift unit-tested; verified on the
bowling cut (visible dissolve, STRIKE overlay stayed on the payoff).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-23 23:41:12 +02:00
JSLMPR 30c832e9d5 Accurate loudness mastering: measure the finished file and correct to target
Single-pass loudnorm in the mix is only ~+/-2 LUFS accurate, so the auto-rendered
output could land quiet (e.g. -18.7 LUFS vs the -16 target). After the mix,
probeIntegratedLoudness measures the file, loudnessGainDb computes the corrective
gain, and masterLoudness applies it with a brickwall limiter for true peak. No-ops
when already on target or when the measurement is implausible; handles MusicGen
loudness variance. loudnessGainDb unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-23 23:09:06 +02:00
JSLMPR 1da0661eea Automatic two-tier highlight director + source-adaptive rendering
Rendering (whole service, driven by source measurements, not constants):
- Orientation-aware geometry: FfmpegClipInspector reads display rotation and
  stores effective dims; HighlightFfmpegRenderer.outputGeometry renders portrait
  sources portrait and skips the 2.39 letterbox on portrait (landscape unchanged).
- Dynamic exposure: probeSourceLuma measures the frames; exposureNormalizationFilter
  maps the mean toward a target; the grade is now exposure-preserving (no crushed
  subjects: bowling final went ~63 -> ~100 mean luma).
- Motion-adaptive in-shot push-in (zoompan), amount from per-shot YDIF.
- Audio mix ducks source audio under the generated score so it leads.
- Highlight duration is no longer capped (validator + config).

Automatic director (plans were hand-authored before):
- Tier 1 HighlightMontageDirector: composes the montage from measured motion (YDIF)
  and audio-energy (RMS) curves -- setup, continuous action/tension, slow-mo payoff
  on the audio climax, resolution button, camera-whip tail trimmed.
- Tier 2 HighlightVisionDirector + tools/vision_caption.py: a local, offline
  vision-language model (moondream2) captions the payoff frame and augments the
  montage with a semantic overlay ("STRIKE") and scene-informed music; fails soft.
- Wired into the scheduler behind auto-director-enabled / vision-director-enabled
  (on in the localpoc profile).

Docs: cinematic-quality-rules.md (R1-R5, R9 both tiers), poc-plan milestones.
Tests: mvn -o verify -> 262 passing, 0 failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
2026-07-23 22:49:05 +02:00
JSLMPR 8b68116dae docs: record Phase 5 next-steps roadmap
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 22:51:18 +02:00
JSLMPR b3718dd0b8 Add cinematic montage mode (break free of the 8-35s highlight windows)
A montage edits many short shots pulled from ANYWHERE in the source (not a few
fixed contiguous highlight windows), sequenced establishing -> quick detail cuts ->
slow-motion hero, over one continuous music bed -- much closer to how a car film is
actually cut.

- MontagePlan model + director/montage.json shot list (per-shot source time,
  duration, punch-in zoom, speed).
- HighlightDirectorFlowService.processMontage: builds one EditPlan from the shot
  list (bypassing the highlight-window validator), one continuous music cue,
  distributed voiceover, and titles; reuses asset prep + worker + renderer.
- Renderer: explicit per-shot framing via a "zoom=" token (falls back to the
  progressive punch-in); pin each segment to its exact target duration with -t so
  frame-quantization drift cannot accumulate across many short shots.

Verified on the DJI source: 18 shots, 16.7 s, -16.5 LUFS, TP -2.8 dBTP, QA green.
The edit now reads as a real montage (varied framing + detail cuts + hero) rather
than slow pans. mvn -o verify green (248). Remaining ceiling is the source footage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 22:35:08 +02:00
JSLMPR daa58a8c8d Dynamic multi-cut edit: tighter framing + progressive punch-in per beat
Turn each beat from one long pan into an edit:
- Flow splits every beat into contiguous cuts (opening 2, rising 3, hero 2) with
  hard cuts between and sequential timeline positions; playback speed is preserved
  so the hero stays slow-motion across its cuts.
- Renderer frames each cut with a tighter center crop that fills the frame and
  hides the mundane location, plus a progressive punch-in (each successive cut of a
  beat steps tighter), so the sequence reads as deliberate.
- Widen the duration QA tolerance to the frozen acceptance value (0.25 s): multi-cut
  and slow-motion accumulate small per-segment frame-quantization drift.

Verified: 7 cuts (2/3/2), final 27.4 s, TP -2.7 dBTP. mvn -o verify green (248).
Known follow-up: integrated loudness runs low (~-18.7 LUFS) on the sparse music-bed
mix; single-pass loudnorm undershoots -- needs two-pass or a louder bed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 19:05:16 +02:00
JSLMPR d3d2a1b14e Soften voiceover entry/exit with short fades
Add afade in (0.12s) / out (0.25s) to each narration line so it eases in and out
instead of hard-cutting -- the hard cut read as un-cinematic. Voice remains
present and leading (measured onset ramp -18 -> -17 dB).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 18:19:27 +02:00
JSLMPR aae92ed7a5 Fix: voiceover was silently dropped from every mix; remove SFX gating
Root cause of "no voiceover": in audioMixCommand the [voice] label was used both
as the sidechaincompress key AND as an amix input. FFmpeg does NOT auto-split a
reused label, so the ducking sidechain consumed the voice entirely and the amix
reference got no audio -- the narration never made it into any final mix (every
render played ducked music with a silent hole where the voice should be).

Fix: explicitly `asplit=2[voice_key][voice_mix]` so one copy keys the duck and one
copy stays in the mix. Measured: voice window went from ~-30..-53 dB (silent) to
~-16 dB (present, leading the bed). Revert the compensating +8 dB over-boost back
to unity; music bed lowered to -14 dB so narration leads cleanly.

Also make SFX direction OPTIONAL in the validator (matching the earlier
voiceover-optional change) so a clean music+narration edit with no sound effects
validates. Regression test added for the asplit. mvn -o verify green (248 tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 18:03:38 +02:00
JSLMPR 235810e253 docs: log Phase 4 cinematic pass progress
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 16:15:02 +02:00
JSLMPR 842f2a7404 Cinematic pass: 2.39 letterbox, film grain, optional voiceover
Push the highlight output toward a genuinely cinematic look after a human review
found it functional-but-not-cinematic:

- HighlightFfmpegRenderer: add a 2.39:1 letterbox and subtle film grain to each
  graded segment; raise overlay placement above the letterbox bar. (A time-based
  crop-zoom push-in was prototyped and removed: FFmpeg crop cannot use the `t`
  variable for width/height; a zoompan push-in is a possible follow-up.)
- HighlightDirectorPlanValidator: make voiceover OPTIONAL so a music-driven edit
  can carry no narration (lines still validated when present).

Combined with a no-voiceover, driving-orchestral, slow-motion-hero director plan,
this yields a letterboxed, richly graded, music-led cinematic cut. mvn -o verify
green (247 tests, 0 failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 16:13:19 +02:00
JSLMPR 3221b9827b docs: add P3.6 acceptance review (frozen thresholds + rubric); Gate A measured PASS
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 11:59:06 +02:00
JSLMPR 6600ded929 docs: P3.5 mix/ducking review findings (calibration deferred to P3.6)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 11:43:10 +02:00
JSLMPR 56205d91f2 Fix local CV visual-analysis client (HTTP/1.1) to enable YOLO selection
The local-cv provider was never exercised end to end (its bootstrap was
prohibited), and hid a latent bug: the JDK HttpClient defaulted to HTTP/2 and
negotiated an h2c cleartext upgrade that the HTTP/1.1-only worker (uvicorn/h11)
mishandled by dropping the request body, so every call returned HTTP 422. Pin the
client to HTTP/1.1.

With this fix the resident YOLOv8 worker (run offline against the existing
yolov8n.pt, no bootstrap script) classifies the sample source as CAR_VLOG at 0.95
with measured OpenCV blur/exposure and a real car label, replacing the previous
filename-keyword GENERIC_VLOG fallback. Provider remains opt-in via runtime
override; the committed localpoc profile keeps the heuristic default.

Also ignore yolov*.pt.license.txt (provenance for the git-ignored weights).
mvn -o verify green (247 tests, 0 failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 11:14:48 +02:00
JSLMPR 9d217c249e docs: log beat-specific grading milestone
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 10:31:41 +02:00
JSLMPR a0b6023b9c Beat-specific cinematic grading for highlights
The highlight renderer now selects a filmic grade per story beat, read from the
EditPlan style key (safeKey: style_<category>_<storyPurpose>), so no change to the
shared EditDecision or its serialization is needed:

- opening_hook: calmer, cooler, softer contrast (linear_contrast, gentle vignette)
- rising_energy / default: balanced base grade (medium_contrast)
- hero_payoff: richest, warmer, stronger S-curve and deeper vignette (strong_contrast)

Threads plan.style() into segmentCommand -> cinematicVisualFilter. Multi-clip
FfmpegEditRenderer untouched. Test added; mvn -o verify green (247 tests, 0 failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 10:19:52 +02:00
JSLMPR 1a1566adda docs: log Phase 3.3 overlay styling milestone
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 10:01:01 +02:00
JSLMPR 24bd1d78f2 Premium highlight overlay styling (P3.3)
Replace the plain hard-cut white caption with a refined 48px caption, a soft drop
shadow, a thin subtle border, and a smooth alpha fade in and out (0.4s ramps)
within each overlay timeline window. Highlight renderer only; the multi-clip
renderer and overlay placement/safe-area logic are unchanged. Test added;
mvn -o verify green (246 tests, 0 failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 09:52:12 +02:00
JSLMPR 0dd64d96cd docs: log Phase 2 render + Phase 3.1/3.2 milestones
Record the operator-approved first render and the verified grade + true-peak
limiter results (TP -1.7/-2.7/-2.8 dBFS, I -16.3 LUFS) with commit references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 09:41:51 +02:00
JSLMPR 0fcfca9283 Improve highlight render: filmic grade + true-peak limiter
Phase 3 quality fixes for the highlight renderer (multi-clip FfmpegEditRenderer
left untouched):

- cinematicVisualFilter: replace the weak fixed eq with a deliberate filmic grade
  (curves medium_contrast S-curve + teal-orange colorbalance + eq + unsharp +
  vignette). Richer blue, warm highlights, tonal contrast. Still one uniform look;
  beat/category-specific grading is future work (needs a validated grade enum).
- audio mix: add a brickwall limiter (alimiter limit=0.72) after loudnorm. The
  first render clipped at 0.0 dBFS true peak; measured re-render now lands
  -1.7/-2.7/-2.8 dBFS per highlight (all within the -1.5 dBTP gate), integrated
  loudness -16.3 LUFS.

Test updated; mvn -o verify green (245 tests, 0 failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 09:38:47 +02:00
JSLMPR 588c652a2f Add opt-in localpoc profile for the local highlight PoC
Activated with --spring.profiles.active=localpoc. Points only at pre-provisioned
local model paths (Piper voice, MusicGen, AudioLDM2), disables bootstrap
auto-start, uses heuristic visual analysis, isolates PoC input/output dirs, and
keeps render disabled + director approval required. Base/production defaults are
untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-22 09:38:47 +02:00
JSLMPR a95d1fa0ac Wire local generative audio models into the highlight asset worker
Replace the unusable audiocraft path (requires xformers, which has no Intel-Mac
build) with runtimes proven to work offline on this machine:
- music: transformers MusicGen (facebook/musicgen-small)
- sfx:   diffusers AudioLDM2 (cvssp/audioldm2), resampled 16k -> 48k
- voiceover: Piper (unchanged), normalized to 48 kHz mono

The worker CLI contract and exit codes are preserved, so the Java
LocalAssetSynthesizer license gate and fail-closed behavior are unchanged.
Add tools/provision_local_models.py to materialize models into models/ from the
local HF cache with no network. Models and their license sidecars live under the
git-ignored models/ dir; both audio models are non-commercial (CC-BY-NC-4.0 /
CC-BY-NC-SA-4.0), recorded for later production review.

Add docs/cinematic-highlight-poc-plan.md tracking the PoC plan and milestones.
mvn -o verify: 245 tests, 0 failures/errors/skips (unchanged).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bg76sLc43Wc3j5ZcLkboYR
2026-07-21 23:00:57 +02:00
JSLMPR f6dc6b8a83 Harden cinematic highlight planning and rendering
- require explicit approval and skip ineligible highlight projects
  - validate timing and deduplicate candidate ranges
  - enforce licensed local assets and fail-closed generation
  - preserve video duration when mixing generated audio
  - refresh skill runbooks and regression coverage
2026-07-21 18:32:35 +02:00
JSLMPR 97ba827d50 add asset generation 2026-07-21 10:03:27 +02:00
156 changed files with 15046 additions and 327 deletions

View File

@ -0,0 +1,99 @@
# AI Handover: Local Cinematic Highlight PoC
Last updated: 2026-07-23 (Europe/Berlin). **This file goes stale — verify before trusting it.**
## FIRST: check for staleness (prevents the 2026-07-23 stale-session incident)
This handover was once resumed from an old commit while the repo had advanced 2 days on another
branch/agent, wasting a whole session re-deriving already-built work. Do NOT let that happen:
```bash
git rev-parse --short HEAD # compare to "Recorded HEAD" below
git log -8 --oneline --decorate
git status --short # uncommitted work may be the real current state
```
- **Recorded HEAD when this was written: `9fb24e8`** ("R6 speed-ramp: ease into slow-motion"). The 2026-07-23/24
session's quality work: `1da0661` two-tier director + source-adaptive rendering, `30c832e` loudness mastering,
`f03d547` R6 crossfades, `67bdf68` R7 bold overlays, `407f6b5` Tier-2 caption-driven selection, `a2a7d7d` R8
music swell, `4d2cea1` fix music-cut-off-early (separate video/audio crossfade passes), `9fb24e8` R6 speed-ramp.
**The cinematic ruleset is now R1R14 (docs/cinematic-quality-rules.md).** 2026-07-25 session added, from a
web review of "what makes video cinematic" + how DJI LightCut/Insta360 auto-edit: R10 24fps cadence, R11
filmic tone-curve grade + optional licensed `lut3d` hook, R12 motion blur, R13 beat-synced cuts
(`HighlightBeatSync` + `tools/beat_detect.py`, librosa), R14 subject-tracking reframe (`HighlightSubjectTracker`
+ `tools/subject_track.py`, YOLO/AGPL). Also fixed a latent 444→420p playback-compat defect (all encode passes
now pin `-pix_fmt yuv420p`). R13/R14 are opt-in flags, ON in localpoc. `mvn verify` = 288 tests green. Bowling
re-render verified: 24fps, yuv420p, 16.0 LUFS, 3 cut boundaries beat-snapped, all 5 shots subject-followed.
NOT yet committed as of this note. **The cinematic ruleset R1R9 baseline remains COMPLETE.**
- **Then production-hardening started (recorded HEAD `fd22412`):** `31efe33` CI (.github/workflows/ci.yml) +
README, `430d71f` render-approval gate on `POST /v1/edit-projects/{id}:render` (config `require-render-approval`,
default true, 409 without `approved.flag`), `452bab6` Dockerfile + .dockerignore (models mounted at runtime;
NOT build-validated — no Docker daemon here), `c323523` Maven Wrapper (pins Maven 3.9.9; CI uses `./mvnw`),
`fd22412` ported the e2e runbook + local-model facts into repo docs (`docs/RUNBOOK-highlight-e2e.md`,
`docs/LOCAL-MODELS.md`) so any AI/human reading the repo has them. `mvn -o verify` (with the tmpdir workaround)
= 271 tests, 0 failures, coverage met.
- **Production-readiness verdict (honest, 2026-07-24):** the output is a technically-clean, much-improved
cinematic DRAFT, NOT a certified production deliverable. Blockers: no blinded human creative Gate-B review has
passed; source is 576p (soft when upscaled); music is generic musicgen-small; VLM weak on distant subjects;
audio models are CC-BY-NC (non-commercial). Do not claim production-ready on technical probes alone.
- **Still remaining (large / decision-gated):** commercial licensing (P5.7), full authN/Spring Security,
PostgreSQL/Testcontainers, container build + no-egress certification, and the formal human Gate-B review (P5.6).
- **Env note (2026-07-24):** the sandbox began denying the default `$TMPDIR` and socket binds mid-session; see
[[highlight-e2e-render-runbook]] for the java.io.tmpdir + dangerouslyDisableSandbox workarounds.
- If HEAD is newer than `8b68116`, or there are unfamiliar commits/uncommitted files, **this file is behind
the repo**. The authoritative live state is: the git log, `git status`, and
`docs/cinematic-highlight-poc-plan.md` (milestone log at its tail). Trust those over this file.
- If a `claude agents` background session is still "running" for this repo, only one agent should mutate the
repo at a time (renders/commits/state). Confirm the others are stopped.
## What exists NOW (built, not aspirational)
The full local single-source cinematic-highlight pipeline is implemented and working end-to-end:
analysis → candidates/category → (manual) director plan → local asset generation → cinematic FFmpeg render
→ QA. Models are provisioned and proven (Piper voice, MusicGen music, AudioLDM2 SFX) — see
[[local-model-runtime-intel-mac]]. The user authorized model downloads on 2026-07-21 (this **reversed** the
earlier no-download stance; that reversal is real and current).
To run source → final render, follow [[highlight-e2e-render-runbook]] — do not re-derive it.
## Delivered 2026-07-23 (whole service, not just PoC) — committed at `1da0661`
**Automatic two-tier director** (plans were hand-authored before; now auto-generated):
- Tier 1 `HighlightMontageDirector`: composes `director/montage.json` from measured motion (YDIF) + audio (RMS)
— setup, continuous action, slow-mo payoff on the audio climax, resolution button, camera-whip tail trimmed.
- Tier 2 `HighlightVisionDirector` + `tools/vision_caption.py`: local moondream2 (offline) captions the payoff
→ semantic overlay ("STRIKE") + scene-informed music; fails soft. See [[local-model-runtime-intel-mac]].
- Wired into the scheduler behind `auto-director-enabled` / `vision-director-enabled` (localpoc on).
**Source-adaptive rendering fixes:**
1. **Portrait/orientation**: `FfmpegClipInspector` now reads rotation side-data/`rotate` tag and stores the
EFFECTIVE (display) width/height. `HighlightFfmpegRenderer.outputGeometry()` renders portrait sources to a
portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait phone footage being
stretched into landscape.
2. **Audio mix**: source audio (`[0:a]`) is now ducked (-20 dB, or -24 with narration) so the generated score
leads instead of being buried; montage music cue raised to -9 dB. In `HighlightFfmpegRenderer.audioMixCommand`
+ the montage cue in `HighlightDirectorFlowService`.
3. **Duration constraint removed**: `HighlightDirectorPlanValidator` no longer enforces min/max highlight
duration (only positive/finite + sane playback speed + candidate containment). Config defaults relaxed
(min 2 / max 3600 as soft candidate hints only). A highlight can be as long as the story needs. Montage path
was already unconstrained.
`mvn -o verify`: 254 tests, 0 failures (JaCoCo gate passes). 6 new tests cover the above. NOT yet committed —
the user has not asked to commit.
## Still open / next levers (from docs/cinematic-highlight-poc-plan.md Phase 5)
- Source footage quality is the ceiling (e.g. 576p phone clips upscale soft). Biggest lever.
- No resident local **director** — plans are hand-authored (`montage.json`). P5.2/P5.5.
- Bigger/continuous music (musicgen-medium is slow on CPU) — P5.3.
- In-shot `zoompan` push-in, beat-synced cut lengths — P5.4.
- Licensing: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL → non-commercial. Production needs
commercially-licensed models — P5.7.
- Production hardening (security, CI/CD, containers, PostgreSQL, no-egress cert) deferred until quality passes.
## Non-negotiables still in force (except downloads, which the user authorized)
No external AI services in the media path; no unlicensed assets shipped as production; no placeholder
silence/tones as success; no rendering without explicit approval; record license/provenance for every model.
Manual approval is fine for the PoC. Do not commit unless the user asks.

View File

@ -0,0 +1,30 @@
# Video Editing Service: Current State
Last verified: 2026-07-23 (Europe/Berlin). **Verify against git before trusting — see ai-handover.md.**
## Snapshot
- Branch `feature/imrovements`; recorded HEAD `8b68116` + uncommitted working-tree fixes (2026-07-23).
- `mvn -o verify`: **254 tests, 0 failures**, JaCoCo gate passes.
- The full local single-source cinematic-highlight pipeline is BUILT and works end-to-end:
analysis → candidates/category → manual `montage.json` director plan → local audio generation
(Piper voice, MusicGen music, AudioLDM2 SFX) → cinematic FFmpeg render (grade, grain, overlays,
multi-cut, slow-mo, orientation-aware geometry) → QA.
- Models provisioned + proven locally; user authorized downloads 2026-07-21 (reversed the no-download stance).
## Authoritative sources (do not duplicate them here)
- Live code/state: `git log`, `git status`.
- Roadmap + milestone log: `docs/cinematic-highlight-poc-plan.md`.
- How to render e2e: memory `highlight-e2e-render-runbook`.
- Model runtime facts: memory `local-model-runtime-intel-mac`.
- Handover + staleness check: `.claude/memory/ai-handover.md`.
## 2026-07-23 fixes (whole service)
- Orientation-aware: portrait sources render portrait (rotation read in `FfmpegClipInspector`,
geometry chosen in `HighlightFfmpegRenderer.outputGeometry`), letterbox landscape-only.
- Audio mix: source audio ducked under the generated score (score now leads).
- Highlight duration no longer capped (validator + config); a highlight can be as long as needed.
## Known limits / next (Phase 5 in poc-plan)
Source footage quality is the ceiling; no resident local director (plans are manual); bigger/continuous
music pending; licensing of local models is non-commercial (CC-BY-NC / AGPL); production hardening deferred
until the creative-quality gate passes. Highlight rendering still defaults OFF; approval still required.

View File

@ -0,0 +1,271 @@
---
name: cinematic-media-engineering-reference
description: Load when designing, debugging, or reviewing this repository's highlight detection, FFmpeg trimming/rendering, scene and visual analysis, audio mixing/mastering, voiceover, music/SFX generation, overlays/color, proxies/thumbnails, or local media-model runtime. Use it to interpret media metrics correctly and to separate current heuristics from production evidence.
---
# Cinematic Media Engineering Reference
Last verified against the repository: **2026-07-21**.
## Use this skill correctly
Use this skill to reason about the media domain and this repository's implementation. Before changing behavior, load `video-editing-change-control`. For commands, artifact locations, and startup safety, load `video-editing-run-and-operate`. For a failing run, load `video-editing-debugging-playbook`. For release evidence and thresholds, load `video-editing-validation-and-qa`. For property defaults, load `video-editing-config-and-flags`. For architecture boundaries, load `video-editing-architecture-contract`.
Do **not** use this skill as:
- permission to render, download a model, access a network, or change a production-facing default;
- proof that an output is cinematic because a heuristic score or `qa-report.json` says so;
- a licensing opinion or permission to use an asset/model;
- a replacement for final-media measurement and an approved human creative review.
## Context-specific operating consequences
The no-waiver policy is authoritative in `video-editing-change-control`. Apply these media-specific consequences before every media or model experiment:
1. Work offline. Pre-provision every executable, Python wheel, model weight, voice, font, LUT, music track, and SFX file through an approved artifact path.
2. Pin the artifact version and SHA-256; record its license, source, approval, target platform, and expected output contract.
3. Fail closed when a required model or asset is missing. Never substitute silence, a sine tone, OS speech, a heuristic provider, or an unrelated library asset.
4. Treat rendering as a behavior-changing operation. Require explicit approval; do not rely on current packaged defaults.
5. Never use external AI services. Model/media inference must be in-process or use approved non-network IPC; loopback HTTP is network and is not a certified path.
6. Route any code, configuration, model, threshold, or render-profile change through `video-editing-change-control`, then through `video-editing-validation-and-qa` before promotion.
Current code still violates parts of this policy. `tools/run_local_cv_worker.sh` installs packages and may fetch `yolov8n.pt`; `tools/run_local_asset_worker.sh` installs unpinned packages; and `SourceVisualAnalyzer` can fall back to filename heuristics. The 2026-07-21 working tree removed asset silence/tone/host-speech fallbacks, made strict asset readiness fail startup, and requires local model paths, but no complete resident asset-model bundle is present. Do not run the bootstrap launchers in a controlled or production environment.
## Map the actual media flow
| Stage | Current implementation | Interpret it as |
|---|---|---|
| Inspect | `FfmpegClipInspector` reads format duration, codecs, width/height, and `r_frame_rate` | Container-level summary, not a complete timing/color/channel contract |
| Sample | `ThumbnailExtractor` takes 5 evenly spaced JPEGs by default; `ProxyGenerator` makes 640-wide H.264 CRF 28/AAC 96k proxies | Review aids; neither is full-source evidence |
| Segment | `ShotSceneSegmenter` uses FFmpeg `select='gt(scene,0.35)',metadata=print`, then rejects sub-1-second boundaries | Frame-difference boundary proposals, not semantic scenes or highlights |
| Analyze audio | `SourceAudioAnalyzer` chains `silencedetect` and `volumedetect` | dBFS mean/maximum and thresholded silence only; not LUFS, true peak, dialogue, beat, or quality analysis |
| Analyze visuals | `LocalCvVisualAnalysisProvider` calls `tools/local_cv_worker.py`; optional fallback is `HeuristicVisualAnalysisProvider` | Sparse-thumbnail technical hints with major blind spots |
| Rank | `HighlightCandidateGenerator` makes shot-aligned or overlapping 12-second coverage windows, ranks with source-level visual scores, scene score, duration, and non-silence, and keeps the configured top 3 by default | Deterministic baseline with explicit fallback downweighting; not semantic or temporally resolved multimodal understanding |
| Direct | `HighlightDirectorPromptGenerator` asks a filesystem-capable model for `director/edit-plan.json`; `HighlightDirectorPlanValidator` gates it | Still an untrusted creative proposal until validated and reviewed; validation proves contract safety, not artistic quality |
| Prepare assets | `HighlightAssetPreparationService`, `HighlightLocalAssetWorker`, `LocalAssetSynthesizer` | File materialization; existence does not prove provenance, intelligibility, fit, or quality |
| Render | `HighlightDirectorFlowService` makes one `EditDecision` per highlight; `HighlightFfmpegRenderer` renders and stream-concats | A single source range per highlight, not an internally multi-shot edit |
| Report | renderer writes `render-manifest.json` and `qa-report.json`; the highlight path probes duration, black ranges, long silence, and sample peaks | Technical evidence only; loudness, true peak, A/V sync, freeze detection, raster text safety, and creative fit remain unmeasured |
The two ranking paths are deliberately distinct. `CinematicHighlightAnalyzer` remains the older multi-clip edit-project ranker and writes root-level files. The single-source `HighlightSourceScheduler` now invokes `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Treat its category as model-supported only when independent local visual labels clear the threshold; metadata/fallback evidence stays low-confidence `generic_vlog` and requires media review.
## Reason about time before touching cuts
Define these terms once:
- **PTS** (presentation timestamp): when a decoded frame/sample is presented.
- **DTS** (decode timestamp): when it is decoded; reordered codecs may have DTS different from PTS.
- **time base**: seconds per timestamp tick. Wall-clock seconds are `timestamp * time_base`.
- **CFR/VFR**: constant/variable frame rate. `r_frame_rate` is not proof of CFR and is not a frame clock.
- **GOP** (group of pictures): frames between independently decodable keyframes; non-keyframes depend on neighbors.
Current plans round seconds to milliseconds. `FfmpegClipInspector` does not retain `time_base`, `start_time`, `avg_frame_rate`, frame count, rotation, sample aspect ratio, or VFR evidence. Do not assume `frame = seconds * r_frame_rate`, do not compare decimal timestamps as if they were original PTS, and do not promise frame-accurate cuts from `analysis/ffprobe.json` alone.
Inspect the missing timing facts without writing media:
```bash
INPUT=path/to/source.mp4
ffprobe -v error -show_entries \
format=start_time,duration,format_name:stream=index,codec_type,codec_name,time_base,start_time,duration,r_frame_rate,avg_frame_rate,nb_frames,width,height,pix_fmt,sample_aspect_ratio,sample_rate,channel_layout \
-of json "$INPUT"
ffprobe -v error -select_streams v:0 -show_frames \
-show_entries frame=best_effort_timestamp_time,key_frame,pict_type \
-of csv=p=0 "$INPUT" | head -n 40
```
Interpret boundaries as half-open ranges `[start,end)`. For every decision, enforce the repository's duration equation:
```text
rendered_duration = (source_end - source_start) / playback_speed
```
`EditPlanValidator` permits only 0.25x through 4.0x and allows 0.05 seconds of equation error. `HighlightDirectorFlowService` independently clamps speed into that range but does not invoke the validator.
### Copy versus re-encode
| Operation | Boundary behavior | Repository use |
|---|---|---|
| `-c copy` trim | Start commonly snaps to dependency/keyframe constraints; no filter can run | Folder clipping may preserve input quality; do not call this an exact creative cut without packet evidence |
| Decode + encode trim | Decoder can discard to requested time and encoder creates a new boundary; still quantized to stream/filter time bases | Both edit renderers trim each segment, filter it, and encode H.264/AAC |
| Concat demuxer + `-c copy` | Requires stream-compatible segments: codec parameters, time bases, stream layout, and ordering must agree | Both renderers join normalized segments this way; mixed audio-presence/channel layouts remain a risk |
| `xfade`/`acrossfade` | Requires overlapping, timestamp-normalized streams and shortens/repositions the combined timeline | Not implemented |
The string `crossfade` is misleading today. Both renderers translate it into a fade-out on one segment plus a fade-in on the next, then concatenate them without overlap. `HighlightDirectorFlowService` itself emits only `cut`, `fade-in`, and `fade-out`.
## Understand codecs, containers, and formats here
Keep three layers separate: a **container** (MP4/MOV/MKV) stores streams; a **codec** (H.264/HEVC/AAC) compresses a stream; a decoded **pixel/sample format** describes raw data.
The current segment contract is MP4 containing libx264 H.264 at configured resolution and frame rate, `yuv420p`, CRF 18, preset `veryfast`, plus optional AAC at the configured bitrate and 48 kHz default. The configured `video-bitrate` is not used by either editing renderer. `ProxyGenerator` separately emits H.264 CRF 28, width 640 by default, and AAC 96k. `ThumbnailExtractor` emits JPEG at `-q:v 2`.
Do not infer production compatibility from the `.mp4` suffix. Before concat or delivery, compare stream count, codec/profile/level, pixel format, dimensions, time base, frame rate behavior, audio sample rate, channel layout, and color metadata. The current inspector cannot do that comparison.
The renderers scale to fit and pad to 1920x1080 by default. This preserves the whole source frame but can letterbox/pillarbox. Any non-`none` visual treatment also applies the same fixed contrast/saturation/brightness, unsharp, vignette, and a 4% center crop in `HighlightFfmpegRenderer`; the treatment text does not select a distinct grade. `HighlightVisualEffectsStage` writes descriptive effect names only. It does not execute LUTs, grain, animated overlays, or prompt-specific effects.
## Read the visual signals without overclaiming
| Signal | Exact current calculation | What it can and cannot mean |
|---|---|---|
| Scene score | FFmpeg `scene` metadata at selected frames; threshold 0.35 | Large frame difference. It can fire on flashes/camera motion and miss semantic changes. `ShotSegment.sceneScore` is the boundary score entering that segment, not its average quality. |
| Cut-density “motion” | `clamp(0.25 + ((segments-1)/duration*60)/20)` | Editing/camera-change density, not optical or subject motion. A static multicamera cut sequence can score high; a continuous action shot can score low. |
| Blur score | Average `clamp(variance(Laplacian(gray))/500)` over readable thumbnails | Really a sparse sharpness/edge-energy score: higher is sharper. Noise, text, and texture inflate it; intentional shallow focus and low-detail frames depress it. |
| Exposure score | Average `clamp(1 - 2*abs(mean(gray)/255 - 0.5))` | Mean proximity to mid-gray. It does not detect clipped highlights, crushed shadows, contrast, skin exposure, HDR transfer, or local exposure. |
| Composition | Average of aspect score (0.8 for 1.70-1.90, else 0.55) and thumbnail-presence score (0.7 or 0.45) | An aspect/file-presence prior, not composition analysis. |
| Face presence | OpenCV frontal-face Haar cascade; any detection yields `faces_detected` | Sparse, frontal, binary presence only. No identity, consent, expression, tracking, screen area, or false-positive calibration. |
| Objects | YOLO max confidence per class over sampled thumbnails, top 10 | Detector confidence on sparse frames, not semantic relevance or continuous presence. Current default `yolov8n.pt` may auto-download. |
| Filename heuristic | Tokens such as `porsche`, `car`, `food`, `family` in clip ID/path | Metadata guess only. Confidence constants are authored values, not calibrated probabilities. |
When OpenCV or thumbnails are unavailable, the worker returns 0.5 quality defaults. When YOLO yields nothing, it returns `unknown` at 0.2. These are sentinel-like values but are not explicitly typed as “unknown”; never aggregate them as measurements. `SourceVisualAnalyzer` marks a local-CV failure with `local_cv_failed_fallback_...`, so gate on `analysisMethod`, not just numeric fields.
### Worked visual example
For 13 detected segments in a 120-second source, the current motion heuristic gives:
```text
cuts_per_minute = (13 - 1) / 120 * 60 = 6
motion_score = 0.25 + 6 / 20 = 0.55
```
This proves only that the scene detector retained about six boundaries per minute at the configured threshold/minimum duration. It does not prove visible motion or highlight value. Discriminate the causes by reviewing frames around each boundary and a proxy interval on both sides.
## Evaluate highlight candidates as evidence
Current `CinematicHighlightAnalyzer`:
1. Classifies from project name/style/clip path substrings.
2. Splits each clip into eight-second windows.
3. Caps at 12 windows and spreads them through long clips.
4. Scores `min(1, 0.35 + 0.12*(max(.1,sharpness)+max(.1,brightness)+max(.1,motion)) + role + category)`.
5. Assigns role from position: first is hook, last is hero, early third is rising action.
Because `FfmpegClipInspector` initializes all three quality fields to zero and no current enrichment replaces them, quality contributes exactly `0.036`. A first car candidate therefore scores `0.35 + 0.036 + 0.25 + 0.20 = 0.836` without inspecting its pixels or sound. Treat the ranking as a coverage scaffold.
For a production candidate evaluator, require each candidate to carry independently measured and normalized features plus their availability masks:
| Dimension | Discriminating evidence | Reject or penalize when |
|---|---|---|
| Technical usability | sharpness distribution, highlight/shadow clipping, shake, occlusion, audio noise/clipping | defect defeats the intended role unless narratively essential |
| Semantic moment | local embeddings/detectors/transcript tied to sampled timestamps | evidence does not support the category/action claim |
| Novelty | distance from already selected shots in visual/audio embedding space | angle/action is redundant |
| Story function | observable setup, action, reaction, reveal, or payoff | role is assigned only from source position |
| Editability | clean handles before/after, usable continuity, compatible motion/audio | cut begins/ends mid-action or cannot bridge coherently |
| Audio value | dialogue/event/ambience confidence and intelligibility | silence/noise is mistaken for low/high drama |
Predict expected metric movement before changing a model or weight. Evaluate on an approved, licensed golden set using timestamp-level relevance, diversity, coverage, false-positive cost, and blinded human pairwise preference. Never tune and certify on the same sources.
## Engineer sound instead of merely attaching files
Define the units:
- **dBFS**: sample amplitude relative to digital full scale; 0 dBFS is the sample ceiling.
- **LUFS-I**: program integrated loudness with perceptual weighting and gating.
- **dBTP**: reconstructed true peak, which can exceed sample peak between samples.
- **LRA**: loudness range, a statistical measure of loudness variation, not simply max minus min.
`SourceAudioAnalyzer` reports `volumedetect` mean/max dBFS and `silencedetect` ranges at -35 dB for 0.5 seconds by default. It does not report any LUFS, dBTP, LRA, clipping count, speech intelligibility, noise floor, music beats, or semantic events. For missing audio it reports `audioPresent=false` but numeric mean/max values of `0`; never interpret those zeros as loud audio.
Measure a final file read-only:
```bash
FINAL=path/to/final.mp4
ffmpeg -hide_banner -nostats -i "$FINAL" \
-af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json -f null -
ffmpeg -hide_banner -nostats -i "$FINAL" \
-af ebur128=peak=true -f null -
ffmpeg -hide_banner -nostats -i "$FINAL" \
-af silencedetect=n=-35dB:d=0.5 -f null -
```
The repository defaults target -16 LUFS-I, -1.5 dBTP, and LRA 11, but these are configuration targets, not certified acceptance thresholds. The renderer uses one-pass `loudnorm`; exact conformance requires measuring the encoded output, and deterministic file normalization normally requires a measured first pass followed by a second pass using the measured fields.
### Current mix graph and its consequences
- Source audio is always referenced as `[0:a]`; a video-only timeline can make mixing fail.
- Music now uses the plan cue's `gainDb`, start, end, trim, and delay. It still needs final-media loudness measurement and a proven scene-fit generation model.
- Each script line is a distinct voiceover asset, trimmed and delayed to its planned `VoiceoverLine` range, then combined into the voice bus at unity gain. Generated waveform duration and intelligibility are not yet measured against that slot.
- Music is sidechain-compressed only when voiceover exists, using configured threshold 0.045, ratio 8, attack 20 ms, release 250 ms by default.
- SFX are trimmed, gain-adjusted in dB, and delayed to their cue start.
- `amix=duration=first` makes the source timeline define output length, then one-pass loudness normalization feeds AAC 192k/48 kHz by default.
- Audio-mix and preview failures fail the highlight render. The mastering check proves that the successful command contained `loudnorm`; it does not measure encoded LUFS-I or true peak.
Therefore asset existence and a passed QA JSON are not proof of audible voiceover/music/SFX. Verify stream presence, timing, loudness, true peak, silence, and content audibly and instrumentally. A max sample peak of 0 dBFS signals risk but does not alone prove intersample clipping; use true-peak analysis.
For voice timing, derive speech duration from the generated waveform, not `characters / constant`. Place each line with `adelay`, trim/pad deliberately, leave breaths and edit handles, and fail when a line cannot fit its approved slot without unacceptable time-stretch. Duck from the aligned voice bus, not an unaligned full-file narration.
## Apply color, LUTs, and overlays safely
Before grading, preserve and inspect transfer, primaries, matrix, range, bit depth, pixel format, and rotation. Current inspection and rendering do not preserve or validate this color contract explicitly. A numeric `eq` preset is not color management and can damage already-graded, log, HDR, or limited-range footage.
Treat a LUT as executable media data: approve its license and checksum; document expected input/output color spaces; test neutral ramps, skin/brand colors, legal range, banding, and highlight roll-off. Although a LUT folder is configured, neither renderer uses it.
Current overlays use FFmpeg `drawtext`, 54 px white type, 2 px dark border, and positions around 6% horizontal/8-10% vertical margins. The `animation` field is validated in the multi-clip path but ignored by both renderers. Fonts are not explicitly selected, so output depends on host font resolution. The highlight QA check rejects out-of-timeline and unsupported placement values, but “safe” placement names are heuristics, not proof against glyph width, multiline wrapping, platform UI, subtitles, crops, or localization.
Require a pinned licensed font, measured text bounds, explicit wrapping, title/action safe policy for target platforms, and raster review at every output aspect ratio. Do not approve an overlay because `text_overlays_safe=true`; it proves plan bounds, not raster safety.
## Make local inference reproducible
For every model runtime, record this minimum manifest next to the approved deployment artifact, not as tribal knowledge:
| Field | Required content |
|---|---|
| Identity | task, model family, exact revision/weight filename, SHA-256 |
| Provenance | authoritative source, retrieval date, license text/version, commercial-use decision, approver |
| Runtime | OS/architecture, CPU/GPU/accelerator, driver, Python, native libraries, framework/wheel hashes |
| Inference | preprocessing, image/audio sample policy, thresholds, seeds, deterministic settings, precision/device |
| Contract | input/output schema, label map/voice identity, expected sample rate/channel/pixel format |
| Evidence | golden-set version/hash, accuracy/quality results, latency, memory, known failure slices |
Use platform-specific artifacts for macOS development, Linux/VPS, and cloud images, but prove semantic parity on the same golden inputs. Bit-identical neural outputs across different accelerators may be unrealistic; define allowed numeric/output drift before testing and compare downstream decisions as well as tensors.
Current gaps are blocking for production reproducibility: `tools/local_asset_requirements.txt` has no versions/hashes; local CV package versions are pinned but not hash-locked; model names are mutable aliases; no model/asset checksum or license manifest is enforced; the runtime can select OS `say`/`espeak`; AudioCraft `get_pretrained` and YOLO name loading can use network/cache state.
Inventory pre-provisioned artifacts without downloading anything:
```bash
find input/highlights/assets -type f -print 2>/dev/null | sort
find . -type f \( -name '*.pt' -o -name '*.pth' -o -name '*.onnx' -o -name '*.bin' \) -print
shasum -a 256 path/to/model-or-asset
```
On Linux, use `sha256sum path/to/model-or-asset`. A hash proves identity, not license or quality.
### Placeholder trap: worked trace
As of 2026-07-21, unavailable AudioCraft/Piper models return failure, the worker does not emit tones, silence, or host speech, `LocalAssetSynthesizer` deletes failed/inaudible output, and requested missing assets block rendering. The highlight path creates script-specific per-line voice files and delays them to planned reading-time windows before combining the voice bus for ducking. Detecting nonzero PCM is still not speech intelligibility or script-faithfulness validation, so keep approved rendering blocked until semantic audio QA and licensed immutable model provenance exist.
## Interpret final output with a measured checklist
Before calling a render production-ready, require all of the following:
- Source selections trace to timestamped evidence and are diverse, technically usable, and story-relevant.
- Every plan range is within the real stream timeline and survives decode-boundary inspection.
- Final streams match an approved codec/container/pixel/audio/color contract.
- Final duration is probed, not copied from plan arithmetic.
- Every required asset has approved provenance, checksum, license, model/runtime manifest, and intended content.
- Voiceover is intelligible, fact-grounded, aligned to its slots, and absent only by explicit creative decision.
- Music and SFX are aligned, mixed, ducked, and measured; no placeholder or silent fallback exists.
- LUFS-I, dBTP, LRA, silence, black frames, freeze/duplication, and A/V sync are measured against approved thresholds.
- Overlays are rendered with pinned fonts and checked for bounds/safe areas on each target.
- Color treatment is appropriate to the input color space and does not clip or shift protected colors.
- A blinded human review compares against the approved baseline; heuristic scores cannot waive this review.
- Promotion follows `video-editing-change-control` and evidence is retained under the process defined by `video-editing-validation-and-qa`.
## Provenance and maintenance
The implementation claims above come from `FfmpegClipInspector`, `ThumbnailExtractor`, `ProxyGenerator`, `WaveformGenerator`, `ShotSceneSegmenter`, `SourceAudioAnalyzer`, `SourceVisualAnalyzer`, `HeuristicVisualAnalysisProvider`, `LocalCvVisualAnalysisProvider`, `CinematicHighlightAnalyzer`, `HighlightSourceScheduler`, `HighlightDirectorPromptGenerator`, `HighlightDirectorFlowService`, `HighlightVisualEffectsStage`, `HighlightFfmpegRenderer`, `FfmpegEditRenderer`, `EditPlanValidator`, `LocalAssetRuntimeVerifier`, `LocalAssetSynthesizer`, `tools/local_cv_worker.py`, `tools/local_asset_worker.py`, both worker shell scripts, both requirements files, and `application.yml`.
Re-verify volatile facts with read-only commands:
```bash
rg -n 'ffmpeg|ffprobe|sceneDetection|silenceThreshold|loudness|ducking|autoStart|strictRuntime|fallback' src/main/java/org/example/videoclips src/main/resources/application.yml
rg -n 'filter|libx264|crf|loudnorm|sidechaincompress|amix|drawtext|concat|copy_post_timeline' src/main/java/org/example/videoclips/editing/*Renderer.java
rg -n 'Laplacian|brightness|cuts_per_minute|Haar|YOLO|yolov8n|fallback|silence|tone|get_pretrained' tools src/main/java/org/example/videoclips/editing
rg -n 'category.json|highlight-candidates.json|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing
sed -n '1,140p' src/main/resources/application.yml
sed -n '1,120p' tools/local_cv_requirements.txt
sed -n '1,120p' tools/local_asset_requirements.txt
ffmpeg -version
ffprobe -version
ffmpeg -hide_banner -filters | rg 'scene|loudnorm|ebur128|xfade|sidechaincompress|lut3d|drawtext'
```
If any command, property, path, formula, or wiring changes, update this skill in the same approved change. Date-stamp the new verification and preserve the distinction between implemented behavior, measurement theory, candidate improvements, and certified evidence.

View File

@ -0,0 +1,382 @@
---
name: video-editing-architecture-contract
description: Load when changing module boundaries, workflow orchestration, domain models, ports/adapters, project-folder protocols, state transitions, render approval, local-model integration, or persistence in the video-editing service; use it to preserve current contracts while moving the code toward a production-grade modular monolith.
---
# Video Editing Architecture Contract
Use this contract before changing a workflow boundary, state transition, persisted JSON shape, project directory, scheduler, renderer, model worker, or adapter. Treat it as a map of the implementation as verified on **2026-07-21**, not as proof that the target architecture already exists.
## Do not use this skill for
| Need | Load instead |
|---|---|
| Decide whether a proposed change is allowed, gated, or ready to merge | `video-editing-change-control` |
| Reproduce a build or repair a workstation/runtime | `video-editing-build-and-env` |
| Look up a property, default, or environment variable | `video-editing-config-and-flags` |
| Diagnose a concrete failure or stuck project | `video-editing-debugging-playbook` |
| Decide whether media output meets its evidence bar | `video-editing-validation-and-qa` |
| Operate schedulers, inputs, projects, or outputs | `video-editing-run-and-operate` |
| Improve highlight selection or cinematic quality | `video-editing-cinematic-highlights-campaign` |
Do not use architecture language to waive a quality gate. Route every behavior-changing proposal through `video-editing-change-control`.
## Define the terms
- **Capability**: a cohesive business responsibility, such as upload-and-clipping or cinematic highlight production.
- **Domain**: business concepts and invariants that can be tested without Spring, HTTP, JPA, FFmpeg, or the filesystem.
- **Application service**: orchestration for one use case; it calls domain behavior and outbound ports.
- **Port**: an interface owned by the core that expresses a required inbound or outbound capability.
- **Adapter**: framework or infrastructure code implementing a port, such as JPA, S3, FFmpeg, or a controller.
- **Filesystem state machine**: a workflow in which directory location and artifact presence act as queue and state signals.
- **Contract artifact**: a persisted file whose name, location, or JSON shape is consumed by another stage.
- **Modular monolith**: one deployable application with enforced capability boundaries and inward-pointing dependencies. One Maven module or one JVM process alone does not make the code modular.
## Start with the honest baseline
As of 2026-07-21, this is one Spring Boot application and one Maven module. It contains four workflows with different contracts. The upload-and-clipping workflow has recognizable ports and adapters. The folder workflow is a small vertical slice. The `editing` package contains multi-clip and single-source highlight domains, orchestration, filesystem persistence, process execution, and model-worker integration together.
The project does **not** currently have Spring Modulith, ArchUnit, automated package-boundary rules, a module diagram in code, or enforced clean-architecture dependency direction. `VideoAssetService` imports HTTP request/response DTOs, and `EditProjectController` invokes `EditRenderer` directly. Do not describe the current tree as a completed hexagonal architecture.
## Map the four workflows
| Workflow | Trigger and orchestrator | Durable state | Main outputs | Representative tests |
|---|---|---|---|---|
| API upload and clipping | `/v1/video-assets`, `/v1/clip-jobs`, and `/v1/clips`; `VideoAssetService`; `ClipJobQueuePort`; `ClipProcessor` | Repository records, object keys, and optionally `queue_messages` | Uploaded clip objects plus `Clip` records and signed download URLs | `VideoAssetControllerTest`, `DatabaseBackedClipJobQueueAdapterTest` |
| Folder segmentation | Poll `input-directory`; `FolderVideoScanScheduler` -> `FolderVideoValidator` -> `FolderFfmpegClipper` | File location: source -> working -> processed/rejected; `.failed` is last-resort quarantine | `output-directory/<source-base>[-N]/clip_%05d.<container>` | `FolderVideoScanSchedulerTest`, `FolderFfmpegIntegrationTest` |
| Multi-clip cinematic edit | REST API or poll `editing.local-director.source-directory`; `LocalDirectorScheduler`, `EditPlanInboxScanner`, `FfmpegEditRenderer` | `project.json`, analysis/plan/manifest JSON, inbox file names, and source-folder location | One project `final.mp4`, `render-manifest.json`, `qa-report.json` | `CinematicEditingIntegrationTest`, `EditPlanInboxScannerTest`, `FfmpegEditRendererTest` |
| Single-source highlights | Poll `editing.highlight-scheduler.source-directory`; `HighlightSourceScheduler`, `HighlightDirectorPlanScanner`, `HighlightDirectorFlowService` | Highlight project tree, `project.json`, director-plan presence, approval marker, final-file presence | Per-highlight final/preview/QA/manifests plus project `final.mp4` | `HighlightSourceSchedulerTest`, `HighlightDirectorFlowServiceTest`, `HighlightFfmpegRendererTest` |
Do not connect two workflows merely because both call FFmpeg or share records. First identify the owner of the use case, state, retry semantics, and output contract.
## Preserve the implemented dependency map
### Upload and clipping
Follow this current call direction while improving it incrementally:
```text
api controllers
-> application.VideoAssetService
-> application.VideoClippingRepository
-> queue.ClipJobQueuePort
-> storage.ObjectStoragePort
queue adapters -> processing.ClipProcessor
processing.ClipProcessor
-> application.VideoClippingRepository
-> processing.VideoClipperPort
-> storage.ObjectStoragePort
outbound implementations:
infrastructure.InMemoryVideoAssetRepository
persistence.JpaVideoClippingRepository
queue.InMemoryClipJobQueueAdapter / DatabaseBackedClipJobQueueAdapter
processing.StubVideoClipperAdapter / FfmpegVideoClipperAdapter
storage.InMemoryObjectStorageAdapter / S3ObjectStorageAdapter
```
Treat `VideoClippingRepository`, `ClipJobQueuePort`, `VideoClipperPort`, and `ObjectStoragePort` as existing contracts. Do not make core business behavior depend on JPA entities, Spring Data repositories, S3 types, or `ProcessBuilder`.
Current exception: `VideoAssetService` consumes types from `api.dto`. Move transport-to-command mapping to the inbound adapter when changing this surface; preserve API compatibility and add characterization tests first.
### Folder segmentation
Treat `org.example.videoclips.folder` as a self-contained file-ingestion capability. It currently calls FFprobe and FFmpeg directly through package-private functional test seams. Do not route folder jobs through the API repository or object storage unless a stated product requirement unifies their ownership and recovery model.
### Editing workflows
Recognize these existing interfaces, but do not mistake their package location for enforced isolation:
| Port-like interface | Current owner and purpose |
|---|---|
| `EditProjectStore` | Multi-clip project tree and top-level JSON artifacts |
| `HighlightProjectStore` | Single-source highlight project tree and nested artifacts |
| `EditRenderer` | Render a saved multi-clip project by project ID |
| `VoiceoverGenerator` | Produce project voiceover from timed lines |
| `EditAssetProvider` / `EditAssetLibrary` | Resolve or select local media assets |
| `VisualAnalysisProvider` | Analyze a clip from metadata, thumbnails, and shots |
Keep application-facing interfaces free of `Process`, HTTP client, and framework-specific return types. Introduce a port only for a concrete substitution, isolation, or testability need.
## Preserve each state machine
### API upload and clipping states
```text
VideoAsset: PENDING_UPLOAD -> UPLOADED -> PROCESSING -> READY
| failure/retry -> UPLOADED
any retained terminal asset -> DELETED
ClipJob: QUEUED -> RUNNING -> SUCCEEDED
| -> FAILED
QUEUED or RUNNING -> CANCEL_REQUESTED -> CANCELLED
```
Enforce these invariants:
- Create a job only for an `UPLOADED` or `READY` asset.
- Keep tenant ownership checks at every read or mutation boundary.
- Preserve idempotency for create-asset and create-job requests.
- Store generated media before exposing a persisted clip as downloadable.
- Return an asset to `UPLOADED` after a failed or cancelled processing attempt.
- Keep DB-queue retry ownership in `DatabaseBackedClipJobQueueAdapter`; do not persist terminal failure from `ClipProcessor` before retries are exhausted.
### Folder file states
```text
source candidate --atomic move--> working
working --valid + FFmpeg success--> processed
working --invalid or processing failure--> rejected
working --rejection move failure--> <name>.failed beside active file
```
Preserve one-file-per-scan behavior, natural filename ordering, refusal to overwrite a destination, and exclusion of hidden/partial/failed files. Atomic moves assume directories share a filesystem; treat cross-device deployment as unsupported until tested and designed.
### Multi-clip edit states
The implemented local-director path normally follows:
```text
CREATED -> ANALYZING -> ANALYZED -> WAITING_FOR_DIRECTOR
-> PLANNED -> RENDERING -> RENDERED
any handled workflow/render failure -> FAILED
```
`PLANNING` exists in `EditProjectStatus` but is not part of the normal scheduler sequence. `EditProjectService.updateProject` does not validate legal transitions. Add transition enforcement before relying on status as a concurrency or authorization boundary.
The source folder also moves `source -> working -> processed/rejected`. An accepted inbox plan moves from `inbox/edit-plan.json` to `inbox/edit-plan.json.accepted`; a rejected plan becomes `.rejected`. Approval is signaled by `inbox/approved.flag` when configured.
### Single-source highlight states
The observed new-project path is:
```text
CREATED -> WAITING_FOR_DIRECTOR -> RENDERING -> RENDERED
invalid empty director plan -> FAILED
```
`ANALYZING` and `PLANNED` exist in `HighlightProjectStatus`, but the primary scheduler/flow does not currently persist them. The render scanner considers a project renderable when `director/edit-plan.json` exists and project `final.mp4` does not. Its in-process `AtomicBoolean` prevents overlapping scans only inside one JVM; it is not a distributed claim.
Do not infer progress solely from the enum. Check artifacts and logs. Before horizontal scaling, add durable claims, leases, idempotent stages, and recovery tests.
## Preserve the two editing contracts separately
### Multi-clip contract
The root is `video-clipping.editing.project-directory` (default `./output/edit-projects`). A project currently uses:
```text
<project-id>/
project.json
analysis.json
cinematic-highlight-analysis.json
category.json
highlight-candidates.json
thumbnails/ contact-sheets/ proxies/ audio/
inbox/
edit-plan.json
final.mp4
render-manifest.json
qa-report.json
```
`EditProject` describes an input directory containing multiple clips. `EditPlan.decisions` can reference multiple clip IDs and builds one combined timeline.
### Single-source highlight contract
The root is `video-clipping.editing.highlight-project-directory` (default `./output/highlight-projects`). `FileSystemHighlightProjectStore` creates:
```text
<project-id>/
project.json
manifest.json
source/
analysis/
ffprobe.json
scene-segments.json
audio-analysis.json
visual-analysis.json
source-analysis.json
frames/ contact-sheets/ proxies/ audio/
director/
director-brief.md
director-prompt.md
edit-plan.json
approved.flag # only when approval is required
assets/{voiceover,music,sfx,overlays}/
highlights/<highlight-id>/
storyboard.md
edit-plan.json
assets/
final.mp4
preview.mp4
render-manifest.json
qa-report.json
final.mp4
render-manifest.json
```
`HighlightFolderContract.standard()` lists intended names such as `shots.json`, `scenes.json`, `transcript.json`, `audio-events.json`, `category.json`, and `highlight-candidates.json`. The source analyzer writes its five technical analysis JSON files; the scheduler's `HighlightCandidateGenerator` separately writes `category.json` and `highlight-candidates.json`. `shots.json`, `scenes.json`, `transcript.json`, and `audio-events.json` remain manifest intentions, so do not claim every listed artifact exists.
`HighlightDirectorPlan` selects several windows from exactly one source and carries story purpose, visual treatment, music/SFX directions, voiceover, and overlays. `HighlightDirectorFlowService` converts each item to a single-decision `EditPlan`, renders per-highlight outputs, then concatenates them.
### Why the contracts remain separate
Keep them separate because they have different aggregates, input cardinality, director schemas, folder protocols, approval locations, output cardinality, scheduler ownership, and recovery signals. Sharing `EditPlan`, `RenderManifest`, or analysis helpers is implementation reuse, not schema equivalence.
Do not merge the stores, statuses, or roots until all of these exist:
- A versioned replacement contract and explicit migration/compatibility decision.
- Golden JSON fixtures for both schemas and backward-read tests.
- Recovery tests from every durable stage boundary.
- A collision strategy for project IDs and artifact names.
- An approval model that cannot be bypassed by REST or scheduler entry points.
- A rollout and rollback plan approved through `video-editing-change-control`.
## Enforce load-bearing invariants
Apply this checklist to every architecture change:
- [ ] Keep project IDs restricted to `[A-Za-z0-9][A-Za-z0-9._-]{0,127}` and reject `..`.
- [ ] Normalize and containment-check every project-relative path before reading or writing it.
- [ ] Never accept an absolute path or backslash-containing relative artifact name in the highlight store.
- [ ] Refuse overwrite when claiming source files; never silently replace an existing processed/rejected input.
- [ ] Treat source media and completed contract artifacts as immutable inputs to later stages.
- [ ] Validate project ID, source identity, time ranges, supported effects, and required licensed assets before rendering.
- [ ] Persist an artifact successfully before advertising the status that promises it exists.
- [ ] Make retries idempotent: never duplicate a rendered highlight, queue message, clip record, or asset charge.
- [ ] Record model identity/version, input hash, asset provenance/license, FFmpeg command, and output hash for reproducibility.
- [ ] Derive acceptance from measured media probes and approved creative review, never file existence or visual impression alone.
- [ ] Keep transport DTOs, JPA entities, and worker-process details outside the domain.
- [ ] Keep framework annotations out of new domain objects unless a decision record justifies them.
## Enforce the local-runtime and render prohibitions
The following are project directives, even where current defaults or code contradict them:
1. Do not download dependencies or models automatically at startup, test time, or job time.
2. Do not call external AI services. Model/media inference must use in-process calls or approved non-network inter-process communication; loopback HTTP is still network and is not a certified path.
3. Do not permit model/media inference or acquisition to use any network, including loopback. Pre-provision and verify dependencies, model weights, and licenses during an approved artifact-build process. Production API, database, storage, and telemetry integrations remain separately approved and secured network boundaries.
4. Do not ingest or generate assets without recorded licensing and provenance.
5. Do not substitute silence, sine tones, or placeholder media for missing voiceover, music, or SFX.
6. Do not render without explicit approval bound to the exact plan and source digest.
7. Do not change production-facing defaults without change-control approval, compatibility analysis, and rollback evidence.
Current disqualifying gaps include:
- `tools/run_local_asset_worker.sh` and `tools/run_local_cv_worker.sh` install Python dependencies in `auto` mode; the CV script can load/download YOLO weights, and AudioCraft `get_pretrained` can fetch models.
- The 2026-07-21 working tree removes worker silence/tone/host-speech fallbacks and makes requested assets mandatory at both renderer boundaries. Standalone bootstrap scripts and local-CV heuristic fallback remain noncompliant gaps.
- `application.yml` enables highlight ingestion but now defaults highlight rendering off and director approval on. Approval remains a bare, non-digest-bound file.
- `POST /v1/edit-projects/{projectId}:render` invokes the renderer without an approval check.
- `HighlightFfmpegRenderer` can copy video forward after audio mixing fails, and several QA checks are hard-coded `true` rather than measured.
Treat these as open architecture defects. Do not operate them as production behavior, and do not hide them with documentation. Repair them only through the change-control and validation skills.
## Know the current weak points
| Weak point | Evidence and consequence | Required direction |
|---|---|---|
| No enforced modules | One Maven module; no ArchUnit/Modulith dependency test | Add capability-oriented boundaries and fail the build on violations |
| Anemic state records | Public records allow unchecked states; update methods accept arbitrary status | Put invariants and legal transitions in domain/application code |
| Filesystem is database and queue | Direct JSON writes, artifact-presence polling, local `AtomicBoolean` guards | Add atomic publish, durable claim/lease, restart recovery, retention, and concurrency tests |
| Highlight analysis is incomplete | New-source analyzer does not call `CinematicHighlightAnalyzer`; prompt tolerates absent category/candidates | Make multimodal candidate production an explicit, measured stage |
| Director plan contract is gated but creative truth is not proven | `HighlightDirectorPlanValidator` validates identity, category/candidate linkage, safe IDs, ranges, configured bounds, and required directions; it does not verify visible claims, asset licenses, or artistic fit | Keep media review and explicit render approval; add provenance and content-grounding evidence before claiming production readiness |
| Local worker boundary is porous | Shell bootstrap downloads; loopback endpoint is configurable; models are named rather than content-addressed | Package locked runtime/model artifacts; verify hashes; replace loopback HTTP with in-process or approved non-network IPC and deny inference networking |
| QA can overstate success | Highlight checks assert duration, overlays, assets, and mastering without measuring output | Probe outputs and fail closed on production-required checks |
| Renderer configuration leaks | Project concatenation uses literal `ffmpeg`/`ffprobe` while other stages use typed properties | Route executables and process policy through an outbound media-tool port |
| OS assumptions leak | macOS `say`, local paths, atomic rename, and worker virtualenvs are runtime details | Prove equivalent macOS dev and Linux/VPS/cloud behavior; remove OS-specific production fallbacks |
| API/core dependency points outward | Application service imports `api.dto`; controller directly calls renderer | Map DTOs at inbound adapters and expose application use cases |
| Local project state blocks horizontal safety | Edit/highlight projects require shared local paths and have no distributed ownership | Define durable object/project storage before claiming cloud scalability |
## Move toward the target modular monolith
Use capabilities, not technical-layer-only packages, as the top-level target. Keep one deployable unless measured requirements justify distribution.
```text
org.example.videoclips
clipping/ # upload sessions, assets, clip jobs, clip publication
folderingestion/ # watched-folder segmentation workflow
multiedit/ # multi-source project, plan, render lifecycle
highlights/ # single-source analysis, direction, assets, render lifecycle
media/ # approved FFmpeg/FFprobe and local-model runtime adapters
platform/ # configuration, persistence wiring, security, observability
```
Within each capability, point dependencies inward:
```text
inbound adapters -> application use cases -> domain
outbound adapters -> application-owned outbound ports
configuration -> all adapters for composition only
domain -> no Spring, HTTP, persistence, filesystem, process, or sibling adapter package
```
Expose cross-capability access through a small documented application API or durable event. Do not import another capability's adapter or persistence entity. Do not create a generic base service, universal repository, universal mapper, CQRS infrastructure, event bus, or microservice without a measured requirement.
### Execute boundary migration safely
1. Characterize the current API, JSON, file names, state changes, and restart behavior with tests.
2. Write an ADR and route it through `video-editing-change-control`.
3. Define the destination capability and its public application API.
4. Move domain types without changing behavior; keep compatibility mapping at the old boundary.
5. Move outbound interfaces inward, then adapt JPA, filesystem, FFmpeg, and model workers outside them.
6. Add ArchUnit or Spring Modulith verification only when its rule corresponds to the documented dependency direction.
7. Run narrow tests, full tests, clean-checkout verification, and artifact contract tests.
8. Remove compatibility code only after persisted-project and API migration evidence is approved.
For macOS development, Linux/VPS production, and cloud deployment, use the same platform-neutral application artifact and contract versions. Build signed platform-specific runtime/model/image bundles, then promote each bundle without rebuild for the same OS/CPU target. Permit environment-specific adapters/configuration, not environment-specific domain behavior.
## Record every major decision
For every module split, new dependency, persistence change, state change, model/runtime choice, or contract version, record:
1. Requirement being addressed.
2. Selected approach.
3. Alternatives considered.
4. Benefits and trade-offs.
5. Operational consequences.
6. Security implications.
7. Automated and manual verification.
8. Revisit conditions.
Label the target as `proposed`, `accepted`, `implemented`, or `verified`. Never call it production-ready until clean-checkout build, security, observability, recovery, load, media-quality, and operational gates all pass.
## Review checklist
- [ ] Name the affected workflow and aggregate.
- [ ] List every changed API, JSON, file, state, metric, and approval contract.
- [ ] Show dependency direction before and after.
- [ ] Keep domain behavior testable without Spring.
- [ ] Add a port only where a real adapter boundary exists.
- [ ] Prove restart and retry behavior at the changed durable boundary.
- [ ] Prove path containment and tenant/project isolation.
- [ ] Prove no startup/job-time downloads or external network calls occur.
- [ ] Prove all assets and models are pre-provisioned, hashed, and licensed.
- [ ] Prove approval binds source, plan, configuration, model, and asset digests.
- [ ] Prove QA from measurements; reject placeholder audio and degraded render fallback.
- [ ] Run the relevant workflow tests plus the full change-control gates.
## Provenance and maintenance
Re-verify the package inventory: `find src/main/java/org/example/videoclips -type d | sort`
Re-verify application wiring and schedulers: `rg -n '@SpringBootApplication|@EnableAsync|@EnableScheduling|@Scheduled|@ConditionalOn' src/main/java`
Re-verify existing ports: `rg -n '^public interface ' src/main/java/org/example/videoclips/{application,processing,queue,storage,editing}`
Re-verify outward application imports: `rg -n '^import org\.example\.videoclips\.(api|persistence|storage|queue|processing)' src/main/java/org/example/videoclips/{domain,application}`
Re-verify statuses: `for f in src/main/java/org/example/videoclips/domain/ClipJobStatus.java src/main/java/org/example/videoclips/editing/{EditProjectStatus,HighlightProjectStatus}.java; do sed -n '1,120p' "$f"; done`
Re-verify filesystem writes and render signals: `rg -n 'writeJson\(|Files\.(move|copy|write)|final\.mp4|approved\.flag' src/main/java/org/example/videoclips/{folder,editing}`
Re-verify highlight artifacts against the declared folder contract: `rg -n 'analysis/.*\.json|director/.*\.json|HighlightFolderContract' src/main/java/org/example/videoclips/editing`
Re-verify prohibited downloads and fallbacks: `rg -n 'pip install|get_pretrained|YOLO\(|write_silence|fallback[_ -]tone|anullsrc|sine=' tools src/main/java src/main/resources`
Re-verify production-facing defaults: `sed -n '1,180p' src/main/resources/application.yml && sed -n '1,120p' src/main/resources/application.properties`
Re-verify architecture enforcement dependencies/tests: `rg -n 'archunit|spring-modulith|ApplicationModules|ArchRule' pom.xml src/test || true`
Re-run focused contract tests after architecture changes: `mvn -o -Dtest=VideoAssetControllerTest,DatabaseBackedClipJobQueueAdapterTest,FolderVideoScanSchedulerTest,EditPlanInboxScannerTest,FileSystemEditProjectStoreTest,FileSystemHighlightProjectStoreTest,HighlightDirectorFlowServiceTest test`

View File

@ -0,0 +1,345 @@
---
name: video-editing-build-and-env
description: Load this skill when reproducing or repairing the build environment on macOS, Linux/VPS, or a cloud build runner; when Maven, Java 21, FFmpeg/ffprobe, Python runtimes, local model files, executable permissions, architecture compatibility, offline dependency resolution, clean-checkout reproducibility, or test working-directory assumptions are involved; or when designing the dependency-locked and network-isolated build supply chain. Do not load it for an already-built service's runtime symptom.
---
# Build and Environment
Use this runbook from the repository root. Treat every path and version below as a verified description of `HEAD` on **2026-07-21**, unless it is explicitly labeled `TARGET`.
## Use this skill for
- Recreating a build environment on macOS, Linux/VPS, or a cloud build runner.
- Deciding whether a failure comes from Java, Maven, FFmpeg, Python, a missing local model, permissions, CPU architecture, or an unclean workspace.
- Running focused tests, the current Maven gate, or a clean archived-checkout experiment.
- Designing the future reproducible, dependency-locked, offline build and runtime supply chain.
## Do not use this skill for
| Need | Load instead |
|---|---|
| Starting schedulers, processing media, locating outputs, or deployment operation | `video-editing-run-and-operate` |
| Changing application properties or worker flags | `video-editing-config-and-flags` |
| Deciding what evidence proves media or creative quality | `video-editing-validation-and-qa` |
| Diagnosing a runtime symptom after the environment is known good | `video-editing-debugging-playbook` |
| Learning why the clean-checkout and local-asset traps exist | `video-editing-failure-archaeology` |
| Promoting a toolchain, dependency, image, model, or default change | `video-editing-change-control` |
Do not use a successful compile or test run as proof of cinematic quality, production readiness, security, or operability.
Terms: **SBOM** means Software Bill of Materials; **OCI** means Open Container Initiative image format.
## Non-negotiable environment boundary
Enforce these rules in development, CI, image construction, and production:
1. Do not allow application startup or validation to install a package, fetch a model, contact an external AI service, or access the network.
2. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` as setup commands. Their default `BOOTSTRAP_MODE=auto` can create virtual environments and invoke `pip`; the CV launcher can also cause Ultralytics to fetch YOLO weights.
3. Provision Maven artifacts, Python wheels, FFmpeg, local model weights, fonts, LUTs, music, and sound effects only in an approved build/image process. Record version, SHA-256 checksum, origin, license, target OS, target CPU architecture, and approval for every non-source artifact.
4. Run the delivered service and its validation offline. Missing dependencies or models must fail closed; do not replace them with downloads, silence, tones, host speech utilities, heuristics, or unlicensed assets.
5. Do not change production-facing defaults to make a build pass. Route such a change through `video-editing-change-control`.
The current source still violates parts of this boundary: the standalone launchers bootstrap automatically and local CV can fall back to heuristics. Asset generation itself now rejects missing local models and placeholder output, and strict asset readiness fails startup. Treat the remaining launcher/CV behavior as open defects, not approved procedures.
## Current baseline and gaps
| Component | Current repository fact | Production interpretation |
|---|---|---|
| Java | `pom.xml` sets `java.version` to `21`; compilation uses Java release 21. | Use a supported Java 21 LTS JDK for the current code. Do not infer vendor or patch level from the POM. |
| Spring Boot | Parent is `org.springframework.boot:spring-boot-starter-parent:3.3.2`. | This is the implemented version, not a claim that it is the latest supported release. Upgrade only through change control with compatibility evidence. |
| Build tool | Maven project with `pom.xml`; no `mvnw`, Maven wrapper metadata, or Gradle files exist. | A host Maven installation is currently required and its version is not repository-controlled. |
| Project version | `org.example:video-editing:1.0-SNAPSHOT`. | The application artifact itself is a snapshot. The POM contains no snapshot/prerelease dependency version, but no automated enforcer proves that property. |
| Dependency management | Spring Boot manages most versions; AWS SDK uses BOM `2.28.29`; JaCoCo is `0.8.12`. | There is no dependency lock, repository mirror policy, SBOM plugin, vulnerability gate, or update automation in the POM. |
| Test runner | Surefire activates Spring profile `test`; JaCoCo checks 100% instruction, line, and branch coverage only for `org.example.videoclips.folder` during `verify`. | `test` does not execute the JaCoCo check. No project-wide coverage threshold is implemented. |
| FFmpeg | Application defaults call `ffmpeg` and `ffprobe` from `PATH`. | Version, build flags, codecs, and checksum are unpinned. Some real-media tests skip when these tools are missing. |
| CV Python | Four top-level packages are version-pinned in `tools/local_cv_requirements.txt`; transitive dependencies and hashes are not locked. | `.venv-local-cv` and YOLO weights are local, platform-specific, untracked state. |
| Asset Python | `torch`, `audiocraft`, `soundfile`, and `numpy` are unpinned in `tools/local_asset_requirements.txt`. | `.venv-local-asset`, Piper, and music/SFX/voice models are not reproducibly provisioned. |
| Packaging/deploy | No Dockerfile, Compose file, Maven wrapper, CI workflow, or deployment manifest is tracked. | There is no certified Linux/VPS/cloud build path or OCI image yet. |
## Inspect before building
Run only read-only checks first:
```bash
pwd
test -f pom.xml
git status --short
java -version
javac -version
mvn -version
python3 --version
command -v ffmpeg
command -v ffprobe
ffmpeg -version | sed -n '1,4p'
ffprobe -version | sed -n '1,3p'
uname -s
uname -m
git ls-files -s tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh
```
Require these observations:
| Check | Accept now | If different |
|---|---|---|
| Repository root | `pom.xml`, `src/`, and `tools/` are present. | Stop and change to the repository root. Relative worker and data paths depend on the process working directory. |
| Java | Runtime and compiler both report major version 21. | Select a Java 21 LTS JDK. Do not silently compile with a newer release. |
| Maven | A host Maven is available. | Stop. There is no wrapper to recover a missing or incompatible Maven. Record the selected Maven version in evidence. |
| Media tools | Both commands resolve and report versions. | Unit tests may still pass, while real FFmpeg integration tests skip. Do not accept that as media validation. |
| OS/CPU | Record, do not normalize away. | Build or select Python wheels, FFmpeg, and model runtime packages for this exact OS/architecture. |
| Script modes | CV launcher is `100755`; asset launcher is `100644` at current `HEAD`. | The asset bootstrap cannot be directly executed by `ProcessBuilder` on POSIX. Do not fix locally with `chmod` and call the environment reproducible; change the tracked mode through change control. |
Inspect source declarations without invoking Maven plugins:
```bash
sed -n '1,180p' pom.xml
sed -n '1,120p' tools/local_cv_requirements.txt
sed -n '1,120p' tools/local_asset_requirements.txt
sed -n '1,120p' tools/run_local_cv_worker.sh
sed -n '1,120p' tools/run_local_asset_worker.sh
```
Do not run `mvn dependency:*` or `mvn help:*` as an offline preflight unless the corresponding plugin is already present in the approved Maven cache. On the observed workstation, `mvn -o help:evaluate` fails because the help plugin is absent from the local cache.
## Platform setup contract
Provision tools outside application startup. This section names the required result; it intentionally does not use `brew`, `apt`, `curl`, `pip`, or a model hub.
### macOS development
Provide all of the following before running Maven:
- A supported Java 21 LTS JDK and a known Maven distribution.
- `ffmpeg` and `ffprobe` from the same approved FFmpeg build.
- Python runtimes and wheel sets matching `uname -m` (`arm64` or `x86_64`). Never reuse a virtual environment created for the other architecture.
- Pre-provisioned, licensed local model files. Configure explicit absolute paths; do not rely on a model name that a library may resolve online.
- A network-denied validation environment.
Do not rely on macOS `say`. The former host-dependent fallback was removed; it is not the approved local voiceover engine and is unavailable on Linux.
### Linux/VPS production
Build and validate on the same Linux CPU architecture and compatible runtime family used in production. Provide Java 21, Maven only in the build stage, FFmpeg/ffprobe, Python runtimes, wheels, Piper, and model files from the approved artifact store. Keep the runtime artifact set read-only where practical.
Do not rely on `espeak`. The former fallback was removed and was never a production voiceover contract. Do not copy a macOS virtual environment, native Python wheel, FFmpeg binary, or Maven build output that embeds host-specific paths into Linux.
### Cloud build and image pipeline
Split acquisition from verification:
| Stage | Network | Required output |
|---|---|---|
| Approved acquisition | Restricted to approved artifact repositories | Versioned Maven cache, hashed Python wheelhouse, hashed FFmpeg distribution, hashed model/asset bundle, licenses, SBOM inputs. |
| Build | Denied | Compiled and tested application from immutable inputs. |
| Runtime image assembly | Denied | Java runtime, application, media tools, Python runtime, models, assets, and manifest for one OS/CPU target. |
| Runtime/smoke validation | Denied | Fail-closed startup and functional evidence; no cache writes or downloads. |
Build distinct artifacts for `linux/amd64` and `linux/arm64` until cross-platform equivalence is demonstrated. GPU, CUDA, Metal/MPS, and CPU model runtimes are not interchangeable. Record device/runtime selection and numerical or quality acceptance results; never assume model availability means equivalent output.
## Safe build commands
All Maven commands write under `target/`. They must run with network access denied. Maven `-o` requests offline resolution but is not itself a network sandbox.
### Compile without executing tests
Use this only to isolate compilation:
```bash
mvn -o -DskipTests package
```
Expected current artifact name after success: `target/video-editing-1.0-SNAPSHOT.jar`. This is not a quality gate.
### Run a focused test
Use a focused class while developing a narrow change:
```bash
mvn -o -Dtest=EditPlanValidatorTest test
```
Replace the class name only after locating it under `src/test/java`. Focused tests are evidence for that scope only. Tests that directly construct production process adapters may invoke host tools even though Surefire activates the `test` profile.
### Run the current full Maven gate
The current repository-wide command is:
```bash
mvn -o verify
```
`verify` compiles main and test sources, runs Surefire tests, applies the folder-package JaCoCo check, and creates a coverage report. It does **not** provide dependency locking, formatting, static analysis, architecture enforcement, vulnerability scanning, secret scanning, API compatibility, container scanning, SBOM generation, signing, or deployment validation.
Do not report this gate as green from the current source. A clean archived checkout fails as documented below. A pass in a long-lived workspace can be caused by untracked `.venv-local-asset` state and may invoke a model library capable of downloading weights.
## Reproduce the clean-checkout gate
Use an archive so untracked virtual environments, models, generated media, and local ignore rules cannot contaminate the result. Point Maven at an explicitly approved, pre-populated cache. The command performs no Git mutation and no source-tree write:
```bash
repo_root="$(git rev-parse --show-toplevel)"
archive_root="$(mktemp -d "${TMPDIR:-/tmp}/video-editing-clean.XXXXXX")"
git -C "$repo_root" archive --format=tar HEAD | tar -xf - -C "$archive_root"
(
cd "$archive_root"
test ! -e .venv-local-cv
test ! -e .venv-local-asset
test -d "${MAVEN_REPO:?set MAVEN_REPO to the approved offline Maven cache}"
mvn -o -Dmaven.repo.local="$MAVEN_REPO" verify
)
```
Interpret the result:
| Observation | Meaning | Action |
|---|---|---|
| Maven cannot resolve an artifact or plugin in offline mode | The approved cache is incomplete. | Fix the approved acquisition manifest/cache. Do not remove `-o` or enable network. |
| Both FFmpeg integration tests are skipped | `ffmpeg` or `ffprobe` is unavailable. | Fail media-capable validation; provision the approved pair. |
| `LocalAssetGenerationStageTest.reusesExistingSharedAssetsAndSynthesizesMissingVoiceover` fails | This is the valid verified clean-archive result when Maven runs from inside the archive. | Do not seed an untracked virtual environment to hide it. Make the test hermetic and make the runtime fail closed through change control. |
| The clean archive passes but the workspace fails | Workspace-generated state or source changes affect the result. | Compare tracked changes and environment; do not delete user work. |
| The workspace passes but the clean archive fails | The workspace relies on untracked state. | Treat clean reproducibility as failed. |
### Verified clean result on 2026-07-21
With Java `21.0.10`, Maven `3.6.3`, FFmpeg/ffprobe `7.1.1`, and a populated host Maven cache, the valid run changed directory into the archived checkout, compiled 157 main sources, and ran 216 tests in 56 test classes. It finished with **one failed assertion**: `LocalAssetGenerationStageTest.reusesExistingSharedAssetsAndSynthesizesMissingVoiceover` expected `voiceover.wav` to exist.
An earlier run reported two failures because it invoked Maven with `-f` from the long-lived workspace. That violated the test's process-working-directory assumption and is not valid clean-checkout evidence. The workspace's 216 tests passed while `.venv-local-asset` existed locally but was untracked. Together these observations establish a reproducibility defect, not the exact behavior of every clean machine. The macOS, Maven, Java, and FFmpeg patch versions are observations from one machine, not repository requirements.
## Working-directory and executable traps
The defaults contain repository-relative paths such as `./tools/run_local_cv_worker.sh`, `./tools/local_asset_worker.py`, and `./.venv-local-asset/bin/python`. Java converts some of these to absolute paths relative to the process working directory.
Follow these rules:
1. Run current development Maven commands from the repository root.
2. Do not start the packaged JAR from an arbitrary directory while retaining relative worker paths.
3. In a production artifact, configure absolute paths inside the immutable runtime layout.
4. Verify every directly executed POSIX script with `test -x`; a shebang does not compensate for a missing execute bit.
5. Invoke `.py` files through the approved Python interpreter. Do not make source scripts executable as an ad hoc workaround.
Use these diagnostics without starting workers:
```bash
test -r tools/local_asset_worker.py
test -x tools/run_local_cv_worker.sh
test -x tools/run_local_asset_worker.sh
bash -n tools/run_local_cv_worker.sh
bash -n tools/run_local_asset_worker.sh
git ls-files .venv-local-cv .venv-local-asset
```
At current `HEAD`, the third command is expected to fail and the final command prints nothing. That is evidence about tracked state, not permission to alter it locally.
## Known test signals and warnings
| Signal | Current interpretation |
|---|---|
| `Standard Commons Logging discovery... remove commons-logging.jar` | Observed classpath warning. Track it as dependency hygiene; do not suppress it as a build fix. |
| Byte Buddy dynamic-agent warning | Mockito currently loads an agent dynamically; a future JDK may disallow this default. Java 21 tests currently continue. |
| Flyway warns that H2 `2.2.224` is newer than its tested H2 support | Current tests use H2 for Spring contexts. This warning does not prove PostgreSQL behavior. |
| `spring.jpa.open-in-view is enabled by default` during a test context | Investigate effective test configuration; main `application.properties` declares it false. Do not dismiss the mismatch. |
| Local-CV fallback warnings in unit tests | Some tests intentionally exercise fallback behavior. They do not certify a production local CV model. |
| FFmpeg test skipped | The two `@EnabledIf` integration classes skip when tools are unavailable. A green count with skips is weaker evidence. |
## TARGET: reproducible, dependency-locked build campaign
Nothing in this section is implemented merely because it is listed. Complete and promote each phase through `video-editing-change-control`.
### Phase 1: pin the build toolchain
- [ ] Add a Maven wrapper with an exact Maven distribution URL and verified distribution checksum.
- [ ] Pin the Java 21 vendor/distribution and patch policy in CI/image metadata; enforce Java major version and Maven version at build start.
- [ ] Centralize deliberate plugin and dependency version overrides; preserve Spring Boot dependency management unless an ADR proves why an override is required.
- [ ] Reject dependency versions containing `SNAPSHOT`, milestone, RC, or other prerelease markers. Treat the project's own `1.0-SNAPSHOT` separately and replace it with traceable release versioning before release.
- [ ] Set reproducible archive metadata and prove two isolated builds of the same commit and inputs produce identical artifact hashes.
Gate: a clean checkout executes one documented wrapper command with no host Maven dependency, and the build records Java, Maven, OS, CPU, commit, and artifact hash.
### Phase 2: freeze dependency acquisition
- [ ] Resolve Maven only through an approved immutable repository mirror; capture the complete dependency/plugin graph and checksums.
- [ ] Produce a Python lock per supported OS/CPU/runtime, including transitive versions and hashes.
- [ ] Build an approved wheelhouse; install with offline and hash-required semantics. Never run unconstrained `pip install` in service startup.
- [ ] Generate an SBOM and license inventory for Java, Python, FFmpeg, native libraries, models, fonts, LUTs, music, and SFX.
- [ ] Add vulnerability and license policy gates. Zero unresolved critical/high findings may ship without documented risk acceptance.
Gate: remove all repository and model-hub access, delete mutable caches, rebuild only from the approved inputs, and obtain the same dependency inventory.
### Phase 3: provision local models and media tools
- [ ] Give every model an immutable identifier, exact file list, SHA-256 checksums, source, license/usage rights, architecture/runtime compatibility, and acceptance evidence.
- [ ] Configure models by explicit local paths. Prohibit library shorthand such as `yolov8n.pt` or `musicgen-small` when it can trigger online resolution.
- [ ] Pin FFmpeg/ffprobe to one approved build per platform, including configure flags and codec/filter inventory.
- [ ] Remove or fail closed on automatic bootstrap, heuristic substitution, host speech fallback, silence, and tone generation.
- [ ] Verify local runtime readiness before accepting work; absence or checksum mismatch must produce a non-ready state and actionable error.
Gate: startup and a representative highlight run succeed with egress denied and read-only dependency/model directories; removing one required model causes a deterministic readiness failure and no placeholder output.
### Phase 4: make tests hermetic and cross-platform
- [ ] Replace `LocalAssetGenerationStageTest` host-process coupling with controlled test doubles or checked test fixtures; separately test the real packaged local runtimes.
- [ ] Require explicit FFmpeg integration execution instead of silently accepting skips in the full gate.
- [ ] Test PostgreSQL behavior with a real approved PostgreSQL test environment; current Spring tests use H2.
- [ ] Run the same build portfolio on macOS developer architectures and supported Linux production architectures.
- [ ] Add architecture tests, formatting, static analysis, API checks, security scans, secrets scanning, and deterministic end-to-end smoke tests only where each closes a stated acceptance criterion.
Gate: the clean archive passes from an empty workspace on every supported target without user-home state, network, host speech tools, or untracked files.
### Phase 5: certify the release artifact
- [ ] Build an OCI-compatible, non-root runtime image or equivalent immutable artifact for each supported architecture.
- [ ] Keep build tools out of the runtime; include only the approved Java runtime, application, FFmpeg/ffprobe, Python runtime, models, and licensed assets needed for operation.
- [ ] Generate provenance linking commit, dependency/model manifests, SBOM, tests, scans, and image digest.
- [ ] Sign the release artifact and verify the signature before deployment.
- [ ] Validate offline startup, graceful shutdown, readiness, resource limits, and representative media execution in the production-like environment.
Gate: promotion consumes the already-built digest; it does not rebuild per environment. Rollback and roll-forward use known digests and compatible data migrations.
## Evidence checklist
Attach this information to any environment or build claim:
- [ ] Commit identifier and `git status --short` output.
- [ ] `java -version`, `javac -version`, and `mvn -version`.
- [ ] OS and CPU architecture.
- [ ] FFmpeg and ffprobe versions plus build configuration.
- [ ] Maven cache manifest/digest and confirmation that offline mode/network denial were active.
- [ ] Python lock/wheelhouse digest and exact interpreter version, if local workers were in scope.
- [ ] Model and licensed-asset manifest/digest, if cinematic processing was in scope.
- [ ] Exact Maven command, test totals, skips, failures, and JaCoCo gate result.
- [ ] Application JAR/image digest and SBOM/provenance references, when implemented.
- [ ] A clear `CURRENT FAILURE`, `CANDIDATE`, or `PROMOTED` label. Never call an unimplemented target production-ready.
## Provenance and maintenance
Re-verify the Java, Spring Boot, AWS BOM, project, and JaCoCo declarations:
```bash
rg -n '<java.version>|spring-boot-starter-parent|aws.sdk.version|<version>1.0-SNAPSHOT</version>|jacoco-maven-plugin|<minimum>1.00</minimum>' pom.xml
```
Re-verify build-system and delivery gaps:
```bash
find . -maxdepth 3 -type f \( -name mvnw -o -name mvnw.cmd -o -name maven-wrapper.properties -o -name 'Dockerfile*' -o -name 'compose*.yml' -o -path './.github/workflows/*' -o -name '*.lock' \) -print
```
Re-verify Python pins, bootstrap/download paths, placeholders, and test coupling:
```bash
rg -n '^[A-Za-z0-9_.-]+(\[.*\])?(==.*)?$|pip install|YOLO\(|get_pretrained|write_silence|write_fallback_tone|fallbackTone|runSpeechFallback|LocalAssetSynthesizer' tools src/main/java src/test/java/org/example/videoclips/editing/LocalAssetGenerationStageTest.java
```
Re-verify relative paths, startup defaults, and test safety overrides:
```bash
rg -n 'auto-start:|enabled:|render-enabled:|fallback-to-heuristic:|bootstrap-script:|python-binary:|script:' src/main/resources/application.yml src/test/resources/application.properties src/main/java/org/example/videoclips/config/VideoClippingProperties.java
```
Re-verify tracked permissions and untracked-runtime assumptions:
```bash
git ls-files -s tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh tools/local_asset_worker.py
git ls-files .venv-local-cv .venv-local-asset
```
Re-run the archived offline experiment from **Reproduce the clean-checkout gate** after build, test, Python, model, or worker changes. Update the dated result only from that isolated evidence; do not copy counts or versions from a long-lived workspace.

View File

@ -0,0 +1,280 @@
---
name: video-editing-change-control
description: Load before proposing, implementing, reviewing, merging, releasing, or rolling back any change to this video-editing repository, especially changes to runtime behavior, configuration defaults, APIs, persistence, FFmpeg rendering, highlight selection, local CV/TTS/music/SFX models, assets, security, deployment, or production promotion. Also load when deciding what evidence and approvals a change requires.
---
# Video Editing Change Control
## Purpose
Control every change from intent through evidence and promotion. Treat **promotion** as moving a change into a more trusted environment or declaring it production-ready. Treat a **gate** as evidence or approval that must exist before promotion.
Use the repository state dated **2026-07-21** as the current baseline. The Fortune 500 requirements below are the target acceptance standard, not a description of the repository today.
## When not to use this skill
- Do not use this as a configuration catalog. Load `video-editing-config-and-flags`.
- Do not use this to diagnose a failure. Load `video-editing-debugging-playbook`, then return here before changing behavior.
- Do not use this as the test procedure. Load `video-editing-validation-and-qa`, then attach its evidence here.
- Do not use this to reconstruct an old failure. Load `video-editing-failure-archaeology`.
- Do not use this to learn module boundaries. Load `video-editing-architecture-contract`.
- Do not use this to operate schedulers or renders. Load `video-editing-run-and-operate`; never use an operating procedure to bypass the approval gates here.
- Do not use this to judge cinematic quality by eye. Load the highlight-quality campaign and proof/analysis skills; bring their measurements back to this gate.
## Establish the baseline first
Run these read-only checks from the repository root before editing:
```bash
git status --short
git branch --show-current
git log -10 --oneline --decorate
git diff --stat
git diff
```
Record the starting commit, existing uncommitted files, scope, and expected observable outcome. Do not overwrite, reset, clean, or reformat unrelated work.
Confirm important current facts rather than trusting a plan checkbox:
```bash
test -f pom.xml
test -f README.md
test -x mvnw
find .github -maxdepth 3 -type f -print 2>/dev/null
find src/test/java -name '*Test.java' | sort
rg -n 'enabled:|auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:|strict-runtime:' src/main/resources
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone' tools src/main
```
As of 2026-07-21, `pom.xml` exists; `README.md`, Maven Wrapper, `.github/` CI, container definitions, and deployment definitions do not. The current working tree passes `mvn -q -o verify` with 245 tests in 62 test classes. Phase 1 also reproduced a clean archived-checkout `mvn verify` failure in `LocalAssetGenerationStageTest` because that revision could depend on an untracked local asset virtual environment or an audible TTS fallback. The working tree removed that fallback dependency, but do not call the build reproducible until a fresh clean, offline run passes on macOS and Linux.
## Apply the non-negotiables
Stop a change immediately when it violates any of the seven user prohibitions below. **These prohibitions have no waiver or risk-acceptance path.** Existing violations are gaps to remove, never precedent. `video-editing-change-control` is the authoritative home for this policy; sibling skills state only workflow-specific consequences.
| Rule | Required behavior | Why and repository evidence |
|---|---|---|
| No automatic dependency or model downloads | Provision signed, checksummed dependencies and model weights before startup. Model/media inference and acquisition must make no network connection, including loopback remote procedure calls (RPCs). Fail startup or the affected job when an approved local artifact is absent. | `tools/run_local_asset_worker.sh` still creates a virtualenv and runs `pip install`; do not invoke it. The working-tree runtime verifier no longer bootstraps and the asset worker accepts only existing local model paths with offline hub flags. |
| No external AI service in the production media path | Keep highlight identification, direction, CV, voiceover, music, and SFX inference within the approved runtime using pre-provisioned local models. | `docs/cinematic-highlight-operator-checklist.md` currently instructs an operator to run Codex, Claude, or another filesystem-capable AI. That is a manual development workflow, not the target production architecture. |
| No unlicensed asset | Require machine-readable origin, license identifier/terms, checksum, and allowed use for every music, SFX, voice, font, LUT, and model artifact before render approval. | `AssetLicensePolicy` now excludes media without a nonblank adjacent `.license.txt`, preserves that sidecar during copies, and render preflight requires licensed requested audio. A sidecar string alone still does not prove origin, checksum, allowed use, or commercial rights. |
| No placeholder silence or tones | Fail closed when requested voiceover, music, or SFX cannot be generated or resolved. Never promote synthetic silence or a diagnostic sine tone as a finished asset. | The 2026-07-21 working tree removes Python/Java silence, tone, and host-speech success paths and tests missing-asset render rejection. Preserve that gate. |
| No unapproved rendering | Require a validated plan, complete licensed assets, measured QA, and an explicit approval artifact or auditable approval record before FFmpeg starts. | Highlight defaults now keep rendering disabled and require `approved.flag` if enabled. The flag is not bound to source/plan/configuration digests, and `POST /v1/edit-projects/{projectId}:render` still has no approval gate, so neither path is production authorization. |
| No unapproved network access | Use in-process calls or explicitly designed non-network inter-process communication for model/media inference. The current loopback CV HTTP worker is noncompliant and cannot be a certified path. Separately, API, PostgreSQL, object-storage, and telemetry connections are production integrations: each requires explicit approval, authentication/authorization where applicable, transport security, least privilege, bounded timeouts, and tested failure behavior. Permit no external AI or unapproved egress. | Local CV currently uses configured loopback HTTP at `127.0.0.1:8091`; S3 and HTTP APIs also exist. Loopback is still network, and network presence is not approval. |
| No silent production-default change | List old value, new value, environments affected, migration, rollback, and operator impact. Require explicit review for every default or `matchIfMissing` change. | `application.yml` currently enables folder scheduling, editing, local workers, the local director, and highlight ingestion by default. Highlight rendering now defaults off, but a normal start can still consume files and start the network-capable CV bootstrap path. |
| No claim without measured evidence | Label plans, generated artifacts, and heuristic checks accurately. A passing test or existing MP4 is not proof of cinematic or production quality. | Commit `5d889b0` explicitly recorded “working version but not cinematic.” Several highlight QA checks in `HighlightFfmpegRenderer.buildQaReport` are hard-coded `true`, including duration, asset, overlay, and mastering claims. |
## Classify the change
Choose the highest applicable class. Splitting a large change does not lower its class when the parts jointly alter behavior.
| Class | Examples here | Minimum route |
|---|---|---|
| C0: knowledge-only | Skills, docs, comments, diagnostic scripts that cannot execute in a build or runtime | Ground-truth review, command/path verification, docs consistency review |
| C1: isolated implementation | Internal refactor with identical API, config, storage, scheduling, render, and media output behavior; focused deterministic test | Focused tests, full offline unit suite, compatibility review |
| C2: behavior or contract | API/DTO/error change, config/flag, dependency/plugin, scheduler state, filesystem contract, queue/retry, observability, performance, or renderer command | C1 plus integration/contract evidence, migration and rollback, security and operations review |
| C3: high-impact | Highlight ranking, director plan semantics, local model/runtime, asset generation/licensing, automatic rendering, auth, persistence migration, deployment, data deletion, production default | All gates; adversarial validation; macOS and Linux evidence; explicit security, data, operations, and product/creative approval |
| Emergency | Active confidentiality, integrity, availability, or legal incident | Use the smallest reversible mitigation, preserve evidence, obtain incident authority, test the failure path, and complete C2/C3 follow-up before normal promotion |
Treat any unknown impact as C3 until discriminating evidence lowers it.
## Write the change record
For every C2/C3 decision, add or update an ADR when the repository has an ADR location. Until that location exists, include this record in the pull request or review artifact; do not invent a docs-of-record location silently.
```text
Requirement:
Selected approach:
Alternatives considered:
Benefits and trade-offs:
Operational consequences:
Security implications:
Verification evidence:
Revisit conditions:
Owner and review date:
Change class and affected environments:
Rollback or roll-forward procedure:
```
State what will not change. Map each claim to a test, measurement, schema comparison, scan, or reviewed artifact.
## Execute the gates
### Gate 1: scope and dependency direction
- Keep the modular monolith unless measured business requirements justify distribution.
- Put business rules in domain/application code; keep controllers, persistence, HTTP, filesystem, FFmpeg, and model processes at adapter boundaries.
- Do not add a dependency, abstraction, repository base class, mapper layer, CQRS infrastructure, cache, circuit breaker, virtual thread, or service merely to satisfy a pattern checklist.
- For each new dependency, record maintenance status, stable version, license, vulnerability result, why Spring Boot dependency management is insufficient if overridden, and removal plan.
- Block snapshots, milestones, release candidates, deprecated dependencies, and undocumented overrides.
- Require automated architecture rules before claiming module boundaries, no cycles, or inward dependencies are enforced. None were found in the 2026-07-21 test inventory.
### Gate 2: forbidden-path scan
Run before review:
```bash
rg -n 'pip install|curl |wget |get_pretrained|from_pretrained|snapshot_download' . --glob '!target/**' --glob '!.git/**'
rg -n 'write_silence|write_fallback_tone|fallback-to-heuristic|fallback.*copy' tools src/main docs
rg -n 'render-enabled:|auto-render|require.*approval|auto-start:|matchIfMissing' src/main src/test docs
rg -n -i 'api[_-]?key|secret|password|token|private[_-]?key' . --glob '!target/**' --glob '!.git/**'
rg -n '<version>.*(SNAPSHOT|M[0-9]+|RC[0-9]*)</version>' pom.xml
```
Classify every hit. Existing violations do not authorize new ones. Any new runtime download, placeholder, approval bypass, secret, or prerelease dependency blocks the change.
### Gate 3: deterministic verification
Use the locally provisioned toolchain without network access:
```bash
mvn -o test
mvn -o verify
git diff --check
```
`mvn verify` currently runs Surefire and JaCoCo; JaCoCo enforces 100% instruction, line, and branch coverage only for `org.example.videoclips.folder`. It does not provide the target 90% domain/application or 80% overall gate. Add focused tests at the lowest useful layer and use real FFmpeg/model/PostgreSQL infrastructure where behavior depends on it.
For clean-checkout evidence without modifying the current tree:
```bash
tmp_dir="$(mktemp -d)"; git archive HEAD | tar -x -C "$tmp_dir"; (cd "$tmp_dir" && mvn -o verify)
```
Run that only after Maven artifacts and approved model/runtime assets have been provisioned outside application startup. Record OS, architecture, Java, Maven, FFmpeg/ffprobe, Python, model checksums, command, duration, and result. A cache-dependent pass is not a reproducible-build proof.
### Gate 4: contract, data, security, and resilience
For affected changes, require all applicable evidence:
- Validate every public endpoint's success, validation, authentication, authorization, idempotency, bounded-collection, and failure cases against a versioned OpenAPI contract.
- Use RFC 9457 Problem Details without internal paths, exceptions, stack traces, storage keys, secrets, or sensitive payloads.
- Exercise Flyway migrations from an empty real PostgreSQL Testcontainer; verify constraints, indexes, transaction/concurrency behavior, backward-compatible rollout, backup/recovery, and roll-forward. H2 compatibility mode is not PostgreSQL evidence.
- Threat-model trust boundaries and apply server-side authorization, least privilege, restrictive CORS/headers, secret management, audit events, and sensitive-log redaction.
- Verify explicit timeouts, bounded safe retries, idempotency, partial failure, and resource limits for each network/process integration.
- Produce dependency, static, secret, container, and Software Bill of Materials (SBOM) evidence. Release with zero unresolved critical/high findings.
The current `pom.xml` has no Spring Security, OpenAPI, Testcontainers, ArchUnit/Spring Modulith, mutation testing, dependency lock, or security scanning integration. Do not mark this gate passed from intent or documentation.
### Gate 5: media and local-model evidence
For highlight, render, or asset changes, require more than structural output:
- Pin every local model and asset by immutable version/checksum and approved license; pre-provision it for macOS development and the target Linux/VPS/cloud runtime.
- Start and run model/media inference with all network access denied, including loopback RPC. Use in-process or approved non-network IPC. Missing or corrupt models must fail closed with a diagnosable, non-sensitive error.
- Validate highlight choices against versioned representative footage and annotations. Report precision/recall or an explicitly defined ranking metric, category slices, negative cases, and regressions.
- Probe actual output duration, codecs, streams, frame rate, resolution, black frames, freeze frames, clipping, integrated loudness, true peak, long silence, asset presence, and A/V synchronization.
- Verify voiceover intelligibility and timing, licensed music/SFX provenance, ducking, and that no silence/tone placeholder entered the final mix.
- Compare against a named baseline using predicted numeric outcomes and a blinded human rubric. Preserve manifests, commands, model versions, inputs or legal fixture references, measurements, and reviewer scores.
- Inspect each `qa-report.json` check's mechanism. Current duration/black/silence/sample-peak checks are measured; asset, mastering-filter, and overlay-bound checks remain structural, and the report is not a creative-quality certificate.
### Gate 6: operability and promotion
Promote the same immutable, platform-neutral application artifact across environments for a given release. Package it into signed, platform-specific runtime/model/image bundles for each supported OS/CPU target. Promote a bundle unchanged between environments with the same target; do not rebuild it per environment.
Require:
- typed validated external configuration; no embedded secret; safe disabled defaults; an environment diff;
- structured logs, bounded metrics, traces, secured health/readiness/liveness, business service-level indicators (SLIs), actionable alerts, and linked runbooks;
- non-root reproducible Open Container Initiative (OCI) image, vulnerability scan, read-only filesystem where practical, graceful shutdown, resource limits, and validated deployment definitions;
- measured workload with p50/p95/p99, throughput, errors, CPU, memory, GC, pools, threads, a 60-minute endurance run, downstream slowdown, and 2x expected peak behavior;
- commit-to-artifact provenance, SBOM, controlled approval, smoke/acceptance tests, and tested rollback or roll-forward;
- named owner, Recovery Time Objective (RTO), Recovery Point Objective (RPO), backup/restore, dependency-outage, migration-failure, and rollback procedures where applicable.
The user-provided default API targets of 200 requests/second/instance, p95 below 200 ms, p99 below 500 ms, error rate below 0.1%, no endurance memory growth, and graceful behavior at 2x peak are candidate baselines. Adjust them to the measured asynchronous video workload through an approved decision; do not claim they are currently met.
## Review the change
Require an independent reviewer for C2 and at least domain/architecture plus security/operations reviewers for C3. Add a creative-quality reviewer for highlight/render/audio changes and a data owner for persistence changes.
Review in this order:
1. Reproduce the original symptom or requirement.
2. Verify that one mechanism explains positive and negative observations.
3. Attempt an adversarial refutation: missing model, denied egress, corrupt asset, invalid plan, duplicate job, process timeout, disk pressure, restart, and approval absence as applicable.
4. Inspect the complete diff, configuration delta, schema/API delta, dependency tree, generated artifacts, and rollback.
5. Re-run the narrow test, then all required gates from a clean checkout.
6. Record residual risk and explicit approval; never infer approval from silence, a passing build, or an existing flag value.
Use the target human rubric as the final release scorecard: score architectural clarity, domain modeling, maintainability, security, tests, API, data, resilience, observability, performance, cloud operation, CI/CD, developer experience, docs, and operational readiness from 0 to 4 with concrete evidence. Block reference-architecture status if any category is below 3, the average is below 3.5, or security, data integrity, testing, or operational readiness is below 3.
## Promote or reject
Promote only when every applicable gate is green and the artifact is approved. The repository does **not** meet the production/reference-architecture definition as of 2026-07-21: the clean build is not reproducible, required security/API/architecture/CI/container/deployment gates are absent, and the production-quality local-model highlight flow has known fail-open behavior.
Reject or retain behind a disabled experimental flag when evidence is incomplete. Give every experimental flag an owner, safe default, expiry, removal criterion, and telemetry. Never change a production-facing default merely to make an experiment run.
For rollback:
- Prefer disabling a new path with an already-reviewed safe flag or rolling forward with a corrective artifact.
- Never edit an already released Flyway migration; write a new recovery migration and test it on production-shaped data.
- Preserve render manifests, audit records, failing inputs where legally permitted, and model/artifact checksums.
- Re-run smoke, integrity, and observability checks after rollback. A process restart alone is not rollback evidence.
## Historical rationale
Use these incidents to challenge recurring shortcuts; load `video-editing-failure-archaeology` for the full chronology.
| Incident | Lesson enforced here |
|---|---|
| Upload completion was initially a metadata flip; commit `7e8a214` routed it through object storage. | A state label must reflect the external side effect, not intention. Test the adapter boundary. |
| Generated clips were metadata-only until commit `66e998e` persisted real object locations. Signed URLs later derived fake paths until commit `7307082` used the persisted `objectKey`. | Verify artifact existence, identity, and retrieval end to end. Do not synthesize locations. |
| Worker files accumulated until commit `8f4804c` added cleanup in a `finally` block. | Test cleanup on success, failure, retry, and interruption; include disk saturation in operations review. |
| Internal storage locations leaked through API/events until commit `0d45f89` removed and tested them. | Treat paths, storage keys, and process output as boundary-sensitive data. |
| Retryable database-queue exceptions became terminal failures until commit `1737d8b` separated retry from DLQ exhaustion. | Model transient and terminal states separately; prove duplicate and retry behavior. |
| Commit `5d889b0` documented a renderer that worked but was not cinematic. Later work added category planning, validation, effects, QA, and approval. | Technical completion is not product quality. Predict and measure creative outcomes independently. |
| Commit `9536928` added approval to the local-director path; later highlight work initially defaulted to rendering without approval. The working tree now defaults highlight rendering off and approval on. | Apply policy to every workflow; similar names and flags do not imply equivalent gates. Keep tests on both property defaults. |
| Commit `97ba827` added automatic bootstrap and local generation with runtime installation/model-resolution and successful silence/tone fallbacks. The working tree removed asset-runtime bootstrap and placeholder success, but standalone bootstrap and the CV path remain network-capable. | “Local” is not the same as offline, provisioned, licensed, or fail-closed. Test absence and denied-egress paths. |
## Definition of done
Check every applicable item; do not weaken the list in a feature branch.
- [ ] Requirement, class, scope, affected environments, and eight-part decision record are explicit.
- [ ] No forbidden runtime download, external AI, unlicensed asset, placeholder, unapproved render, egress dependency, or silent default change remains.
- [ ] Core business rules remain independent of adapters and automated architecture tests enforce the intended boundaries.
- [ ] Clean offline one-command build passes on supported macOS and Linux from committed sources plus separately provisioned approved artifacts.
- [ ] Focused, full, architecture, API, persistence, integration, security, resilience, and end-to-end tests pass as applicable.
- [ ] API/schema/config compatibility, migration, idempotency, failure semantics, and rollback are verified.
- [ ] Dependency, static, secret, container, vulnerability, license, SBOM, signing/provenance, and reproducibility evidence passes as applicable.
- [ ] Logs, metrics, traces, health, SLI/dashboard/alert, capacity, shutdown, recovery, and runbooks are verified in production-like staging.
- [ ] Media/model output passes objective probes, offline/fail-closed tests, baseline comparison, representative/negative fixtures, and blinded human review.
- [ ] The immutable application artifact and target-specific signed bundle are approved and traceable from commit through controlled promotion without per-environment rebuild.
- [ ] Each target rubric category has linked evidence; no score is below 3 and the average is at least 3.5.
- [ ] Docs, OpenAPI, diagrams, configuration reference, threat model, testing/deployment/operations guides, and ADRs match implementation.
## Provenance and maintenance
This skill was grounded in repository source, configuration, tests, docs, and Git history on **2026-07-21**. Re-verify volatile facts before relying on them:
```bash
git log -12 --oneline --decorate
```
```bash
find src/test/java -name '*Test.java' | wc -l; awk -F'[,: ]+' '/Tests run:/{sum+=$3;fail+=$5;err+=$7;skip+=$9} END{print "tests="sum,"failures="fail,"errors="err,"skipped="skip}' target/surefire-reports/*.txt 2>/dev/null
```
```bash
for p in README.md mvnw .mvn .github Dockerfile compose.yml docker-compose.yml; do test -e "$p" && echo "present $p" || echo "absent $p"; done
```
```bash
rg -n 'enabled:|auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:|strict-runtime:' src/main/resources/application*.yml
```
```bash
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|local_asset_runtime_degraded' tools src/main
```
```bash
rg -n 'duration_matches_timeline|required_assets_resolved|text_overlays_safe|audio_mastering_applied' src/main/java/org/example/videoclips/editing/*Renderer.java
```
```bash
rg -n 'spring-security|springdoc|testcontainers|archunit|modulith|dependency-check|cyclonedx|pitest' pom.xml
```
```bash
mvn -o verify
```

View File

@ -0,0 +1,476 @@
---
name: video-editing-cinematic-highlights-campaign
description: "Load this skill when planning, implementing, debugging, or promoting the single-source cinematic highlight campaign: local-only highlight identification and editing with production visuals, music, SFX, and voiceover; especially for candidate generation, an offline director, model packaging, director-plan safety, measured media QA, resumability, source-to-final tests, or macOS/Linux parity."
---
# Cinematic Highlights Campaign
Status verified against the repository on **2026-07-21**. Treat every path or component marked
`[PROPOSED]` as work that does not exist yet. Introduce it only through
`video-editing-change-control`.
## Use this campaign correctly
Use this skill to drive the hardest live problem: turn one source video into correctly selected,
high-cinematic-quality highlights with production visuals, music, sound effects (SFX), and
voiceover, using only pre-provisioned local models resident in the service runtime.
Do **not** use it for:
| Need | Use instead |
|---|---|
| A routine, already-understood patch | `video-editing-change-control` |
| A symptom with no established root cause | `video-editing-debugging-playbook` |
| Media theory or FFmpeg fundamentals | `cinematic-media-engineering-reference` |
| General test evidence or existing test inventory | `video-editing-validation-and-qa` |
| Existing measurement commands and artifact inspection | `video-editing-diagnostics-and-tooling` |
| Post-triage causal proof or first-principles derivation | `video-editing-proof-and-analysis-toolkit` |
| Environment setup or normal operation | `video-editing-build-and-env` or `video-editing-run-and-operate` |
Never use this campaign to route around approval, licensing, security, or change control.
## Terms and hard constraints
| Term | Meaning here |
|---|---|
| Certified fixture | Rights-cleared source media plus immutable annotations, checksum, provenance, and an assigned development or locked-holdout split. |
| Temporal IoU (tIoU) | Intersection duration divided by union duration for a predicted and annotated highlight interval. |
| Recall@3 | Fraction of annotated highlights matched by one of the top three predictions at the stated tIoU. |
| Macro-F1 | Unweighted mean of per-category F1 scores, so a large category cannot hide a weak one. |
| nDCG@3 | Normalized discounted cumulative gain for the top three ranked candidates; higher relevance near rank one receives more credit. |
| SLO | Service-level objective: an approved target for a measured service-level indicator over a stated workload/window. |
| SBOM | Software Bill of Materials: an inventory of shipped software components and their identities. |
| Fail closed | Stop before rendering or promotion when required evidence, models, assets, approval, or QA is missing. |
| Local model | A model whose weights, loader, tokenizer, and runtime dependencies are pre-provisioned in the deployed artifact or mounted runtime bundle; loading it performs no network operation. |
| Creative rubric | A blinded 0-4 human score for story, selection, pacing, visual craft, sound design, voiceover, and factual grounding. |
The no-waiver rules are authoritative in `video-editing-change-control`; these are their campaign consequences:
- Make no automatic dependency or model downloads. Do not call external AI services.
- Make no network connection during the certified offline run, including loopback and model
resolution. The current CV worker uses loopback HTTP; replace that transport under change control
before certification.
- Use only assets with recorded licenses and permitted production use.
- Produce no placeholder silence, tones, generic assets, or unrequested fallback render.
- Render only after explicit approval. Do not change packaged production-facing defaults.
- Do not treat a successful FFmpeg exit or a JSON field set to `true` as quality evidence.
- Keep every experiment isolated behind an explicit, disabled-by-default campaign control. Promotion
still follows `video-editing-change-control`.
## Current truth: do not design from the plan documents
| Repository fact | Evidence and consequence |
|---|---|
| `HighlightSourceScheduler` calls `HighlightCandidateGenerator` after source analysis and before `HighlightDirectorPromptGenerator`. | The single-source flow now persists `analysis/category.json` and `analysis/highlight-candidates.json`; missing files mean candidate generation failed or the project predates this change. |
| `HighlightCandidateGenerator` is a deterministic baseline; `CinematicHighlightAnalyzer` remains the older multi-clip ranker. | The single-source generator ranks shot/coverage windows from source-level visual scores, FFmpeg silence sections, scene score, duration, and position. It explicitly downweights heuristic/fallback visual evidence; do not call it semantic highlight understanding. |
| No local director executes the generated highlight prompt. | `director/director-brief.md` tells an operator to use Codex, Claude, or another agent; rendering begins only when `director/edit-plan.json` appears. |
| The highlight plan has a dedicated `HighlightDirectorPlanValidator`; it does not use the multi-clip `EditPlanValidator`. | It requires persisted category/candidates, safe unique IDs, candidate-contained time ranges, configured duration bounds, and nonblank visual/music/SFX/voiceover direction before asset work. |
| Plan-derived `highlightId` and mutable asset-request paths are validated. | `HighlightLocalAssetWorker` requires exact project/highlight identity, supported types, safe keys, bounded duration, contract-exact request/target paths, project containment, and no symlink component before materialization. Adjacent nonblank asset-license sidecars are required and preserved; authenticity, checksum, origin, allowed-use validation, and request-file integrity signatures remain open. |
| Standalone bootstrap scripts remain network-capable. | Both worker shell scripts run `pip install` in `auto` mode; the CV script defaults to `yolov8n.pt`. `LocalAssetRuntimeVerifier` no longer invokes the asset bootstrap, but operators must still not run either launcher in a certified environment. |
| Strict asset readiness now fails startup. | The 2026-07-21 working tree requires pre-provisioned Piper, MusicGen, and AudioGen paths and throws when strict readiness is incomplete. No such three-model bundle is present in this repository. |
| Placeholder asset success was removed. | The Python and Java asset workers now return failure and delete invalid output instead of emitting OS speech, silence, or tones. Missing requested music, SFX, or voiceover blocks rendering. |
| Audio and preview failures now fail the highlight render. | The renderer no longer copies the pre-mix timeline or final output after those failures. Prepared SFX keys and rendered SFX keys are shared. |
| Highlight QA now includes selected media probes but is not a creative-quality gate. | Final duration, black ranges, long silence, and sample peaks are probed. Integrated loudness, true peak, A/V sync, freeze detection, raster safe area, asset semantics, and human creative acceptance remain open. |
| The source-to-final test is incomplete. | `HighlightDirectorFlowServiceTest` uses a mocked renderer and dummy files. `CinematicEditingIntegrationTest` covers the separate multi-clip flow. |
| Processing is not stage-resumable. | The scanner skips only projects with root `final.mp4`; there is no per-stage journal, input digest, lease, or verified checkpoint. |
Reconfirm those claims before opening a campaign change:
```bash
rg -n "HighlightCandidateGenerator|category.json|highlight-candidates.json" \
src/main/java/org/example/videoclips/editing/{HighlightSourceScheduler,HighlightCandidateGenerator,HighlightDirectorPromptGenerator}.java
rg -n "planValidator|HighlightDirectorPlanValidator|highlightId\(\)|targetPath\(\)" \
src/main/java/org/example/videoclips/editing/{HighlightDirectorFlowService,HighlightDirectorPlanValidator,HighlightLocalAssetWorker}.java
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded|duration_matches_timeline" \
tools src/main/java/org/example/videoclips/editing
```
Expected on the 2026-07-21 working tree: matches in all three commands; explicit scheduler wiring to
`HighlightCandidateGenerator`; no placeholder-success implementation in the asset synthesizers; network-capable
launcher matches remain non-zero. Request target strings remain in the code but are now checked against exact
project-local paths before use. If that changes, update this campaign before proceeding.
## Campaign gate map
Do not start a later phase until the preceding gate is green.
| Phase | Deliverable | Numeric gate |
|---:|---|---|
| 0 | Frozen offline baseline | 245 tests in 62 test classes; the full offline Maven suite has 0 failures, 0 errors, and 0 skips |
| 1 | Certified fixture and annotation set | 4 current categories represented; 2 independent annotations per source; 100% checksums/licenses/splits present |
| 2 | Single-source category and candidate wiring | Macro-F1 and Recall@3 thresholds calibrated and recorded; every processed fixture emits exactly 1 category file and 1 candidate file |
| 3 | Resident local director and safe plan | 0 network operations; 100% schema-valid plans; all adversarial plans rejected before filesystem/render work |
| 4 | Fail-closed model and asset runtime | 100% manifest/checksum/license/canary pass; any one missing capability prevents readiness and render |
| 5 | Production render semantics | Every requested effect/audio/VO action either has execution evidence or fails the render; 0 silent degradations |
| 6 | Measured technical and creative QA | 0 ERROR checks; creative average >=3.5/4 and no dimension <3; paired-preference lower 95% bound >50% |
| 7 | Resumability and real source-to-final test | 1 final per requested highlight, 0 duplicate side effects, and exact recovery after every injected stage interruption |
| 8 | Offline platform, security, performance, operations | macOS and Linux/VPS outputs pass identical gates; 0 attempted network calls; 0 path escapes; workload SLOs met |
| 9 | Controlled promotion | All change-control evidence present; human reference-architecture rubric has no score <3 and average >=3.5 |
## Phase 0: freeze an offline baseline
1. Work from the repository root. Record, but do not mutate, source and tool state:
```bash
git status --short
git rev-parse HEAD
java -version
mvn -version
ffmpeg -version | head -n 1
ffprobe -version | head -n 1
printf 'test_methods='; rg -n '^\s*@Test\b' src/test/java | wc -l
```
2. Run the current tests that do not invoke the network-capable model/asset bootstrap paths:
```bash
mvn -q -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
```
Gate 0 expectations for the current working tree: Maven exits `0`. Treat generated test media as
disposable, not certified evidence. A 2026-07-21 `mvn -q -o verify` run executed all 245 tests in 62 test classes with
zero failures, errors, or skips and passed the bound JaCoCo gate. Maven offline mode does not prevent
subprocess networking, so call this offline dependency-resolution evidence, not no-egress certification.
- If offline Maven reports a missing artifact, branch to `video-editing-build-and-env`; pre-provision a
locked artifact repository or build cache. Do not enable network.
- If a test fails because local voice/TTS behavior differs, branch to Phase 4. The current test suite
can consume an untracked local environment and is not yet clean-checkout reproducible.
- If the worktree is dirty, preserve unrelated changes. Record their paths and isolate campaign
evidence; never reset them.
## Phase 1: create certified fixtures and annotations
The repository has no certified creative golden inventory. Do not promote existing `input/` or
`output/` media to goldens by assumption.
Under change control, add `[PROPOSED]` fixture metadata and small rights-cleared media, or an
authenticated internal fixture-bundle mount whose immutable digest is recorded. The metadata must
contain: fixture ID, SHA-256, byte size, duration, codecs, category, rights/license ID, permitted use,
provenance, split, and annotation version. Keep weight files and large/proprietary footage out of Git.
For each source, collect two independent annotations containing category, positive highlight
intervals, excluded intervals, story role, visible facts allowed in voiceover, unsafe/blurred frames,
speech intervals, and music/SFX intent. An adjudicator resolves disagreement without seeing model
output. Lock the holdout split before tuning.
Starting floor (a campaign minimum, not a SOTA claim): at least 12 sources, at least 3 each for
`car_vlog`, `food_vlog`, `family_vlog`, and `generic_vlog`; at least one no-audio and one low-quality
negative source; 2 annotators per source. Record a larger statistically powered sample before making
external quality claims.
Gate 1 `[PROPOSED]` command; implement a repository script only through change control:
```bash
test -x tools/validate-highlight-fixtures && tools/validate-highlight-fixtures --offline --strict
```
Expected: exit `0`, `sources>=12`, `categories=4`, `annotators_per_source>=2`,
`missing_checksums=0`, `missing_licenses=0`, `split_overlap=0`. If the script is absent, stop: do not
replace it with visual inspection. If rights are unclear, quarantine the fixture and branch to legal/
asset governance.
## Phase 2: wire category and candidate generation
Implement a single-source application use case between `HighlightSourceAnalyzer.analyze` and prompt
generation. It must consume persisted source, scene, audio, and visual analysis; classify using actual
local evidence; generate shot-boundary-aligned candidates; and atomically write exactly:
- `analysis/category.json`
- `analysis/highlight-candidates.json`
Do not merely copy the metadata-keyword classifier from the multi-clip path. First define a
deterministic heuristic baseline, then compare local learned scorers against it. Candidate IDs must be
stable for the same source/config/model digests. Durations must honor the configured 8-35 second
range; at most the configured top 3 proceed to planning. Uncertain classification is a review state,
not permission to render generic content.
Before running, predict category confusion, candidate Recall@3 at `tIoU>=0.5`, and ranking nDCG@3 on
development fixtures. Record predicted numbers, then run once. Set final thresholds from baseline plus
confidence intervals in an ADR; until that ADR exists, use the campaign floors: macro-F1 `>=0.85`,
Recall@3 `>=0.80`, nDCG@3 `>=0.80`, and invalid/out-of-range candidates `=0`. These are candidate
promotion thresholds, not current results.
Gate 2 `[PROPOSED]`:
```bash
mvn -o -Dtest=HighlightSourceCandidatePipelineTest,HighlightCandidateEvaluationTest test
test -x tools/evaluate-highlight-selection && \
tools/evaluate-highlight-selection --fixtures certified --split holdout --offline
```
Expected: both exit `0`; one category and one candidate artifact per source; the four numeric floors
above pass. If category passes but Recall@3 fails, branch to segmentation/fusion experiments. If Recall
passes but nDCG fails, keep candidates and work only on ranking. If all offline metrics pass but human
selected moments remain weak, inspect annotation agreement before changing the model.
## Phase 3: implement a resident local director and validate plans
Replace the operator-mediated external-agent step with an explicit application port and a local
adapter. The adapter must use a pre-provisioned model path, constrained JSON decoding, fixed schema
version, bounded context/output, deterministic seed where supported, timeout, cancellation, and no
socket or model-hub resolution. Preserve human approval; the model may propose but never approve.
Validate `HighlightDirectorPlan` before creating storyboard, asset requests, or directories:
| Required validation | Reject when |
|---|---|
| Identity | Project/source mismatch; unknown candidate; duplicate/unsafe highlight ID |
| Numeric safety | NaN/infinity; negative/reversed/out-of-source time; target outside configured range |
| Structural bounds | Zero highlights; more than configured maximum; empty/oversized strings or arrays |
| Semantics | Unknown category, story role, effect, transition, render profile, voice, or asset type |
| Grounding | Voiceover asserts a fact absent from certified visible/transcript evidence |
| Path safety | Absolute path, `..`, separator, symlink escape, or caller-supplied target path |
| Asset closure | A requested music/SFX/voice/font/LUT capability has no licensed local resolution |
Derive target paths server-side from validated IDs. Resolve against the project root, normalize,
verify containment, reject symlinks at trust boundaries, and create through the store. Make the JSON
schema version explicit and validate it during the Maven build.
Gate 3 `[PROPOSED]`:
```bash
mvn -o -Dtest=LocalHighlightDirectorContractTest,HighlightDirectorPlanValidatorTest,HighlightPathSafetyTest test
```
Expected: exit `0`; valid plans accepted `=100%`; malformed, unknown-field, timestamp, NaN/infinity,
duplicate-ID, `../`, absolute-path, backslash, symlink, oversized-text, unknown-effect, and missing-asset
cases rejected `=100%`; filesystem writes after a rejected plan `=0`; network operations `=0`.
If constrained decoding still emits invalid JSON, do not repair it silently: record the failure and
stop. If factual grounding cannot be established, omit the line or route to human review.
## Phase 4: package and preflight all local capabilities
Add `[PROPOSED]` typed configuration for a runtime model root and a signed/immutable model manifest;
do not overload the current free-form model-name properties. For every CV, ASR, director, TTS, music,
and SFX model record: purpose, relative path, exact bytes, SHA-256, loader/runtime version, tokenizer/
config paths, license identifier and license-text digest, approved use, attribution, memory estimate,
and a deterministic canary input/output predicate.
Pre-provision Python wheels and weights in the build/deployment supply chain. Use locked versions and
hashes. Production startup must never run `pip`, `YOLO("yolov8n.pt")`, or
`get_pretrained("facebook/...")`. A readiness preflight must verify regular-file containment,
checksums, licenses, loader compatibility, capacity, and one inference canary for every enabled
capability. Any failure makes readiness false and blocks project claims and rendering.
Gate 4 `[PROPOSED]`:
```bash
test -x tools/verify-local-model-bundle && \
tools/verify-local-model-bundle --manifest "${VIDEO_EDITING_MODEL_MANIFEST:?set manifest}" --offline --strict
mvn -o -Dtest=LocalModelManifestTest,OfflineCapabilityPreflightTest,FailClosedAssetGenerationTest test
```
Expected: `models_checked>=6`, `checksum_mismatches=0`, `license_failures=0`, `canary_failures=0`,
`network_attempts=0`; Maven exits `0`. Re-run after corrupting one byte in a disposable test copy:
expected preflight exit non-zero, readiness false, renders started `=0`. If a license is absent or
ambiguous, the model/asset is unavailable, not experimental.
Remove production paths to `say`, `espeak`, tones, and silence. Keep negative tests proving these
cannot satisfy a required asset. Do not edit packaged defaults during the campaign; activate new
components only with explicit environment configuration in approved environments.
## Phase 5: make render instructions real
Implement an allowlisted renderer capability registry. A plan may name only behavior the renderer
can prove it executed. Today, any non-empty visual treatment maps to the same crop/grade and fades are
per-segment. Audio and preview failures now propagate; keep regression tests on that fail-closed behavior.
Do not relabel the remaining generic visual behaviors as cinematic.
| Plane | Required semantics and evidence |
|---|---|
| Selection/edit | Multiple decisions when the story requires them; source/timeline arithmetic verified; true inter-shot transitions rather than independent fades |
| Visual | Explicit color transform, stabilization, reframing, speed ramp, overlay/font, and transition parameters; command/filter evidence plus output measurements |
| Music | Licensed/generated asset digest; deliberate in/out points, trim/loop policy, gain envelope, and final mix contribution |
| SFX | One existing local asset per cue; sample-accurate or measured timing; no missing-path attempt |
| Voiceover | One grounded script version, approved local voice/model, line timing/alignment, intelligibility measurement, and ducking evidence |
| Master | Two-pass or otherwise measured loudness normalization; output codec, geometry, cadence, duration, and channel layout probed from the file |
Make every requested production asset blocking. On FFmpeg, preview, or mix failure, mark the attempt
failed and publish no `final.mp4`. Write to a temporary name and atomically promote only after QA.
Record tool versions, complete argv, exit codes, input/output hashes, models/assets/licenses, timing,
and effective config in `render-manifest.json`.
Gate 5 `[PROPOSED]`:
```bash
mvn -o -Dtest=HighlightRendererIntegrationTest,HighlightAudioMixIntegrationTest,HighlightEffectContractTest test
```
Expected: exit `0`; requested actions `= executed actions`; unresolved actions `=0`; silent
degradations `=0`; failure injection for each FFmpeg stage publishes final files `=0`. If a filter is
unsupported on macOS or Linux, do not substitute one; branch to Phase 8 and either standardize FFmpeg
or remove the capability through change control.
## Phase 6: replace asserted QA with observed QA
Each QA check must include expected value, observed value, unit, tool/algorithm version, command or
method, artifact path/digest, severity, and pass/fail. Probe the actual final file. At minimum measure:
stream presence, duration delta, 1920x1080 geometry, 30 fps cadence, H.264/AAC codecs, 48 kHz audio,
black/freeze spans, silence spans, clipping/true peak, integrated loudness, overlay bounds, asset
contribution, voice intelligibility/alignment, and requested-effect execution.
Use the current configured mastering targets as initial technical gates: integrated loudness
`-16 LUFS +/-1`, true peak `<=-1.5 dBTP`, duration error `<=0.25 s`, missing requested streams/assets
`=0`, and ERROR checks `=0`. Calibrate black/silence/freeze tolerances by content and annotations;
do not apply a universal threshold that rejects intentional black or quiet moments.
Operator spot-check commands (diagnostic only, not a replacement for automated parsing):
```bash
FINAL="${FINAL:?set final mp4}"
ffprobe -v error -show_entries format=duration:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$FINAL"
ffmpeg -hide_banner -i "$FINAL" -vf "blackdetect=d=0.25:pix_th=0.10" -af "silencedetect=n=-45dB:d=1.0" -f null -
ffmpeg -hide_banner -nostats -i "$FINAL" -filter_complex "ebur128=peak=true" -f null -
```
Gate 6 technical command is `[PROPOSED]`; implement the parser through change control rather than
copying console text into `qa-report.json`:
```bash
test -x tools/verify-highlight-render && \
tools/verify-highlight-render --final "$FINAL" --manifest "${MANIFEST:?set render manifest}" --strict
```
Expected: exit `0`, `error_checks=0`, `missing_requested_assets=0`, `duration_error_seconds<=0.25`,
`integrated_lufs` in `[-17,-15]`, and `true_peak_dbtp<=-1.5`. If the tool is absent, Gate 6 is red.
Run blinded creative review against the frozen deterministic baseline. Use at least 3 reviewers per
output; score the seven rubric dimensions 0-4; randomize A/B order; retain disagreements. Gate:
average `>=3.5`, every dimension median `>=3`, factual errors `=0`, and the 95% confidence interval
lower bound for preference over baseline `>50%`. If technical QA passes but creative QA fails, do not
tune FFmpeg blindly: branch according to reviewer tags (selection -> Phase 2, story/VO -> Phase 3,
craft/audio -> Phase 5).
## Phase 7: add resumability and a real source-to-final test
Add a durable stage journal with state, attempt, lease owner/expiry, started/completed timestamps,
code/config/model/input digests, output paths/digests, and failure classification. Required stages:
claim, analyze, classify, rank, direct, validate, approve, prepare-assets, render, QA, publish. Commit
each checkpoint atomically. Resume only when all input digests match; otherwise invalidate that stage
and every dependent stage. Never infer completion from file existence alone.
Build a real source-to-final test using a tiny rights-cleared fixture and pre-provisioned test models/
assets. Do not mock the analyzer, director, asset generator, renderer, FFmpeg, or FFprobe. Test restart
after every stage, duplicate scans, stale leases, corrupt checkpoints, changed model digest, invalid
plan, missing asset, FFmpeg failure, and crash between temporary output and atomic publish.
Gate 7 `[PROPOSED]`:
```bash
mvn -o -Dtest=HighlightSourceToFinalIT,HighlightWorkflowResumeIT,HighlightWorkflowFailureIT test
mvn -o verify
```
Expected: exit `0`; final outputs equal requested approved highlights; duplicate renders/uploads `=0`;
published partial files `=0`; each injected interruption resumes at the first invalid/incomplete stage;
changing any input digest recomputes all dependent stages. If the test uses a mock renderer or dummy
text as MP4, it does not satisfy this gate.
## Phase 8: prove offline parity, security, performance, and operations
Run the same immutable fixture bundle, model bundle, service artifact, config digest, and FFmpeg build
on macOS development and Linux/VPS. Cloud deployment may mount the same pre-provisioned bundle but
must not fetch it at application startup. Container work is `[PROPOSED]`; the repository has no current
container definition, so do not claim `--network none` validation until one is approved.
Define the representative workload before measuring: source minutes, resolution/codecs, highlight
count, concurrent jobs, hardware/accelerator, warm/cold state, and storage. Record per-stage p50/p95/
p99, real-time factor (wall seconds/source seconds), CPU, accelerator, RSS, disk peak, queue depth,
failures, and output bytes. Establish SLOs from measured capacity and business demand; never import the
200-RPS synchronous-API example into a media pipeline.
Gate 8 `[PROPOSED]`:
```bash
test -x tools/run-offline-highlight-certification && \
tools/run-offline-highlight-certification --fixtures certified --models "${VIDEO_EDITING_MODEL_MANIFEST:?}" --strict
mvn -o -Dtest=HighlightOfflineSecurityIT,HighlightCapacityIT,HighlightGracefulShutdownIT test
mvn -o verify
```
Expected on both platforms: technical and creative Gate 6 pass; `network_attempts=0`; path escapes
and symlink escapes `=0`; checksum/license failures `=0`; orphan child processes `=0`; corrupt or
partial finals `=0`; measured p95 and resource peaks remain within the approved workload SLO. Compare
semantic decisions and QA outcomes, not bit-identical encoded video unless the codec/toolchain is
fully standardized. If platform results diverge, capture tool/model/config digests and stop promotion.
Security review must threat-model malicious media, decompression/resource exhaustion, hostile model
output, path traversal/symlinks, command injection, poisoned weights, unlicensed assets, prompt
injection in transcripts/metadata, denial of service, and sensitive media/log leakage. FFmpeg commands
remain argv lists, never shell strings. Run workers with least privilege, bounded CPU/memory/time/disk,
read-only model mounts, project-scoped writable storage, and no network capability.
Operations must expose stage duration/failure, queue age/depth, active jobs, model readiness,
candidate counts, approval waits, render real-time factor, QA failures, disk pressure, and worker
restarts without source filenames, prompt text, or project IDs as metric labels. Define alerts,
retention, cancellation, graceful shutdown, recovery, and model rollback before production.
## Ranked solution menu
Choose with measured evidence, not model popularity.
| Rank | Candidate | Use when | Required derivation and disproof |
|---:|---|---|---|
| 1 | Local multimodal director over deterministic shot/audio candidates | Resident model fits capacity and materially improves grounded story decisions | Predict memory, latency, Recall@3-to-plan loss, grounding error, and creative preference. Disprove with locked holdout, ablation without visual/audio inputs, adversarial transcripts, and local-only trace. |
| 2 | Local feature models + deterministic ranker + smaller constrained text director | A full multimodal director is too slow or weakly grounded | Derive feature normalization, fusion weights, calibration, context budget, and failure states. Ablate each feature; require one mechanism to explain positive and negative fixtures. |
| 3 | Local learned temporal scorer + template/constrained planner | Director variance blocks schema safety but selection can be learned | Prove scorer improves Recall/nDCG and templates improve repeatability without lowering blinded creative scores. Reject if category styles collapse to generic edits. |
| 4 | Deterministic heuristics and fixed renderer | Establish the baseline and retain an emergency diagnostic comparator only | Specify every weight/rule and sensitivity. Never promote as “high cinematic quality” without independently passing all creative gates. |
For any chosen option, the decision record must state: requirement; selected approach; alternatives;
benefits/trade-offs; operational consequences; security implications; verification; revisit condition.
No new database, queue, service, framework, or model is justified merely by this campaign.
## Known wrong paths: fence these off
- Do not use filename/project-style keywords as proof of visual content category.
- Do not ask an external agent to write the plan and call the workflow local-only.
- Do not let model libraries resolve names such as `yolov8n.pt` or `facebook/musicgen-small` at runtime.
- Do not interpret a non-strict readiness log as certification. Strict mode now fails startup, but its
sidecar and file-presence checks do not establish provenance authenticity, checksums, allowed use, or quality.
- Do not accept silence, sine tones, OS `say`/`espeak`, missing optional assets, or source-audio-only
copies when the approved plan requests production audio.
- Do not use `EditPlanValidator` unchanged for highlight projects; it reads the multi-clip store and
analysis contract. Reuse validation concepts, not the wrong persistence contract.
- Do not resolve untrusted plan IDs or request target paths directly with `Path.resolve`/`Path.of`.
- Do not add more prompt adjectives to fix missing analysis, weak selection, or unsupported effects.
- Do not call a fixed 4% crop a dynamic crop, per-segment fades a crossfade, or hard-coded `true` QA.
- Do not tune on the locked holdout, promote unlabeled output, or judge cinematic quality by eye.
- Do not retry permanent validation/model/license failures; do not resume from file existence alone.
- Do not alter packaged production defaults to expose campaign behavior.
## Phase 9: promotion protocol
Route every implementation and final promotion through `video-editing-change-control`. Promotion
requires one evidence bundle containing baseline commit/tool digests; fixture/annotation version and
rights; model/dependency SBOM, hashes, licenses, and signatures; predictions made before experiments;
selection/director/renderer/QA results including negatives and ablations; offline network-denial proof;
macOS/Linux parity; performance/capacity; threat model; recovery/graceful-shutdown evidence; human
review data; runbook/alerts; rollback; and ADRs.
The project is not campaign-complete until every gate passes on the locked holdout, the Fortune 500
human-review rubric has no category below 3 and average at least 3.5, security/testing/operational
readiness are each at least 3, and no critical behavior relies on an undocumented assumption. A model
upgrade restarts Phases 4, 6, 7, and 8. A schema/renderer change restarts Phases 3, 5, 6, 7, and 8.
## Provenance and maintenance
Primary code: `HighlightSourceScheduler`, `HighlightSourceAnalyzer`,
`HighlightDirectorPromptGenerator`, `HighlightDirectorPlanScanner`, `HighlightDirectorFlowService`,
`HighlightDirectorPlan`, `HighlightLocalAssetWorker`, `LocalAssetSynthesizer`,
`LocalAssetRuntimeVerifier`, `HighlightFfmpegRenderer`, `FileSystemHighlightProjectStore`, and the two
worker scripts. Planning documents are context, not implementation truth.
Re-verify volatile facts in one line each:
```bash
rg -n '^\s*@Test\b' src/test/java | wc -l
rg -n "category.json|highlight-candidates.json" src/main/java/org/example/videoclips/editing
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded" tools src/main/java/org/example/videoclips/editing
rg -n "duration_matches_timeline|required_assets_resolved|audio_mastering_applied" src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java
mvn -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
```
Update the dated truth table, baseline counts, thresholds/ADR links, fixture inventory, model manifest
contract, and platform/SLO evidence whenever those commands or campaign decisions change.

View File

@ -0,0 +1,418 @@
---
name: video-editing-config-and-flags
description: Load when inspecting, setting, adding, renaming, reviewing, or debugging video-clipping.*, Spring profiles, scheduler/render guards, adapter selectors, filesystem paths, retention, local CV or local asset worker settings, their environment-variable mappings, or configuration-default drift. Use before any configuration or production-default change.
---
# Video Editing Configuration and Flags
Use this runbook to determine the effective configuration before starting the service or changing behavior. Treat every value below as repository state verified on **2026-07-21**, not as a production endorsement.
## Scope and routing
Use this skill for property ownership, precedence, defaults, environment mappings, validation, and flag-change review.
Do **not** use it to:
- start or operate a workflow; use `video-editing-run-and-operate`;
- rebuild local environments or provision models; use `video-editing-build-and-env`;
- diagnose a failed render or worker; use `video-editing-debugging-playbook` and `video-editing-diagnostics-and-tooling`;
- decide whether a behavior change may ship; use `video-editing-change-control`;
- judge output quality; use `video-editing-validation-and-qa`.
Definitions used here:
| Term | Meaning |
|---|---|
| Java default | Field initializer in `VideoClippingProperties`; applies when no higher-precedence source supplies the key. |
| Packaged effective default | Value obtained from packaged `application.properties` plus `application.yml` with no profile and no environment override. This is what a plain service start sees. |
| Guard | Spring condition or runtime test that enables a component or destructive action. |
| Adapter selector | String property used by `@ConditionalOnProperty` to choose one implementation. |
| Pre-provisioned | Dependency, executable, model, and licensed asset already present in the runtime image/filesystem; startup must not fetch it. |
| Fail-closed | Missing or invalid required capability prevents processing or promotion; it must not silently substitute lower-quality output. |
## Non-negotiable configuration policy
Enforce these user-supplied constraints for every environment:
1. Prohibit automatic dependency or model downloads, external AI services, network access, unlicensed assets, placeholder silence or tones, and unapproved rendering.
2. Do not change a production-facing default without `video-editing-change-control`, an ADR-style decision record, tests, rollback instructions, and environment impact analysis.
3. Provision local models and dependencies before runtime. Pin their identity and checksum outside behavioral configuration, then verify them before accepting work.
4. Production startup now fails under `strict-runtime=true` when Piper, MusicGen, AudioGen, their local paths, or the Python stack are absent. Keep that behavior and add license/checksum manifest verification before calling the runtime certified.
5. Keep all automatic render controls disabled and require an approval artifact at the render boundary.
6. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` with their current default `BOOTSTRAP_MODE=auto` in a network-prohibited runtime. Both scripts can create virtualenvs and invoke `pip`; the CV script can also make Ultralytics resolve/download `yolov8n.pt`.
## Resolve the effective value
Use Spring Boot precedence: command-line arguments and environment-derived properties override packaged config; profile files override non-profile packaged files. This repository also embeds explicit `${CUSTOM_ENV:default}` placeholders in `application.yml`.
Follow this procedure:
1. Record active profiles and all `VIDEO_EDITING_*`, `FOLDER_SCHEDULER_*`, `LOCAL_ASSET_*`, and `LOCAL_CV_*` variables without exposing secrets.
2. Locate the key in `application-<profile>.*`, then `application.yml`, then `application.properties`, then the Java initializer.
3. Check the component guard below. A bound value can exist while its component is absent.
4. Check whether Java and packaged defaults differ. Packaged configuration wins in a normal application start.
5. For a generic Spring environment override, convert dots and hyphens to underscores and uppercase, for example `video-clipping.cleanup.enabled` to `VIDEO_CLIPPING_CLEANUP_ENABLED`. Prefer an explicit placeholder already present in `application.yml`; it documents the supported operational name.
Read configuration without starting the service:
```bash
sed -n '1,220p' src/main/resources/application.properties
sed -n '1,220p' src/main/resources/application.yml
sed -n '1,220p' src/main/resources/application-jpa.properties
sed -n '1,160p' src/main/resources/application-cinematic-editing-local.yml
sed -n '1,160p' src/main/resources/application-folder-scheduler-local.yml
sed -n '1,220p' src/test/resources/application.properties
```
## Critical drift: plain startup is unsafe
The packaged effective defaults activate multiple independent consumers and local bootstrap paths:
| Axis | Java default | Packaged effective default | Risk/status |
|---|---:|---:|---|
| folder scheduler enabled | `false` | `true` | Consumes files under `./input/source`. Unsafe for an exploratory start. |
| editing enabled | `true` | `true` | Creates editing components and starts runtime checks. |
| local director enabled | `true` | `true` | Polls editing input. Auto-render remains off in base config. |
| highlight scheduler enabled | `true` | `true` | Polls and moves highlight sources. |
| highlight render enabled | `false` | `false` | Render scan is fail-closed by default. |
| highlight approval required | `true` | `true` | Requires the configured flag when rendering is explicitly enabled; the bare flag is not production authorization. |
| visual provider | `heuristic` | `local-cv` | Packaged YAML selects the local HTTP adapter. |
| visual heuristic fallback | `true` | `true` | Quality may degrade without failing. Not production-ready. |
| local CV auto-start | `false` | `true` | Launches bootstrap script, which may install/download. Prohibited at runtime. |
| local asset auto-start | `false` | `true` | Runs resident-runtime verification at application startup; it no longer invokes the bootstrap script. |
| local asset strict runtime | `false` | `true` | Throws at startup when required local binaries, Python imports, model paths, or adjacent model-license sidecars are missing. |
| local director auto-render | `false` | `false` | Safe in base config. |
| local director approval required | `true` | `true` | Approval applies only if auto-render is enabled. |
The `cinematic-editing-local` profile sets local-director auto-render to `true` while requiring `approved.flag`. Treat the profile as a local workflow convenience, not a production profile. The `folder-scheduler-local` profile enables folder segmentation. Never activate either without checking the input directories.
## Complete application property catalog
Notation: `J` is the Java default; `P` is the packaged effective default when it differs. `@Min` is the only bean-validation constraint currently declared. Values without `@Min` generally lack startup validation.
### API, adapters, S3, queue, and processing
| Property | J / P default | Meaning, values, guard |
|---|---|---|
| `video-clipping.max-file-size-bytes` | `53687091200` | Max declared source size; `@Min(1)`. |
| `video-clipping.multipart-part-size-bytes` | `104857600` | Multipart chunk size; `@Min(5242880)`. |
| `video-clipping.upload-url-ttl-minutes` | `60` | Upload URL lifetime; `@Min(1)`. |
| `video-clipping.download-url-ttl-minutes` | `15` | Download URL lifetime; `@Min(1)`. |
| `video-clipping.allowed-content-types` | `video/mp4`, `video/quicktime`, `video/webm` | Boundary allow-list; no non-empty validation. |
| `video-clipping.repository` | no Java field / `memory` | Conditional selector only: `memory` (also match-if-missing) or `jpa`. It is present in packaged properties but absent from `VideoClippingProperties`. |
| `video-clipping.storage` | `memory` | `memory` (also match-if-missing) or `s3`; other values leave no storage adapter. |
| `video-clipping.queue` | `memory` | `memory` (also match-if-missing) or `db`; `db` enables polling and queue metrics. |
| `video-clipping.processing` | `stub` | `stub` (also match-if-missing) or `ffmpeg`; `stub` is not production output. |
| `video-clipping.s3.region` | `us-east-1` | S3 client region. |
| `video-clipping.s3.bucket` | `video-clipping-dev` | Object bucket; development name is not a production default. |
| `video-clipping.s3.endpoint` | unset | Optional endpoint override. |
| `video-clipping.s3.path-style` | `true` | Path-style addressing. |
| `video-clipping.database-queue.poll-interval-ms` | `1000` | Database queue scheduled delay; `@Min(100)`. |
| `video-clipping.database-queue.batch-size` | `5` | Max messages claimed per poll; `@Min(1)`. |
| `video-clipping.database-queue.visibility-timeout-ms` | `900000` | Expired-claim threshold; `@Min(1000)`. |
| `video-clipping.database-queue.max-attempts` | `3` | Dead-letter threshold; `@Min(1)`. |
| `video-clipping.database-queue.retry-backoff-ms` | `5000` | Retry delay; `@Min(0)`. |
| `video-clipping.quotas.max-active-jobs-per-tenant` | `5` | Active-job admission cap; `0` is permitted and rejects all new active work; `@Min(0)`. |
`application-jpa.properties` selects `repository=jpa`, `queue=db`, and `processing=stub`, then configures an H2 database in PostgreSQL compatibility mode with Flyway and Hibernate validation. It is not a PostgreSQL production profile.
Packaged Spring/platform settings are also configuration axes:
| Property | Packaged value | Status |
|---|---|---|
| `spring.application.name` | `video-clipping-service` | Service identity. |
| `spring.mvc.problemdetails.enabled` | `true` | Enables Spring Problem Details handling. |
| `management.endpoints.web.exposure.include` | `health,info,metrics,prometheus` | Exposure is not the same as authorization; production must secure management endpoints. |
| `spring.jpa.open-in-view` | `false` | Deliberate persistence boundary. |
| `logging.level.org.example.videoclips.folder` | `INFO` | Base YAML; explicit env `FOLDER_SCHEDULER_LOG_LEVEL`. |
| `spring.datasource.url/driver-class-name/username/password` | H2 memory URL / H2 driver / `sa` / empty | `application-jpa.properties` only; development/test convenience. |
| `spring.jpa.hibernate.ddl-auto` / `spring.jpa.show-sql` | `validate` / `false` | JPA profile schema validation and SQL logging. |
| `spring.flyway.enabled` | `true` | JPA profile migration control. |
Classify selectors and modes conservatively:
| Classification | Current values |
|---|---|
| Development/test only | `repository=memory`, `storage=memory`, `queue=memory`, `processing=stub`, H2 JPA profile, heuristic analysis, all local convenience profiles. |
| Experimental/unsafe for promoted output | heuristic fallback, network-capable local-CV auto-bootstrap, named/defaulted YOLO analysis, non-strict asset readiness, and any explicitly enabled render path whose approval is not authenticated and digest-bound. Historical Python/Java placeholder-audio fallbacks are removed in the 2026-07-21 working tree. |
| Production candidates requiring evidence | `repository=jpa` with real PostgreSQL, `storage=s3`, `queue=db`, `processing=ffmpeg`, pre-provisioned local CV/assets/director. Candidate means not certified by this repository. |
### API FFmpeg workspace, cleanup, and retention
| Property | J / P default | Meaning and validation |
|---|---|---|
| `video-clipping.ffmpeg.ffmpeg-binary` | `ffmpeg` | API clipper executable. |
| `video-clipping.ffmpeg.input-directory` | `./tmp/ffmpeg-input` | Materialized source workspace. |
| `video-clipping.ffmpeg.output-directory` | `./tmp/ffmpeg-output` | Generated clip workspace. |
| `video-clipping.ffmpeg.exact-preset` | `veryfast` | FFmpeg encoding preset used by the API clipper; absent from packaged config, so the Java default applies. |
| `video-clipping.ffmpeg.cleanup-local-files` | `true` | Per-job worker cleanup control. |
| `video-clipping.cleanup.enabled` | `true` | Runtime check inside both scheduled cleanup jobs; jobs still wake when false. |
| `video-clipping.cleanup.local-artifact-poll-interval-ms` | `300000` | Local cleanup delay; `@Min(30000)`. |
| `video-clipping.cleanup.local-artifact-retention-hours` | `24` | Age for FFmpeg and fixed temporary roots; `@Min(1)`. |
| `video-clipping.cleanup.retention-poll-interval-ms` | `3600000` | storage/repository cleanup delay; `@Min(30000)`. |
| `video-clipping.cleanup.source-retention-hours` | `168` | Expiry assigned to new source assets; `@Min(1)`. |
| `video-clipping.cleanup.clip-retention-hours` | `168` | Expiry assigned to clips; `@Min(1)`. |
Local artifact cleanup also targets fixed `tmp/in-memory-storage` and `tmp/stub-output`; those roots are not configurable. Relative paths resolve from the process working directory. Validate mount ownership, capacity, backups, retention, and path separation on macOS, Linux/VPS, and cloud runtimes.
### Folder segmentation scheduler
| Property | J / P default | Meaning and validation |
|---|---|---|
| `video-clipping.folder-scheduler.enabled` | J `false`; P `true` | Creates initializer, validator, clipper, and poller. |
| `.input-directory` | J `/input/source`; P `./input/source` | Inbox; scheduler moves candidates. Env: `FOLDER_SCHEDULER_INPUT_DIRECTORY`. |
| `.output-directory` | J `/output/clips`; P `./output/clips` | Segmented output. Env: `FOLDER_SCHEDULER_OUTPUT_DIRECTORY`. |
| `.processed-directory` | J `/input/processed`; P `./input/processed` | Successful source destination. Env: `FOLDER_SCHEDULER_PROCESSED_DIRECTORY`. |
| `.rejected-directory` | J `/input/rejected`; P `./input/rejected` | Failed source destination. Env: `FOLDER_SCHEDULER_REJECTED_DIRECTORY`. |
| `.working-directory` | J `/input/working`; P `./input/working` | Claimed source directory. Env: `FOLDER_SCHEDULER_WORKING_DIRECTORY`. |
| `.poll-interval-ms` | `5000` | Initial/fixed delay; `@Min(1000)`. Env: `FOLDER_SCHEDULER_POLL_INTERVAL_MS`. |
| `.segment-duration-seconds` | `8` | Segment target; `@Min(1)`; no explicit env placeholder. |
| `.ffmpeg-binary` / `.ffprobe-binary` | `ffmpeg` / `ffprobe` | Executables; no explicit env placeholders. |
| `.output-container` / `.exact-preset` | `mp4` / `veryfast` | FFmpeg output choices; no enum/allow-list validation. |
| `.preserve-input-quality` | `true` | Selects preservation behavior. Env: `FOLDER_SCHEDULER_PRESERVE_INPUT_QUALITY`. |
`FOLDER_SCHEDULER_ENABLED` controls the packaged guard; `FOLDER_SCHEDULER_LOG_LEVEL` controls only `org.example.videoclips.folder` logging.
### Editing, render, analysis, and assets
All keys below start with `video-clipping.editing`.
| Suffix | J / P default | Meaning / explicit environment variable |
|---|---|---|
| `.enabled` | `true` | Master component guard; `VIDEO_EDITING_ENABLED`. |
| `.project-directory` | `./output/edit-projects` | Multi-clip projects; `VIDEO_EDITING_PROJECT_DIRECTORY`. |
| `.highlight-project-directory` | `./output/highlight-projects` | Single-source projects; `VIDEO_EDITING_HIGHLIGHT_PROJECT_DIRECTORY`. |
| `.ffmpeg-binary` / `.ffprobe-binary` | `ffmpeg` / `ffprobe` | Editing executables; `VIDEO_EDITING_FFMPEG_BINARY`, `VIDEO_EDITING_FFPROBE_BINARY`. |
| `.thumbnail-count-per-clip` / `.contact-sheet-columns` | `5` / `5` | Inspection density; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
| `.proxy-enabled` / `.proxy-width` | `true` / `640` | Proxy generation and width; width `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
| `.scene-detection-threshold` | `0.35` | FFmpeg scene threshold; no range validation; `VIDEO_EDITING_SCENE_DETECTION_THRESHOLD`. |
| `.minimum-scene-duration-seconds` | `1.0` | Segment merge threshold; no validation; `VIDEO_EDITING_MINIMUM_SCENE_DURATION_SECONDS`. |
| `.silence-threshold-db` / `.silence-minimum-duration-seconds` | `-35.0` / `0.5` | Source audio analysis; no validation; matching `VIDEO_EDITING_*` variable. |
| `.target-duration-seconds` | `600` | Multi-clip target; `@Min(1)`; `VIDEO_EDITING_TARGET_DURATION_SECONDS`. |
| `.output-width` / `.output-height` / `.output-frame-rate` | `1920` / `1080` / `30` | Render geometry/rate; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
| `.audio-sample-rate` | `48000` | Render audio rate; `@Min(1)`; `VIDEO_EDITING_AUDIO_SAMPLE_RATE`. |
| `.video-bitrate` / `.audio-bitrate` | `12000k` / `192k` | FFmpeg bitrate strings; no format validation; matching `VIDEO_EDITING_*` variable. |
| `.voiceover-provider` | `local` | Bound and tested but not consumed by main Java code; it is not a working remote-provider selector. Env: `VIDEO_EDITING_VOICEOVER_PROVIDER`. External AI is prohibited. |
| `.loudness-target-i` / `.loudness-true-peak` / `.loudness-range` | `-16.0` / `-1.5` / `11.0` | `loudnorm` parameters; no range validation; matching `VIDEO_EDITING_*` variable. |
| `.music-ducking-threshold` / `.music-ducking-ratio` | `0.045` / `8.0` | `sidechaincompress` parameters; no range validation; matching `VIDEO_EDITING_*` variable. |
| `.music-ducking-attack-ms` / `.music-ducking-release-ms` | `20` / `250` | Ducking timing; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
| `.assets.music-folder` | `./input/highlights/assets/music` | Intended location for approved, licensed local music; directory membership is not license proof. `VIDEO_EDITING_ASSETS_MUSIC_FOLDER`. |
| `.assets.sfx-folder` | `./input/highlights/assets/sfx` | Intended location for approved, licensed local SFX; directory membership is not license proof. `VIDEO_EDITING_ASSETS_SFX_FOLDER`. |
| `.assets.fonts-folder` | `./input/highlights/assets/fonts` | Intended location for approved, licensed local fonts; directory membership is not license proof. `VIDEO_EDITING_ASSETS_FONTS_FOLDER`. |
| `.assets.luts-folder` | `./input/highlights/assets/luts` | Intended location for approved, licensed local LUTs; directory membership is not license proof. `VIDEO_EDITING_ASSETS_LUTS_FOLDER`. |
| `.assets.voiceover-folder` | `./output/highlight-projects/_voiceover-cache` | Generated voiceover cache; `VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER`. |
Treat render, loudness, ducking, scene, and duration changes as output behavior changes. Require objective measurements and representative media evidence; configuration binding tests alone are insufficient.
Exact environment names for grouped editing rows:
| Property suffixes | Environment variables, in the same order |
|---|---|
| `.thumbnail-count-per-clip`, `.contact-sheet-columns` | `VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP`, `VIDEO_EDITING_CONTACT_SHEET_COLUMNS` |
| `.proxy-enabled`, `.proxy-width` | `VIDEO_EDITING_PROXY_ENABLED`, `VIDEO_EDITING_PROXY_WIDTH` |
| `.silence-threshold-db`, `.silence-minimum-duration-seconds` | `VIDEO_EDITING_SILENCE_THRESHOLD_DB`, `VIDEO_EDITING_SILENCE_MINIMUM_DURATION_SECONDS` |
| `.output-width`, `.output-height`, `.output-frame-rate` | `VIDEO_EDITING_OUTPUT_WIDTH`, `VIDEO_EDITING_OUTPUT_HEIGHT`, `VIDEO_EDITING_OUTPUT_FRAME_RATE` |
| `.video-bitrate`, `.audio-bitrate` | `VIDEO_EDITING_VIDEO_BITRATE`, `VIDEO_EDITING_AUDIO_BITRATE` |
| `.loudness-target-i`, `.loudness-true-peak`, `.loudness-range` | `VIDEO_EDITING_LOUDNESS_TARGET_I`, `VIDEO_EDITING_LOUDNESS_TRUE_PEAK`, `VIDEO_EDITING_LOUDNESS_RANGE` |
| `.music-ducking-threshold`, `.music-ducking-ratio` | `VIDEO_EDITING_MUSIC_DUCKING_THRESHOLD`, `VIDEO_EDITING_MUSIC_DUCKING_RATIO` |
| `.music-ducking-attack-ms`, `.music-ducking-release-ms` | `VIDEO_EDITING_MUSIC_DUCKING_ATTACK_MS`, `VIDEO_EDITING_MUSIC_DUCKING_RELEASE_MS` |
### Visual analysis and local CV worker
All keys start with `video-clipping.editing.visual-analysis`.
| Suffix | J / P default | Meaning / explicit environment variable |
|---|---|---|
| `.provider` | J `heuristic`; P `local-cv` | Recognized routing is `local-cv`; any other value uses heuristic. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER`. |
| `.endpoint` | `http://127.0.0.1:8091/v1/analyze-visuals` | Current local HTTP endpoint. Loopback is network, so this is a noncompliant implementation gap, not an approved certified path. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT`. |
| `.timeout-ms` | `30000` | Total HTTP call timeout; `@Min(1)`. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS`. |
| `.fallback-to-heuristic` | `true` | On local-CV failure, return weaker heuristic analysis. Must be false in the fail-closed production target. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC`. |
| `.local-cv-worker.auto-start` | J `false`; P `true` | Starts `script` only when provider is `local-cv`. Runtime auto-start/bootstrap is prohibited. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START`. |
| `.local-cv-worker.script` | `./tools/run_local_cv_worker.sh` | Launched executable. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT`. |
| `.local-cv-worker.startup-wait-ms` | `0` | `0` skips health waiting; otherwise timeout; `@Min(0)`. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS`. |
| `.local-cv-worker.health-path` | `/health` | Health URI path. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH`. |
| `.local-cv-worker.health-check-interval-ms` | `1000` | Readiness poll and per-call cap; `@Min(1)`. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS`. |
Worker-only environment variables are not Spring properties:
| Variable | Default/source | Effect and policy |
|---|---|---|
| `LOCAL_CV_HOST`, `LOCAL_CV_PORT` | `127.0.0.1`, `8091`; manager derives both from endpoint | Uvicorn bind address/port. Keep local to the service trust boundary. |
| `LOCAL_CV_BOOTSTRAP_MODE` | `auto` | `always` or first-use `auto` invokes pip. Runtime must use a pre-provisioned path/mode that cannot install. |
| `LOCAL_CV_PRELOAD_ONLY` | `false` | Exit after bootstrap/model preload. Preload can still download. |
| `LOCAL_CV_YOLO_MODEL` | `yolov8n.pt` | Ultralytics model path/name. A name can trigger model retrieval; production must supply a verified local path. |
| `LOCAL_CV_DISABLE_YOLO` | `false` | Skips YOLO and returns weaker/default labels. Experimental/debug only, never production-quality success. |
| `LOCAL_CV_LOG_LEVEL` | `INFO` | Python worker logging level. |
The worker health response reports library availability but returns HTTP 200 even if OpenCV or YOLO is absent. Do not equate `/health` success with production analysis readiness.
### Local asset runtime
All keys start with `video-clipping.editing.local-asset-worker`.
| Suffix | J / P default | Meaning / explicit environment variable |
|---|---|---|
| `.auto-start` | J `false`; P `true` | Runs bootstrap verification, not a persistent asset server. Must be false at production runtime under the no-download rule. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START`. |
| `.strict-runtime` | J `false`; P `true` | Makes startup fail when the pre-provisioned local asset runtime is incomplete. It requires adjacent nonblank model-license sidecars but does not verify authenticity, origin, checksums, or allowed use. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_STRICT_RUNTIME`. |
| `.bootstrap-script` | `./tools/run_local_asset_worker.sh` | Startup verifier command; `VIDEO_EDITING_LOCAL_ASSET_BOOTSTRAP_SCRIPT`. |
| `.script` | `./tools/local_asset_worker.py` | Per-asset CLI worker; `VIDEO_EDITING_LOCAL_ASSET_WORKER_SCRIPT`. |
| `.startup-wait-ms` | `0` | Bound and tested but not consumed by current asset verifier/synthesizer; `@Min(0)`. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_STARTUP_WAIT_MS`. |
| `.health-path` | `/health` | Bound and tested but no local asset HTTP server consumes it. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_PATH`. |
| `.health-check-interval-ms` | `1000` | Bound and tested but unused; `@Min(1)`. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_CHECK_INTERVAL_MS`. |
| `.python-binary` | `./.venv-local-asset/bin/python` | Python used for import readiness and worker invocation; `VIDEO_EDITING_LOCAL_ASSET_PYTHON_BINARY`. |
| `.piper-binary` | `piper` | Voice model executable; `VIDEO_EDITING_LOCAL_ASSET_PIPER_BINARY`. |
| `.piper-model-path` | empty | Required in practice for Piper; `VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH`. |
| `.music-model` | `musicgen-small` | AudioCraft model name; `VIDEO_EDITING_LOCAL_ASSET_MUSIC_MODEL`. A short name is normalized to `facebook/<name>` and may download. |
| `.sfx-model` | `audiogen-medium` | AudioCraft model name; `VIDEO_EDITING_LOCAL_ASSET_SFX_MODEL`. A short name may download. |
Worker-only environment variables:
| Variable | Current behavior |
|---|---|
| `LOCAL_ASSET_BOOTSTRAP_MODE` | Script default `auto`; `auto`/`always` invokes pip. The Java verifier forcibly sets `auto`, overriding an inherited safer value. This blocks compliant runtime auto-start today. |
| `LOCAL_ASSET_PRELOAD_ONLY` | Java verifier sets `true`; script exits after dependency bootstrap. |
| `LOCAL_ASSET_PIPER_BINARY` | Java verifier passes configured binary; Python CLI also reads it by default. |
| `LOCAL_ASSET_PIPER_MODEL_PATH` | Java verifier exports it when nonblank; generation passes the configured model explicitly. |
| `LOCAL_ASSET_HOST`, `LOCAL_ASSET_PORT` | Script reads defaults `127.0.0.1:8092` but never uses them; there is no HTTP asset server. |
The Python and Java fallback success paths were removed in the 2026-07-21 working tree. Continue rejecting any generated asset whose production provenance, local model identity, checksum, license, and semantic fit are not verified; an audible WAV alone is insufficient.
### Local director and highlight scheduler
| Property suffix | Default | Meaning / explicit environment variable |
|---|---|---|
| `.local-director.enabled` | `true` | With editing enabled, polls multi-clip inbox; `VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED`. |
| `.local-director.source/working/processed/rejected-directory` | `./input/editing/<state>` | Move-based lifecycle; matching `VIDEO_EDITING_LOCAL_DIRECTOR_*_DIRECTORY`. |
| `.local-director.poll-interval-ms` | `5000` | Initial/fixed delay; `@Min(1000)`; `VIDEO_EDITING_LOCAL_DIRECTOR_POLL_INTERVAL_MS`. |
| `.local-director.director-prompt-file-name` | `ai-director-prompt.md` | Prompt artifact; `VIDEO_EDITING_LOCAL_DIRECTOR_PROMPT_FILE_NAME`. |
| `.local-director.expected-plan-file-name` | `edit-plan.json` | Plan inbox filename; `VIDEO_EDITING_LOCAL_DIRECTOR_EXPECTED_PLAN_FILE_NAME`. |
| `.local-director.auto-render-when-plan-appears` | `false` | Enables render after plan scan; `VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER`. Keep false unless explicitly approved. |
| `.local-director.require-approval-before-render` | `true` | Requires approval file when auto-rendering; `VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL`. |
| `.local-director.approval-file-name` | `approved.flag` | Approval artifact name; `VIDEO_EDITING_LOCAL_DIRECTOR_APPROVAL_FILE_NAME`. |
| `.highlight-scheduler.enabled` | `true` | With editing enabled, enables both source and plan/render scanners; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED`. |
| `.highlight-scheduler.source/working/processed/rejected-directory` | `./input/highlights/<state>` | Move-based source lifecycle; matching `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_*_DIRECTORY`. |
| `.highlight-scheduler.poll-interval-ms` | `5000` | Shared by source and render scanners; `@Min(1000)`; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS`. |
| `.highlight-scheduler.render-enabled` | `false` | Runtime render scan gate; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED`. Keep false until the approval workflow is proven. |
| `.highlight-scheduler.require-director-approval` | `true` | Requires director approval file before flow; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL`. The bare file is not digest-bound authorization. |
| `.highlight-scheduler.approval-file-name` | `approved.flag` | Approval artifact under director directory; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME`. |
| `.highlight-scheduler.max-highlights-per-source` | `3` | Plan item cap; `@Min(1)`; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE`. |
| `.highlight-scheduler.highlight-min-duration-seconds` | `8.0` | Lower duration clamp; `@Min(1)`; matching environment variable. |
| `.highlight-scheduler.highlight-max-duration-seconds` | `35.0` | Upper duration clamp; `@Min(1)`; matching environment variable. No validation ensures max >= min. |
Exact environment names for grouped scheduler rows:
| Property suffixes | Environment variables, in the same order |
|---|---|
| `.local-director.source/working/processed/rejected-directory` | `VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_WORKING_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_PROCESSED_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_REJECTED_DIRECTORY` |
| `.highlight-scheduler.source/working/processed/rejected-directory` | `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_SOURCE_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_WORKING_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY` |
| `.highlight-scheduler.highlight-min/max-duration-seconds` | `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS` |
## Guards and coupled settings
| Behavior | Required condition | Additional gate |
|---|---|---|
| Folder segmentation | `folder-scheduler.enabled=true` | None beyond input eligibility; it moves files. |
| Editing API/core | `editing.enabled=true` or missing | Many editing beans use `matchIfMissing=true`. |
| Multi-clip director polling | editing and local-director enabled | Directory candidate rules. |
| Multi-clip auto-render | above plus `auto-render-when-plan-appears=true` | Approval file only when `require-approval-before-render=true`. |
| Highlight ingestion | editing and highlight scheduler enabled | Moves one eligible video at a time. |
| Highlight plan rendering | same | `render-enabled=true`; approval only when `require-director-approval=true`. |
| Local CV process | editing enabled | provider exactly `local-cv` and worker auto-start true. |
| Local CV fallback | provider `local-cv` fails | Returns heuristic only when fallback is true; otherwise throws. |
| Local asset readiness check | editing enabled | Local asset auto-start true; strict mode fails startup when readiness is incomplete. The bootstrap script is not invoked by this verifier. |
| Cleanup | scheduled unconditionally | Work skipped when cleanup enabled is false. |
| DB queue poller | `queue=db` | Requires JPA repository/infrastructure coherence. |
Before changing one setting, inspect all settings in its row. In particular, enabling an approval boolean without disabling render during rollout leaves a race window unless deployment/configuration is atomic.
## Profiles and tests
| Source | Overrides | Classification |
|---|---|---|
| `application.properties` | API defaults, adapters, S3, DB queue, FFmpeg workspace, quotas, cleanup, actuator exposure | Packaged base; currently development-oriented. |
| `application.yml` | Folder scheduler and complete editing tree | Packaged base; current all-on behavior is unsafe. |
| `application-jpa.properties` | JPA/db queue/stub plus H2, Flyway, Hibernate validation | Local/integration convenience, not production PostgreSQL. |
| `application-folder-scheduler-local.yml` | Enables local folder scheduler paths | Local only. |
| `application-cinematic-editing-local.yml` | Disables folder scheduler; enables editing/director auto-render with approval | Local only; rendering side effects. |
| `src/test/resources/application.properties` | Disables folder scheduler, director, highlight scheduler, CV auto-start, asset auto-start; sets asset strict false | Test safety override. |
| `src/test/resources/application-test.properties` | Disables folder scheduler only | Narrow test-profile override; do not assume it disables other consumers. |
`VideoClippingPropertiesTest` verifies most editing Java defaults and representative binding overrides. It does not exhaustively assert every top-level, folder, cleanup, queue, or highlight setting. Profile tests parse YAML values but do not prove safe runtime composition.
## Production target, not current implementation
For macOS development, Linux/VPS production, and cloud infrastructure, converge through change control on this behavior:
- Ship one immutable, platform-neutral application artifact, then create signed platform-specific runtime/model/image bundles. Promote the same bundle unchanged between environments for the same OS/CPU target; inject paths and secrets externally.
- Disable every scheduler, worker auto-start, and render path by default. Enable only the deployed capability.
- Run local model workers as pre-provisioned supervised processes or explicit adapters with no installation/download code in the request/startup path.
- Replace model-worker HTTP with in-process calls or explicitly designed non-network IPC. Deny model/media inference networking, including loopback. Treat API, database, storage, and telemetry as separately approved production integrations.
- Require verified local model paths, checksums, licenses, writable/output mounts, capacity limits, and health that validates model readiness.
- Make missing FFmpeg, FFprobe, Python stack, Piper model, AudioCraft model, CV model, or licensed asset fail startup/readiness for any enabled dependent capability.
- Disable heuristic, silence, tone, macOS `say`, and `espeak` substitutions for production results.
- Require approval for every render path and record the approver/artifact; a bare filesystem flag is a current mechanism, not sufficient Fortune 500 authorization evidence.
- Use distinct durable/shared project storage or enforce single-instance scheduling. Current filesystem queues/projects and move-based inboxes are not automatically horizontally scalable.
Do not silently implement this target by editing defaults. Record each gap as a controlled change with migration and rollback.
## Add or change a property
Complete every checkbox:
- [ ] State the requirement, owner, consumers, supported values/units, safe default, production classification, expiry/revisit condition, and whether it changes output or side effects.
- [ ] Search for an existing property before adding one; avoid aliases and speculative flags.
- [ ] Add the typed field under `VideoClippingProperties`; use an immutable value type where practical and add Jakarta validation for every meaningful range, nonblank path, enum, URI, or cross-field invariant.
- [ ] Decide whether omission must fail. Never give production-critical model, license, secret, endpoint, or approval settings an insecure fallback.
- [ ] Add the packaged property exactly once. If operations need an explicit environment name, use a documented placeholder and preserve Spring relaxed-binding consistency.
- [ ] Classify the default change through `video-editing-change-control`. Treat enabling a scheduler/worker/render, changing an adapter, endpoint, path, retention, quality threshold, model, codec, or fallback as production-facing.
- [ ] Map environment impact for macOS, Linux/VPS, and cloud: paths/mounts, permissions, filesystem persistence, process supervision, network policy, resources, secrets, rollout ordering, backward compatibility, and rollback value.
- [ ] Check coupled guards and all profiles. Make production fail closed; keep runtime downloads, external AI, network fetching, unlicensed assets, placeholders, and unapproved renders impossible.
- [ ] Add default, valid-override, invalid-value, and application-context guard tests. Add a cross-field test for min/max or mutually dependent settings.
- [ ] Add behavior tests proving the consumer uses the property. For output changes, add objective QA evidence; binding alone is not proof.
- [ ] Update this catalog and the configuration reference/documents of record. Explain every deliberate Spring Boot default deviation.
- [ ] Run the focused tests, full clean build prescribed by `video-editing-validation-and-qa`, configuration drift scans, and `git diff --check`.
- [ ] Provide rollout order, observable signals, rollback command/value, and removal date for temporary flags. Do not leave permanent dual paths without ownership.
## Review a proposed environment
Reject the configuration if any answer is “no”:
- [ ] Are only the intended scheduler and adapter beans enabled?
- [ ] Are model/dependency downloads impossible after image/build promotion?
- [ ] Do all model identifiers resolve to verified local artifacts rather than remote names?
- [ ] Are external AI and arbitrary remote worker endpoints impossible?
- [ ] Are all input, working, processed, rejected, project, cache, and temporary paths distinct, mounted, permissioned, capacity-limited, and retained intentionally?
- [ ] Does any missing model or asset fail closed without heuristic, silence, tone, `say`, or `espeak` substitution?
- [ ] Is every render disabled until authenticated approval is present?
- [ ] Are queue/repository/storage selectors coherent and production adapters, not `memory`/`stub`/H2 conveniences?
- [ ] Are cleanup and retention consistent with recovery, audit, and legal requirements?
- [ ] Are local worker health checks capability-aware, and are timeout/resource limits proven?
- [ ] Is the same application artifact and same signed runtime bundle used without rebuild across environments for each platform target, with auditable external configuration?
## Provenance and maintenance
Primary sources: `VideoClippingProperties`, packaged/profile/test configuration, all `@ConditionalOnProperty`/`@ConditionalOnExpression` guards, scheduler annotations, local worker managers, shell launchers, Python workers, and configuration/profile tests. Re-verify volatile facts after any configuration, worker, or dependency change.
```bash
# List declared Java defaults and validation annotations.
rg -n '^ public static class|^[[:space:]]+@(Min|Max|NotBlank|NotNull|Pattern)|^[[:space:]]+private ' src/main/java/org/example/videoclips/config/VideoClippingProperties.java
# Regenerate the packaged/profile/test key inventory.
rg -n '^[[:space:]]*[A-Za-z0-9_.-]+[=:]' src/main/resources src/test/resources
# Find configuration consumers and guards.
rg -n '@ConditionalOn(Property|Expression)|@Scheduled|video-clipping\.' src/main/java src/test/java
# Regenerate explicit environment mappings and worker-only variables.
rg -n '\$\{[A-Z][A-Z0-9_]*:|LOCAL_(ASSET|CV)_[A-Z0-9_]+' src/main/resources tools src/main/java
# Recheck fallback and download hazards without executing them.
rg -n 'pip install|get_pretrained|YOLO\(|fallback|write_silence|write_fallback_tone|strictRuntime|strict-runtime' tools src/main/java src/main/resources
# Run focused binding/profile tests only after confirming test resources disable side effects.
mvn -o -Dtest=VideoClippingPropertiesTest,CinematicEditingLocalProfileTest,FolderSchedulerLocalProfileTest,FolderSchedulerSpringContextTest test
# Check documentation formatting and the repository diff without mutation.
wc -l .claude/skills/video-editing-config-and-flags/SKILL.md
git diff --check -- .claude/skills/video-editing-config-and-flags
```

View File

@ -0,0 +1,330 @@
---
name: video-editing-debugging-playbook
description: "Load when a concrete video-editing symptom needs symptom-to-branch triage: startup/readiness failure, missing or stalled highlight output, inconsistent project state, missing category/candidate artifact, rejected director plan, FFmpeg/audio/codec failure, scheduler quarantine, API queue/DLQ/storage/JPA incident, profile drift, or clean-checkout failure. Use it to choose discriminating read-only experiments; move to the proof toolkit only after triage identifies competing causal mechanisms."
---
# Debug the Video Editing Service
Use this runbook from the repository root. Treat code, tests, runtime artifacts, and structured logs as evidence; do not judge media quality by eye alone.
Facts marked **current** were verified against `main` on 2026-07-21. **Target** means the requested Fortune 500 production architecture, not implemented behavior.
## Safety boundary
The no-waiver policy lives in `video-editing-change-control`. During triage, make no dependency/model download, external-AI call, model/media network call (including loopback), unlicensed-asset use, placeholder acceptance, unapproved render, or production-default change. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh`: both can install packages automatically, and the CV script can fetch a YOLO model.
Keep the first pass read-only. A local FFprobe or decode-to-null command measures an existing file; it does not render or publish a replacement.
Route every proposed code, dependency, configuration, asset, migration, redrive, or render change through `video-editing-change-control`. Use `video-editing-failure-archaeology` before revisiting a historical fix. Never mutate queue rows during triage.
## Know the terms
| Term | Meaning here |
|---|---|
| Highlight project | Single-source filesystem workflow rooted by default at `output/highlight-projects/<project-id>/`. |
| Edit project | Multi-clip filesystem workflow rooted by default at `output/edit-projects/<project-id>/`. It has a different plan scanner and validator. |
| Candidate | Ranked source time range proposed for a highlight. Its expected single-source file is `analysis/highlight-candidates.json`. |
| Director plan | Human/local-director-authored `director/edit-plan.json` consumed by the single-source renderer. |
| Strict runtime | `video-clipping.editing.local-asset-worker.strict-runtime`; when asset auto-start is enabled, incomplete resident model/runtime readiness throws during startup. |
| DLQ | Dead-letter queue; terminal database-queue state after the configured maximum attempts. |
| Marker | A failed folder input renamed in place to `<name>.failed` when movement to the rejected directory also fails. |
| Discriminating experiment | A check whose possible outcomes select different causes or next actions. |
## Start with a read-only evidence bundle
Set shell variables only to shorten commands; do not infer project IDs from display names.
```bash
project_id='<exact-project-directory-name>'
project="output/highlight-projects/$project_id"
logs='<path-to-captured-service-log>'
git status --short
git rev-parse --short HEAD
java -version
mvn -version
ffmpeg -version | sed -n '1,3p'
ffprobe -version | sed -n '1,3p'
find "$project" -maxdepth 4 -type f -print | sort
sed -n '1,220p' "$project/project.json"
rg 'event=(highlight|local_cv|local_asset|visual_analysis)' "$logs" | tail -n 250
```
If a command reports a missing path, preserve that as evidence. Do not create the path to make the check pass.
## Fast symptom map
| Symptom | First discriminator | If X, branch to | If not X, branch to |
|---|---|---|---|
| Startup hangs/fails | Search `local_cv_worker_*` and `local_asset_*` events | Bootstrap/readiness | Spring/profile/config |
| Source disappears; no final | Inventory source, working, processed, rejected, project | Scheduler claim/analysis | Wrong directory/profile |
| `CREATED` persists | Check `analysis/source-analysis.json` and failure logs | Partial analysis | Prompt/status persistence |
| `WAITING_FOR_DIRECTOR` persists | Check plan, approval, asset request files | Plan/approval/assets | Scanner disabled or wrong root |
| `RENDERING` persists | Check renderer logs and per-highlight `final.mp4` | FFmpeg/publish failure | Stale status/artifact mismatch |
| `FAILED` | Read `failureMessage` and correlate the same project | Empty highlight list | Unrecorded exception; do not invent cause |
| No category/candidates | Compare single-source artifacts with generator wiring | Known pipeline gap | Corrupt/mislocated JSON |
| Folder input is rejected/marked | Inspect validator and movement events | Invalid media/collision | FFmpeg failure |
| API jobs stall/fail | Confirm queue adapter and queue row states | DB queue/DLQ | Memory queue/process-local loss |
| Works locally, clean build fails | Run isolated offline archive build | Untracked runtime dependency | Toolchain/cache mismatch |
## Triage startup and local models
### Local CV worker
**Current:** `application.yml` selects `local-cv`, enables auto-start, sets `startup-wait-ms: 0`, and permits heuristic fallback. A zero wait logs `local_cv_worker_ready` without an HTTP health check. The launch script defaults `LOCAL_CV_BOOTSTRAP_MODE=auto`, installs packages when its virtualenv is incomplete, and loads `yolov8n.pt`, which may trigger a model fetch.
Run only local, non-starting checks:
```bash
test -x .venv-local-cv/bin/python && echo python-present || echo python-missing
.venv-local-cv/bin/python -c 'import fastapi,uvicorn,cv2,numpy,ultralytics; print("cv-imports-ready")'
test -f yolov8n.pt && stat -f '%N %z bytes' yolov8n.pt 2>/dev/null || stat -c '%n %s bytes' yolov8n.pt 2>/dev/null
rg 'local_cv_worker_(start|health|process)|local_cv_visual_analysis_(completed|fallback|failed)' "$logs"
```
- If imports or the model are missing, stop. The runtime image/environment is not preprovisioned; use `video-editing-build-and-env`. Do not invoke bootstrap.
- If `local_cv_worker_ready` appears with `local_cv_worker_health_wait_skipped`, readiness is unproved.
- If `local_cv_visual_analysis_fallback` appears, inspect `analysis/visual-analysis.json`. An `analysisMethod` beginning `local_cv_failed_fallback_` proves degraded heuristic output.
- If the worker exited unexpectedly, inspect preceding `local_cv_worker_output`; do not merely restart it.
- **Target:** immutable, licensed, checksummed model artifacts inside the approved runtime; startup must prove readiness without downloads.
### Local asset runtime
**Current working tree:** application startup no longer invokes asset bootstrap. Strict readiness requires existing Piper, MusicGen, and AudioGen paths and throws when incomplete. Java/Python generation returns failure and removes invalid output instead of using host speech, silence, or tones. The standalone launcher still runs `pip install` in `auto` mode; never use it for triage or certified operation.
```bash
test -x .venv-local-asset/bin/python && echo python-present || echo python-missing
.venv-local-asset/bin/python -c 'import torch,audiocraft,soundfile,numpy; print("audio-imports-ready")'
test -n "${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:-}" && test -f "$VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH" && echo piper-model-present || echo piper-model-missing
command -v piper || true
rg 'local_asset_(bootstrap|runtime|python_stack|worker|generation|voiceover)' "$logs"
```
- If `local_asset_runtime_check_failed` appears, do not render or promote. In strict mode the application must fail startup; in non-strict mode it logs and continues.
- The current asset verifier does not invoke its bootstrap script. Treat any bootstrap log as a historical revision or a separately invoked prohibited shell launcher.
- Successful Python imports do not prove model weights are present. Require the approved runtime inventory/checksum evidence; do not discover availability by invoking `get_pretrained`.
- If logs say `local_asset_worker_completed`, verify the waveform; exit zero is not proof of a model-generated asset.
- **Target:** missing or invalid required local models fail startup; every produced asset carries model/provenance evidence; placeholders are impossible.
## Detect prohibited silence and tones
Apply to every generated voiceover, music, and SFX file. First correlate generation events; legacy `strategy=silence`, `strategy=fallback-tone`, `model=silence`, or `model=ffmpeg-sine` in an older artifact is an immediate failure. The current worker must fail instead of emitting those strategies.
```bash
audio='<generated-audio-file>'
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels,duration -of default=noprint_wrappers=1 "$audio"
ffmpeg -hide_banner -nostats -i "$audio" -af 'astats=metadata=1:reset=0' -f null - 2>&1 | rg 'Peak level dB|RMS level dB|Zero crossings rate'
ffmpeg -hide_banner -nostats -i "$audio" -af 'silencedetect=noise=-50dB:d=0.5' -f null - 2>&1 | rg 'silence_(start|end|duration)'
```
- If peak/RMS is `-inf`, reject as silence.
- If generation failed and a narrow periodic signal appears, treat it as suspected fallback tone; confirm against the log strategy or compare its dominant behavior with the documented 110 Hz music / 880 Hz SFX implementation. Do not approve by listening.
- If speech is audible but came from macOS `say` or `espeak`, it is a fallback, not proof that the configured Piper model ran.
- Route deterministic asset certification to `video-editing-validation-and-qa`.
**Costly trap:** commit `97ba827` added strict runtime after placeholder-producing paths already existed. The 2026-07-21 working tree now fails strict startup and removes placeholder success, but file existence, a license sidecar, or exit code zero still does not prove production audio quality, provenance authenticity, or allowed use.
## Diagnose the single-source highlight flow
### Establish the artifact frontier
```bash
for f in project.json analysis/source-analysis.json analysis/category.json analysis/highlight-candidates.json director/director-prompt.md director/edit-plan.json final.mp4 render-manifest.json; do
test -f "$project/$f" && printf 'present %s\n' "$f" || printf 'missing %s\n' "$f"
done
find "$project/highlights" -maxdepth 3 -type f -print 2>/dev/null | sort
```
Interpret the last present artifact as the frontier, then inspect the first missing producer. Do not use downstream absence as a root cause.
### State-specific branches
| State | Current meaning | Discriminating branch |
|---|---|---|
| `CREATED` | Project record was written. Source analysis may not have completed. | If `analysis/source-analysis.json` is missing and source moved to rejected, inspect `highlight_analysis_step_failed`. If analysis exists but state remains `CREATED`, prompt generation/status update did not complete. |
| `ANALYZING` | Enum exists, but the single-source scheduler does not currently write it. | Treat it as external/stale data unless code history proves otherwise. |
| `WAITING_FOR_DIRECTOR` | Prompt generation updated status; no validated plan has entered the flow. | If a plan exists, inspect approval and validator rejection in `failureMessage`/logs. |
| `PLANNED` | The director plan passed `HighlightDirectorPlanValidator`; required assets may still be pending. | Inspect asset request JSON, immutable local model inventory, and worker failure logs. Do not send it back to director unless the plan itself changes. |
| `RENDERING` | Flow set status immediately before renderer invocation. | Missing final plus FFmpeg exception means interrupted render; current flow does not catch all exceptions to mark `FAILED`. |
| `RENDERED` | Project-level final and manifest were written before status update. | Still probe both media and manifest; status alone is insufficient QA. |
| `FAILED` | Single-source flow explicitly writes this for an empty highlight list. | Read `failureMessage`; other exceptions may escape without this state. |
### No category or candidates
**Current behavior:** `HighlightSourceAnalyzer` writes FFprobe, scene, audio, visual, and source analysis. `HighlightSourceScheduler` then invokes `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Missing files on a newly created project are therefore a failed/incomplete candidate stage; older projects may predate this wiring.
```bash
rg -n 'HighlightCandidateGenerator|category\.json|highlight-candidates\.json' src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java src/main/java/org/example/videoclips/editing/HighlightCandidateGenerator.java src/main/java/org/example/videoclips/editing/HighlightDirectorPromptGenerator.java
rg -n 'writeJson.*(category|highlight-candidates)' src/main/java/org/example/videoclips/editing
```
- If files are absent on a new project, inspect `highlight_candidates_started`, `highlight_candidate_scored`, and `highlight_candidates_completed`; do not continue to director or render.
- If files exist, parse and validate their project ID, category confidence, nonempty candidate list, bounds, and scores against `analysis/source-analysis.json`.
- Do not copy multi-clip root files (`category.json`, `highlight-candidates.json`) into the single-source `analysis/` tree as a workaround.
### Director plan, approval, and pending assets
```bash
sed -n '1,260p' "$project/director/edit-plan.json"
rg 'highlight_flow_(waiting_for_approval|waiting_for_assets|started|completed)|highlight_render_scan' "$logs" | rg "$project_id"
find "$project" -path '*/requests/*.json' -type f -print -exec sed -n '1,180p' {} \;
```
- If plan is absent, remain `WAITING_FOR_DIRECTOR`; do not fabricate one during incident response.
- If `require-director-approval=true` and the configured approval file is absent, this is an intentional gate.
- If request JSON remains pending, inspect the matching target path and worker logs. Never satisfy it with placeholder media.
- If the plan has an empty `highlights` array, the flow marks `FAILED`.
- If JSON is malformed, deserialization throws. Semantic validation is performed by `HighlightDirectorPlanValidator`, which requires category/candidate artifacts, safe IDs, candidate-contained ranges, duration bounds, and the production creative directions. A rejection marks the project `FAILED`; read `failureMessage` before changing the plan.
- If `contentCategory` is missing or unknown, rendering silently maps it to `GENERIC_VLOG`; treat this as invalid production input, not successful classification.
**Costly trap:** history labeled commit `5d889b0` “working version but not cinematic.” A completed render proved mechanical execution, not creative correctness. Require candidate evidence, certified assets, media probes, and the QA rubric.
### Multi-clip manual plans are different
Use this branch only for `output/edit-projects/<project-id>/inbox/edit-plan.json`, not highlight projects.
```bash
edit_project='output/edit-projects/<project-id>'
find "$edit_project" -maxdepth 2 -type f -print | sort
rg 'event=edit_plan_(inbox_detected|rejected|imported|saved)|event=edit_render_waiting' "$logs" | tail -n 100
```
- Invalid JSON or validation failure moves the inbox plan to `edit-plan.json.rejected`; read the associated rejection message.
- A valid import writes root `edit-plan.json`, renames the inbox file `.accepted`, and writes `PLANNED` before any approved automatic render.
- The validator checks project/style identity, source and timeline bounds, playback speed, transitions, audio cues, voiceover, overlays, target duration, and minimum cinematic richness. It also requires referenced SFX files to exist.
- If auto-render is disabled, `PLANNED` is expected. If approval is required, the configured file belongs in the project `inbox/` directory.
- Do not move `.rejected` back or edit it in place during triage. Correct the producer through change control and submit a new reviewed plan.
## Diagnose FFmpeg, filters, audio, and codecs
Probe the source and output before changing commands:
```bash
media='<source-or-output-media>'
ffprobe -v error -show_entries format=duration,size:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$media"
ffmpeg -v error -i "$media" -f null -
```
- If FFprobe fails, isolate corrupt/unreadable media or an unsupported container before inspecting filter graphs.
- If FFprobe succeeds but decode-to-null fails, isolate codec/decoder or damaged packets.
- If decode succeeds and rendering fails, capture the exact logged FFmpeg command/output. Test that exact command only in an approved disposable workspace; do not simplify it until the failing filter is identified.
- If failure names `drawtext`, verify the selected font file and escaping. If it names `loudnorm`, `amix`, or `sidechaincompress`, inventory actual audio inputs and durations. If concat copy fails, compare codecs, time bases, resolution, and audio layouts across each per-highlight final.
- If per-highlight outputs exist but project `final.mp4` does not, inspect concat. The project concatenates with `-c copy`, so incompatible streams are a separate cause from segment rendering.
- A `qa-report.json` pass is not a creative certificate. Existing QA contains structural checks and warning-level probes; use `video-editing-validation-and-qa` for acceptance evidence.
## Diagnose folder scheduler quarantine
```bash
find input/source input/working input/processed input/rejected -maxdepth 1 -type f -print 2>/dev/null | sort
find output/clips -maxdepth 2 -type f -print 2>/dev/null | sort
rg 'event=(candidate|validation|ffprobe|ffmpeg|processing|failed_source|failure_)' "$logs" | tail -n 200
```
- Source only: candidate may be ignored because it is hidden, unsupported, `.tmp`, `.part`, `.download`, or `.failed`, or scheduler/profile is disabled.
- Working then rejected: validation or clipping failed; use event ordering to distinguish them.
- Rejected collision: the scheduler refuses overwrite. If quarantine then also fails, it renames the working file to `.failed`; an existing marker causes `failure_marker_skipped` and leaves the source unchanged.
- Processed collision after clips were created: source is moved to rejected. Inventory the output directory before retrying to avoid duplicate artifacts.
- `ffmpeg_completed exit_code=0` with zero clips is still failure.
Do not delete markers or move quarantined files back during diagnosis. Route recovery through `video-editing-run-and-operate` and change control.
## Diagnose API queue, DLQ, storage, and JPA
First identify adapters from resolved configuration/logs. **Current defaults:** memory repository, memory storage, memory queue, and stub processing. `application-jpa.properties` opts into JPA and DB queue but uses H2 in PostgreSQL compatibility mode; it is not a production PostgreSQL profile.
```bash
rg -n '^video-clipping\.(repository|storage|queue|processing)=|^spring\.datasource|^spring\.jpa|^spring\.flyway' src/main/resources/application*.properties
rg 'DatabaseBackedClipJobQueueAdapter|InMemoryClipJobQueueAdapter|JpaVideoClippingRepository|S3ObjectStorageAdapter' "$logs"
```
For an approved read-only PostgreSQL session, run:
```sql
select status, count(*) from queue_messages group by status order by status;
select id, job_id, status, attempt_count, available_at, claimed_at, last_error
from queue_messages where status in ('PENDING','PROCESSING','DLQ') order by available_at limit 50;
select id, asset_id, status, attempt_count, progress_percent, error_message
from clip_jobs where id = '<job-id>';
select event_type, message, created_at from job_events
where aggregate_id = '<job-id>' order by created_at;
```
- Rising `PENDING`: worker capacity/polling or a downstream bottleneck.
- Old `PROCESSING`: compare age with `visibility-timeout-ms` (current default 900000 ms); reclaim is expected only after expiry.
- Retry below max attempts: job should return to `QUEUED`, not terminal `FAILED`.
- `DLQ`: current code marks the job `FAILED` and records `DLQ` and `FAILED` events. Fix cause before any controlled redrive.
- Memory queue: state is process-local and disappears at restart; DB runbooks do not apply.
- Storage metadata without a retrievable object is failure. Verify the persisted `source_object_key`/clip `object_key` against the configured adapter using an approved provider procedure; do not infer an object key from an ID.
- Actuator custom health indicators currently report adapter identity/capability without exercising repository, queue, or storage. `UP` is not dependency reachability evidence.
**Historical traps:** commits `66e998e` and `7307082` fixed metadata-only clip artifacts and fabricated download paths; `1737d8b` fixed retryable DB failures being marked terminal on the first exception. Read `video-editing-failure-archaeology` before changing these contracts.
## Detect profile and documentation drift
Resolve behavior from code plus the active profile, never from a plan checkbox alone.
```bash
rg -n 'spring\.profiles\.active|application-[^.]+\.(yml|properties)' pom.xml src/main/resources src/test/resources
rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml
rg -n '@ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
git log -1 --format='%h %cI %s' -- src/main/resources/application.yml docs
```
- Tests force the `test` profile via Surefire; test resources disable schedulers and worker auto-start. A passing test does not prove default application startup safety.
- Base `application.yml` currently enables the folder scheduler, editing, local CV auto-start, local asset auto-start, local director, and highlight ingestion; highlight rendering defaults off and director approval defaults on.
- `application-cinematic-editing-local.yml` changes multi-clip local-director behavior, not the entire single-source highlight contract.
- When docs and executable code differ, record the discrepancy and route doc correction to `video-editing-docs-and-writing`; do not silently choose the more convenient claim.
## Reproduce clean-checkout build failures without network
**Current as of 2026-07-21:** the working tree passes 245 tests in 62 test classes with `mvn -q -o verify`. An earlier archived clean tree failed `LocalAssetGenerationStageTest` because it depended on untracked local synthesis/host fallback; the test is now deterministic and fail-closed, but a new clean-archive run is still required. The repository has no Maven wrapper or CI configuration.
Use an already populated Maven cache and forbid network:
```bash
tmp="$(mktemp -d)"
git archive HEAD | tar -x -C "$tmp"
(cd "$tmp" && mvn -o -q verify)
git ls-files mvnw .mvn .github .gitlab-ci.yml README.md
git check-ignore -v .venv-local-asset .venv-local-cv target
```
- Workspace passes, archive fails: find untracked runtime/tool/asset assumptions; do not copy them into the archive as a fix.
- Both fail identically: diagnose the first failing test, not the final Maven summary.
- Offline dependency resolution fails before tests: report the cache limitation separately; it does not prove source failure.
- Archive passes: compare JDK, Maven, FFmpeg, locale, and environment variables with the failing environment.
- Route reproducibility changes to `video-editing-build-and-env` and change control.
## Do not use this skill when
- Designing a fix or changing defaults: use `video-editing-change-control` and `video-editing-architecture-contract`.
- Cataloging configuration: use `video-editing-config-and-flags`.
- Running an approved production operation, redrive, or render: use `video-editing-run-and-operate`.
- Certifying cinematic/audio quality: use `video-editing-validation-and-qa`.
- Advancing candidate identification and production local-model rendering: use `video-editing-cinematic-highlights-campaign`.
- Researching whether an old approach was rejected: use `video-editing-failure-archaeology`.
## Exit checklist
- [ ] Record commit, active profile, resolved adapters, tool versions, project/job ID, and timestamps.
- [ ] Identify the first missing or invalid artifact, not merely the last visible symptom.
- [ ] Correlate state, artifact, and log evidence; note contradictions explicitly.
- [ ] Run at least one experiment that separates the leading hypotheses.
- [ ] Reject downloads, external calls, unlicensed assets, placeholders, unapproved renders, and default changes.
- [ ] State current behavior separately from the production target.
- [ ] Route any mutation through change control with a test and rollback plan.
## Provenance and maintenance
Re-verify state writers: `rg -n 'HighlightProjectStatus\.|markStatus|markFailed' src/main/java/org/example/videoclips/editing`
Re-verify scheduler/default guards: `rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml src/test/resources/application*.properties`
Re-verify prohibited bootstrap behavior: `rg -n 'pip install|YOLO\(|write_silence|write_fallback_tone|fallbackTone|writeSilence' tools src/main/java/org/example/videoclips/editing`
Re-verify single-source artifact producers: `rg -n 'writeJson.*(source-analysis|category|highlight-candidates)|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing`
Re-verify queue states and retry rules: `rg -n 'PENDING|PROCESSING|COMPLETED|DLQ|maxAttempts|visibilityTimeout|markRetry|markTerminal' src/main/java/org/example/videoclips/queue src/main/java/org/example/videoclips/processing`
Re-verify build/profile behavior: `rg -n 'spring.profiles.active|maven-surefire|jacoco|java.version|spring-boot-starter-parent' pom.xml && git ls-files mvnw .mvn .github .gitlab-ci.yml README.md`
Re-verify historical incidents: `git show --stat --oneline 1737d8b 7307082 66e998e bd620fc 5d889b0 97ba827`

View File

@ -0,0 +1,283 @@
---
name: video-editing-diagnostics-and-tooling
description: "Load this skill when you need a read-only measurement tool: inventory edit/highlight project state, verify a render manifest or QA report against media, run FFprobe/FFmpeg probes, lexically audit runtime hazards/defaults, inspect existing logs/metrics/health captures, or rerun an existing benchmark harness offline. Do not load it for interactive symptom-to-branch triage or for causal proof after triage."
---
# Video Editing Diagnostics and Tooling
Use measurement before changing code. Treat every `passed` field, health response, log line, and benchmark document as a claim with a known evidence boundary.
Verified against the repository on **2026-07-21**.
## Do not use this skill when
| Need | Load this sibling instead |
| --- | --- |
| Classify, approve, implement, or promote a behavior change | `video-editing-change-control` |
| Start schedulers, workers, or the service; deploy or recover it | `video-editing-run-and-operate` |
| Look up a configuration property or add a flag | `video-editing-config-and-flags` |
| Choose acceptance thresholds or certify a render/release | `video-editing-validation-and-qa` |
| Triage a known symptom interactively | `video-editing-debugging-playbook` |
| Explain a settled failure or rejected fix | `video-editing-failure-archaeology` |
| Prove a new algorithm or quality mechanism | `video-editing-proof-and-analysis-toolkit` |
Do not use a diagnostic result to bypass change control. Diagnostics may falsify a claim; they do not authorize a render, default change, dependency installation, model acquisition, or production promotion.
## Diagnostic safety boundary
The canonical prohibitions are in `video-editing-change-control`; the configuration catalog is `video-editing-config-and-flags`, and acceptance evidence is defined by `video-editing-validation-and-qa`. Never perform these actions from a diagnostic session:
- Download a dependency or model automatically.
- Call an external AI service or use network transport for model/media work, including loopback.
- use an asset without recorded license/provenance evidence.
- Accept silence, a generated tone, or an OS speech fallback as production voiceover/music/SFX.
- Trigger rendering without recorded approval.
- change a production-facing default.
- Start the application merely to inspect project files. Current defaults enable the folder scheduler, editing, local director, highlight ingestion, and two local-worker auto-start paths; highlight rendering itself defaults off.
The local worker shell scripts are **not safe diagnostic commands**. `tools/run_local_asset_worker.sh` and `tools/run_local_cv_worker.sh` run `pip install` in their default `auto` bootstrap mode. The CV script also constructs an Ultralytics `YOLO` model from the default `yolov8n.pt`, which may initiate model acquisition when the file is absent. The asset runtime verifier no longer calls its bootstrap script and strict readiness now fails startup when resident model paths are missing.
## Start here
Run these from the repository root. They are offline and read-only.
```bash
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py --repo . --fail-on never
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py --repo . --fail-on never
```
Then verify a specific render:
```bash
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py \
--repo . \
--project-dir output/edit-projects/PROJECT_ID
```
For a per-highlight render, point `--project-dir` to its directory:
```bash
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py \
--repo . \
--project-dir output/highlight-projects/PROJECT_ID/highlights/HIGHLIGHT_ID \
--deep
```
`--deep` reads the entire media file three times through FFmpeg. Run it on a production copy or isolated worker, not on a latency-sensitive service volume.
## Shipped tools
| Tool | Question answered | Dependencies | Exit codes |
| --- | --- | --- | --- |
| `scripts/inventory_projects.py` | Which edit/highlight projects are partial, inconsistent, stale, or backed only by asserted QA? | Python 3 standard library | `0` no finding at selected threshold; `2` findings; `3` invocation/read error |
| `scripts/verify_render.py` | Does a manifest agree with the real output duration and streams? Were QA checks measured or asserted? | Python 3, `ffprobe`; `ffmpeg` with `--deep` | `0` no error finding; `2` verification error; `3` tool/input error |
| `scripts/audit_runtime_safety.py` | Where do forbidden downloads, placeholders, auto-start, auto-render, approval bypass, or asset-license gaps remain? | Python 3 standard library | `0` no selected-severity finding; `2` finding; `3` invocation/read error |
All output ordering is deterministic. JSON mode is suitable for later CI integration:
```bash
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py \
--repo . --stale-hours 24 --format json --fail-on warning
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py \
--repo . --format json
```
`--stale-hours` compares file modification times with the invocation time. Omit it when reproducibility across dates matters. An asset request count means request JSON files exist; it does not prove that every request remains unresolved.
## Interpret project state
The edit project root defaults to `output/edit-projects`; the highlight project root defaults to `output/highlight-projects`. Override either root explicitly when diagnosing a mounted production copy:
```bash
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py \
--repo . \
--edit-root /mounted-copy/edit-projects \
--highlight-root /mounted-copy/highlight-projects \
--format table --fail-on error
```
The tool only treats a child as a project when `project.json` exists. It ignores caches such as `_voiceover-cache`.
| Observation | Meaning | Next discriminating check |
| --- | --- | --- |
| `RENDERED_OUTPUT_MISSING` | Persisted state claims completion, but `final.mp4` is absent. | Inspect storage/copy logs; do not repair state by hand. |
| `OUTPUT_STATUS_MISMATCH` | A final file exists while project state is nonterminal/failed. | Verify the file and manifest, then inspect the state-transition failure. |
| `MANIFEST_OUTPUT_MISSING` | Provenance exists for an absent local output. | Resolve whether cleanup, relocation, or failed publication occurred. |
| `DIRECTOR_PLAN_MISSING` | A planned/rendering/rendered highlight has no director plan at the folder-contract path. | Check move/copy history and `highlight_render_scan_*` events. |
| `HIGHLIGHT_QA_ASSERTED` | A highlight QA report exists but did not run FFmpeg media probes. | Run `verify_render.py --deep`; do not certify from the report. |
| `NONTERMINAL_STALE` | No project file changed inside the chosen time window. | Identify the expected scanner/worker and correlate its last event. |
| `RENDERED_QA_MISSING` | A render has no persisted QA report. | Verify it independently; absence is not a pass or failure by itself. |
Statuses are `CREATED`, `ANALYZING`, `ANALYZED` (edit only), `WAITING_FOR_DIRECTOR`, `PLANNING` (edit only), `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`. Only `RENDERED` and `FAILED` are terminal for stale detection.
## Separate measured QA from asserted QA
This distinction is load-bearing.
| Renderer | Current report behavior | Evidence actually present |
| --- | --- | --- |
| `FfmpegEditRenderer` | Runs `blackdetect`, `silencedetect`, and `astats`; also checks files, timeline arithmetic, assets, overlays, command recording, and configured mastering filters. | Media probes plus structural assertions. A warning does not make overall `passed` false; only failed `ERROR` checks do. |
| `HighlightFfmpegRenderer` | Marks duration, assets, overlays, and mastering checks true from plan/file assumptions; runs no black/silence/peak probe in `buildQaReport`. | Assertions only. Its `passed: true` is not measured cinematic, visual, or audio quality. |
The older edit duration check compares the edit plan to the duration value passed into report construction. It does **not** probe the rendered container duration. Always compare `render-manifest.json.durationSeconds` with FFprobe independently.
`verify_render.py` checks:
- manifest `outputPath` resolution;
- FFprobe readability;
- manifest duration versus container duration, default tolerance `0.10s`;
- presence of video and audio streams;
- codec, dimensions, pixel format, frame-rate expression, sample rate, channels, and file size;
- declared QA failure and measured-versus-asserted classification;
- optionally, black ranges, silence ranges, and audio peaks using the repository's current thresholds.
It does **not** prove that highlights are correctly selected, pacing is cinematic, speech is intelligible, music/SFX are appropriate or licensed, overlays are readable, colors are intentional, or source-to-output continuity is correct. Load `video-editing-validation-and-qa` for certification.
## Direct media measurements
Use an explicit input path. These commands read media and write only standard output/error.
### Container and streams
```bash
ffprobe -v error -show_streams -show_format -of json -- path/to/final.mp4
```
Interpretation checklist:
- Confirm exactly the stream topology required by the acceptance contract; this project normally emits H.264 video and optional/expected AAC audio depending on the flow.
- Compare `format.duration` with the manifest and planned timeline. Container duration, stream duration, and planned duration can differ.
- Confirm dimensions, pixel format, average frame rate, audio sample rate, channels, and codec rather than inferring them from a filename/profile string.
- Treat successful decode metadata as structural evidence, not creative-quality evidence.
### Black, silence, clipping, freeze, and loudness
Repository-equivalent probes:
```bash
ffmpeg -hide_banner -v info -i path/to/final.mp4 -vf 'blackdetect=d=0.5:pic_th=0.98' -an -f null -
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'silencedetect=noise=-45dB:d=2' -vn -f null -
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'astats=metadata=1:reset=1' -vn -f null -
```
Additional investigative probes, not current application gates:
```bash
ffmpeg -hide_banner -v info -i path/to/final.mp4 -vf 'freezedetect=n=-60dB:d=2' -an -f null -
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'ebur128=peak=true' -vn -f null -
```
Interpret the event timestamps, not just whether a word appears. Intentional fades can trigger black detection; pauses can trigger silence detection; a static composition can trigger freeze detection. Compare each interval with the edit plan and source. `astats` peaks at or above `-0.1 dBFS` fail the older renderer's audio-peak predicate. The configured mastering targets are integrated loudness `-16 LUFS`, true peak `-1.5 dBTP`, and loudness range `11 LU`; the renderer invokes `loudnorm`, but only measurement can show the result.
## Logs: event IDs and correlation
Application logs are SLF4J messages containing stable-looking `event=<id>` key/value tokens. They are not configured as JSON in the repository. Do not call them machine-readable structured logs until a formatter/parser contract is implemented and tested.
Search source-defined IDs without guessing:
```bash
rg -o 'event=[a-zA-Z0-9_{}-]+' src/main/java | sed 's/.*event=//' | sort -u
```
| Investigation | Start events | Completion/failure events | Correlation fields present in relevant paths |
| --- | --- | --- | --- |
| Edit analysis | `edit_analysis_started` | `edit_analysis_completed` | `project_id`, elapsed/count fields |
| Edit render | `edit_render_started` | `edit_render_completed`, `edit_render_failed` | `project_id`, elapsed, output, duration, size/error |
| Highlight ingest/analysis | `highlight_scan_started`, `highlight_analysis_started` | `highlight_scan_completed`, `highlight_processing_failed`, `highlight_analysis_completed` | `scan_id`, `project_id`, source/candidate fields vary by event |
| Highlight render flow | `highlight_flow_started`, `highlight_render_started` | `highlight_flow_completed`, `highlight_render_completed` | `flow_id`, `scan_id`, `project_id`, `highlight_id` across relevant events |
| Local CV | `local_cv_request_started`, `local_cv_worker_starting` | `local_cv_visual_analysis_completed`, `local_cv_visual_analysis_failed`, worker lifecycle events | clip/endpoint/status/elapsed fields vary |
| Local assets | `local_asset_runtime_check_started`, `local_asset_generation_started` | `local_asset_generation_completed`, `local_asset_generation_failed` | project/type/model/target/strategy fields vary |
Discriminating searches:
```bash
rg 'event=highlight_(flow|render|asset)|event=local_asset_' path/to/application.log
rg 'project_id=PROJECT_ID|flow_id=PROJECT_ID:' path/to/application.log
rg 'fallback|degraded|pending|failed|silence|ffmpeg-sine' path/to/application.log
```
Do not assume trace correlation exists. The build has no Micrometer Tracing/OpenTelemetry/Brave dependency and source has no tracer instrumentation as of the verification date. Documentation says `traceId` “if available”; that is not implementation evidence.
Never put media paths containing sensitive names, prompt bodies, transcript/voiceover text, tokens, tenant/user IDs, or exception payloads into newly added broad logs. Existing logs include paths and failure messages; assess redaction before production use.
## Metrics, dashboards, and alerts
Actuator exposes `health`, `info`, `metrics`, and `prometheus`. Prometheus metric names translate dots to underscores.
| Metric family in code | Type | Interpretation |
| --- | --- | --- |
| `video.clipping.queue.pending`, `.processing`, `.dlq`, `.oldest.pending.age.seconds` | DB-queue-only gauges | Backlog state; absent when queue mode is not `db`. |
| `video.clipping.edit.projects.created` | Counter | Process-local project creations since restart. |
| `video.clipping.edit.analysis.duration` | Timer | Analysis attempts that reached `analysisCompleted`; failures before it may be absent. |
| `video.clipping.edit.analysis.clip.count`, `.error.count` | Distribution summaries | Per-completed-analysis counts, not current gauges. |
| `video.clipping.edit.render.duration` | Timer | Both completed and failed renders record duration. |
| `video.clipping.edit.render.failures` | Counter | Render exceptions observed by instrumented renderers. |
| `video.clipping.edit.render.output.duration.seconds`, `.size.bytes` | Distribution summaries | Completed render output claims passed to observability, not FFprobe measurements. |
| `video.clipping.edit.director.prompt.tokens`, `.asset.generation.cost.usd` | Distribution summaries | Estimates, not tokenizer/provider invoices. |
The checked-in Grafana dashboard currently covers DB queue depth/age/DLQ/processing, API request rate and p95, CPU, and heap. It does not visualize edit/highlight render metrics or creative/business quality. The checked-in alert document covers queue age, DLQ, API 5xx, target down, and process CPU. It explicitly records missing job-failure, FFmpeg-exit, storage-failure, signed-URL, and disk metrics.
Do not invent a render success ratio: there is no render-success counter. A candidate alert such as `increase(video_clipping_edit_render_failures_total[15m]) > 0` can detect recorded exceptions, but it misses silent audio/preview fallbacks and needs change-control approval, deployment labels, threshold evidence, and a linked runbook before adoption.
### Cardinality rules
- Never tag metrics with `project_id`, `highlight_id`, `flow_id`, `scan_id`, file path, prompt, model path, tenant/user ID, request ID, trace ID, or raw exception message.
- Put per-project correlation in logs/traces; use metrics only for bounded dimensions such as workflow, outcome, stage, or a reviewed error class.
- Enumerate every allowed tag value in the design review. Reject free-form strings.
- Verify emitted series count and actual Prometheus names in a production-like scrape before adding panels or alerts.
- Do not expose health/details or Prometheus publicly. Exposure in `application.properties` is endpoint availability, not proof of authorization or network policy.
## Health is not readiness proof
`RepositoryHealthIndicator`, `QueueHealthIndicator`, and `ObjectStorageHealthIndicator` always return `UP` with adapter/capability details. They do not execute a repository query, publish/claim a message, access object storage, invoke FFmpeg/FFprobe, verify disk capacity, or check local models. Consequently:
- `UP` proves only that Spring constructed those adapter beans and called the indicator.
- The DB-queue gauges query repository state, but their presence is not an end-to-end queue test.
- Local asset/CV worker `/health` handling is separate from Actuator health.
- No repository evidence establishes distinct liveness/readiness groups or an authenticated actuator policy.
Treat a health response as one signal. Correlate it with queue age, real project progress, dependency metrics, logs, and an approved synthetic transaction.
## Benchmark harnesses
These Maven tests are reproducible local planning harnesses. They write reports under `target/benchmarks`; they are not read-only and they are not production load tests.
| Command | What it measures | What it does not prove |
| --- | --- | --- |
| `mvn -q -o -Dtest=FfmpegPresetBenchmarkHarness test` | Real FFmpeg preset wall time/output bytes on generated 16s 720p input | Production throughput or highlight quality |
| `mvn -q -o -Dtest=WorkerBenchmarkHarness test` | Synchronous stub worker CPU/wall/local/uploaded bytes for modeled 600s input | Real FFmpeg or object-store behavior |
| `mvn -q -o -Dtest=ObjectStorageBandwidthBenchmarkHarness test` | Local copy and throttled transfer throughput | Live S3/network performance |
| `mvn -q -o -Dtest=CostModelBenchmarkHarness test` | Formula output using checked-in assumptions | Current vendor pricing or invoice forecast |
Run a harness only after confirming it cannot trigger forbidden model/dependency acquisition in the prepared environment. Record hardware, OS, Java, Maven, FFmpeg, source commit, warmup, sample count, input, and raw report. Never compare current output to checked-in numbers without normalizing those variables. No checked-in load generator or executed production-like end-to-end load result exists as of 2026-07-21.
## Evidence checklist for a diagnostic report
- [ ] Record repository commit and dirty-worktree state without mutating Git.
- [ ] Record OS/architecture, Java, Maven, FFmpeg, and FFprobe versions.
- [ ] State whether files are originals or read-only copies and identify the project/highlight IDs.
- [ ] Attach inventory JSON and render-verifier JSON.
- [ ] Distinguish an asserted application field from an independently measured value.
- [ ] State thresholds before running the measurement.
- [ ] Preserve negative observations and failed commands.
- [ ] Redact secrets, personal data, transcripts, voiceover text, signed URLs, and sensitive paths.
- [ ] Route every proposed behavior/config/instrumentation change through `video-editing-change-control`.
- [ ] Route certification through `video-editing-validation-and-qa`; never certify by eye or from `qa-report.json` alone.
## Provenance and maintenance
Re-verify volatile facts from the repository root; none of these commands starts the application or accesses the network:
```bash
rg -n 'auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:' src/main/resources/application*.yml
rg -n 'pip install|YOLO\(' tools src/main/java
rg -n 'fallbackTone|writeSilence|anullsrc|ffmpeg-sine|strategy=silence' src/main/java tools
rg -n 'RenderQaCheck|blackdetect|silencedetect|astats' src/main/java/org/example/videoclips/editing/{FfmpegEditRenderer,HighlightFfmpegRenderer}.java
rg -n 'registry\.(counter|timer|summary)|Gauge.builder' src/main/java
rg -n 'micrometer-tracing|opentelemetry|zipkin|brave|ObservationRegistry|Tracer' pom.xml src/main || true
rg -n 'management.endpoints.web.exposure.include' src/main/resources
jq -r '.panels[]? | [.title,.type] | @tsv' dashboards/video-clipping-overview-grafana.json
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py --help
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py --help
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py --help
```

View File

@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""Lexically audit known runtime safety hazards without executing project code."""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
RULES = (
("BLOCK", "AUTO_DEPENDENCY_INSTALL", re.compile(r"\bpip\s+install\b"), ("tools/*.sh",)),
("BLOCK", "REMOTE_MODEL_RESOLUTION", re.compile(r"\bget_pretrained\s*\("), ("tools/*.py",)),
("BLOCK", "NAMED_YOLO_ACQUISITION", re.compile(r"(?:LOCAL_CV_YOLO_MODEL.*:-|os\.getenv\([^,]+,)\s*[\"']yolo[^\"']*\.pt|YOLO\([^\n]*LOCAL_CV_YOLO_MODEL"), ("tools/*.sh", "tools/*.py")),
("WARN", "YOLO_LOAD_REQUIRES_REVIEW", re.compile(r"\bYOLO\s*\("), ("tools/*.py",)),
("BLOCK", "PLACEHOLDER_TONE", re.compile(r"fallbackTone|fallback-tone|ffmpeg-sine"), ("src/main/java/**/*.java",)),
("BLOCK", "PLACEHOLDER_TONE", re.compile(r"write_fallback_tone\s*\("), ("tools/*.py",)),
("BLOCK", "PLACEHOLDER_SILENCE", re.compile(r"writeSilence|strategy=silence|anullsrc="), ("src/main/java/**/*.java",)),
("BLOCK", "PLACEHOLDER_SILENCE", re.compile(r"write_silence\s*\("), ("tools/*.py",)),
("BLOCK", "AUTO_START_DEFAULT_TRUE", re.compile(r"auto-start:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
("BLOCK", "RENDER_DEFAULT_TRUE", re.compile(r"render-enabled:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
("BLOCK", "APPROVAL_DEFAULT_FALSE", re.compile(r"require-director-approval:\s*\$\{[^}:]+:false\}"), ("src/main/resources/application*.yml",)),
("BLOCK", "HEURISTIC_FALLBACK_DEFAULT_TRUE", re.compile(r"fallback-to-heuristic:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
("WARN", "CONFIGURED_HTTP_NON_LOOPBACK", re.compile(r"https?://(?!127\.0\.0\.1|localhost)", re.IGNORECASE), ("src/main/resources/application*",)),
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Read-only lexical audit for runtime hazards; findings require human classification and a clean result is not compliance proof."
)
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root (default: current directory).")
parser.add_argument("--format", choices=("text", "json"), default="text")
parser.add_argument(
"--fail-on", choices=("block", "warn", "never"), default="block",
help="Exit 2 on BLOCK findings, on BLOCK or WARN findings, or never because of findings (default: block).",
)
return parser.parse_args()
def paths_for(repo: Path, globs: tuple[str, ...]) -> list[Path]:
found: set[Path] = set()
for pattern in globs:
found.update(path for path in repo.glob(pattern) if path.is_file())
return sorted(found, key=lambda path: str(path.relative_to(repo)))
def main() -> int:
opts = parse_args()
repo = opts.repo.resolve()
if not (repo / "pom.xml").is_file():
print(f"error: not a recognized repository root (pom.xml missing): {repo}", file=sys.stderr)
return 3
findings: list[dict] = []
try:
for severity, code, pattern, globs in RULES:
for path in paths_for(repo, globs):
for number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
if pattern.search(line):
findings.append({"severity": severity, "code": code,
"path": str(path.relative_to(repo)), "line": number,
"evidence": line.strip()[:240]})
asset_root = repo / "input" / "highlights" / "assets"
if asset_root.is_dir():
license_markers = list(asset_root.glob("LICENSE*")) + list(asset_root.glob("**/*.license*"))
media = [p for p in asset_root.rglob("*") if p.is_file() and p.suffix.lower() in {".wav", ".mp3", ".flac", ".aif", ".aiff", ".ttf", ".otf", ".cube"}]
if media and not license_markers:
findings.append({"severity": "WARN", "code": "ASSET_LICENSE_MARKER_MISSING",
"path": str(asset_root.relative_to(repo)), "line": 0,
"evidence": f"{len(media)} media/font/LUT files and no LICENSE* or *.license* marker; directory membership and marker files are not license proof"})
except OSError as exc:
print(f"error:goal f {exc}", file=sys.stderr)
return 3
findings.sort(key=lambda item: (item["path"], item["line"], item["code"]))
if opts.format == "json":
print(json.dumps({"repo": str(repo), "findings": findings}, indent=2, sort_keys=True))
else:
for item in findings:
location = f"{item['path']}:{item['line']}" if item["line"] else item["path"]
print(f"{item['severity']} {item['code']} {location} {item['evidence']}")
print(f"SUMMARY block={sum(f['severity'] == 'BLOCK' for f in findings)} warn={sum(f['severity'] == 'WARN' for f in findings)}")
severities = {item["severity"] for item in findings}
if opts.fail_on == "block" and "BLOCK" in severities:
return 2
if opts.fail_on == "warn" and severities & {"BLOCK", "WARN"}:
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""Inventory edit/highlight project state without modifying it."""
from __future__ import annotations
import argparse
import json
import sys
import time
from pathlib import Path
TERMINAL = {"RENDERED", "FAILED"}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Inventory partial, inconsistent, and optionally stale edit/highlight projects."
)
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root (default: current directory).")
parser.add_argument("--edit-root", type=Path, help="Override edit-project root.")
parser.add_argument("--highlight-root", type=Path, help="Override highlight-project root.")
parser.add_argument(
"--allow-missing-roots", action="store_true",
help="Do not report missing edit/highlight roots as errors (use only when the absent workflow is intentional).",
)
parser.add_argument("--stale-hours", type=float, help="Flag nonterminal projects not modified within this many hours.")
parser.add_argument("--format", choices=("table", "json"), default="table")
parser.add_argument(
"--fail-on", choices=("error", "warning", "never"), default="error",
help="Return 2 at or above this finding severity (default: error).",
)
return parser.parse_args()
def load_json(path: Path) -> tuple[dict | None, str | None]:
try:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
return None, "top-level JSON value is not an object"
return value, None
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
return None, str(exc)
def qa_mode(path: Path) -> str:
if not path.is_file():
return "missing"
value, error = load_json(path)
if error or value is None:
return "invalid"
names = {str(check.get("name")) for check in value.get("checks", []) if isinstance(check, dict)}
measured = {"black_frames", "long_silence", "audio_clipping"}
return "measured" if measured <= names else "partial"
def finding(severity: str, code: str, detail: str) -> dict:
return {"severity": severity, "code": code, "detail": detail}
def inspect_project(kind: str, directory: Path, now: float, stale_hours: float | None) -> dict:
project_file = directory / "project.json"
project, error = load_json(project_file)
findings: list[dict] = []
if error:
findings.append(finding("error", "PROJECT_JSON_INVALID", error))
project = {}
status = str(project.get("status", "UNKNOWN"))
final = directory / "final.mp4"
render_manifest = directory / "render-manifest.json"
qa_report = directory / "qa-report.json"
if status == "RENDERED":
if not final.is_file():
findings.append(finding("error", "RENDERED_OUTPUT_MISSING", "status is RENDERED but final.mp4 is absent"))
if not render_manifest.is_file():
findings.append(finding("error", "RENDERED_MANIFEST_MISSING", "status is RENDERED but render-manifest.json is absent"))
if kind != "highlight" and not qa_report.is_file():
findings.append(finding("warning", "RENDERED_QA_MISSING", "status is RENDERED but qa-report.json is absent"))
elif final.is_file():
findings.append(finding("error", "OUTPUT_STATUS_MISMATCH", f"final.mp4 exists while status is {status}"))
if render_manifest.is_file() and not final.is_file():
findings.append(finding("error", "MANIFEST_OUTPUT_MISSING", "render-manifest.json exists but final.mp4 is absent"))
if status == "FAILED" and not (project.get("failureReason") or project.get("failureMessage")):
findings.append(finding("warning", "FAILURE_REASON_MISSING", "FAILED project has no recorded failure reason"))
if stale_hours is not None and status not in TERMINAL:
try:
newest = max((path.stat().st_mtime for path in directory.rglob("*") if path.is_file()), default=directory.stat().st_mtime)
age_hours = (now - newest) / 3600.0
if age_hours >= stale_hours:
findings.append(finding("warning", "NONTERMINAL_STALE", f"newest artifact is {age_hours:.1f} hours old"))
except OSError as exc:
findings.append(finding("error", "STAT_FAILED", str(exc)))
pending_assets = 0
highlight_count = 0
qa_modes: list[str] = []
if kind == "highlight":
director_plan = directory / "director" / "edit-plan.json"
if status in {"PLANNED", "RENDERING", "RENDERED"} and not director_plan.is_file():
findings.append(finding("error", "DIRECTOR_PLAN_MISSING", f"status is {status} but director/edit-plan.json is absent"))
highlights = directory / "highlights"
if highlights.is_dir():
for child in sorted((p for p in highlights.iterdir() if p.is_dir()), key=lambda p: p.name):
highlight_count += 1
mode = qa_mode(child / "qa-report.json")
if mode != "missing":
qa_modes.append(mode)
requests = child / "assets" / "requests"
if requests.is_dir():
pending_assets += sum(1 for p in requests.glob("*.json") if p.is_file())
if status == "RENDERED" and not qa_report.is_file():
if qa_modes:
findings.append(finding(
"warning", "HIGHLIGHT_AGGREGATE_QA_MISSING",
"project-root aggregate qa-report.json is absent; per-highlight QA exists but does not certify the concatenated final.mp4",
))
else:
findings.append(finding(
"warning", "RENDERED_QA_MISSING",
"no project-root aggregate or per-highlight qa-report.json was found",
))
if "partial" in qa_modes:
findings.append(finding("warning", "HIGHLIGHT_QA_PARTIAL", "per-highlight QA lacks one or more expected FFmpeg media probes"))
if "invalid" in qa_modes:
findings.append(finding("error", "HIGHLIGHT_QA_INVALID", "one or more per-highlight QA reports are invalid"))
else:
mode = qa_mode(qa_report)
if mode != "missing":
qa_modes.append(mode)
return {
"kind": kind,
"projectId": str(project.get("id") or directory.name),
"status": status,
"path": str(directory),
"finalExists": final.is_file(),
"manifestExists": render_manifest.is_file(),
"qaModes": sorted(set(qa_modes)),
"highlightCount": highlight_count,
"assetRequestCount": pending_assets,
"findings": findings,
}
def scan_root(kind: str, root: Path, now: float, stale_hours: float | None) -> list[dict]:
if not root.exists():
return []
if not root.is_dir():
raise ValueError(f"{kind} root is not a directory: {root}")
projects = []
for directory in sorted((p for p in root.iterdir() if p.is_dir()), key=lambda p: p.name):
if (directory / "project.json").is_file():
projects.append(inspect_project(kind, directory, now, stale_hours))
return projects
def print_table(projects: list[dict]) -> None:
print("KIND\tPROJECT\tSTATUS\tFINAL\tMANIFEST\tQA\tASSET_REQUESTS\tFINDINGS")
for item in projects:
codes = ",".join(f["severity"][0].upper() + ":" + f["code"] for f in item["findings"]) or "-"
print("\t".join((item["kind"], item["projectId"], item["status"], str(item["finalExists"]).lower(),
str(item["manifestExists"]).lower(), ",".join(item["qaModes"]) or "-",
str(item["assetRequestCount"]), codes)))
def main() -> int:
args = parse_args()
if args.stale_hours is not None and args.stale_hours < 0:
print("error: --stale-hours must be nonnegative", file=sys.stderr)
return 3
repo = args.repo.resolve()
if not (repo / "pom.xml").is_file():
print(f"error: not a recognized repository root (pom.xml missing): {repo}", file=sys.stderr)
return 3
roots = {
"edit": (args.edit_root or repo / "output" / "edit-projects").resolve(),
"highlight": (args.highlight_root or repo / "output" / "highlight-projects").resolve(),
}
try:
now = time.time()
missing_roots = [finding("error", "PROJECT_ROOT_MISSING", f"{kind} root is absent: {root}")
for kind, root in roots.items() if not root.exists()]
if args.allow_missing_roots:
missing_roots = []
projects = scan_root("edit", roots["edit"], now, args.stale_hours)
projects += scan_root("highlight", roots["highlight"], now, args.stale_hours)
except (OSError, ValueError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 3
if args.format == "json":
print(json.dumps({"roots": {k: str(v) for k, v in roots.items()}, "rootFindings": missing_roots,
"projects": projects}, indent=2, sort_keys=True))
else:
for item in missing_roots:
print(f"ROOT\t-\t-\t-\t-\t-\t-\t{item['severity'][0].upper()}:{item['code']} {item['detail']}")
print_table(projects)
severities = {f["severity"] for f in missing_roots}
severities.update(f["severity"] for p in projects for f in p["findings"])
if args.fail_on == "error" and "error" in severities:
return 2
if args.fail_on == "warning" and severities & {"warning", "error"}:
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""Verify render manifests and media using local ffprobe/ffmpeg only."""
from __future__ import annotations
import argparse
import json
import re
import shutil
import subprocess
import sys
from pathlib import Path
def args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Verify one edit or highlight render without modifying it.")
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root used to resolve manifest paths.")
parser.add_argument("--project-dir", type=Path, required=True, help="Directory containing render-manifest.json.")
parser.add_argument("--manifest", type=Path, help="Manifest override (default: PROJECT_DIR/render-manifest.json).")
parser.add_argument("--qa-report", type=Path, help="QA report override (default: PROJECT_DIR/qa-report.json).")
parser.add_argument("--media", type=Path, help="Media override; otherwise use manifest outputPath.")
parser.add_argument("--duration-tolerance", type=float, default=0.10, help="Allowed manifest/probe difference in seconds.")
parser.add_argument("--deep", action="store_true", help="Also run read-only black, silence, and audio-peak FFmpeg probes.")
parser.add_argument("--format", choices=("text", "json"), default="text")
return parser.parse_args()
def read_object(path: Path) -> dict:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise ValueError(f"top-level JSON is not an object: {path}")
return value
def run(command: list[str]) -> subprocess.CompletedProcess[str]:
return subprocess.run(command, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
def resolve_media(value: str, repo: Path, project: Path) -> tuple[Path | None, list[str], list[str]]:
raw = Path(value)
# Current manifests record either an absolute path, a repository-relative path such as
# output/edit-projects/<id>/final.mp4, or a project-relative path such as final.mp4.
candidates = [raw] if raw.is_absolute() else [repo / raw, project / raw, project / raw.name]
normalized: list[Path] = []
for candidate in candidates:
resolved = candidate.resolve()
if resolved not in normalized:
normalized.append(resolved)
existing = [candidate for candidate in normalized if candidate.is_file()]
chosen = existing[0] if len(existing) == 1 else None
return chosen, [str(path) for path in normalized], [str(path) for path in existing]
def qa_summary(path: Path) -> dict:
if not path.is_file():
return {"present": False, "mode": "missing", "declaredPassed": None, "failedChecks": []}
report = read_object(path)
checks = [c for c in report.get("checks", []) if isinstance(c, dict)]
names = {str(c.get("name")) for c in checks}
expected = {"black_frames", "long_silence", "audio_clipping"}
mode = "measured" if expected <= names else "partial"
return {
"present": True,
"mode": mode,
"declaredPassed": report.get("passed"),
"failedChecks": [str(c.get("name")) for c in checks if c.get("passed") is False],
"invalidChecks": [str(c.get("name")) for c in checks if not isinstance(c.get("passed"), bool)],
"missingProbes": sorted(expected - names),
}
def deep_probes(ffmpeg: str, media: Path) -> tuple[dict, list[dict]]:
findings: list[dict] = []
probes: dict = {}
commands = {
"black": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-vf", "blackdetect=d=0.5:pic_th=0.98", "-an", "-f", "null", "-"],
"silence": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-af", "silencedetect=noise=-45dB:d=2", "-vn", "-f", "null", "-"],
"peaks": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-af", "astats=metadata=1:reset=1", "-vn", "-f", "null", "-"],
}
for name, command in commands.items():
result = run(command)
output = result.stdout + result.stderr
probes[name] = {"exitCode": result.returncode}
if result.returncode != 0:
findings.append({"severity": "error", "code": f"FFMPEG_{name.upper()}_FAILED", "detail": f"exit={result.returncode}"})
continue
if name == "black":
count = len(re.findall(r"black_start:", output))
probes[name]["rangeCount"] = count
if count:
findings.append({"severity": "error", "code": "BLACK_RANGE_DETECTED", "detail": f"ranges={count}; threshold d=0.5,pic_th=0.98"})
elif name == "silence":
count = len(re.findall(r"silence_start:", output))
probes[name]["rangeCount"] = count
if count:
findings.append({"severity": "warning", "code": "LONG_SILENCE_DETECTED", "detail": f"ranges={count}; threshold -45dB for 2s"})
else:
peaks = []
for value in re.findall(r"Peak level dB:\s*([^\s]+)", output):
try:
peaks.append(float(value))
except ValueError:
pass
maximum = max(peaks) if peaks else None
probes[name]["maximumPeakDbfs"] = maximum
if maximum is None:
findings.append({"severity": "warning", "code": "AUDIO_PEAK_UNMEASURED", "detail": "astats emitted no numeric peak"})
elif maximum >= -0.1:
findings.append({"severity": "warning", "code": "AUDIO_PEAK_UNSAFE", "detail": f"maximum={maximum} dBFS"})
return probes, findings
def main() -> int:
opts = args()
if opts.duration_tolerance < 0:
print("error: --duration-tolerance must be nonnegative", file=sys.stderr)
return 3
repo = opts.repo.resolve()
project = opts.project_dir.resolve()
manifest_path = (opts.manifest or project / "render-manifest.json").resolve()
qa_path = (opts.qa_report or project / "qa-report.json").resolve()
findings: list[dict] = []
try:
manifest = read_object(manifest_path)
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 3
try:
qa = qa_summary(qa_path)
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
qa = {"present": True, "mode": "invalid", "declaredPassed": None, "failedChecks": [],
"invalidChecks": [], "missingProbes": []}
findings.append({"severity": "error", "code": "QA_REPORT_INVALID", "detail": str(exc)})
if qa["mode"] == "missing":
findings.append({"severity": "error", "code": "QA_REPORT_MISSING", "detail": str(qa_path)})
elif qa["mode"] == "partial":
findings.append({"severity": "warning", "code": "QA_PARTIAL", "detail": f"missing probes={','.join(qa['missingProbes'])}"})
if qa["present"] and not isinstance(qa["declaredPassed"], bool):
findings.append({"severity": "error", "code": "QA_PASSED_INVALID", "detail": "passed must be a JSON boolean"})
if qa.get("invalidChecks"):
findings.append({"severity": "error", "code": "QA_CHECK_RESULT_INVALID", "detail": ",".join(qa["invalidChecks"])})
if qa["declaredPassed"] is True and qa["failedChecks"]:
findings.append({"severity": "error", "code": "QA_RESULT_INCONSISTENT", "detail": "passed=true with failed checks: " + ",".join(qa["failedChecks"])})
if qa["declaredPassed"] is False:
findings.append({"severity": "error", "code": "QA_DECLARED_FAILED", "detail": ",".join(qa["failedChecks"]) or "report passed=false"})
media_value = str(opts.media) if opts.media else str(manifest.get("outputPath") or "")
if not media_value:
print("error: no --media and manifest outputPath is empty", file=sys.stderr)
return 3
if opts.media:
explicit = opts.media.resolve()
media, candidates, existing = (explicit if explicit.is_file() else None), [str(explicit)], ([str(explicit)] if explicit.is_file() else [])
else:
media, candidates, existing = resolve_media(media_value, repo, project)
if len(existing) > 1:
findings.append({"severity": "error", "code": "MEDIA_RESOLUTION_AMBIGUOUS", "detail": "multiple candidates exist: " + ",".join(existing)})
if media is None:
if not existing:
findings.append({"severity": "error", "code": "MANIFEST_OUTPUT_MISSING", "detail": f"candidates={','.join(candidates)}"})
report = {"manifest": str(manifest_path), "media": None, "resolutionCandidates": candidates,
"qa": qa, "probe": None, "deepProbes": None, "findings": findings}
if opts.format == "json":
print(json.dumps(report, indent=2, sort_keys=True))
else:
for item in findings:
print(f"{item['severity'].upper()} {item['code']} {item['detail']}")
return 2
ffprobe = shutil.which("ffprobe")
if not ffprobe:
print("error: ffprobe is not available on PATH", file=sys.stderr)
return 3
result = run([ffprobe, "-v", "error", "-show_streams", "-show_format", "-of", "json", str(media)])
if result.returncode != 0:
print(f"error: ffprobe exited {result.returncode}: {result.stderr.strip()}", file=sys.stderr)
return 3
try:
probe = json.loads(result.stdout)
duration = float(probe.get("format", {}).get("duration"))
except (TypeError, ValueError, json.JSONDecodeError) as exc:
print(f"error: invalid ffprobe result: {exc}", file=sys.stderr)
return 3
expected = manifest.get("durationSeconds")
if isinstance(expected, bool) or not isinstance(expected, (int, float)) or float(expected) < 0:
findings.append({"severity": "error", "code": "MANIFEST_DURATION_INVALID", "detail": f"durationSeconds={expected!r}"})
elif abs(float(expected) - duration) > opts.duration_tolerance:
findings.append({"severity": "error", "code": "DURATION_MISMATCH", "detail": f"manifest={expected} probe={duration:.6f} tolerance={opts.duration_tolerance}"})
streams = probe.get("streams", [])
video_streams = [s for s in streams if s.get("codec_type") == "video"]
audio_streams = [s for s in streams if s.get("codec_type") == "audio"]
if not video_streams:
findings.append({"severity": "error", "code": "VIDEO_STREAM_MISSING", "detail": "ffprobe found no video stream"})
if not audio_streams:
findings.append({"severity": "warning", "code": "AUDIO_STREAM_MISSING", "detail": "ffprobe found no audio stream"})
deep = None
if opts.deep:
ffmpeg = shutil.which("ffmpeg")
if not ffmpeg:
print("error: ffmpeg is not available on PATH", file=sys.stderr)
return 3
deep, deep_findings = deep_probes(ffmpeg, media)
findings.extend(deep_findings)
summary = {
"durationSeconds": duration,
"sizeBytes": int(probe.get("format", {}).get("size", media.stat().st_size)),
"video": [{k: s.get(k) for k in ("codec_name", "width", "height", "pix_fmt", "avg_frame_rate")} for s in video_streams],
"audio": [{k: s.get(k) for k in ("codec_name", "sample_rate", "channels", "channel_layout")} for s in audio_streams],
}
report = {"manifest": str(manifest_path), "media": str(media), "resolutionCandidates": candidates,
"qa": qa, "probe": summary, "deepProbes": deep, "findings": findings}
if opts.format == "json":
print(json.dumps(report, indent=2, sort_keys=True))
else:
print(f"media={media}")
print(f"duration_seconds={duration:.6f} video_streams={len(video_streams)} audio_streams={len(audio_streams)} qa_mode={qa['mode']}")
for item in findings:
print(f"{item['severity'].upper()} {item['code']} {item['detail']}")
if not findings:
print("OK manifest and measured media properties agree")
return 2 if any(item["severity"] == "error" for item in findings) else 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,400 @@
---
name: video-editing-docs-and-writing
description: Load when creating, correcting, reviewing, or retiring this repository's README, plans, runbooks, diagrams, ADRs, API/configuration references, release evidence, operational documents, prompts, or milestone checklists; also load when a command, path, status, production-readiness claim, or implementation description in docs may have drifted from source.
---
# Video Editing Documentation And Writing
## Purpose
Write documentation that a zero-context mid-level engineer or Sonnet-class model can execute without guessing. Separate implemented behavior from target architecture, and make every important claim traceable to source, a test, a measurement, or an approved decision.
Treat repository facts in this skill as verified on **2026-07-21**. The Fortune 500 reference-architecture requirements are the target acceptance standard; they do not describe the repository's current state.
Compact glossary: **SBOM** is Software Bill of Materials; **SLI** is service-level indicator; **SLO** is service-level objective; **RTO** is Recovery Time Objective; **RPO** is Recovery Point Objective; **OCI** is Open Container Initiative image format.
## When not to use this skill
| Need | Load instead |
|---|---|
| Decide whether a behavior/default may change or a production claim may be promoted | `video-editing-change-control` |
| Find the implemented architecture and load-bearing invariants | `video-editing-architecture-contract` |
| Catalog or change properties, profiles, flags, and guards | `video-editing-config-and-flags` |
| Recreate the toolchain or prove a clean build | `video-editing-build-and-env` |
| Start schedulers, render media, or locate runtime artifacts | `video-editing-run-and-operate` |
| Define tests, creative-quality evidence, or promotion thresholds | `video-editing-validation-and-qa` |
| Diagnose a symptom or interpret diagnostic output | `video-editing-debugging-playbook` and `video-editing-diagnostics-and-tooling` |
| Explain why an old decision or failure happened | `video-editing-failure-archaeology` |
Do not use documentation to authorize a change, bypass approval, or turn intent into evidence.
## Apply the documentation safety boundary
Enforce these rules in every document, example, diagram, and generated artifact:
1. Do not instruct runtime startup to download dependencies or model weights. Do not prescribe external AI services, unapproved network access, unlicensed assets, placeholder silence/tones, or unapproved rendering.
2. Do not present a production-facing default change as a documentation correction. Route the implementation and its documentation through `video-editing-change-control`.
3. Mark the current automatic bootstrap/download, heuristic fallback, placeholder audio, and approval-bypass behavior as defects or development-only behavior, never as approved production practice.
4. Do not paste secrets, credentials, private hosts, user home paths, local virtual environments, proprietary footage, or unredacted production payloads into documentation.
5. Do not claim production-ready, secure, scalable, reproducible, cinematic, licensed, validated, signed off, or complete without the required evidence and approval.
6. Treat documentation-only work as C0 only when it cannot change runtime/build behavior. A command, configuration, API, schema, deployment, or operational contract correction that reveals or requires behavior change inherits the applicable C2/C3 route.
## Start from the current repository truth
Run these read-only checks from the repository root before writing:
```bash
git status --short
git log -10 --oneline --decorate
git ls-files 'docs/**' 'src/main/resources/**' 'tools/**' 'dashboards/**' pom.xml | sort
find .claude/skills -maxdepth 2 -name SKILL.md -print | sort
find src/main/java src/test/java -type f | sort
```
As of 2026-07-21:
| Current fact | Documentation consequence |
|---|---|
| There is no tracked `README.md`, contributor guide, OpenAPI document, ADR set, architecture diagram, CI workflow, container definition, or deployment definition. | List each as missing target work. Never write “see” or “validated by” one of these artifacts. |
| `docs/` contains plans, prompts, runbooks, operating checklists, benchmark baselines, and readiness templates without an explicit authority hierarchy. | Classify each document before relying on it. A plan is not an implementation contract. |
| `pom.xml` uses Java 21 and Spring Boot 3.3.2; no Maven Wrapper is tracked. | State implemented versions and current prerequisites. Do not call Spring Boot 3.3.2 “latest” without a separately approved, date-stamped primary-source check. |
| Base `application.yml` enables ingestion schedulers and local-worker auto-start, while highlight rendering defaults off and approval defaults on; current CV/bootstrap scripts may install packages or resolve models. | Do not recommend a plain startup as a harmless documentation-verification command. Use the operating/build siblings and explicit safe overrides. |
| `application-jpa.properties` uses H2 in PostgreSQL compatibility mode. | Do not describe current repository integration tests as real PostgreSQL/Testcontainers evidence. |
| `production-cinematic-highlight-editing-plan.md` checks all milestones, while `highlight-rendering-gap-closure-plan.md` remains unchecked and `cinematic-highlight-operator-checklist.md` says rendering is not wired. Current source now contains `HighlightDirectorFlowService` and `HighlightFfmpegRenderer`. | Label the documents contradictory/stale until source, tests, and an executed flow establish one current description. Do not settle the conflict by trusting the newest prose or a checkbox. |
| `video-clipping-service-implementation-plan.md` checks “Load-test signoff,” while `load-test-signoff.md` says execution evidence is pending and no load generator/results are checked in. | State “documentation template exists; load signoff unvalidated.” |
| `input-folder-scheduler-plan.md` names a nonexistent `FolderSchedulerProperties.java`; properties are nested under `VideoClippingProperties`. | Correct or label the proposed path; do not send an engineer to it. |
| Existing benchmark documents describe local baselines and generated files under `target/`. | Preserve environment/date/commit qualifiers. Never convert a workstation baseline into an SLO or production capacity claim. |
## Use the authority hierarchy
Resolve conflicts in this order. Higher authority establishes current behavior; it does not automatically prove quality or production readiness.
| Rank | Authority | What it can establish |
|---:|---|---|
| 1 | Executable source, migrations, packaged resource files, build manifest | What the checked-out artifact is designed to do and its configured defaults. |
| 2 | Deterministic tests and retained command output tied to a commit/environment | What was exercised, within the test's actual assertions and prerequisites. |
| 3 | Versioned generated contract or measured report tied to its generator and inputs | The generated schema/measurement, if regeneration matches. `target/` alone is disposable, not a record. |
| 4 | Approved ADR, API contract, operations record, or release evidence | The accepted decision/contract for its stated scope and date. These record approval, not code existence. |
| 5 | Current reference/runbook checked against ranks 1-4 | How to understand or operate the verified behavior. |
| 6 | Plan, prompt, issue-shaped note, checklist, proposed diagram | Intent. A checked box still needs evidence. |
| 7 | Historical or retired document | Context only; never an operating instruction. |
When code and an approved contract differ, report the mismatch. Do not silently rewrite the contract to match code or treat the code as approved.
## Give each fact one home
Keep the detailed fact in one canonical home. Cross-reference it elsewhere with a short reason; do not copy tables that will drift.
| Fact | Current/target canonical home | Cross-reference rule |
|---|---|---|
| Entry point, prerequisites, safe local quick start, repository map | **TARGET:** `README.md` (currently absent) | Other docs link to the relevant README heading once it exists. Until then, state that the entry-point doc is missing. |
| Dependency versions and build plugin behavior | `pom.xml`; explanation in build/environment docs | Never duplicate a version without “verified YYYY-MM-DD” and a recheck command. |
| Runtime property defaults and profile overrides | Source/resources; `video-editing-config-and-flags` is the maintained explanatory catalog | Runbooks name only the few overrides needed for that procedure and link to the catalog. |
| REST resources, schemas, status/errors, compatibility | **TARGET:** versioned OpenAPI document (currently absent) | Controller prose is an implementation note, not the public contract. |
| Database schema and migration order | `src/main/resources/db/migration/` | Data/runbook docs link to migrations and explain operation; do not reproduce full DDL. |
| Module responsibilities and dependency direction | **TARGET:** architecture overview plus context/module diagrams (currently absent) | ADRs explain decisions; code maps and README link to the overview. |
| Important design decision and alternatives | **TARGET:** one immutable-numbered ADR per decision (currently absent) | Plans and PRs link to the ADR; do not restate the decision history. |
| Alert response and recovery procedure | `docs/runbooks.md` or one linked specialized runbook | Alert definitions link to the procedure; dashboards do not duplicate it. |
| Test and quality acceptance | `video-editing-validation-and-qa` until a tracked testing strategy exists | Plans list intended coverage only; releases link to retained evidence. |
| Historical failure | `video-editing-failure-archaeology` | Current docs link to the incident/lesson rather than retelling it. |
| Change gates and promotion | `video-editing-change-control` | Every behavior-changing plan routes to it. |
| Release-specific versions, evidence, approvals, known limits | **TARGET:** immutable release record (currently absent) | README/runbooks link to the current supported release record. |
Propose any new canonical path through documentation review. Do not create a second home merely because the target home is absent.
## Label status and evidence explicitly
Put this block near the top of every maintained reference, runbook, plan, diagram, or report:
```text
Status: CURRENT | TARGET | CANDIDATE | HISTORICAL
Scope: <workflow, environment, and audience>
Last verified: YYYY-MM-DD
Applies to: commit <full-or-short-sha> | release <identifier> | proposed
Owner: <team or role; UNKNOWN if not assigned>
Evidence: <source paths, tests, measurement record, ADR, or NONE>
Next review: YYYY-MM-DD | event that forces review
```
Use labels strictly:
| Label | Meaning | Required wording |
|---|---|---|
| `CURRENT` | Matches source and has evidence for the bounded claim. | Say exactly what was verified, where, when, and on which environment. |
| `TARGET` | Accepted requirement not yet demonstrated. | Use “must,” “target,” or “not implemented”; never “supports” or “provides.” |
| `CANDIDATE` | Proposed/hypothesized and awaiting decision or evidence. | Name owner, experiment/gate, and expiry or revisit condition. |
| `HISTORICAL` | Superseded, rejected, or incident context. | Name replacement/status and prohibit operational use. |
Qualify evidence separately: `SOURCE-INSPECTED`, `TESTED`, `MEASURED`, `REVIEWED`, or `UNVERIFIED`. A source-inspected branch is not tested; a passing structure test is not cinematic quality; a document checklist is not execution evidence.
Never use a bare `[x]` for a milestone. Use a table:
```markdown
| Milestone | Status | Evidence | Last verified | Remaining gate |
|---|---|---|---|---|
| Local-model voiceover | TARGET | `NONE` | 2026-07-21 | Offline licensed model; intelligibility and timing measurements; approval |
```
## Follow the maintenance workflow
1. **Classify the document.** Choose reference, runbook, contract, decision, plan, evidence report, prompt, or historical record. Do not combine target design and current operation without visibly separate sections.
2. **Select one fact owner.** Use the table above. If the home is missing, add it to the backlog; do not scatter temporary copies.
3. **Build a claim ledger.** For each normative/current sentence, record claim, status, evidence, verification command, and volatility.
4. **Inspect implementation.** Read the relevant source, resources, tests, migrations, and history. A class name proves existence only, not wiring or success.
5. **Write in imperative order.** State purpose and safety boundary, prerequisites, exact steps, expected observations, failure branches, rollback/stop condition, and evidence to retain.
6. **Synchronize all affected surfaces.** Use the matrices below. If an expected contract is absent, state the gap instead of inventing it.
7. **Verify every literal.** Check paths, class names, commands, flags, defaults, endpoint mappings, filenames, event names, and output fields against the current tree.
8. **Run non-mutating audits.** Do not start schedulers, bootstrap workers, download dependencies/models, render, use network, or mutate Git merely to validate prose.
9. **Route claims.** Send behavior/default changes and production-readiness claims through `video-editing-change-control`; attach validation evidence rather than declaring success in prose.
10. **Record maintenance.** Date volatile facts and add one-line re-verification commands at the end.
## Synchronize contracts
### Configuration
For any property statement, compare all applicable locations:
```bash
rg -n 'class VideoClippingProperties|@ConfigurationProperties|enabled|auto-start|render-enabled|require.*approval|fallback|strict-runtime' src/main/java/org/example/videoclips/config src/main/resources src/test
rg -n 'VIDEO_EDITING_|FOLDER_SCHEDULER_|video-clipping\.' docs tools src/main/resources src/main/java
```
Check the Java default, packaged default, profile override, environment mapping, validation, component guard, tests, operational risk, and rollback. Defer the full catalog to `video-editing-config-and-flags`.
### API
For any endpoint statement, compare controllers, DTOs, exception mapping, application service, and API tests:
```bash
rg -n '@(RequestMapping|GetMapping|PostMapping|PutMapping|PatchMapping|DeleteMapping)|ProblemDetail' src/main/java/org/example/videoclips/api
find src/main/java/org/example/videoclips/api/dto src/test/java/org/example/videoclips/api -type f | sort
find . -maxdepth 3 -type f \( -iname '*openapi*' -o -iname '*swagger*' \) -print
```
The last command currently returns nothing. Do not claim OpenAPI publication, build validation, compatibility checking, authentication, pagination, or idempotency unless separately evidenced.
### Runtime file layout
For any input/output path, compare packaged configuration, folder-contract/constants, initializer, store, scheduler, tests, and the runbook:
```bash
rg -n 'Path |resolve\(|createDirectories|source-directory|working-directory|processed-directory|rejected-directory|project-directory|highlight-project-directory' src/main src/test
rg -n 'input/|output/|target/|project.json|edit-plan.json|render-manifest.json|qa-report.json' docs
```
Distinguish repository-relative defaults, configurable paths, generated/disposable `target/` output, and production volume contracts. Never use a private absolute path.
### Operations, metrics, and deployment
Compare event/metric source with alert, dashboard, and runbook names:
```bash
rg -n 'event=|Counter|Timer|Gauge|Observation|MeterRegistry|HealthIndicator' src/main/java
rg -n 'video_clipping_|http_server_|event=|/actuator/' docs dashboards
find . -maxdepth 3 -type f \( -iname 'Dockerfile*' -o -iname '*compose*' -o -iname '*helm*' -o -iname '*k8s*' \) -print
```
The deployment-file search currently returns nothing. Label Kubernetes, HPA/KEDA, cloud, backup, restore, and rollout material as target guidance or environment procedure unless executed evidence exists.
## Run the documentation audits
### Broken or stale path audit
```bash
git ls-files | sort > /tmp/video-editing-tracked-files.txt
rg -n '`(src|docs|tools|dashboards)/[^`]+`|`pom.xml`|`README.md`' docs
rg -n 'FolderSchedulerProperties|README|openapi|swagger|ADR|Dockerfile|compose|helm|k8s' docs
find docs src/main/resources tools dashboards -type f -print | sort
```
Inspect every literal path against the tracked-file list. Treat placeholders such as `<project-id>` as patterns, not files. Flag references to generated `target/` artifacts unless the document provides a reproducible generator and explicitly calls them disposable.
### Milestone and overclaim audit
```bash
rg -n '^- \[[ xX]\]|complete|implemented|production[- ]ready|validated|verified|signed off|supports|automatic today' docs
rg -n 'pending|not implemented|not yet|still manual|limitation|known gap|unchecked|heuristic|fallback|placeholder|proposed|intended' docs
git log --format='%h %ad %s' --date=short -- docs
```
Pair every positive claim with contrary qualifiers and implementation evidence. Mandatory current findings include the highlight-plan/rendering contradiction and the load-signoff contradiction listed above. Preserve old milestones as `HISTORICAL` when useful; do not erase a failed or superseded approach.
### Source drift audit
```bash
git diff --name-only HEAD -- pom.xml src/main src/test tools dashboards docs
rg -n '<java.version>|spring-boot-starter-parent|<version>|<artifactId>' pom.xml
rg -n '^video-clipping:|^[[:space:]]+[a-z0-9-]+:' src/main/resources/application*.yml
find src/main/resources/db/migration -maxdepth 1 -type f -print | sort
find src/test/java -name '*Test.java' -print | sort
```
When code changes without docs, identify the canonical document affected. When docs change without code, verify that they remain intent-only or accurately describe existing behavior. Do not use line/test counts as stable claims unless date-stamped and regenerated.
### Command and example audit
For each command, verify executable, working directory, required environment, side effects, network behavior, expected exit/result, and cleanup. Prefer a focused read-only or test command. Never recommend `mvn spring-boot:run` as a documentation lint because current defaults can consume input, start workers, install dependencies, resolve models, and render.
Use these syntax/content checks after editing Markdown:
```bash
git diff --check -- .claude/skills docs README.md 2>/dev/null
rg -n '^```' .claude/skills docs
rg -n '[/]Users/|[/]home/|[A-Za-z]:\\\\|api[_-]?key|secret|password|token' .claude/skills docs
```
Review fence pairs manually or with an approved Markdown linter if one is later added. No Markdown lint/build is configured in `pom.xml` today.
## Use these document templates
### ADR
```markdown
# ADR-NNN: <Decision>
Status: CANDIDATE | ACCEPTED | SUPERSEDED
Date: YYYY-MM-DD
Owners: <roles>
## Context
Requirement, constraints, current evidence, and trust boundaries.
## Decision
Selected approach and dependency direction.
## Alternatives Considered
Option, rejection reason, and evidence.
## Consequences
Benefits, trade-offs, operational consequences, and security implications.
## Verification
Tests, measurements, scans, and approval required.
## Revisit Conditions
Specific signal, date, dependency change, or failed threshold.
```
For every major decision include the requirement, selected approach, alternatives, benefits/trade-offs, operational consequences, security implications, verification, and revisit conditions. Do not mark an ADR accepted without the authority required by change control.
### Incident record
```markdown
# Incident: <Observed symptom>
Status: OPEN | MITIGATED | RESOLVED
Window/Environment/Build: <UTC times, environment, commit/release>
Impact: <measured user/data/media effect>
Detection: <alert, report, or operator observation>
Evidence preserved: <logs, traces, metrics, manifests, checksums; redacted>
Timeline: <UTC event/action/result>
Root cause: <mechanism, or UNKNOWN>
Contributing conditions: <bounded facts>
Rejected hypotheses: <experiment and result>
Mitigation and recovery: <approved action and verification>
Corrective actions: <owner, due date, gate>
Recurrence test: <exact automated or drill evidence>
```
Move the durable symptom -> cause -> evidence -> status lesson into `video-editing-failure-archaeology`; keep sensitive operational data outside the repository.
### Operational runbook
```markdown
# <Alert/Symptom> Runbook
Status/Owner/Last verified/Applies to: <metadata>
Trigger and user impact: <measurable conditions>
Safety boundary: <actions requiring approval; data to preserve>
Prerequisites: <role, environment, tools>
Triage: <read-only command -> expected result -> branch>
Mitigation: <smallest approved reversible action>
Verification: <metrics, traces, state, and user journey>
Escalation: <threshold, role, evidence bundle>
Rollback/Recovery: <procedure and stop condition>
Known gaps: <explicit target work>
```
Never place an unapproved database mutation, file deletion, scheduler start, render, dependency/model download, or production-default change in a “quick fix.”
### Diagram
Use text-source diagrams that can be reviewed in diffs. Label every box as implemented, target, or external; show trust boundaries, protocols, state stores, local model processes, ownership, and dependency arrows. A context diagram shows users/external systems. A module/container diagram shows deployable/process/module boundaries. A runtime-flow diagram shows ordering, state transitions, failure paths, approval, retry/idempotency, and observability.
Validate every node against source or an accepted ADR. Do not draw target PostgreSQL, OAuth/OIDC, OpenTelemetry, containers, Kubernetes, model registry, or cloud services as current.
### Release record
```markdown
# Release <immutable identifier>
Commit/artifact digest/build provenance/SBOM: <references>
Environment promotions and approvals: <records>
Schema/API/config/default deltas: <old -> new>
Model/asset versions, checksums, origins, licenses: <inventory>
Security and dependency scan result: <evidence>
Test/quality/load/resilience evidence: <evidence and thresholds>
Known limitations/risk acceptances/expiry: <records>
Migration/rollback/smoke verification: <commands and results>
Owners/RTO/RPO/support window: <facts>
```
Do not create release notes from commit subjects alone. Verify user-visible behavior and retained evidence.
## Maintain the Fortune 500 documentation backlog
Record these as **TARGET/MISSING**, not complete. Create them only through reviewed work; keep one fact home as defined above.
| Priority | Required artifact | Current evidence/gap |
|---:|---|---|
| P0 | README with prerequisites, safe local setup, architecture map, test/config/deploy links, representative API requests, and troubleshooting | No `README.md`. |
| P0 | Versioned OpenAPI plus build validation and breaking-change gate | Controllers/tests exist; no OpenAPI artifact/plugin. |
| P0 | Architecture overview: context, modules/dependency direction, and critical runtime flows | No architecture diagram or enforced module document. |
| P0 | Security/threat-model summary: trust boundaries, assets, threats, mitigations, auth, secrets, audit/redaction | No threat model; Spring Security is not in the current POM. |
| P0 | Testing strategy and evidence inventory: unit, architecture, slices, PostgreSQL/Testcontainers, contracts, E2E, mutation, coverage | Tests exist, but no comprehensive strategy or required enterprise gates. |
| P0 | Local development guide for macOS and reproducible Linux/VPS/cloud build/runtime | Existing runbooks are workflow-specific; clean-checkout reproducibility is not established. |
| P0 | Deployment/operations guide: immutable OCI artifact, config/secrets, health, graceful shutdown, resources, rolling deploy, rollback/roll-forward | No image or deployment definitions; existing operational docs are templates/guidance. |
| P0 | Local-model/media supply-chain and licensing record | Current workers can bootstrap/download/fallback; model and asset provenance is not production-certified. |
| P1 | Configuration reference generated/checked against typed properties, resources, profiles, guards, and secret policy | Use `video-editing-config-and-flags` now; no product-facing reference artifact. |
| P1 | ADR index and ADRs for modular-monolith boundaries, persistence/queue/storage, local model isolation, approval gate, file contracts, observability, and deployment | No ADRs. |
| P1 | Observability/SLI/alert/runbook map and dashboard validation | Docs/dashboard exist; traces, deployed evidence, owners, and full business SLIs remain incomplete. |
| P1 | Data operations: schema strategy, immutable migrations, UTC/concurrency/index decisions, backup/restore, retention, RTO/RPO | Flyway V1-V6 and procedure prose exist; no real PostgreSQL restore evidence or assigned RTO/RPO. |
| P1 | CI/CD, quality-gate, dependency-update, branch-review, SBOM/signing/provenance, and promotion policy | No CI workflow or release provenance record. |
| P1 | Performance/capacity methodology and executed workload reports | Local baselines exist; no checked-in production-like load generator/signoff. |
| P2 | Ownership/contribution policy, glossary, support/escalation map, changelog/release records, and documentation review cadence | No contributor/ownership/release docs found. |
Treat existing plans as inputs to this backlog, not substitutes for these records.
## Apply the house style
- Address the operator directly with imperative verbs: “Run,” “Verify,” “Stop,” and “Escalate.”
- Define each project/domain term once. Link to its owner thereafter.
- Put prerequisites before commands and expected observations immediately after them.
- Give discriminating branches: “If X, do Y; if not, stop and collect Z.”
- Use repository-relative paths and exact case. Put literals in backticks.
- Use UTC for incidents/releases; use ISO `YYYY-MM-DD` for volatile fact dates.
- Separate `CURRENT`, `TARGET`, `CANDIDATE`, and `HISTORICAL` content visibly.
- Use tables for catalogs/comparisons and checklists for actions, not for proof.
- State negative facts plainly: “No OpenAPI artifact exists,” not “OpenAPI is forthcoming.”
- Keep examples realistic but sanitized. Label illustrative IDs, hosts, values, and outputs.
- Explain rationale and evidence; do not narrate obvious syntax or advertise the architecture.
- Preserve limitations, failed experiments, and uncertainty. Write `UNKNOWN` rather than guessing.
## Definition of done for a documentation change
- [ ] The document has one type, owner, status, scope, date, applicable commit/release, evidence, and review trigger.
- [ ] Every current claim is bounded and evidenced; every target/candidate statement is labeled.
- [ ] The fact has one canonical home and sibling documents cross-reference rather than duplicate it.
- [ ] Paths, commands, config values, endpoints, event/metric names, file layouts, versions, and tests were rechecked.
- [ ] Commands state prerequisites, side effects, expected observations, failure branch, and avoid prohibited actions.
- [ ] API/config/schema/file-layout/operations surfaces were synchronized or an explicit drift issue was recorded.
- [ ] Security, licensing, privacy, approval, and network boundaries are explicit where applicable.
- [ ] Behavior/default/production claims went through `video-editing-change-control` with evidence from the correct sibling skill.
- [ ] No private path, secret, proprietary input, generated-only source, or unverified production claim was embedded.
- [ ] `git diff --check` and the audits above pass or each remaining finding is recorded with owner/status.
## Provenance and maintenance
Repository baseline and volatile findings were source-inspected on **2026-07-21**. Re-verify before use:
```bash
git log -1 --format='%H %cs %s'
test -f README.md; find .github -maxdepth 3 -type f -print 2>/dev/null; find . -maxdepth 3 -type f \( -iname '*openapi*' -o -iname '*adr*' -o -iname 'Dockerfile*' -o -iname '*compose*' \) -not -path './target/*' -print
rg -n '^- \[[ xX]\]|production[- ]ready|signed off|execution evidence pending|still manual|not wired|limitation|known gap' docs
rg -n 'FolderSchedulerProperties|HighlightDirectorFlowService|HighlightFfmpegRenderer|render-enabled|require-director-approval' docs src/main src/test
rg -n '@(RequestMapping|GetMapping|PostMapping|PutMapping|PatchMapping|DeleteMapping)|ProblemDetail' src/main/java/org/example/videoclips/api
rg -n '<java.version>|spring-boot-starter-parent|jacoco-maven-plugin' pom.xml
find src/main/resources/db/migration -maxdepth 1 -type f -print | sort
git diff --check -- .claude/skills docs README.md 2>/dev/null
```

View File

@ -0,0 +1,318 @@
---
name: video-editing-external-positioning
description: Load this skill when drafting or reviewing public claims, release notes, papers, architecture case studies, model cards, benchmark comparisons, production-readiness statements, Fortune 500 reference-architecture assessments, or claims that this video-editing service is cinematic, local, offline, secure, scalable, reproducible, or state of the art. Use it to assemble the evidence package and prevent repository capabilities from being overstated.
---
# Position This Project Externally
## Purpose
Turn repository facts into claims that an independent reviewer can reproduce. Treat code,
plans, tests, executed measurements, and accepted claims as different evidence levels.
Use the repository state and rules dated **2026-07-21**:
- The target is a production-grade reference architecture suitable for a Fortune 500
engineering organization.
- The hardest live problem is identifying highlights and editing them to high cinematic
quality with production-ready visuals, sound, music, and voiceover using models packaged
inside the service runtime.
- Automatic dependency or model downloads, external AI services, unlicensed assets,
placeholder silence or tones, unapproved rendering, network access, and changes to
production-facing defaults are prohibited.
- Required environments are macOS development, Linux/VPS production, and cloud
infrastructure.
Do not weaken these rules to make a release or paper easier to announce.
## When Not To Use This Skill
| Need | Load this sibling instead |
|---|---|
| Classify, gate, or approve a code/config change | `video-editing-change-control` |
| Decide whether tests and measurements meet acceptance thresholds | `video-editing-validation-and-qa` |
| Turn a hypothesis into an experiment and survive adversarial review | `video-editing-research-methodology` |
| Plan work on cinematic highlight quality | `video-editing-cinematic-highlights-campaign` |
| Document build prerequisites or reproduce the runtime | `video-editing-build-and-env` |
| Maintain repository documentation rather than external claims | `video-editing-docs-and-writing` |
Do not use this skill to approve a claim about your own change. Require an independent
reviewer and the gates below.
**SBOM** means Software Bill of Materials.
## Define The Claim Vocabulary
Use these terms consistently:
| Term | Meaning here |
|---|---|
| Technique | A known engineering method used by the repository, such as FFmpeg filtering, Flyway migrations, ports/adapters, or Micrometer metrics. Its presence is not novelty. |
| Capability | Behavior demonstrated by code plus an executable test or retained run artifact for a named commit and environment. |
| Candidate | A plausible design, result, or novelty that has not met its evidence gate. |
| Claim | A bounded statement with named scope, baseline, metric, threshold, commit, environment, and evidence owner. |
| Evidence package | Immutable inputs, commands, outputs, measurements, provenance, negative results, and independent review needed to audit a claim. |
| Production-ready | Verified against the entire production definition of done, not merely able to start or render an MP4. |
| Local | Computation executes on the declared host or contained runtime. This does not imply offline, bundled models, no telemetry, or no download. |
| Offline | A clean, isolated run completes with egress denied and all dependencies, weights, and assets supplied by an approved artifact. |
| Reproducible | An independent operator can recreate a declared result from an immutable revision and artifact set within stated tolerances. Do not imply byte-for-byte identity unless tested. |
| Cinematic quality | A preregistered technical and blinded-human evaluation passes. File validity or the presence of effects is insufficient. |
| State of the art (SOTA) | A statistically defensible improvement over current, relevant external baselines established from approved primary evidence. |
## Start From The Current Claim Ledger
Treat this table as the default posture until newer retained evidence replaces it.
| Area | Repository fact | Allowed wording | Forbidden inference |
|---|---|---|---|
| Platform | `pom.xml` declares Java 21, Spring Boot 3.3.2, Maven, PostgreSQL, Flyway, S3, Actuator, Prometheus, H2, and test dependencies. | "The repository currently declares ..." | "Latest," "supported," "locked," "hardened," or "vulnerability-free." No approved current external version/vulnerability evidence exists. |
| Editing | Code contains source analysis, candidate/director flow, FFmpeg renderers, local asset integration, manifests, and QA report types. | "Implements an in-repository cinematic highlight pipeline candidate." | "Produces production-ready cinematic edits." |
| Visual models | A local HTTP CV adapter and worker exist; configuration defaults to `local-cv` and permits heuristic fallback. | "Supports a local CV worker adapter with configurable heuristic fallback." | "All highlight decisions are model-driven," "fully local," or "offline." |
| Audio models | `tools/local_asset_worker.py` invokes Piper when configured and AudioCraft model loaders for music/SFX. | "Contains adapters for local TTS and generative audio runtimes." | "Models reside in the service runtime," "no downloads," or "production voice/music/SFX." |
| Fallbacks | Historical output may contain silence, tones, or host speech. The 2026-07-21 working tree removes those success paths and rejects missing requested assets. | Identify the exact source revision and asset provenance. | Any cinematic or real-asset claim for an older output or an output without model/asset manifests. |
| Downloads | Worker bootstrap scripts run unpinned `pip install`; model APIs can resolve named weights. Local CV documentation says startup may download dependencies/YOLO weights. | "Current bootstrap paths may access package/model networks." | "Hermetic," "offline," "dependency-locked," or "no automatic downloads." |
| Approval | Highlight rendering now defaults off and approval on; the separate local-director profile also requires a flag. The REST edit-render endpoint and bare flag design still lack production authorization. | State the exact workflow, effective configuration, and approval limitation. | "All rendering requires authenticated, digest-bound human approval." |
| QA | `HighlightFfmpegRenderer.buildQaReport` now probes duration, black ranges, long silence, and sample peaks; asset/mastering and overlay checks remain structural or plan-level. | "Writes a technical QA report with measured defect probes and structural checks." | "QA-certified," "validated loudness," "raster-safe overlays," or "measured cinematic quality." |
| Tests | The suite contains unit, Spring, and FFmpeg-backed tests. A 2026-07-21 offline working-tree `mvn -q -o verify` run passed 245 tests in 62 test classes. | State the command, tree state, date, and attached report. | "Clean-checkout reproducible," "network-isolated," or cross-platform. An earlier clean archived checkout failed because an asset-generation test depended on untracked runtime state or an audible TTS fallback. |
| Operations | Plans, benchmark harnesses, runbooks, alerts, and a Grafana dashboard exist. `docs/load-test-signoff.md` explicitly says execution evidence is pending. | "Operational planning artifacts exist." | "Load-tested," "cloud-ready," "operationally signed off," or "scalable." |
| Enterprise controls | No checked-in CI definition, Maven wrapper, container/deployment manifest, OpenAPI document, architecture-boundary gate, Testcontainers setup, SBOM/signing gate, Spring Security dependency, or threat model was found in Phase 1. | List each as a current gap. | "Secure," "Fortune 500 reference architecture," "cloud-native," or "release governed." |
| Licensing | Asset lookup can copy adjacent `.license.txt` files, but the repository has no complete model/asset/license inventory. | "Supports an adjacent license-file convention for selected assets." | "All assets/models are licensed" or "commercially usable." |
| Novelty | The repository combines known local analysis, planning, asset, and FFmpeg techniques around filesystem project contracts. | "Repository-specific integration candidate." | "Novel architecture," "research contribution," or "SOTA." No novelty search or external comparison is approved. |
Plans and checked boxes prove intent or implementation bookkeeping only. They do not replace
executed evidence. Generated files under `target/`, `input/`, `output/`, or local virtual
environments are not certified evidence unless an evidence manifest names and hashes them.
## Apply The Claim Ladder
Promote wording one level at a time. Never skip a level.
| Level | Required support | Permitted verbs |
|---|---|---|
| 0: Intended | Plan or prompt only | "plans," "targets," "proposes" |
| 1: Implemented | Reviewed code at an immutable commit | "contains," "implements," "configures" |
| 2: Tested | Deterministic automated test passes from a controlled checkout | "tests demonstrate," with scope |
| 3: Measured | Retained representative artifacts and preregistered metrics pass | "measured," with numbers and uncertainty |
| 4: Independently reproduced | Separate reviewer/environment recreates the result | "reproduced on ..." |
| 5: Production qualified | Every applicable enterprise gate passes and operations approve | "production-qualified for [declared scope]" |
Reject adjectives without a level and scope. Replace "fast" with throughput/latency under a
named workload; replace "high quality" with rubric scores and confidence intervals; replace
"secure" with the verified control set and unresolved risk count.
## Gate High-Risk Claims
Build the evidence listed below before using the claim in a paper, release, presentation, or
README. A result can satisfy one row without satisfying another.
| Claim | Minimum evidence package |
|---|---|
| **Production-ready output** | Immutable source rights; no placeholder assets; model/asset provenance; approved director plan; measured media QA; blinded human review; failure/recovery results; retained final, manifest, logs, config, and checksums. |
| **Cinematic quality** | Preregistered category-balanced dataset; baseline outputs; randomized blinded review by declared reviewers; shot relevance, narrative coherence, pacing, visual finish, speech intelligibility, music/SFX fit, mix quality, and overall preference metrics; inter-rater agreement; confidence intervals; failure examples; technical QA. |
| **Local models in runtime** | Image/package inventory; exact model IDs, weight hashes, licenses, paths, memory/accelerator needs; startup/run traces showing local processes; egress-denied success; no host-only fallback; no runtime installation or model resolution. |
| **Offline** | Clean environment with network namespace/egress deny; preloaded approved bundle; packet/connection evidence; cold-start plus full source-to-final run; proof that caches were not inherited; failure if an undeclared artifact is removed. |
| **Licensed assets** | Source, owner, SPDX or verbatim license, commercial/derivative rights, territory/expiry if applicable, file hash, attribution obligation, and approval for every source, font, LUT, music, SFX, voice, model code, and weight. Generated output also requires model/weight/output-use analysis. |
| **Secure** | Threat model; authentication/authorization tests; OWASP ASVS 5.0 scope and evidence; secret/static/dependency/container scans; zero unresolved critical/high findings or recorded risk acceptance; least-privilege runtime; data/log redaction; audit evidence; secured actuator surface. |
| **Scalable** | Documented representative workload and hardware; p50/p95/p99, throughput, error rate, CPU, memory, GC, disk, threads, pools, queue age, and model saturation; 60-minute endurance; 2x peak; slowed-dependency test; horizontal-scale evidence; bottleneck and capacity limits. Do not inherit the user's example API numbers without adapting them to media workloads. |
| **Reproducible** | Clean checkout; one documented command; locked Java/Maven/Python/native/model inputs; checksums; no network for model/media work; macOS and Linux/VPS runs; one platform-neutral application artifact plus signed target-specific runtime/model/image bundles promoted without rebuild; deterministic seeds where applicable; declared numeric/media tolerance; independent reproduction. |
| **SOTA** | Approved primary-source review with dated search protocol; strongest relevant public baseline reproduced or obtained from auditable artifacts; same rights-cleared dataset and budget; preregistered metrics; uncertainty/significance; ablations; negative results; external review. |
| **Fortune 500 reference architecture** | Entire definition of done passes. Score all 15 human-review categories 0-4 with concrete evidence: architecture, domain, maintainability, security, testing, API, data, resilience, observability, performance, cloud operation, CI/CD, developer experience, documentation, and operations. No score below 3; average at least 3.5; security, data, testing, and operations each at least 3. |
For the Fortune 500 claim, also require clean build/run, architecture gates, API contract,
database migration from empty PostgreSQL, logs/metrics/traces/health/dashboards, load and
resilience results, secure container evidence, deployment and rollback exercises, ADRs, and
zero critical behavior dependent on undocumented assumptions.
## Build A Defensible Cinematic Comparison
Do not choose only flattering footage. Freeze the evaluation before rendering.
1. Define intended users, distribution format, input constraints, output duration, content
categories, hardware budget, latency budget, and prohibited content.
2. Create a rights-cleared, immutable, checksum-indexed dataset covering family, food, car,
generic, low light, sparse events, noisy audio, no audio, long takes, and failure cases.
3. Freeze these internal baselines where applicable:
- Source excerpt or uniform-window selection: measures whether highlight identification adds value.
- Heuristic visual provider: measures the local CV model's contribution.
- Deterministic plan: measures the director's contribution.
- Source-audio-only render: measures generated or licensed audio assets.
- Historical `5d889b0` ("working version but not cinematic"): use only after its runtime and
exact outputs are reproducibly reconstructed.
4. Add an external baseline only after approved primary evidence and artifacts are vendored or
cited. Network access is prohibited during this workflow; do not rely on memory for current
products, versions, papers, benchmarks, or licenses.
5. Match source clips, output duration, compute budget, allowed assets, and human-review process.
6. Preregister metrics, thresholds, exclusions, stopping rule, statistical method, and all
hypotheses. Predict expected numbers before execution under
`video-editing-research-methodology`.
7. Randomize and blind human review. Hide system names, filenames, and render order. Retain raw
scores and rejected samples.
8. Run ablations: remove local CV, director, generated music, SFX, voiceover, and each visual
treatment separately. Require one mechanism to explain positive and negative observations.
9. Report distributions and confidence intervals, not only averages or a montage of best cases.
10. Send the result through `video-editing-validation-and-qa`, then change control, before
promotion.
No cinematic acceptance threshold is currently agreed or evidenced. Keep the claim at
"candidate" until the campaign establishes thresholds and passes them.
## Record Artifact, Model, And License Provenance
Create one immutable manifest per evidence package. Store it in the approved evidence location
defined by change control; do not silently add repository paths from this skill.
```yaml
claim_id: <stable-id>
claim_text: <bounded proposed wording>
claim_level: <0-5>
commit: <40-character-sha>
created_utc: <timestamp>
owner: <team-or-role>
independent_reviewer: <team-or-role>
environments:
- os: <macOS-or-Linux-distribution-and-version>
architecture: <arm64-or-amd64>
cpu: <model-and-count>
memory_bytes: <integer>
accelerator: <model-driver-runtime-or-none>
commands:
- <exact-command>
network_policy: <egress-denied-evidence-reference>
inputs:
- path: <logical-artifact-name>
sha256: <hex>
source: <approved-origin>
license: <SPDX-or-verbatim-reference>
models:
- id: <model-and-revision>
weights_sha256: <hex>
code_license: <reference>
weights_license: <reference>
output_use_review: <approval-reference>
outputs:
- path: <logical-artifact-name>
sha256: <hex>
metrics: <raw-and-summary-result-reference>
negative_results: <retained-result-reference>
limitations: <known-boundaries>
decision: <rejected-candidate-or-approved>
```
Reject provenance that says only "local," "open source," "free," or a mutable model alias. A
cache hit is not proof that a model is bundled, approved, or available on a clean machine.
## Prove Cross-Environment Reproducibility
Use the same immutable application and model/asset bundle in all environments.
| Environment | Required proof |
|---|---|
| macOS development | Clean checkout; documented toolchain; no IDE dependency; egress denied; cold cache; source-to-final workflow; test and artifact logs. Record Apple/Intel architecture. |
| Linux/VPS production | Non-root runtime; explicit CPU/RAM/GPU/disk; egress denied; read-only filesystem except declared work volumes; signal/shutdown behavior; cold start; repeated render; recovery after restart. |
| Cloud | Same release artifact; externalized configuration/secrets; least privilege; readiness/liveness; resource limits; telemetry; rolling deploy and rollback; dependency outage and scale tests; no environment rebuild. |
Define reproducibility tolerance before execution. For media output, compare timeline decisions,
streams/codecs, duration, frames or perceptual hashes, loudness, clipping/silence, and human
scores as appropriate. Never promise byte identity across FFmpeg, codec, CPU, or accelerator
variants without proving it.
## Review Every Claim
Copy this checklist into the claim review:
- [ ] Quote the exact proposed sentence and intended audience.
- [ ] Bound the workflow, environment, content categories, commit, and date.
- [ ] Assign a claim-ladder level and an evidence owner.
- [ ] Separate repository facts, measurements, interpretations, and candidates.
- [ ] Name the fairest baseline and equalize inputs, compute, time, and rights.
- [ ] Preregister metrics, thresholds, hypotheses, exclusions, and stopping rules.
- [ ] Attach exact commands, raw logs, configs, immutable inputs/outputs, and SHA-256 hashes.
- [ ] Inventory every dependency, binary, model, weight, font, LUT, source, music, SFX, and voice license.
- [ ] Prove no automatic download, external service, network access, placeholder, or unapproved render occurred.
- [ ] Include negative results, known failure cases, uncertainty, and scope limits.
- [ ] Reproduce on every environment named by the claim.
- [ ] Obtain independent factual, security/license, validation, and change-control approval.
- [ ] Make release wording no stronger than the weakest required evidence item.
Reject the claim if any required box is unknown. "Unknown" is a result, not permission to infer.
## Write Release Notes Without Oversell
Use this template:
```markdown
## <Release or experiment name>
Revision: `<commit>`
Status: <experimental | candidate | qualified for bounded scope>
### Changed
- <observable behavior, not aspiration>
### Verified
- <command, environment, dataset, measured result, threshold, evidence ID>
### Not verified
- <production, security, scale, offline, license, quality, or platform gaps>
### Models and assets
- <immutable IDs/hashes, licenses, approval, whether bundled>
### Compatibility and operations
- <configuration/API/data change, rollout, observability, rollback>
### Negative results and limitations
- <failures, excluded cases, uncertainty, known weak points>
### Claim decision
- Proposed wording: <exact sentence>
- Evidence level: <0-5>
- Reviewers: <roles>
- Decision: <rejected | candidate | approved for exact scope>
```
Do not describe a plan, prompt, unchecked runtime path, synthetic fixture, or mocked test as an
end-user result. Link a release to retained evidence, not to a moving working directory.
## Route Claim Promotion
1. Open the candidate under `video-editing-research-methodology` when the claim depends on a
quality, novelty, mechanism, or comparative result.
2. Classify the implementation and any production-default impact under
`video-editing-change-control`. Never change defaults to make an evaluation pass.
3. Generate the complete evidence package without network access or prohibited fallbacks.
4. Ask `video-editing-validation-and-qa` for threshold and evidence review.
5. Assign an adversarial reviewer to reproduce the result and seek counterexamples.
6. Run security/license review for any public, customer, or commercial use.
7. Approve only the exact bounded wording. Record rejected wording and the reason.
8. Publish limitations and negative results beside the claim.
9. Reopen the claim when dependencies, models, datasets, runtime environments, defaults, or
relevant external baselines change.
## Provenance and maintenance
This skill was grounded on repository revision `97ba827d507292e38387c188a4b88157fd119ddb`
and the project owner's criteria on **2026-07-21**. Primary repository evidence included
`pom.xml`, `src/main/resources/application.yml`, local worker scripts, editing/rendering code,
tests, operational documents, highlight plans/checklists, and Git history. No network research
was performed; current external SOTA, version support, vulnerabilities, and licenses remain
unverified.
Re-verify volatile facts before every claim review:
```bash
git rev-parse HEAD
sed -n '1,260p' pom.xml
sed -n '1,180p' src/main/resources/application.yml
rg -n "pip install|get_pretrained|write_silence|fallback_tone|fallbackTone|writeSilence" tools src/main/java/org/example/videoclips/editing
rg -n "buildQaReport|new RenderQaCheck" src/main/java/org/example/videoclips/editing
find . -maxdepth 3 -type f \( -name 'mvnw*' -o -iname '*docker*' -o -iname '*compose*' -o -iname '*openapi*' -o -iname '*sbom*' \) -print
find . -maxdepth 3 -type f \( -path './.github/*' -o -name '.gitlab-ci.yml' -o -name 'Jenkinsfile' \) -print
rg -n "spring-security|SecurityFilterChain|ArchUnit|Modulith|Testcontainers|CycloneDX|dependency-check|OpenTelemetry" pom.xml src
sed -n '120,190p' docs/load-test-signoff.md
git log --oneline --all --decorate -80
```
Run build, media, security, load, offline, and cross-platform commands only through their
sibling runbooks. Retain their logs and hashes; a command printed here is not evidence that it
passed.

View File

@ -0,0 +1,388 @@
---
name: video-editing-failure-archaeology
description: Load this skill when investigating whether a video-editing failure, tempting fix, stale generated artifact, completed checklist item, deleted test, or unusual Git object has already been encountered in this repository. Use it before reopening storage, queue/DLQ, local-file cleanup, FFmpeg segment timing, cinematic-quality, local-model asset, port/configuration, or single-source highlight incidents, and before treating historical output as proof.
---
# Video Editing Failure Archaeology
## Purpose
Reconstruct what the repository actually proves before changing behavior. Treat a **root cause** as the mechanism demonstrated by code, a patch, a test, or a reproducible observation. Treat a commit subject, plan checkbox, and generated media as leads rather than proof.
Facts and workspace observations in this skill were checked on **2026-07-21**.
## Use This Skill When
- A symptom resembles an earlier storage, queue, filesystem, FFmpeg, render, or local-model problem.
- A plan says an item is complete but production evidence is unclear.
- An ignored or committed runtime artifact appears to prove a workflow works.
- Git history contains a deletion, rewrite, unreachable commit, or apparently abandoned approach.
- A proposed fix repeats an earlier partial fix.
## Do Not Use This Skill When
- Triaging a live symptom from scratch: use `video-editing-debugging-playbook` first, then return here after identifying the subsystem.
- Changing code, configuration, dependencies, defaults, or runtime behavior: use `video-editing-change-control`; this skill never authorizes a change.
- Deciding whether output meets acceptance thresholds: use `video-editing-validation-and-qa`.
- Looking up current defaults or environment wiring: use `video-editing-config-and-flags`.
- Operating or recovering a deployed instance: use `video-editing-run-and-operate`.
## Non-Negotiable Investigation Rules
1. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` during archaeology. They can create virtual environments, install packages, and download model weights.
2. Do not use network access, external AI services, unlicensed assets, placeholder silence or tones, or unapproved rendering to reproduce a result.
3. Do not change a production-facing default to test a theory.
4. Do not mutate Git history. Use `git log`, `git show`, `git diff`, and `git fsck`; never reset, rebase, clean, or delete artifacts.
5. Do not promote ignored media, a manifest, or a green structural test as cinematic-quality evidence.
6. Route every behavior change through `video-editing-change-control`, including a retry change, port change, fallback change, or FFmpeg flag change.
## Evidence Labels
Use these labels in incident notes and pull requests:
| Label | Meaning | Required evidence |
|---|---|---|
| `FIXED` | The demonstrated mechanism was changed and regression coverage exists. | Patch plus current code/test location. |
| `ACTIVE` | The mechanism remains in current code or reproducibly fails. | Current source or dated experiment. |
| `PARTIAL` | One failure mode was fixed, but adjacent obligations remain. | Explicit boundary and remaining gap. |
| `HISTORICAL` | The evidence describes an earlier tree, not necessarily current behavior. | Commit hash and date. |
| `OPEN` | Evidence shows a concern but not its cause or intent. | Known facts and the missing discriminator. |
| `NOT EVIDENCE` | The artifact cannot support the claimed conclusion. | Reason it is non-representative or unverified. |
## Fast History Triage
Run read-only checks from the repository root:
```bash
git status --short --ignored
git log --oneline --decorate --all --reverse
git log --all --name-status -- src/main src/test docs
git fsck --no-reflogs --unreachable --no-progress
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|open question|fallback|silence|tone' docs src tools
```
Before accepting a historical claim, inspect the exact patch:
```bash
export COMMIT="${COMMIT:?set COMMIT to a full commit ID}"
export FILE_PATH="${FILE_PATH:?set FILE_PATH to a repository-relative path}"
export OLDER_COMMIT="${OLDER_COMMIT:?set OLDER_COMMIT to a full commit ID}"
export NEWER_COMMIT="${NEWER_COMMIT:?set NEWER_COMMIT to a full commit ID}"
git show --format=fuller --stat "$COMMIT"
git show --format=fuller --no-ext-diff "$COMMIT" -- "$FILE_PATH"
git diff "$OLDER_COMMIT" "$NEWER_COMMIT" -- "$FILE_PATH"
```
## Chronicle At A Glance
| Date | Symptom or risk | Demonstrated mechanism | Status on 2026-07-21 |
|---|---|---|---|
| 2026-07-08/09 | Upload completion and clips existed as state/metadata without a complete storage path. | Storage operations and persisted object keys were absent from earlier paths. | `FIXED` for the implemented adapters; production S3 proof remains separate. |
| 2026-07-09 | Internal object locations leaked through normal API responses and logs. | `objectKey` and staged local path were returned/recorded directly. | `FIXED` by `0d45f89`; regression assertions remain. |
| 2026-07-09 | Worker files accumulated after success or failure. | Per-job staging/output lacked guaranteed deletion. | `PARTIAL`: finally cleanup and scheduled local cleanup exist; errors are swallowed and other editing trees are outside that job. |
| 2026-07-09 | Retryable DB-queue failure became terminal on the first exception. | `ClipProcessor` persisted `FAILED` before the queue decided retry versus DLQ. | `FIXED` by `1737d8b` for the DB queue contract. |
| 2026-07-10 | Folder clips could preserve quality or hit exact boundaries, but not both. | Stream copy cuts on source keyframes; exact cuts force keyframes by transcoding. | `ACTIVE` design trade-off; default selects preservation. |
| 2026-07-11 | Renderer worked but was explicitly “not cinematic.” | Exact cause was not recorded; one patch added a uniform grade, followed by broader planning/render/QA work. | `OPEN`: no certified creative-quality evidence. |
| 2026-07-12 onward | Single-source projects stopped at `CREATED` or `WAITING_FOR_DIRECTOR`. | Local artifacts lack a director plan/final output; current scanner requires `director/edit-plan.json`. | `ACTIVE` missing local-director capability; artifacts alone do not prove a regression. |
| 2026-07-21 | Workspace build passed while clean archived checkout failed local-asset tests. | Tests invoke the default untracked Python runtime and real host fallbacks instead of a hermetic fake. | `ACTIVE`; clean-checkout reproducibility is not established. |
## Case 1: Metadata-Only Storage Grew Into A Real Data Path
**Symptom.** API state could say upload or clip work completed while the service did not yet prove a durable source-to-generated-object path.
**Demonstrated mechanism.** The sequence is visible in the patches:
| Commit | Added missing responsibility |
|---|---|
| `7e8a214` | `uploads:complete` called `ObjectStoragePort.completeMultipartUpload` instead of only changing metadata. |
| `9df9736` | `VideoAsset.sourceObjectKey` was persisted and passed to clipping. |
| `bd620fc` | Source objects were materialized into local staging before FFmpeg. |
| `66e998e` | Generated clips carried local paths, were uploaded, and persisted an `objectKey`. |
| `7307082` | Signed download URLs used the persisted clip key instead of deriving a fake key from `clipId`. |
**Rejected partial fixes.** Do not reintroduce any of these:
- Marking upload complete without completing the provider upload.
- Saving clip rows before uploading the generated files.
- Deriving storage keys from public IDs when the persisted key is authoritative.
- Treating the in-memory adapter's no-op completion as proof that S3 completion works.
**Current invariant.** `ClipProcessor` must follow materialize source -> generate local clips -> upload each clip -> save each clip record -> mark the job successful. `ObjectStoragePort` owns materialization, upload, delete, and signed-download operations.
**Discriminate metadata drift from storage failure.** Read the job events and compare the clip record with backend existence. Do not expose the key in a client response merely to debug it. Inspect code with:
```bash
git show --stat 7e8a214 9df9736 bd620fc 66e998e 7307082
rg -n 'materializeSourceObject|uploadGeneratedClip|createClipDownloadUrl' src/main/java
rg -n 'source_object_key|object_key' src/main/resources/db/migration
```
**Status.** `FIXED` at the code-contract level. No checked-in production S3 end-to-end evidence makes the external system production-certified.
## Case 2: Object-Key And Local-Path Disclosure
**Symptom.** Normal API payloads exposed `sourceObjectKey`/clip `objectKey`, and a processing event included the staged local path.
**Root cause.** Commit `66e998e` initially added `objectKey` to `toClipResponse`; earlier asset mapping also exposed the source key. Internal routing identifiers crossed the API boundary. The materialization event also included a filesystem path.
**Fix.** Commit `0d45f89` removed those fields from normal responses and removed the path from the event message. Current assertions in `VideoAssetControllerTest` require `sourceObjectKey` and clip `objectKey` not to exist. Clients obtain a signed URL through the download endpoint.
**Wrong path.** Do not return a key “temporarily,” put it in Problem Details, or log the full local path. Diagnose through authorized backend tooling and redacted correlation identifiers.
```bash
git show --no-ext-diff 0d45f89 -- src/main/java src/test/java
rg -n 'sourceObjectKey.*doesNotExist|objectKey.*doesNotExist' src/test/java/org/example/videoclips/api
rg -n 'toAssetResponse|toClipResponse' src/main/java/org/example/videoclips/application/VideoAssetService.java
```
**Status.** `FIXED` for the tested response paths. Any new DTO, log, manifest, or error mapping must repeat the disclosure review through change control.
## Case 3: Retry And DLQ State Split
**Symptom.** A transient DB-queue processing exception marked the clip job `FAILED` on the first attempt even though the queue intended to retry it.
**Root cause.** The processor owned terminal failure persistence while the queue adapter owned attempt policy. The lower layer finalized state before the policy layer knew whether the attempt was terminal.
**Fix.** Commit `1737d8b` added `processNow(jobId, false)` for DB-queue attempts. The queue adapter now increments attempts, schedules a non-terminal `PENDING` retry with `RETRY_SCHEDULED`, and calls `markTerminalFailure` only when `max-attempts` is reached; terminal events are `DLQ` and `FAILED`.
**Do not repeat.** Do not mark a job failed in an adapter-independent catch block when a caller owns retry policy. Do not blindly redrive DLQ rows. `docs/dlq-redrive-procedure.md` requires root-cause correction, source existence, a small sample, and transactional row changes; the repository has no built-in redrive command.
**Discriminating observations.** Before the last attempt, expect message `PENDING`, job `QUEUED`, and a `RETRY_SCHEDULED` event. At exhaustion, expect message `DLQ`, job `FAILED`, then `DLQ` and `FAILED` events. A job already `FAILED` while its message is retryable indicates the old split-brain symptom or another writer.
```bash
git show --no-ext-diff 1737d8b -- src/main/java/org/example/videoclips/processing/ClipProcessor.java src/main/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java
rg -n 'RETRY_SCHEDULED|markTerminalFailure|setStatus\("DLQ"\)' src/main/java src/test/java
sed -n '1,320p' docs/dlq-redrive-procedure.md
```
**Status.** `FIXED` for the database-backed queue. The memory queue and any future external queue require their own semantics and tests.
## Case 4: Worker Temporary-File Accumulation
**Symptom.** Staged sources and generated local clips accumulated across successful and failed jobs.
**Root cause.** The processing path had no guaranteed cleanup boundary.
**Fixes.** Commit `8f4804c` put job-local deletion in `ClipProcessor.finally`, gated by `video-clipping.ffmpeg.cleanup-local-files`. Commit `3d961f2` added scheduled age-based cleanup for FFmpeg input/output, in-memory storage, and stub output.
**Limitations.** `LocalArtifactCleanupJob` catches and ignores `IOException`; deletion failure can be silent. Its roots do not include the cinematic edit and highlight project trees. Retention of durable storage/database records belongs to `RetentionCleanupJob`, not this local sweep.
**Wrong path.** Do not disable cleanup to make a failing test inspectable in production. Preserve a controlled diagnostic copy outside the worker path under an approved procedure.
```bash
git show --stat 8f4804c 3d961f2
rg -n 'cleanupLocalFiles|cleanupLocalArtifacts|local-artifact-retention' src/main/java src/main/resources src/test/java
du -sh tmp input output 2>/dev/null
find tmp -type f -mtime +1 -print 2>/dev/null | sed -n '1,100p'
```
**Status.** `PARTIAL`: job-local cleanup and a safety sweep exist, but cleanup observability and all editing artifact retention remain open.
## Case 5: Preserve Quality Versus Exact Segment Timing
**Symptom.** An “8-second clip” expectation conflicts with clips cut at nearby keyframes.
**Mechanism.** `FolderFfmpegClipper` has two mutually exclusive command shapes:
| `preserve-input-quality` | FFmpeg mode | Consequence |
|---|---|---|
| `true` (packaged default) | `-c copy` plus segment muxer | No generation loss; boundaries can drift to source keyframes. |
| `false` | `libx264`, CRF 20, forced keyframes, AAC | More exact segment boundaries; re-encode cost and generation loss. |
The trade-off is documented in `docs/input-folder-scheduler-plan.md`. The plan still asks whether exact timing or keyframe alignment is required, so “exact” is not a settled product requirement for this workflow.
**Wrong paths.** Do not call stream-copy output exact. Do not switch the default to transcoding to satisfy one fixture. Do not infer visual preservation solely from container/codec names.
```bash
rg -n 'preserve-input-quality|force_key_frames|segment_time|qualityMode' src/main docs/input-folder-scheduler-plan.md
git show --no-ext-diff aaacd7f -- src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java
```
**Status.** `ACTIVE` intentional trade-off. Any default or acceptance-threshold decision is a behavior change and must pass change control and measured validation.
## Case 6: “Working Version But Not Cinematic”
**Symptom.** Commit `5d889b0` records the only direct historical judgment: the render worked but was not cinematic.
**What the patch proves.** That commit added a fixed contrast/saturation/sharpen/vignette filter for non-empty visual treatments, plus compatibility/test wiring. It does not record a controlled comparison, reviewer rubric, or measured root cause.
**What followed.** Later commits added category-aware planning (`5006dfd`), stricter plan validation (`3348ae8`), local asset selection (`376bc41`), overlays (`43fccb7`), audio mastering (`bb23ac8`), dynamic crops (`2d0c331`), QA reports (`9b56e89`), and approval gating (`9536928`). This sequence identifies areas the project invested in; it does not prove each was causally necessary or that current output is cinematic.
**Rejected explanation.** “It needed a LUT/grade” is not established. A uniform grade was the immediate change, yet the repository then required planning, sound, motion, QA, and approval work.
**Current status.** `OPEN`. Structural render success, `qa-report.json`, and a committed manifest do not establish creative quality. Use the cinematic campaign and `video-editing-validation-and-qa`; require a licensed asset inventory, local-model provenance, objective media probes, and scored human review. Never fill missing audio with silence or tones.
```bash
git show --no-ext-diff 5d889b0 -- docs src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
git log --oneline 5d889b0..9536928
git show --stat 5006dfd 3348ae8 376bc41 43fccb7 bb23ac8 2d0c331 9b56e89 9536928
```
## Case 7: Single-Source Highlight Projects Stalled
**Observed on 2026-07-21.** Ignored workspace projects under `output/highlight-projects/` include one `CREATED` project and one `WAITING_FOR_DIRECTOR` project from 2026-07-12. Neither contains a final output. The later project contains analysis and a director prompt but no `director/edit-plan.json`.
**Separate stale-artifact trap.** Git tracks `output/edit-projects/source-clips/project.json` and `render-manifest.json`, which claim `RENDERED` and refer to `final.mp4`; Git does not track that media file. A clean checkout therefore contains success metadata without the claimed output. The `/input/`, `/output/`, and `/tmp/` ignore rules live in clone-local `.git/info/exclude`, not a shared repository ignore file. Do not use either the tracked manifest or this clone's ignored media as certified evidence.
**Current gate.** `HighlightDirectorPlanScanner` only selects a project with `director/edit-plan.json` and without `final.mp4`. `HighlightDirectorFlowService` returns `missing_director_plan` without that file and may also return `approval_missing` or `assets_pending`.
**Evidence boundary.** These projects were created after `618ba9a` and before `97ba827`. Their state is consistent with the documented missing-plan gate, so it is not proof that the flow implementation regressed. Current code generates a prompt and watches for a plan file; no runtime local-director model consumes the prompt and writes that plan. Their `manifest.json` also declares analysis filenames that do not exactly match every generated filename. Treat them as diagnostic snapshots, not goldens.
**Discriminate the gate without rendering.** Inspect only metadata and file presence:
```bash
find output/highlight-projects -maxdepth 3 -type f -print 2>/dev/null | sort
find output/highlight-projects -name project.json -exec sed -n '1,100p' {} \; 2>/dev/null
find output/highlight-projects -path '*/director/edit-plan.json' -o -name final.mp4 2>/dev/null
rg -n 'missing_director_plan|approval_missing|assets_pending|findNextRenderableProject' src/main/java/org/example/videoclips/editing
git ls-files output input tmp
```
**Status.** Artifact state is `HISTORICAL`; the absence of an in-runtime local director is `ACTIVE`, and end-to-end production readiness is `OPEN`. Reproduce only with approved, licensed local inputs and pre-provisioned local models. Do not let startup download anything or render without approval.
## Case 8: Local Asset Build Is Not Clean-Checkout Hermetic
**Observed on 2026-07-21.** The workspace suite passed, and its Surefire report shows `LocalAssetGenerationStageTest` invoked `./.venv-local-asset/bin/python`. That virtual environment exists locally but is untracked. A valid run from inside a clean archived checkout ran 216 tests and failed one assertion, `reusesExistingSharedAssetsAndSynthesizesMissingVoiceover`. An earlier two-failure run used `mvn -f` from the workspace and is invalid clean-checkout evidence because it violated the test working-directory assumption.
**Root cause.** The archived tests constructed the production `LocalAssetSynthesizer` with default configuration. They expected missing voiceover/SFX to become render-ready but did not provide a fake process. The default Python path pointed into the untracked virtual environment, host `say`/`espeak` availability affected voiceover, and the former worker could return success after writing silence or tones. **Status 2026-07-21:** the working tree removed those success fallbacks, deletes failed/inaudible output, and makes the affected test deterministic and fail-closed.
**Consequences.** The historical workspace pass was environment-coupled. The 2026-07-21 working tree removes placeholder success and makes strict readiness fail, but the bootstrap script can still install dependencies automatically and no complete resident model bundle is present. Re-run the clean archive before declaring the incident closed.
**Wrong paths.** Do not commit the virtual environment, enable network during tests, weaken strict mode, accept placeholder audio, or skip the test. Make tests hermetic and make production readiness fail closed through change control.
```bash
git ls-files .venv-local-asset tools/local_asset_worker.py tools/run_local_asset_worker.sh
rg -n 'new LocalAssetSynthesizer|readyForRender' src/test/java/org/example/videoclips/editing/LocalAssetGenerationStageTest.java
rg -n 'write_silence|write_fallback_tone|strict_runtime|strict-runtime|pip install' tools src/main src/main/resources
sed -n '1,120p' target/surefire-reports/org.example.videoclips.editing.LocalAssetGenerationStageTest.txt 2>/dev/null
```
**Status.** `ACTIVE`, release-blocking under the clean-checkout and no-placeholder requirements. Do not re-run bootstrap as a “fix.”
## Case 9: Port And Runtime-Shape Drift
**Observed configuration.** API examples assume Spring Boot's default port `8080`. The local CV HTTP worker defaults to `127.0.0.1:8091`, and the process manager derives host/port from its configured endpoint. `tools/run_local_asset_worker.sh` defines `LOCAL_ASSET_HOST` and `LOCAL_ASSET_PORT` defaulting to `8092`, but it launches a one-shot CLI script, not an HTTP server; those variables are unused by `local_asset_worker.py`.
**Interpretation.** This is verified semantic drift, not a documented outage. Do not diagnose local asset generation by probing port 8092, and do not “fix” it by moving another service to that port. First identify whether the component is HTTP (`local-cv`) or process-invoked (`local-asset`).
```bash
rg -n '8080|8091|8092|LOCAL_CV_PORT|LOCAL_ASSET_PORT|server.port' docs src tools
lsof -nP -iTCP -sTCP:LISTEN | rg ':(8080|8091|8092)\b'
rg -n 'ProcessBuilder|local_asset_worker.py|uvicorn' src/main/java/org/example/videoclips/editing tools
```
**Status.** `ACTIVE` documentation/configuration debt. No causal history establishes that a port collision caused a project failure.
## Case 10: Deleted Tenant Quota Test
**Evidence.** `TenantQuotaControllerTest` was added with tenant quota work and existed when `1737d8b` reported its pass. Commit `7fd92a7` deleted it while converting folder-scheduler configuration from properties to YAML. The commit message does not explain the deletion. A file named `input/source/TenantQuotaControllerTest.java.failed` is ignored local runtime debris, not the deleted test and not proof of intent.
**Conclusion.** Label the deletion `OPEN`. Do not claim the behavior was obsolete, flaky, or intentionally replaced unless a maintainer or equivalent coverage proves it.
```bash
git log --all --follow --format='%h %ad %s' --date=iso-strict -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
git show 7fd92a7^:src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
git show --no-ext-diff 7fd92a7 -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
rg -n 'quota|QuotaExceeded' src/test src/main
```
**Status.** `OPEN` test-coverage ambiguity. Restoring or replacing coverage requires change control; the historical file is evidence for intended behavior, not automatically the correct present test.
## Case 11: Rewritten Commits, Not Dead Feature Branches
**Observed on 2026-07-21.** `git fsck` reports six unreachable commits around the signed-download and disclosure fixes: `19ef4d3`, `5919210`, `65a29a4`, `84e6694`, `eb06021`, and `ba3f988`. Several share subjects, trees, parents, or near-identical changes with reachable `7307082` and `0d45f89`.
**Conclusion.** The evidence supports local amend/rewrite debris. It does not support a story about a reverted architecture or abandoned branch. The reachable history contains no commit whose subject is an explicit revert and only the main branch is present in the inspected repository.
```bash
git fsck --no-reflogs --unreachable --no-progress
git show -s --format='%H%nparents=%P%ntree=%T%nauthor=%ad%nsubject=%s%n' --date=iso-strict <unreachable-commit>
git branch -a -vv
git log --all --oneline --grep='^Revert'
```
**Status.** `NOT EVIDENCE` for a rejected solution. If unreachable objects disappear after Git maintenance, this conclusion should rely on the recorded hashes here and reachable patches, not object availability.
## Case 12: Completed Plans Are Not Production Signoff
**Symptom.** `docs/video-clipping-service-implementation-plan.md` marks every repository-scoped milestone complete, including backup/restore validation and load-test signoff.
**What the docs actually say.** The same plan says real backup/restore and production-like load execution remain. `docs/load-test-signoff.md` says `Documentation complete` and `Execution evidence pending`; it explicitly says no checked-in end-to-end environment load evidence exists. The plan's recommended baseline also differs from current `pom.xml` and the current Fortune 500 modular-monolith objective.
**Wrong path.** Do not translate `[x]` into deployed, exercised, secure, scalable, or production-ready. A document, dashboard JSON, benchmark harness, or procedure proves only its own existence until execution evidence is attached.
```bash
sed -n '688,760p' docs/video-clipping-service-implementation-plan.md
sed -n '28,180p' docs/load-test-signoff.md
rg -n 'not prove|pending|not executed|still must|template' docs/backup-and-restore-validation.md docs/load-test-signoff.md docs/dlq-redrive-procedure.md
rg -n 'spring-boot-starter-parent|<java.version>' pom.xml
```
**Status.** `ACTIVE` evidence-labeling risk. Use current code/configuration and dated execution records as truth; route documentation corrections through change control.
## Investigation Record Template
Use this structure in a new incident note or pull request:
```text
Symptom:
First known good / first known bad:
Scope and environment:
Evidence label: FIXED | ACTIVE | PARTIAL | HISTORICAL | OPEN | NOT EVIDENCE
Candidate mechanism:
Prediction before experiment:
Read-only or approved experiment:
Observed result:
Negative observations the mechanism also explains:
Historical analog and commit:
Rejected paths and why:
Behavior/config/security consequences:
Change-control classification:
Validation required before promotion:
Remaining uncertainty:
```
Do not fill the root-cause field until one mechanism explains the positive and negative observations. Record a disconfirming result; it prevents the next engineer from repeating the dead end.
## Provenance and maintenance
Primary evidence: reachable Git history through `97ba827`, current source/tests/configuration, planning and operations documents, current `.git/info/exclude`, Surefire reports, and dated ignored workspace artifacts, all inspected on 2026-07-21. Workspace artifacts and unreachable Git objects are volatile and are never load-bearing proof.
Re-verify the reachable incident sequence:
```bash
git log --oneline --all --reverse | rg '7e8a214|9df9736|bd620fc|66e998e|7307082|0d45f89|8f4804c|3d961f2|1737d8b|5d889b0'
```
Re-verify current storage, queue, cleanup, and segmentation contracts:
```bash
rg -n 'materializeSourceObject|uploadGeneratedClip|RETRY_SCHEDULED|markTerminalFailure|cleanupLocalFiles|preserve-input-quality|force_key_frames' src/main src/test src/main/resources
```
Re-verify local-model prohibitions and hermeticity risks without starting workers:
```bash
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|strict-runtime|auto-start|fallback-to-heuristic' tools src/main src/test src/main/resources
```
Re-verify volatile project artifacts and ignore rules:
```bash
git status --short --ignored
git ls-files input output tmp .venv-local-asset .venv-local-cv
sed -n '1,160p' .git/info/exclude
```
Re-verify ambiguity around rewrites and the deleted test:
```bash
git fsck --no-reflogs --unreachable --no-progress
git log --all --follow --name-status -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
```
Re-verify documentation claims against their own caveats:
```bash
rg -n 'All repository-scoped|Execution evidence pending|no checked-in evidence|Remaining work' docs
```

View File

@ -0,0 +1,440 @@
---
name: video-editing-proof-and-analysis-toolkit
description: "Load this skill after symptom triage when a video-editing change or incident must be proved from first principles: state transitions, timeline or keyframe accuracy, FFmpeg filter/audio graphs, loudness and true peak, highlight-ranking precision/recall/normalized discounted cumulative gain (nDCG), local-model output versus fallback, queue retry/DLQ/idempotency, filesystem atomicity/path confinement, capacity/cost, reproducibility, or causal debugging. Use it to establish or refute a causal mechanism before accepting a benchmark, QA report, cinematic-quality claim, fallback, or production-readiness claim as evidence."
---
# Video Editing Proof And Analysis Toolkit
## Purpose
Turn a claim into a prediction, a discriminating experiment, and an auditable conclusion. A **mechanism** is a causal account that explains the successful cases, failures, and negative controls with the same rules. A green test, an existing file, or a plausible story is not a mechanism.
Repository facts in this skill were verified on **2026-07-21**. Numeric baselines in `docs/` are historical workspace observations, not current acceptance thresholds.
## Use And Routing
Use this skill to derive what an observation should be and to distinguish competing causes. Use these siblings for the surrounding work:
| Need | Load instead or next |
|---|---|
| Classify, approve, implement, or promote a behavior change | `video-editing-change-control` |
| Triage a live symptom quickly | `video-editing-debugging-playbook` |
| Reconstruct an earlier incident or rejected fix | `video-editing-failure-archaeology` |
| Decide the complete acceptance suite or human review | `video-editing-validation-and-qa` |
| Interpret codec, color, audio, timing, or editing theory | `cinematic-media-engineering-reference` |
| Run approved local diagnostics | `video-editing-diagnostics-and-tooling` |
| Change flags or inspect effective configuration | `video-editing-config-and-flags` |
| Execute the hardest highlight-quality program | `video-editing-cinematic-highlights-campaign` |
Do **not** use this skill to authorize rendering, production mutation, DLQ redrive, dependency installation, or model acquisition. Analysis never routes around change control.
## Hard Safety Boundary
1. Work offline. Do not call external AI services or enable network access.
2. Use only pre-provisioned, licensed local models and licensed assets whose identity and checksum are recorded.
3. Do not invoke `tools/run_local_cv_worker.sh`, `tools/run_local_asset_worker.sh`, or application startup as an experiment. Current defaults can bootstrap runtimes, download packages/models, scan inputs, and render.
4. Reject placeholder silence and synthetic fallback tones as production assets. “A file exists” and “samples are nonzero” do not establish voice, music, or SFX validity.
5. Probe only existing approved media. Do not generate or render media without the approval gate in `video-editing-change-control`.
6. Never change a production-facing default to test a hypothesis. Use an isolated fixture and explicit test configuration after approval.
7. Keep diagnostic output in `target/` or an OS temporary directory. Never alter `input/`, `output/`, migrations, source media, or Git history.
8. Run Maven with `-o` during proof work so Maven fails on a missing cached dependency. Maven offline mode is not egress denial and does not prevent test code or subprocesses from networking; enforce network denial independently.
## The Proof Record
Write this record **before** running an experiment:
```text
Claim:
Competing mechanisms (at least two):
Assumptions and controlled variables:
Predicted numeric/structural observation for each mechanism:
Positive control:
Negative control:
Command and input checksums:
Observed result and uncertainty:
Which mechanisms were falsified:
Adversarial refutation still required:
Decision: reject | retain as candidate | accept for stated scope
```
Apply these rules:
- Predict a number, interval, ordering, state tuple, or file set before execution. “It should improve” is invalid.
- Change one causal factor at a time. Use an **ablation**: remove one component while holding all others fixed.
- Preserve negative results. One mechanism must explain both the expected positive and the expected absence under the negative control.
- Separate **structural validity** (schema, file, command, state) from **semantic validity** (correct highlight, intelligible voice, appropriate sound) and **operational validity** (load, crash, recovery).
- Report the denominator, sample selection, hardware, binary/model checksums, and uncertainty. Never promote a single clip or a single warm run.
## Recipe 1: Prove State-Machine Invariants
**Question.** Can an entity reach only legal states, and do queue message state and job state remain consistent?
**Assumptions.** Define the entity, authoritative writer, transaction boundary, and terminal states. Current enums are `ClipJobStatus`, `EditProjectStatus`, and `HighlightProjectStatus`; enum membership alone does not enforce transitions.
**Derive before running.** Draw allowed edges and forbidden edges. For every operation predict `(before, event, after, durable artifacts)`. Require terminal-state monotonicity unless an explicit, audited recovery operation exists.
**Repo-safe experiment.** Inspect all writers, then run focused offline tests:
```bash
rg -n 'ClipJobStatus\.|EditProjectStatus\.|HighlightProjectStatus\.|setStatus\(' src/main/java src/test/java
mvn -o -q -Dtest=DatabaseBackedClipJobQueueAdapterTest,HighlightDirectorFlowServiceTest,LocalDirectorSchedulerTest test
```
**Interpretation.** A test that reaches the expected final state does not prove forbidden edges are rejected. Add transition-table tests through change control. Current highlight code writes `CREATED -> WAITING_FOR_DIRECTOR -> RENDERING -> RENDERED` and failures, while `ANALYZING`/`PLANNED` exist without a centralized transition guard; treat the model as partially asserted, not enforced.
**Adversarial refutation.** Attempt stale replay, duplicate completion, failure after side effect, and terminal-to-active transition in an isolated test. Require the same rule to reject all forbidden sources, not controller-only validation.
**Worked history.** Before commit `1737d8b`, one processing exception could make the job `FAILED` while the DB queue still intended `PENDING` retry. The mechanism was split ownership of terminal policy. Moving terminal failure to the queues max-attempt decision explains both non-terminal retries and DLQ exhaustion.
## Recipe 2: Prove Timeline And Duration Arithmetic
**Question.** Does the planned timeline equal the rendered media timeline?
**Assumptions.** Fix the edit plan version, source identity, frame rate/timebase, audio sample rate, transition semantics, and output binary. Do not compare a stale plan to a newer render.
**Derive before running.** For decision `i`:
```text
source_i = sourceEnd_i - sourceStart_i
rendered_i = source_i / playbackSpeed_i
planned_i = timelineEnd_i - timelineStart_i
timelineStart_1 = 0
timelineStart_i = timelineEnd_(i-1)
T_plan = timelineEnd_last
```
Current `EditPlanValidator` permits `|rendered_i - planned_i| <= 0.05 s`, contiguity error `<= 0.001 s`, final target error `<= 1.0 s`, and speed `[0.25, 4.0]`. These are implementation tolerances, not certified media-quality thresholds. Predict accumulated timing error from frame duration, audio sample period, transitions, and mux timebase before probing.
**Repo-safe experiment.** For an existing approved plan/output pair:
```bash
PLAN=output/edit-projects/PROJECT/edit-plan.json
MEDIA=output/edit-projects/PROJECT/final.mp4
test -f "$PLAN" && test -f "$MEDIA"
ffprobe -v error -show_entries format=start_time,duration -show_entries stream=index,codec_type,time_base,start_time,duration,nb_frames -of json "$MEDIA"
rg -n 'EPSILON|TARGET_TOLERANCE_SECONDS|renderedDuration|plannedDuration' src/main/java/org/example/videoclips/editing/EditPlanValidator.java
```
Do not infer media duration from `RenderManifest.durationSeconds`; the multi-clip and per-highlight render paths derive that field from the plan, while the project-level highlight flow sums ffprobe results. Probe the file used by the claim.
**Interpretation.** Distinguish plan arithmetic error, encoder/mux quantization, and transition overlap. A constant offset on every output implicates command/timebase behavior; error growing per segment implicates accumulation or rounding.
**Adversarial refutation.** Include fractional frame rates, nonzero stream start time, audio shorter/longer than video, speeds at both limits, and a final partial segment.
**Worked history.** Current highlight QA sets `duration_matches_timeline=true` with the explanation that duration is based on the plan. That is asserted QA: it cannot detect a muxed output whose actual duration differs. The discriminating evidence is plan-derived `T_plan` versus ffprobe-derived stream/container durations.
## Recipe 3: Bound Keyframe And Segment Error
**Question.** Is a cut frame-accurate, or merely aligned to an existing keyframe?
**Assumptions.** `FolderFfmpegClipper` and `FfmpegVideoClipperAdapter` stream-copy in preservation/FAST paths; exact paths re-encode with forced keyframes. Stream copy preserves encoded packets and cannot create a keyframe at an arbitrary boundary.
**Derive before running.** Let requested boundary be `b`, surrounding keyframe timestamps `k_prev <= b <= k_next`, and maximum observed keyframe gap `G`. Predict the tool-specific chosen boundary from command shape; the empirical error must be compared with `|b-k_prev|` and `|k_next-b|`. Do not claim a universal direction without observing the muxer result. For forced-keyframe re-encode, predeclare a bound from output frame duration/timebase and then measure it.
**Repo-safe experiment.** Probe source keyframes and existing clips only:
```bash
SOURCE=/path/to/approved/source.mp4
CLIP=/path/to/existing/clip.mp4
ffprobe -v error -select_streams v:0 -skip_frame nokey -show_frames -show_entries frame=best_effort_timestamp_time,pkt_dts_time -of csv=p=0 "$SOURCE"
ffprobe -v error -show_entries format=start_time,duration -show_entries stream=time_base,start_time,duration -of json "$CLIP"
rg -n 'force_key_frames|segment_time|preserveInputQuality|AccuracyMode.EXACT' src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java
```
**Interpretation.** Boundary error tracking source keyframe distance supports keyframe alignment. Error within a predeclared output-frame/timebase bound only in re-encode mode supports forced-keyframe accuracy. Codec names or clip-count metadata do not prove either.
**Adversarial refutation.** Test long GOP, variable frame rate, B-frames, nonzero start time, and audio packet boundaries. Compare first/last decoded frame content, not container duration alone.
**Worked history.** The folder workflow explicitly retains a quality-versus-exactness trade-off: packaged `preserve-input-quality=true` uses `-c copy`; false uses libx264/AAC and forced keyframes. Calling both modes “exact” would contradict the mechanism.
## Recipe 4: Prove FFmpeg Filter And Audio Graphs
**Question.** Does every intended stream reach the output with the expected timebase, duration, gain, and label?
**Assumptions.** Freeze the exact command, input stream inventory, asset presence, channel layouts, and current editing properties. Treat optional input syntax separately from filter-label requirements.
**Derive before running.** Draw a directed graph: input stream -> trim/delay/rate -> gain -> sidechain -> mix -> loudness -> mapped output. For every label predict channel count, sample rate, start/end time, and whether absence is legal. Check video `setpts=PTS/speed` against audio `atempo=speed`.
Current renderer facts:
| Graph behavior | Current implementation consequence |
|---|---|
| Segment audio is optional at `-map 0:a?` | A video-only timeline can be created. |
| Mix graph starts with `[0:a]` | A video-only timeline makes the mix graph fail even when music/voice/SFX inputs exist. |
| Highlight audio-mix failure is caught | The renderer copies the post-timeline, potentially publishing without intended assets. |
| `amix duration=first` | Output audio length is governed by timeline audio, not longest asset. |
| Music+voice uses sidechain compression | Music is the compressed signal; voice is the detector and is also mixed. |
**Repo-safe experiment.** Inspect command construction and probe an existing output:
```bash
rg -n 'audioMixCommand|sidechaincompress|amix=|loudnorm=|setpts=|atempo=|fallback.*copy' src/main/java/org/example/videoclips/editing
MEDIA=/path/to/existing/approved/final.mp4
ffprobe -v error -show_streams -show_format -of json "$MEDIA"
ffmpeg -hide_banner -nostats -v error -i "$MEDIA" -map 0:v:0 -f null - -map 0:a:0 -f null -
```
**Interpretation.** Command presence proves intent, not application. Require output streams plus measured signal. An audio-less result after a logged mix failure is explained by the catch-and-copy path; a successful graph with wrong loudness needs a different mechanism.
**Adversarial refutation.** Use video-only source, mono/stereo assets, missing one optional asset, asset longer than timeline, and SFX beyond source-audio end. Never “fix” a graph by inserting prohibited silence.
**Worked history.** Commit `5d889b0` records “working version but not cinematic.” A valid FFmpeg graph explained renderability but not creative quality; later filters/assets/QA are hypotheses whose contribution still needs ablation and review.
## Recipe 5: Measure Loudness, Dynamics, And True Peak
**Question.** Does mastered audio meet a declared delivery envelope without clipping or masking speech?
**Definitions.** `LUFS-I` is integrated program loudness; `LRA` is loudness range; `dBTP` is inter-sample true peak. Sample peak from `volumedetect` is not true peak.
**Assumptions.** Freeze the exact encoded output, channel layout, measurement tool/version, delivery target, tolerance, and speech windows. Measure after the final lossy encode.
**Derive before running.** Declare target and tolerance through validation/change control. Packaged filter targets are `I=-16`, `TP=-1.5`, `LRA=11`; they are current defaults, not measured acceptance. Predict integrated loudness, maximum true peak, LRA, silence proportion, and voice/music delta for the fixture.
**Repo-safe experiment.** Measure an existing approved output; this decodes to a null sink and does not render a file:
```bash
MEDIA=/path/to/existing/approved/final.mp4
ffmpeg -hide_banner -nostats -i "$MEDIA" -map 0:a:0 -af 'loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json' -f null - 2>&1
ffmpeg -hide_banner -nostats -i "$MEDIA" -map 0:a:0 -af 'ebur128=peak=true' -f null - 2>&1
rg -n 'loudness-target-i|loudness-true-peak|loudness-range|clippingDetectCommand|volumedetect' src/main/resources/application.yml src/main/java/org/example/videoclips/editing
```
**Interpretation.** Compare measured `input_i`, `input_tp`, and `input_lra` to the preregistered envelope. Current QAs `max_volume < -0.1 dBFS` clipping check is a sample-peak warning, not proof of the configured `-1.5 dBTP` limit. One-pass `loudnorm` filter presence is not measurement.
**Adversarial refutation.** Include silence, isolated transient, codec round-trip, mono/stereo, and speech-over-music windows. A global LUFS pass can coexist with unintelligible voice; test local speech windows and human intelligibility separately.
**Worked history.** Commit `bb23ac8` added mastering filters. That establishes graph construction, not that every output hits targets, especially when highlight audio failure falls back to copying the timeline.
## Recipe 6: Prove Highlight Ranking
**Question.** Does the ranker find the moments reviewers label as highlights, in useful order?
**Assumptions.** Build a licensed, versioned evaluation set with independent timestamp labels and category strata. The repository currently has no certified creative golden set. `CinematicHighlightAnalyzer` uses metadata keyword classification, fixed windows, position roles, and clip-level quality scores repeated across every window; its tests prove bounds/order, not relevance.
**Derive before running.** Freeze temporal intersection-over-union threshold `tau`, top `K`, and split before scoring:
```text
IoU(candidate, truth) = overlap_seconds / union_seconds
precision@K = matched_predictions_in_top_K / K
recall@K = uniquely_matched_truth_windows / truth_window_count
DCG@K = sum((2^relevance_i - 1) / log2(i + 1)); NDCG@K = DCG@K / ideal_DCG@K
```
Use one-to-one matching so duplicate overlapping candidates cannot inflate recall. Report per category and macro average. Predict metric deltas and confidence intervals before each change; never select `tau` or `K` on the test set.
**Repo-safe experiment.** First prove what current tests do and do not cover:
```bash
sed -n '1,220p' src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java
sed -n '1,180p' src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java
rg -n 'golden|ground.truth|precision|recall|NDCG|IoU' src/test docs
```
Then evaluate exported candidate JSON against separately reviewed labels with a deterministic evaluator added only through change control. Record exact-match policy, ties, missing categories, and bootstrap seed.
**Ablations.** Hold candidates fixed and remove one feature family at a time: source position, visual scores, audio events, scene boundaries, semantic/local-model features. A useful feature must predict a preregistered metric change and survive category-stratified negatives.
**Interpretation.** High precision with low recall means conservative discovery; high recall with low precision burdens the director. NDCG distinguishes useful ordering from an unordered candidate set. Do not aggregate away a failing content category.
**Adversarial refutation.** Include long clips where the 12-window cap changes step size, generic filenames, repeated scenes, quiet emotional moments, high-motion non-events, and near-duplicate windows.
**Worked history.** Category-aware planning (`5006dfd`) followed the “not cinematic” result, but history contains no controlled ranking comparison. Treat category boosts as candidates until labeled-set precision/recall/NDCG and ablations support them.
## Recipe 7: Distinguish Model Signal From Fallback
**Question.** Did a declared local model produce the semantic asset, or did the system emit a placeholder or host fallback?
**Assumptions.** Freeze the requested asset type/prompt, declared local model checksum, runtime, seed/config, and expected duration. Do not infer provenance from a filename or directory.
**Derive before running.** Predict all three layers: provenance `(model id, local path, checksum, runtime, seed/config)`, signal `(duration, sample rate, channels, RMS/silence, spectrum)`, and semantics `(speech transcript/intelligibility or licensed music/SFX rubric)`. Require all three. File existence is zero-layer proof.
**Repo-safe experiment.** Do not invoke the workers. Inspect current fallback paths and measure only an existing candidate asset:
```bash
rg -n 'write_silence|write_fallback_tone|fallbackTone|writeSilence|speech-fallback|strict_runtime|strict-runtime' tools/local_asset_worker.py src/main/java src/main/resources
AUDIO=/path/to/existing/candidate.wav
ffprobe -v error -show_streams -show_format -of json "$AUDIO"
ffmpeg -hide_banner -nostats -i "$AUDIO" -af 'silencedetect=noise=-60dB:d=0.2,astats=metadata=0:reset=0' -f null - 2>&1
```
Historical artifacts may contain exact silence from the former Python voiceover fallback, a `110 Hz` low-level music tone, an `880 Hz` short SFX tone, Java-generated tones, or host `say`/`espeak` speech. Those success paths were removed from the 2026-07-21 working tree. Keep detecting the signatures in old outputs; do not use absence of a known signature as a provenance whitelist.
**Interpretation.** Silence proves failure. Nonzero samples disprove only digital silence; they do not prove speech or model generation. A stable simple tone matching a known path plus absent model provenance supports fallback. Real speech from a host tool is still not the approved embedded model.
**Adversarial refutation.** Test a valid model asset, digital silence, known tone, corrupted WAV, unrelated audible audio, and host TTS. The mechanism must classify positives and negatives without relying on filename.
**Worked history.** On 2026-07-21 the workspace build passed while a clean archive failed `LocalAssetGenerationStageTest`; tests depended on an untracked virtual environment/host fallback. The working tree now makes that test deterministic and strict readiness fail closed. Environment-coupled success explains the old observations; only a new clean archive can confirm closure.
## Recipe 8: Prove Retry, DLQ, Idempotency, And Leases
**Question.** Does each logical job have bounded attempts and at-most-one durable effect despite crashes and redelivery?
**Assumptions.** Fix queue mode, `max-attempts`, retry backoff, visibility timeout, transaction boundary, idempotency key, and side effects. The DB-queue proof does not generalize to the memory queue or a future broker.
**Derive before running.** For configured maximum `M`, predict attempt sequence `1..M`; failures `1..M-1` yield message `PENDING`, job `QUEUED`, and one `RETRY_SCHEDULED`; failure `M` yields message `DLQ`, job `FAILED`, and terminal events. Predict side-effect cardinality by idempotency key. Let visibility timeout `V` exceed a measured runtime bound or provide lease extension; otherwise predict overlapping work after `V`.
**Repo-safe experiment.** Use existing mocks and read-only inspection:
```bash
mvn -o -q -Dtest=DatabaseBackedClipJobQueueAdapterTest test
rg -n 'attemptCount|maxAttempts|visibilityTimeout|RETRY_SCHEDULED|markTerminalFailure|DLQ|Idempotency' src/main/java src/test/java src/main/resources
sed -n '1,260p' docs/dlq-redrive-procedure.md
```
**Interpretation.** State correctness does not prove effect idempotency. Compare object key, clip index, database uniqueness constraints, and event counts after duplicate delivery. Current queue claims and processing occur in one transaction method, but FFmpeg/object-storage effects are not rolled back with a database transaction.
**Adversarial refutation.** Fail after upload/before row save, after row save/before completion, at `V-epsilon` and `V+epsilon`, and with two pollers. Do not redrive real DLQ rows as a test.
**Worked history.** Commit `1737d8b` fixed premature terminal job failure. It does not by itself prove exactly-once external effects; that requires idempotency and crash-boundary tests.
## Recipe 9: Prove Filesystem Atomicity And Path Confinement
**Question.** Can a reader observe a partial file, or can untrusted names escape the configured root?
**Assumptions.** Fix the configured root, filesystem/mount, writer/reader concurrency, crash point, and symlink policy. Lexical normalization and physical filesystem confinement are separate claims.
**Derive before running.** For publication require write-to-sibling-temp -> fsync as required -> atomic rename on the same filesystem. Predict reader observations as `{old, new}`, never partial. For confinement require validated token, `root.resolve(name).normalize()`, `startsWith(root)`, rejection of absolute paths, separators, and `..`; decide symlink policy explicitly.
**Repo-safe experiment.** Run current confinement tests and inspect publication sites:
```bash
mvn -o -q -Dtest=FileSystemEditProjectStoreTest,FileSystemHighlightProjectStoreTest,EditPlanInboxScannerTest test
rg -n 'ATOMIC_MOVE|writeValue\(|writeString\(|Files.copy|startsWith\(|validateProjectId|validate.*File' src/main/java/org/example/videoclips/editing
```
**Interpretation.** Current stores validate lexical paths but write JSON directly to final files; path confinement tests do not prove crash-safe publication. Highlight source claiming and inbox archival use `ATOMIC_MOVE`, which can fail across filesystems rather than silently degrade. `startsWith` on normalized paths does not settle symlink traversal.
**Adversarial refutation.** Test absolute paths, both separators, `..`, Unicode/confusable names if accepted, symlink inside root to outside, concurrent readers, process death mid-write, and cross-filesystem move. Keep these in JUnit temporary directories.
**Worked history.** Existing `project.json`/manifest files can claim rendered output while media is absent. That history shows why metadata publication and durable artifact publication must be one explicit invariant, not two independent existence checks.
## Recipe 10: Build Resource, Capacity, And Cost Models
**Question.** What arrival rate, concurrency, disk, memory, model residency, and cost can the service sustain?
**Assumptions.** Freeze workload mix, source durations/codecs, warm/cold model state, hardware, worker topology, storage/database limits, retention, retry rate, and dated prices. State every omitted cost.
**Derive before running.** Use measured distributions, not a single average:
```text
real_time_factor = processing_seconds / source_seconds
service_rate_per_worker = 1 / mean_job_seconds
required_concurrency ~= arrival_rate_per_second * target_system_time_seconds (Little's Law)
scratch_bytes >= concurrency * (source + intermediates + output) * safety_factor
compute_cost/source_min = worker_seconds/source_min / 3600 * worker_cost/hour
transfer_seconds/source_min = transferred_MiB/source_min / measured_MiB_per_second
```
Add fixed local-model memory once per worker process, peak temporary tensors per concurrent inference, FFmpeg threads, JVM heap, and database/object-storage limits. Predict saturation resource and queue-growth slope before load.
**Repo-safe experiment.** Inspect historical models without treating them as current:
```bash
sed -n '1,260p' docs/cost-model-by-video-minute.md
sed -n '1,220p' docs/ffmpeg-preset-benchmark-baseline.md
sed -n '1,220p' docs/object-storage-bandwidth-benchmark-baseline.md
sed -n '1,260p' docs/load-test-signoff.md
```
The checked-in cost harness hardcodes example prices and prior benchmark values. Replace assumptions with dated target-environment measurements only after an approved benchmark. Never claim scalability from the stub worker baseline or the documentation-only load signoff.
**Interpretation.** Validate predictions against p50/p95/p99 service time and resource saturation. If measured queue slope differs, the arrival rate, service-time distribution, concurrency constraint, or retry amplification is missing from the model.
**Adversarial refutation.** Include cold model load, warm inference, mixed durations, 2x burst, slow storage, disk pressure, codec mix, retry amplification, and a 60-minute endurance run. A model must predict both steady state and backlog growth when `arrival_rate > capacity`.
**Worked history.** `docs/load-test-signoff.md` says execution evidence is pending. Existing local baselines explain planning inputs, not production API/worker throughput or cloud cost.
## Recipe 11: Prove Reproducibility With Checksums
**Question.** Can another approved macOS/Linux/VPS/cloud runtime reproduce the result from the same declared inputs?
**Assumptions.** Define whether the claim is byte reproducibility or metric-bounded reproducibility. Freeze source/config/model/asset checksums, toolchain, locale/timezone, device, and randomness.
**Derive before running.** Predict which bytes must be identical and which measurements may vary. Media encodes may differ across FFmpeg/library/hardware builds; in that case require identical inputs/config and bounded semantic/technical metrics, not a fabricated byte-equality promise.
**Repo-safe experiment.** Capture source tree identity, tool versions, and existing artifact hashes without starting the service:
```bash
git rev-parse HEAD
java -version 2>&1
mvn -version
ffmpeg -version | sed -n '1,12p'
ffprobe -version | sed -n '1,5p'
git ls-files -s pom.xml src tools
ARTIFACT=/path/to/existing/approved/artifact
if command -v sha256sum >/dev/null 2>&1; then sha256sum "$ARTIFACT"; else shasum -a 256 "$ARTIFACT"; fi
```
For a local model record model-file checksum, license, runtime package lock, device, precision, seed, and inference parameters. A model name such as `musicgen-small` is not an immutable identity.
**Interpretation.** An identical artifact hash proves byte identity for that artifact only. A mismatch requires localization by input, intermediate, and toolchain hashes; it does not automatically prove semantic regression.
**Adversarial refutation.** Repeat from a clean archive with network disabled and no untracked virtual environments; compare macOS and Linux. Separate dependency-cache absence from test failure. Do not solve a clean failure by enabling downloads or committing a virtualenv.
**Worked history.** The workspaces 216 tests passed while a clean archived checkout failed in local-asset generation. The untracked `.venv-local-asset`/host behavior is the discriminating variable, so “`mvn test` passed here” is not reproducible-build proof.
## Recipe 12: Perform Causal Debugging
**Question.** Which single mechanism explains all observations, including negatives?
**Assumptions.** Freeze the incident time window, build, inputs, configuration, environment, and observation reliability. Mark unknowns rather than silently filling them.
**Derive before running.** Build a causal graph from input identity -> configuration -> state -> external/local side effects -> persisted metadata -> published artifact -> measured quality. Mark observation points. For each candidate cause, write a prediction matrix before touching code:
| Experiment | Mechanism A predicts | Mechanism B predicts | Observation |
|---|---|---|---|
| Positive control | number/state/signature | number/state/signature | pending |
| Negative control | absence/different value | same value | pending |
| One-factor ablation | directional delta | no delta | pending |
| Crash/boundary case | durable state/effects | different tuple | pending |
**Repo-safe experiment.** Locate the actual writers and historical change before selecting one minimal discriminator:
```bash
rg -n 'objectKey|markTerminalFailure|duration_matches_timeline|audio_mastering_applied|fallback' src/main src/test
git log --oneline --all -- src/main/java/org/example/videoclips
git show --stat 7307082 1737d8b 5d889b0 9b56e89
```
**Interpretation.** Accept only a mechanism whose predictions match the positive, negative, and boundary cases. Correlation with a changed component is insufficient when another mechanism predicts the same observation.
Prefer the smallest experiment that makes predictions diverge. Reject explanations that merely restate the symptom. Preserve logs, checksums, commands, and negative results; do not patch before discrimination unless containing an incident.
**Worked examples.** Use the historical mechanism, not the headline:
| Symptom | Mechanism that explained positives and negatives | Remaining boundary |
|---|---|---|
| Signed download failed despite a clip row | Derived fake key differed from persisted `objectKey`; commit `7307082` made persisted key authoritative. | Production S3 end-to-end proof remains separate. |
| First transient error looked terminal | Processor finalized failure before queue retry policy; `1737d8b` moved terminal decision to exhaustion. | External side-effect idempotency still needs crash tests. |
| Renderer worked but was not cinematic | Renderability measured graph execution, not semantic highlight/edit quality. | No certified creative golden set or controlled ablation exists. |
| Workspace build passed, clean archive failed | Untracked runtime/host fallback affected local-asset tests. | Hermetic fake/runtime contract is still open. |
| Highlight QA passed duration/mastering checks | Some checks are constants or command-presence assertions rather than output measurements. | Probe-derived gates must be added through change control. |
**Adversarial refutation.** Before accepting a mechanism, assign a reviewer to construct the strongest counterexample and a test where the mechanism predicts no effect. If it cannot explain that negative, retain it as a candidate only.
## Promotion Checklist
- [ ] The proof record was written before execution.
- [ ] Inputs, commit, configuration, binaries, models, assets, licenses, and checksums are recorded.
- [ ] Commands ran offline and did not bootstrap, download, render without approval, mutate production, or use external AI.
- [ ] Structural, semantic, security, and operational claims are separated.
- [ ] Positive, negative, boundary, and failure controls passed.
- [ ] Ranking work reports denominators, category strata, matching policy, uncertainty, and ablations.
- [ ] Media work reports probe-derived duration, streams, loudness, true peak, silence, and human-review evidence where semantic judgment is required.
- [ ] One mechanism explains all observations, including negatives; an adversarial reviewer tried to falsify it.
- [ ] Unproven conclusions remain labeled `OPEN` or `CANDIDATE`.
- [ ] Any behavior/default/dependency/model/asset/render/production change proceeds through `video-editing-change-control` and the validation sibling.
## Provenance and maintenance
Sources of record: current Java/Python implementation and tests; `application.yml`/`application.properties`; benchmark/runbook documents; Git commits `66e998e`, `7307082`, `1737d8b`, `5d889b0`, `5006dfd`, `bb23ac8`, and `9b56e89`; and the clean-checkout investigation recorded in `video-editing-failure-archaeology`.
Re-verify volatile facts before each use:
```bash
git rev-parse HEAD
rg -n 'loudness-target-i|loudness-true-peak|loudness-range|strict-runtime|fallback-to-heuristic|max-attempts|visibility-timeout|preserve-input-quality' src/main/resources
rg -n 'duration_matches_timeline|audio_mastering_applied|write_silence|write_fallback_tone|fallbackTone|ATOMIC_MOVE|writeValue\(' src/main tools
rg -n 'precision|recall|NDCG|ground.truth|golden|Execution evidence pending' src/test docs
git show --stat 66e998e 7307082 1737d8b 5d889b0 5006dfd bb23ac8 9b56e89
```

View File

@ -0,0 +1,285 @@
---
name: video-editing-research-frontier
description: Load when proposing, ranking, starting, evaluating, retiring, or positioning research that could advance this repository beyond its current cinematic-highlight baseline, especially offline local multimodal ranking, constrained directing, audiovisual QA, local voice/music/SFX generation, causal ablations, efficient model packaging, or restartable deterministic rendering. Do not load for ordinary defect repair, production hardening, or execution of the current highlight campaign.
---
# Video Editing Research Frontier
Use this runbook to turn an open technical question into a falsifiable, repo-local research program. Treat all landscape statements below as hypotheses, not claims about the external state of the art (SOTA). The repository does not contain a reviewed literature survey or competitive benchmark.
**Status date:** 2026-07-21.
## Choose The Correct Skill
| Need | Use instead |
|---|---|
| Fix a known defect, remove a prohibited fallback, make the clean build reproducible, or harden production | `video-editing-change-control` and the relevant engineering skill |
| Execute the hardest live objective with decision gates | `video-editing-cinematic-highlights-campaign` |
| Form hypotheses, pre-register predictions, assign adversarial review, or retire an idea | `video-editing-research-methodology` |
| Derive a score, prove an invariant, design an ablation, or calculate uncertainty | `video-editing-proof-and-analysis-toolkit` |
| Certify an output, add a golden, or define release evidence | `video-editing-validation-and-qa` |
| Make a paper, novelty, benchmark, or release claim | `video-editing-external-positioning` |
| Debug an observed failure | `video-editing-debugging-playbook` |
Do **not** use this skill to label missing production controls as research. Adding authentication, immutable model packaging, fail-closed startup, a clean build, render approval, or measured QA is engineering backlog. Research begins only when a controlled experiment compares mechanisms under those controls.
## Enforce The Research Perimeter
Apply these rules before any experiment:
- Run without network access. Never call an external AI service. Never let Maven, Python, a model library, or a worker download dependencies or weights.
- Provision every dependency and model before runtime through the approved artifact process. Record artifact name, exact version, SHA-256, license, approval, platform, and expected resource envelope.
- Use only footage, voices, music, sound effects, fonts, LUTs, and model outputs whose experiment and redistribution rights are recorded. Treat generated media as licensable artifacts, not automatically safe assets.
- Reject missing or invalid models and assets. Never substitute silence, tones, heuristic analysis, host `say`, host `espeak`, or an untracked local asset.
- Require explicit render approval. Keep every experiment flag off by default and leave production-facing defaults unchanged.
- Keep raw media and model prompts out of logs. Treat faces, voices, transcripts, filenames, and embeddings as sensitive data until the threat model says otherwise.
- Execute experiments in an isolated project/output root with quotas. Do not point a scheduler at production input or output directories.
- Route all source changes, model additions, data collection, and promotion through `video-editing-change-control`. A research result is not production approval.
The checked-in launchers violate the target perimeter if used as written: `tools/run_local_cv_worker.sh` can install Python packages and load the default `yolov8n.pt`; `tools/run_local_asset_worker.sh` can install packages; and `tools/local_asset_worker.py` calls `get_pretrained`. Do not run those launchers for research until an offline, pinned, checksum-verified mode exists.
## Separate Backlog From Research
Complete or explicitly gate this backlog before interpreting creative results:
| Known gap | Why it is not research | Required gate |
|---|---|---|
| Highlight rendering now defaults off and approval on, but approval is a bare file rather than authenticated evidence bound to source/plan/configuration digests | This is an authorization and provenance gap, not a hypothesis | Keep rendering disabled until digest-bound approval and recovery behavior are implemented and tested |
| `SourceVisualAnalyzer` can fall back to `HeuristicVisualAnalysisProvider` | Silent mechanism substitution invalidates attribution | Set fallback off and persist the exact analyzer identity |
| Generated audio still lacks certified semantic-fit and provenance evaluation | Placeholder rejection is implemented, but file audibility is not creative or licensing evidence | Add model/asset manifests and held-out scene-to-audio preference evaluation |
| No licensed, checksummed resident asset-model bundle is present | Packaging and provenance are engineering prerequisites | Provision and verify Piper, MusicGen, and AudioGen artifacts without runtime acquisition |
| Highlight QA measures only selected technical defects | Duration, black ranges, long silence, and sample peaks are probed, while loudness, true peak, A/V sync, freezes, raster text safety, semantic fit, and human correlation remain open | Extend the versioned probe portfolio before using certification labels |
| No licensed, versioned creative corpus or human-scored golden inventory is checked in | An experiment cannot generalize from anecdotal outputs | Approve a corpus manifest and annotation protocol; store media outside Git |
| Python audio dependencies are unpinned and model artifacts are not locked | Results cannot be reproduced | Build an immutable offline runtime with a Software Bill of Materials (SBOM) and checksums |
Verify the current guardrail-sensitive defaults without starting the application:
```bash
rg -n "render-enabled:|require-director-approval:|fallback-to-heuristic:|auto-start:|strict-runtime:" src/main/resources/application.yml
rg -n "pip install|get_pretrained|YOLO\(|write_silence|write_fallback_tone|fallbackTone" tools src/main/java/org/example/videoclips/editing
```
## Establish One Evaluation Contract
Create one versioned corpus manifest and one immutable evaluation split before comparing ideas. Do not select examples after seeing results.
Define these terms once:
- **Temporal IoU:** intersection divided by union of a predicted and reference time range.
- **nDCG@3:** normalized discounted cumulative gain for the first three ranked candidates; it rewards relevant moments near the top.
- **Pairwise win rate:** fraction of blinded A/B judgments preferring the candidate over the baseline; ties count as one half.
- **Spearman rho:** rank correlation between an automated score and human ordering.
- **RTF:** real-time factor, processing seconds divided by source-media seconds; lower is faster.
- **Equivalence margin:** the largest quality loss accepted when testing that a faster method is practically no worse.
- **Confidence interval (CI):** an uncertainty interval computed by the pre-registered method; do not report a point estimate alone.
For every frontier item, write a dated experiment record with:
```text
question; mechanism; baseline commit; corpus-manifest hash; split hash
model/dependency hashes and licenses; hardware/OS; experiment flag
primary metric; prediction made before running; equivalence margin
negative controls; adversarial reviewer; stop/retire rule; raw artifact root
```
Use blinded review with multiple reviewers and report disagreement. Category, source identity, variant, and generation seed belong in the analysis table, not metric labels or normal application logs.
## Frontier Portfolio
Rank work in this order unless evidence changes the dependency graph.
| Rank | Program | Research question | Depends on |
|---:|---|---|---|
| 1 | Offline multimodal highlight ranking | Can local temporal evidence find human-valued moments better than positional windows? | Corpus, fail-closed local inference |
| 2 | Controllable cinematic planning | Can a constrained local director produce valid, intentional timelines rather than one-range templates? | Ranked moments, plan semantics |
| 3 | Human-correlated audiovisual QA | Can measurements predict reviewer judgment and block technical defects? | Scored outputs, real probes |
| 4 | Fail-closed local generative audio and voiceover | Can locally packaged models create licensed, intelligible, context-fit assets that pass gates? | Immutable model runtime, QA |
| 5 | Causal edit-component evaluation | Which effects actually cause preference changes, and for which categories? | Deterministic variants, blinded review |
| 6 | Quality-preserving local inference efficiency | Can cascades reduce runtime and memory without meaningful quality loss? | Stable quality metrics, hardware matrix |
| 7 | Restartable, reproducible rendering | Can content-addressed stages resume safely and reproduce equivalent outputs? | Manifested inputs, deterministic stage contracts |
## 1. Offline Multimodal Highlight Ranking
**Repo-grounded problem.** In the older edit-project flow, `CinematicHighlightAnalyzer` creates fixed eight-second windows, caps each clip at twelve candidates, classifies from project names and paths, and adds source-position bonuses. The single-source flow now has a safer deterministic baseline: `HighlightCandidateGenerator` persists shot-aligned or overlapping coverage windows, ranks them with source-level visual quality, scene score, duration fit, and FFmpeg non-silence, and downweights heuristic/fallback visual evidence. The visual scores are not temporally resolved, `unclassified_audio` is not semantic audio understanding, and there is no checked-in transcript or temporal audio-event classifier. The baseline therefore produces review candidates, not content-aware highlight discovery.
**Hypothesis.** A fully local, temporally aligned ranker using shot-level visual embeddings, speech/transcript evidence, audio events, motion, novelty, and narrative role will improve top-three relevance across car, food, family, and generic footage. This is a repo hypothesis, not an external SOTA claim.
**Concrete asset.** Reuse `ShotSceneSegmenter`, extracted frames/contact sheets, `SourceAudioAnalysis`, the `VisualAnalysisProvider` port, `HighlightCandidate`, persisted analysis JSON, and the tested `HighlightCandidateGenerator` as the frozen single-source baseline. Do not call that baseline a research result.
**First three steps.**
1. Approve a licensed corpus manifest; have blinded reviewers mark worthwhile ranges, category, narrative role, and 0-4 relevance. Freeze source-disjoint train/development/test splits and compute baseline nDCG@3 and temporal IoU.
2. Add a disabled experimental provider behind an explicit port. Feed per-shot features only from preprovisioned local models; persist timestamps, model hashes, feature schema, confidence, and missing-modality state. Fail if a required modality is missing.
3. Compare positional baseline, each single modality, the complete model, shuffled-timestamp control, and metadata-only control. Use the same candidate budget and report per-category CIs.
**Prediction before running.** Predict at least `+0.10` absolute nDCG@3 and `+0.10` absolute best-match temporal IoU over the frozen baseline, with no category losing more than `0.03` nDCG@3. Lock these numbers in the experiment record before inference.
**You have a result when:** the held-out, source-disjoint test CI excludes zero for the primary improvement; shuffled timestamps materially reduce performance; and blinded reviewers prefer montages built from the new top candidates by at least 60% pairwise win rate. This proves a repo result, not SOTA.
**Retire or narrow when:** the CI includes zero after the pre-registered sample size, gains vanish on unseen sources, the metadata control matches the full model, or any required model lacks approved offline packaging or rights.
**Risks and obligations.** Control transcript leakage and repeated scenes; measure demographic/category error; encrypt or isolate embeddings; bound CPU/GPU memory and RTF; never send media to a remote endpoint.
## 2. Controllable Cinematic Planning
**Repo-grounded problem.** `HighlightDirectorFlowService` consumes a manually supplied and contract-validated `director/edit-plan.json`, converts each highlight to one `EditDecision`, schedules per-line voiceover from estimated reading time, spaces overlays evenly, and chooses mostly cut/fade transitions. The plan contains direction text, but the renderer still has a much smaller executable vocabulary.
**Hypothesis.** A local planner that combines a typed capability graph, hard timeline constraints, and a learned or search-based objective can produce more intentional and controllable edits than the current one-range template without allowing invalid plans.
**Concrete asset.** Reuse `HighlightDirectorPlan`, `EditPlan`, `EditPlanValidator`, `HighlightVisualEffectsStage`, render manifests, and FFmpeg rendering. Treat every renderer feature as unavailable until a contract test proves it executable.
**First three steps.**
1. Inventory the exact executable vocabulary: trims, speed bounds, transitions, overlays, audio cues, assets, and output profiles. Add round-trip contract fixtures for valid, invalid, and unsupported plans.
2. Define measurable planning controls: target duration, shot diversity, narrative roles, pacing curve, dialogue preservation, asset budget, effect density, and forbidden combinations. Predict all constraint values before rendering.
3. Generate blinded A/B plans from the frozen template and the experimental local planner over the same ranked moments and assets. Render only after approval; score validity, control adherence, repetition, story coherence, and preference.
**Prediction before running.** Predict 100% hard-constraint validity, at least 95% requested-control adherence, and at least 60% pairwise preference over the template, with median rendered duration error no greater than 50 ms.
**You have a result when:** every held-out plan validates and renders, the preference CI is above 50%, changing one requested control changes its measured output while unrelated controls remain within pre-registered tolerances, and unsupported capabilities fail before rendering.
**Retire or narrow when:** preference is indistinguishable from the template, the planner depends on prompt wording rather than typed controls, renderer behavior cannot implement the plan, or constraint repairs erase the preference gain.
**Risks and obligations.** Prevent prompt/filename injection, fabricated factual voiceover, unsafe overlay text, excessive flash/cut rates, and unbounded plan complexity. Persist planner/model hash, seed, constraint decisions, repairs, and approval identity.
## 3. Human-Correlated Audiovisual QA
**Repo-grounded problem.** `HighlightFfmpegRenderer.buildQaReport` now probes encoded duration, black ranges, long silence, and sample peaks, while asset resolution, mastering-filter presence, and overlay plan bounds are structural checks. Neither renderer establishes integrated loudness, true peak, A/V sync, freeze detection, raster text safety, cinematic quality, or correlation with human judgment.
**Hypothesis.** A transparent scorecard combining technical defects, speech intelligibility, dialogue/music balance, cut/beat alignment, shot repetition, exposure/blur continuity, text safety, and artifact provenance will predict human accept/reject decisions well enough to gate review.
**Concrete asset.** Reuse `RenderQaReport`, `RenderQaCheck`, FFmpeg command capture, `RenderManifest`, source analysis, and the existing black/silence/peak probe patterns.
**First three steps.**
1. Define a 0-4 blinded human rubric with mandatory technical rejects and separate creative dimensions. Label real outputs plus deliberately injected black frames, silence, clipping, desynchronization, repetition, unsafe text, and missing provenance.
2. Implement each metric as a versioned probe that emits raw numbers, threshold source, evidence path, and failure semantics. Never infer success from a command string or planned value.
3. Fit thresholds on development data, freeze them, then report defect sensitivity/specificity, Spearman rho by dimension, calibration, and false-pass cases on held-out sources.
**Prediction before running.** Predict at least 95% sensitivity to injected blocking defects, no more than 5% false passes for mandatory rejects, and Spearman `rho >= 0.65` between the composite score and overall human rank.
**You have a result when:** held-out results meet all three predictions, reviewer disagreement is reported, every blocking decision links to reproducible measurements, and removing any claimed useful metric causes the pre-registered degradation.
**Retire or narrow when:** correlation fails on a category, thresholds drift across render profiles, a proxy rewards visibly worse outputs, or human disagreement makes the target unidentifiable. Keep reliable technical probes even if the creative composite is retired.
**Risks and obligations.** Do not hide category-specific failures in an average. Avoid sensitive identifiers in metrics. Pin FFmpeg and probe versions. Treat automated creative scoring as review prioritization until change control explicitly promotes it.
## 4. Fail-Closed Local Generative Audio And Voiceover
**Repo-grounded problem.** The 2026-07-21 working tree removed silence/tone/host-speech success fallbacks, requires existing local AudioCraft model paths with offline resolution, deletes failed/inaudible output, and blocks rendering when requested assets are missing. The runtime still lacks a checked-in immutable licensed Piper/MusicGen/AudioGen bundle, semantic music/SFX fit measurement, speech intelligibility measurement, and artifact-level license attestations. Demonstrating locally generated production quality remains research.
**Hypothesis.** Preprovisioned local models plus artifact-level validation can generate script-faithful voiceover and context-fit music/SFX that reviewers prefer to source-audio-only edits without placeholders or unverifiable assets.
**Concrete asset.** Reuse typed `HighlightAssetRequest`, `ResolvedEditAsset`, category-aware libraries, audio cues, ducking/loudness filters, and render manifests.
**First three steps.**
1. Define an immutable model/voice registry with checksums, licenses, approved uses, voice consent, sample rates, languages, seeds, resource limits, and a zero-network startup test. Make missing entries block the experiment.
2. Add pre-mix asset gates: non-silence/non-tone detection, duration, clipping, script fidelity, pronunciation review, loudness, model provenance, and license completeness. Preserve rejected files as quarantined evidence, never render inputs.
3. Run approved, seed-controlled A/B tests for source-only, licensed-library, and local-generated variants. Score intelligibility, script faithfulness, semantic fit, distraction, mix balance, preference, generation RTF, and failure rate.
**Prediction before running.** Predict zero placeholder false passes, at least 95% reviewer-rated script faithfulness for voiceover, at least 60% pairwise preference over source-only edits, and no unresolved provenance or license field.
**You have a result when:** all outputs are traceable to approved immutable artifacts, blocking gates catch every injected silence/tone/wrong-script control, preference clears its CI gate on held-out prompts, and an offline cold start neither attempts network access nor degrades to another mechanism.
**Retire or narrow when:** licensing blocks deployment, voice consent is absent, faithfulness misses target, generation is unstable across seeds, reviewers prefer source-only audio, or resource use violates the declared service envelope. Fall back only to an explicitly planned source-only edit, never a placeholder asset.
**Risks and obligations.** Threat-model voice impersonation, prompt injection, harmful content, model supply chain, copyrighted training/output concerns, GPU denial of service, and retention of generated voices. Record security and legal approval before promotion.
## 5. Causal Edit-Component Evaluation
**Repo-grounded problem.** The renderer applies coupled treatments such as contrast, saturation, sharpening, vignette, crop, transitions, overlays, music, SFX, and voiceover. Current tests prove command construction, not which component changes human judgment.
**Hypothesis.** Controlled counterfactual renders can identify category- and context-specific effects that cause preference changes, enabling a smaller and safer cinematic vocabulary.
**Concrete asset.** FFmpeg is a deterministic command backend; `EditPlan`, `HighlightVisualEffectsStage`, and manifests can generate paired variants from identical source decisions.
**First three steps.**
1. Freeze edit decisions and assets, then define one-factor removals and a small pre-registered interaction set. Hash all non-varied inputs and randomize blinded presentation order.
2. Render paired variants with identical codec settings. Verify duration, frames outside the treatment, and audio outside the treatment remain within declared equivalence tolerances.
3. Estimate per-category effect sizes and CIs; repeat the strongest result on new sources and assign an adversarial reviewer to search for confounding.
**Prediction before running.** Predict that at least one component has an absolute pairwise preference effect of 10 percentage points or more in one category, while at least one currently available component is neutral or harmful.
**You have a result when:** the effect repeats on held-out sources, the negative control remains neutral, the changed mechanism explains all observed differences, and the effect survives correction for the pre-registered comparisons.
**Retire or narrow when:** effects reverse across sources without an explanatory moderator, equivalence checks show unintended differences, or reviewer blinding fails. Remove unsupported default treatments through change control; do not universalize a category-specific result.
**Risks and obligations.** Guard against photosensitive flash, abrupt loudness, reviewer fatigue, and source memorization. Store assignments and analysis code; do not store reviewer identity in application telemetry.
## 6. Quality-Preserving Local Inference Efficiency
**Repo-grounded problem.** Local CV is an HTTP worker loading YOLO; local audio generation loads Python/Torch/AudioCraft. The repository has no fixed model artifact set, cross-platform hardware matrix, per-stage memory limits, or quality-versus-runtime benchmark for macOS development, Linux/VPS production, and cloud deployment.
**Hypothesis.** An uncertainty-gated cascade using cheap temporal features first and expensive local models only for ambiguous segments can reduce RTF and peak memory while remaining within a pre-registered quality-equivalence margin.
**Concrete asset.** Reuse proxy media, thumbnails, shots, process timing logs, the visual provider boundary, local worker health contracts, and benchmark-harness conventions under `src/test/java/org/example/videoclips/perf`.
**First three steps.**
1. Declare reproducible macOS CPU, Linux/VPS CPU, and approved cloud CPU/GPU profiles; pin thread counts, model hashes, warm/cold state, corpus split, and power mode. Measure per-stage RTF, peak resident memory, throughput, failures, and quality.
2. Implement a disabled cascade that emits uncertainty and escalation reasons. Bound queue depth, concurrency, memory, timeout, and cancellation; never substitute a different analyzer after failure.
3. Sweep thresholds on development data, freeze one operating point per hardware profile, and run equivalence tests on held-out sources plus 2x load and 60-minute endurance tests.
**Prediction before running.** Predict at least 30% lower median analysis RTF and 25% lower peak memory than always-expensive inference, with nDCG@3 loss no greater than `0.02` and no category loss greater than `0.03`.
**You have a result when:** both efficiency CIs clear their targets, quality stays inside the equivalence margins, memory does not grow continuously, and slowdown or failure does not cause unbounded work accumulation.
**Retire or narrow when:** quality equivalence fails, uncertainty is miscalibrated on unseen categories, GPU gains disappear after transfer/queue overhead, or separate artifacts create an unmaintainable supply chain. Keep platform-specific results platform-specific.
**Risks and obligations.** Enforce container/device least privilege, resource quotas, model license compatibility, SBOM coverage, predictable startup, and telemetry without high-cardinality media IDs.
## 7. Restartable, Reproducible Rendering
**Repo-grounded problem.** The highlight flow loops through assets and renders, then publishes final files and manifests. It has no checked-in content-addressed stage graph or failure-injection evidence for restart. Runtime timestamps, temporary prompt filenames, model randomness, host tools, and codecs can change bytes.
**Hypothesis.** A content-addressed stage contract with atomic publication can resume after failure, avoid repeated work, and reproduce exact bytes on one pinned platform plus semantically equivalent media across approved platforms.
**Concrete asset.** Reuse the folder contract, JSON stores, `RenderManifest`, recorded FFmpeg commands, stable project/highlight IDs, and deterministic edit decisions.
**First three steps.**
1. Define a stage key over normalized plan, source/asset/model/tool hashes, render profile, seed, and schema version. Separate immutable stage output from atomic publication; never include wall-clock time or absolute workspace paths in the key.
2. Add crash points before/after analysis, asset generation, segment render, mix, QA, and publication. Restart and record reused stages, recomputed stages, orphan cleanup, final status, and manifest lineage.
3. Compare clean and resumed runs on each pinned platform. Require exact hashes on the same platform/toolchain; define measured semantic tolerances for duration, frame/audio similarity, loudness, and synchronization across platforms.
**Prediction before running.** Predict at least 50% less recomputation after a failure beyond the midpoint, zero published partial finals, exact same-platform final hashes for deterministic assets, and all cross-platform measurements inside pre-registered tolerances.
**You have a result when:** every injected failure resumes or fails terminally without corrupting prior evidence, cache invalidation responds to every declared input change, same-platform hashes repeat, and cross-platform equivalence passes independent probes.
**Retire or narrow when:** hidden inputs repeatedly invalidate reproducibility, cache validation costs approach recomputation, nondeterministic models cannot expose seeds, or hardware codecs cannot meet exactness. Narrow exact reproducibility to pinned software encoding and retain semantic equivalence elsewhere.
**Risks and obligations.** Prevent cache poisoning, cross-tenant artifact reuse, path traversal, stale-license reuse, secret inclusion in keys, and disk exhaustion. Encrypt or isolate sensitive stage outputs and apply retention policy.
## Promote Or Retire
At the end of every experiment:
- Publish raw measurements, exclusions, failed predictions, negative controls, reviewer disagreement, licenses, hashes, and resource profiles.
- Ask the assigned adversarial reviewer to explain all positive and negative observations with one mechanism and attempt to refute it.
- Mark the idea `candidate`, `replicated`, `retired`, or `inconclusive`; never call it production-ready from one experiment.
- Require replication on held-out sources and at least the applicable macOS and Linux/VPS profiles before proposing a production change.
- Route adoption through `video-editing-change-control`, certification through `video-editing-validation-and-qa`, and any novelty/SOTA statement through `video-editing-external-positioning`.
- Preserve a retired idea's hypothesis, commands, evidence, and stop reason so the next engineer does not repeat it unchanged.
## Provenance and maintenance
This skill was derived from repository source, tests, configuration, plans, and Git history available on 2026-07-21. Predictions and thresholds are proposed falsifiable targets, not observed results. No external SOTA claims were verified.
Re-verify volatile facts with no network and no application startup:
```bash
rg -n "DEFAULT_CANDIDATE_SECONDS|MAX_CANDIDATES_PER_CLIP|source position|Project metadata" src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java
rg -n "analyzer.analyze|directorPromptGenerator.generate|CinematicHighlightAnalyzer" src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java
rg -n "List.of\(decision\)|step = targetDuration|transitionIn|transitionOut|missing_director_plan" src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java
rg -n "duration_matches_timeline|required_assets_resolved|text_overlays_safe|audio_mastering_applied" src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java
rg -n "blackDetectCommand|silenceDetectCommand|clippingDetectCommand" src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
rg -n "get_pretrained|write_silence|write_fallback_tone|say|espeak" tools/local_asset_worker.py src/main/java/org/example/videoclips/editing/LocalAssetSynthesizer.java
rg -n "pip install|YOLO\(|yolov8n.pt" tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh tools/local_cv_worker.py
rg -n "^torch$|^audiocraft$|^soundfile$|^numpy$" tools/local_asset_requirements.txt
git log --oneline --all -- src/main/java/org/example/videoclips/editing tools docs/production-cinematic-highlight-editing-plan.md
mvn -o -Dtest=CinematicHighlightAnalyzerTest,HighlightDirectorFlowServiceTest,LocalCvVisualAnalysisProviderTest test
```
The final command passed locally on 2026-07-21 with eight tests. Offline Maven fails closed if required artifacts are absent; do not remove `-o` to make it pass.

View File

@ -0,0 +1,366 @@
---
name: video-editing-research-methodology
description: "Load when turning an observation or hunch in this video-editing repository into a controlled experiment, preregistering numeric predictions, designing baselines/counterfactuals/ablations, assigning adversarial review, managing a disabled experiment flag, replicating offline on macOS and Linux, deciding whether a result is a candidate for adoption, or retiring an inconclusive or disproven idea without losing the negative evidence."
---
# Video Editing Research Methodology
## Purpose
Turn a hunch into an auditable result without weakening production controls. Require one causal
**mechanism** (the rules by which a cause produces an outcome) to explain positive observations,
negative observations, boundary cases, and failures. Accept it only after an assigned reviewer tries
to refute it with a prediction that differs from competing explanations.
Use repository facts verified on **2026-07-21**. Treat numeric targets from planning documents or
sibling skills as candidate thresholds until `video-editing-change-control` approves and freezes them.
## Route The Work
| Need | Use this skill? | Exact sibling |
|---|---:|---|
| Form a hypothesis, preregister numbers, run a research lifecycle, or retire an idea | Yes | This skill |
| Select an open problem or make an external novelty claim | No | `video-editing-research-frontier` or `video-editing-external-positioning` |
| Derive a metric, invariant, causal discriminator, or uncertainty calculation | No | `video-editing-proof-and-analysis-toolkit` |
| Execute the current single-source cinematic-quality program | No | `video-editing-cinematic-highlights-campaign` |
| Define release evidence, goldens, or the acceptance portfolio | No | `video-editing-validation-and-qa` |
| Diagnose an unexplained live failure | No | `video-editing-debugging-playbook`, then `video-editing-failure-archaeology` |
| Change code, dependencies, configuration, defaults, runtime, or promotion status | No | `video-editing-change-control` |
| Look up an existing property or add a configuration axis | No | `video-editing-config-and-flags` |
Do not use this skill for routine implementation, incident triage, release certification, or current-campaign execution; use the exact sibling named in the table.
Do not label missing authentication, offline packaging, fail-closed behavior, reproducible builds,
measured QA, or deployment controls as research. Those are engineering obligations. Use research only
when two or more plausible mechanisms need a controlled comparison.
## Define The Terms Once
| Term | Definition in this project |
|---|---|
| Hypothesis | A falsifiable claim that names a mechanism and predicts measured outcomes before data collection. |
| Baseline | The frozen current or simplest approved method against which a candidate is compared. |
| Counterfactual | An otherwise equivalent case in which the proposed cause is absent or changed. |
| Ablation | A counterfactual made by removing exactly one candidate component while holding the rest fixed. |
| Negative control | A case where the mechanism predicts no effect; it detects leakage, confounding, and false positives. |
| Slice | A preregistered subgroup such as source category, speech presence, lighting, codec, platform, or failure mode. |
| Holdout | Data frozen before tuning and opened only for the confirmatory run. |
| Uncertainty | A stated interval or error bound around an estimate, produced by a preregistered method. |
| Replication | A repeat on independently provisioned runtime or unseen sources, not a rerun of the same warm process. |
| Adversarial reviewer | An independent person assigned before results to construct the strongest counterexample and try to falsify the mechanism. |
| Candidate | A replicated research result eligible for change-control review; it is not production-ready. |
| Retirement | A documented decision to stop an idea while preserving its prediction, evidence, and revisit condition. |
| nDCG | Normalized discounted cumulative gain, a ranking metric that rewards relevant candidates more when they appear earlier. |
## Enforce The Research Perimeter
Stop before experimentation when any item fails:
- [ ] Run with network access denied. Do not call external AI services, model hubs, package indexes,
remote telemetry, or loopback model services during certified experiments.
- [ ] Pre-provision every dependency and model. Record immutable version, SHA-256, source, license,
approved use, platform, runtime loader, and resource envelope. Never bootstrap or download at runtime.
- [ ] Use only rights-cleared footage, voices, music, sound effects, fonts, LUTs, and model outputs.
Record consent or voice rights where applicable. Generated output is not automatically licensed.
- [ ] Reject missing capabilities. Never replace requested voiceover, music, or SFX with silence,
tones, host TTS, generic media, another model, or another analysis mechanism.
- [ ] Obtain explicit render approval before FFmpeg creates a candidate render. Analysis permission is
not render permission.
- [ ] Leave every production-facing packaged default unchanged. Activate research only in an isolated,
approved environment with a dedicated experiment control whose safe default is off.
- [ ] Keep research inputs and results out of production queues, tenant data, normal application logs,
and uncontrolled metrics. Preserve privacy and retention classifications.
- [ ] Route any executable code, flag, schema, fixture, dependency, or behavior change through
`video-editing-change-control` before use.
Do not run `tools/run_local_cv_worker.sh`, `tools/run_local_asset_worker.sh`, or ordinary application
startup as an experiment. On the verified baseline, those paths can install packages, resolve model
weights, scan inputs, or render. Inspect first:
```bash
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
rg -n 'auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:' src/main/resources/application.yml
```
## Use The Research Lifecycle
Never skip a gate. A failed gate produces a negative result or retirement record, not an improvised
change to the method.
| Stage | Required artifact | Gate to advance |
|---:|---|---|
| 0. Observe | Observation record with evidence label | The symptom is reproducible or explicitly marked historical/open. |
| 1. Hypothesize | Preregistered experiment record | Mechanism, alternatives, numeric predictions, controls, slices, and stop rules are frozen. |
| 2. Isolate | Approved disabled experiment control and evidence location | Default is off; scope, owner, expiry, cleanup, and production isolation are verified. |
| 3. Calibrate | Frozen baseline on development data | Metric definitions, denominators, uncertainty method, and data exclusions are fixed. |
| 4. Discriminate | Positive, negative, boundary, counterfactual, and ablation results | Competing mechanisms make different predictions and the observations select among them. |
| 5. Replicate | Unseen-source and platform replication packet | Result repeats offline on supported macOS development and target Linux/VPS runtime. |
| 6. Refute | Signed adversarial review | Independent reviewer cannot produce an unexplained counterexample within the preregistered scope. |
| 7. Decide | Adoption candidate or retirement record | Evidence is complete; no result is silently left behind an expiring flag. |
| 8. Govern | Change-control decision | Adopt through normal gates or remove/disable and preserve the negative ledger. |
### Stage 0: Record The Observation
Distinguish an observation from an explanation. Record the exact source commit, input identity,
effective configuration, environment, and command that exposed it. Label a generated file, plan
checkbox, or hard-coded QA boolean `NOT EVIDENCE` when it cannot support the claim.
Before inventing a new explanation, search the established history:
```bash
git log --oneline --decorate --all
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|fallback|silence|tone' docs src tools
```
Use `video-editing-failure-archaeology` to determine whether the mechanism is already fixed, active,
partial, historical, or open. Do not rerun a retired path unchanged.
### Stage 1: Preregister A Numeric Hypothesis
Write and freeze the experiment record before inference, rendering, or opening the holdout. Make the
hypothesis predict numbers, not adjectives. Include expected direction, minimum effect, allowed
regression, denominator, interval method, and expected values for every negative control.
When no trustworthy baseline exists, run a labeled **exploratory baseline study** first. Use its data
only to design a later confirmatory study. Do not set a threshold from a dataset and claim success on
that same dataset.
For highlight work, name metrics such as nDCG@3, Recall@3 at a stated temporal IoU, false-positive
rate on no-highlight footage, per-category effects, and blinded preference. For audio/render work,
name probe-derived duration, loudness, true peak, silence/tone rejection, A/V synchronization,
technical failure rate, and blinded rubric dimensions. Get formulas and current campaign floors from
the proof toolkit and cinematic campaign; do not duplicate or silently revise them here.
### Stage 2: Isolate With A Safe Experiment Control
The verified repository has no dedicated experiment registry. Current scheduler, fallback, model,
and render properties are operational controls, not research assignment controls. Do not repurpose
them and do not treat `render-enabled` or `fallback-to-heuristic` as experiment flags.
Introduce any experiment control only through change control. Require this contract:
| Field | Required value |
|---|---|
| Identifier | Stable, descriptive, and unique; never a user or media identifier. |
| Owner | Named accountable team/person and independent reviewer. |
| Safe default | `false` in every packaged configuration and `matchIfMissing`; missing means disabled. |
| Scope | Approved fixtures/environments only; never implicit all-traffic assignment. |
| Start and expiry | UTC dates with a short expiry and automated or reviewed removal checkpoint. |
| Activation | Explicit test/experiment configuration outside production defaults. |
| Observability | Bounded, low-cardinality experiment ID; no source path, prompt, secret, or personal data. |
| Failure | Fail the experiment; never fall back to baseline and label the result candidate. |
| Removal | Delete or permanently disable after adoption/retirement; owner verifies no stale path remains. |
Do not create a speculative flag without a funded experiment, owner, decision date, and deletion
condition. A flag is not approval to render or bypass asset/model preflight.
### Stage 3: Freeze Baselines, Inputs, And Analysis
Use a source-disjoint development/holdout split. Compute a corpus-manifest SHA-256 and split SHA-256.
Freeze the baseline commit, candidate commit, toolchain, feature schema, random seeds, render plan,
models, assets, licenses, and analysis script before the confirmatory run.
| Design element | Required practice | Common invalid result |
|---|---|---|
| Baseline | Run the same inputs, candidate budget, render profile, and reviewer protocol. | Candidate gets more clips, assets, compute, or review time. |
| Counterfactual | Change only the claimed cause and verify output equivalence elsewhere. | Two renders differ in timing, encoding, or mix beyond the intended factor. |
| Ablation | Remove one component per variant; preregister multiple-comparison handling. | A kitchen-sink comparison cannot assign causality. |
| Negative control | Include shuffled timestamps, no-highlight footage, silence/tone, missing model, or unrelated audio as appropriate. | Positive-only examples hide leakage or fallback. |
| Slice | Report each preregistered category/platform/failure slice and its denominator. | Aggregate improvement hides a harmed category. |
| Uncertainty | Report confidence interval or error bound and method; include reviewer disagreement. | A point estimate or one clip is called a result. |
| Exclusion | Freeze objective exclusion rules before results; publish every exclusion count/reason. | Failed outputs disappear from the denominator. |
### Stage 4: Execute Once, Preserve Everything
Run only the exact approved commands in the record. Capture stdout/stderr, exit codes, timestamps,
resource measurements, raw per-item results, failed outputs, and hashes. Do not edit the hypothesis,
threshold, split, or exclusion rule after seeing results. Record deviations and classify the run
`invalid`; preregister a replacement instead of repairing history.
Add every failed prediction, neutral ablation, negative control, and platform difference to the
negative-result ledger. Preserve a false start even when a later variant works.
### Stage 5: Replicate Offline On macOS And Linux
Use the same committed source and immutable corpus, split, dependency, model, and asset manifests.
Provision them before execution through the approved artifact process. Deny egress for the whole
process tree. Record OS version, architecture, Java, Maven, Python runtime if applicable, FFmpeg,
ffprobe, CPU/GPU, memory, precision, thread/concurrency settings, seeds, and hashes.
Define replication before running as either:
- **Byte replication:** identical artifact SHA-256 is required; or
- **Metric-bounded replication:** deterministic bytes are not expected, so every preregistered metric
must remain within an approved tolerance and every categorical gate must agree.
Do not average platform disagreement away. Mark the result platform-specific or inconclusive when
one supported platform fails. A warm-cache repeat on one machine is not replication.
### Stage 6: Assign Adversarial Refutation
Assign the reviewer before results. The author, experiment operator, fixture annotator, and promoter
must not be the sole adversarial reviewer. Blind media variants and randomize order where practical.
Require the reviewer to:
1. State the strongest competing mechanism.
2. Identify one case where the proposed mechanism predicts an effect and the competitor does not.
3. Identify one negative case where the proposed mechanism predicts no effect.
4. Look for data leakage, source overlap, hidden fallbacks, changed budgets, reviewer unblinding,
selective exclusions, uncontrolled randomness, and platform-specific behavior.
5. Recompute a sample from raw data and verify hashes/licenses for a sample of inputs/models/assets.
6. Explain every positive, negative, boundary, and failed observation with the proposed mechanism.
7. Return `supported`, `refuted`, or `inconclusive`; never return “looks good.”
Refutation is not an approval gate substitute. A supported result still goes through change control,
validation, security, operations, licensing, and render approval.
### Stage 7: Adopt Or Retire
Classify the idea exactly once at the decision date:
| Status | Meaning | Required next action |
|---|---|---|
| `candidate` | First controlled result met its preregistered gate. | Replicate; do not promote. |
| `replicated-candidate` | Held-out and platform replication passed adversarial review. | Submit the eight-part decision packet to change control. |
| `inconclusive` | Evidence cannot distinguish mechanisms or uncertainty crosses the gate. | Retire temporarily or fund a newly preregistered discriminator. |
| `refuted` | A prediction failed or a counterexample broke the mechanism. | Retire; remove/disable the experiment path. |
| `retired` | Work stopped for evidence, cost, rights, safety, operability, or priority reasons. | Preserve the ledger, stop reason, and revisit condition. |
For an adoption candidate, provide: requirement; selected approach; alternatives; benefits/trade-offs;
operational consequences; security implications; verification; and revisit conditions. Then follow
`video-editing-change-control`. Research status never changes a production default.
## Use These Records
Keep each record in the approved review artifact or a repository docs-of-record location established
through change control. Do not invent a new production path or store secrets, proprietary model
weights, personal data, or unlicensed media in the record.
### Experiment Record Template
```text
Experiment ID / title / UTC date:
Owner / operator / independent adversarial reviewer:
Observation and evidence label:
Question:
Mechanism:
Strongest competing mechanisms:
Baseline commit and exact method:
Candidate commit and safe-disabled experiment control:
Flag owner / safe default=false / start / expiry / deletion condition:
Corpus-manifest hash / split hash / annotation version / licenses:
Dependency, binary, model, asset hashes and approved-use records:
macOS and Linux runtime/hardware/resource envelope:
Primary metric, denominator, formula, predicted baseline and candidate numbers:
Minimum effect / allowed regression / uncertainty method / alpha or interval:
Slice predictions and minimum sample counts:
Positive / boundary / negative controls and predicted numbers:
Counterfactuals / one-factor ablations / multiple-comparison method:
Randomization / blinding / seeds / exclusion rules:
Exact offline commands and egress-denial evidence:
Render approval reference, if rendering is required:
Stop rules / retirement rule / revisit condition:
Raw evidence location and retention/classification:
Observed results, deviations, failed predictions, and uncertainty (after run):
Replication outcome by platform (after run):
Disposition and change-control reference (after review):
```
### Adversarial Review Template
```text
Reviewer / UTC date / independence conflicts:
Mechanism reviewed:
Strongest alternative:
Differentiating positive prediction:
Required negative prediction:
Counterexample attempted and exact command/data:
Leakage/fallback/blinding/exclusion/platform checks:
Raw result recomputation and hash/license sample:
Unexplained positive, negative, boundary, or failure observations:
Verdict: supported | refuted | inconclusive
Required follow-up and change-control blockers:
```
### Negative-Result Ledger Template
```text
Experiment ID / UTC date / commit:
Hypothesis and preregistered numeric prediction:
Observed number and uncertainty:
Failed slice/control/ablation/platform:
Evidence hashes and exact command:
Why the mechanism failed or remains unresolved:
Decision: refuted | inconclusive | retired
Experiment control removed/disabled by / date:
Do not retry unchanged because:
Revisit only when this falsifiable condition changes:
```
## Stop And Revisit Discipline
Stop immediately for a rights gap, checksum mismatch, unexpected network attempt, automatic download,
external AI call, missing model, fallback, placeholder audio, unapproved render, path escape, sensitive
data leak, production-default drift, invalid blinding, split contamination, or corrupted evidence.
Stop the idea at the preregistered resource cap, sample size, failure-rate limit, effect threshold, or
expiry date. Do not extend until significance appears. Revisit a retired idea only when its recorded
condition changes, such as a newly licensed local model, a larger source-disjoint corpus, a corrected
measurement defect, a new discriminating mechanism, or a documented resource-envelope change.
## Learn From The Repository's Actual Origins
| Historical origin | Methodological rule it established |
|---|---|
| Real storage behavior arrived across commits beginning `7e8a214`; earlier state/metadata was not a durable media path. | Measure the real side effect and artifact existence. A successful state transition is not delivery evidence. |
| Commit `1737d8b` moved terminal failure ownership to the DB queue's retry/DLQ policy. | Put the decision at the layer that owns the mechanism; test transient, terminal, duplicate, and boundary attempts. |
| Commit `5d889b0` said “working version but not cinematic.” | Separate renderability from semantic selection and creative quality; require blinded baselines, rubrics, and ablations. |
| Category-aware planning and richer rendering followed in commits such as `5006dfd`, but no controlled causal comparison is recorded. | Treat richer analysis, effects, and assets as candidates until each contribution survives counterfactual testing. |
| Commit `9b56e89` added structural render QA; the current highlight path adds selected duration/black/silence/peak probes while other checks remain structural. | Diagnostics must distinguish measured outputs from plan/command assertions and predict reviewer-relevant failures. |
| Commit `97ba827` added local asset generation with runtime install/model resolution and silence/tone paths. | “Local” does not mean offline, licensed, immutable, or fail-closed; test missing and corrupted capabilities as negative controls. |
Use `video-editing-failure-archaeology` for the full chronicle. History suggests research questions;
it does not prove a current mechanism.
## Completion Checklist
- [ ] Observation, mechanism, alternatives, and numeric predictions were frozen before the run.
- [ ] Baseline, counterfactuals, one-factor ablations, negative controls, boundary cases, slices,
denominators, exclusions, and uncertainty were specified.
- [ ] Dependencies, models, assets, inputs, splits, licenses, approvals, and checksums are complete.
- [ ] The experiment was isolated behind an approved safe-disabled control with owner and expiry.
- [ ] No download, network, external AI, unlicensed asset, placeholder, unapproved render, fallback, or
production-default change occurred.
- [ ] The result replicated offline on supported macOS and target Linux/VPS, or is labeled otherwise.
- [ ] An independent reviewer attempted adversarial refutation and explained positive and negative cases.
- [ ] Failed predictions and neutral results are in the negative-result ledger.
- [ ] The idea became a replicated candidate routed through change control, or was documented and retired.
## Provenance and maintenance
Facts and volatile repository observations were rechecked on **2026-07-21** against
`application.yml`, current source/tests, the six cited commits, and the completed sibling skills.
Re-verify the research perimeter and operational-control defaults:
```bash
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
rg -n -C 3 'fallback-to-heuristic:|strict-runtime:|local-director:|highlight-scheduler:|render-enabled:|require-director-approval:' src/main/resources/application.yml
```
Re-verify historical claims and measurement implementations:
```bash
git show -s --format='%h %cs %s' 7e8a214 1737d8b 5d889b0 5006dfd 9b56e89 97ba827
rg -n 'RETRY_SCHEDULED|markTerminalFailure|duration_matches_timeline' src/main src/test
```
Re-verify the absence or presence of a dedicated experiment registry before documenting a flag:
```bash
rg -n -i 'experiment|feature.?flag' pom.xml src/main src/test src/main/resources docs
```
If any command changes its result, update this skill and the authoritative sibling in the same
knowledge-only change. Do not reinterpret drift as approval to alter behavior.

View File

@ -0,0 +1,345 @@
---
name: video-editing-run-and-operate
description: "Operate the video-editing service safely: choose and start exactly one of its REST clipping, folder segmentation, multi-clip cinematic edit, or single-source highlight workflows; perform preflight, ingest and approve work, locate artifacts, inspect health and queue state, quarantine failures, and stop or resume jobs. Load this skill for runtime commands, operator incidents, local-model readiness, output locations, scheduler behavior, or deployment-readiness questions."
---
# Run and Operate Video Editing
Use this runbook from the repository root. It describes code verified on **2026-07-21**.
## Scope and safety boundary
Operate only an explicitly approved workflow. The checked-in `application.yml` still enables folder ingestion, editing, both editing schedulers, local worker auto-start, and heuristic fallback, while highlight rendering now defaults off and director approval defaults on. A plain `mvn spring-boot:run` or packaged-JAR start can therefore claim input and invoke the network-capable local-CV bootstrap path. The `cinematic-editing-local` profile explicitly enables the separate local-director auto-render path behind its approval flag.
`video-editing-change-control` is authoritative for the seven no-waiver prohibitions. Their operating consequences are:
- automatic dependency or model downloads;
- no network use for model/media inference or acquisition, including loopback RPC, and no external AI; use in-process or approved non-network IPC with pre-provisioned artifacts;
- unlicensed music, SFX, fonts, LUTs, models, or footage;
- silence, generated tones, or heuristic analysis presented as production assets or evidence;
- rendering without a reviewed plan and approval artifact;
- a production-facing default change outside change control.
**Current implementation warning:** strict asset readiness now fails startup and the Java/Python asset workers no longer accept silence, tones, or host speech as successful fallbacks. The repository still lacks resident Piper, MusicGen, and AudioGen model artifacts, and the standalone bootstrap script remains prohibited. Do not start the production profile until approved local paths are provisioned.
### When not to use this skill
| Need | Use instead |
|---|---|
| Change a flag or add a profile | `video-editing-config-and-flags` |
| Install tools or reconstruct the build | `video-editing-build-and-env` |
| Diagnose a failure after locating its state | `video-editing-debugging-playbook` |
| Decide whether an output passes | `video-editing-validation-and-qa` |
| Change runtime behavior or promote it | `video-editing-change-control` |
| Measure FFmpeg, queue, storage, or media behavior | `video-editing-diagnostics-and-tooling` |
## Definitions
| Term | Meaning here |
|---|---|
| Claim | An atomic move from a source directory to a working directory. The source name is not available for a second worker after the move. |
| Publish | Copy or upload a completed artifact to its externally consumed location. |
| Approval artifact | The current plain `approved.flag` scheduler signal. Its contents are not validated, so it is not production authorization. |
| Quarantine | Move failed input to `rejected`, or leave/rename it in `working` when that move fails. |
| Local model | Model weights provisioned before startup on the same runtime host; no request may fetch weights or call an external inference service. |
| Project store | A filesystem tree below `output/edit-projects` or `output/highlight-projects`; it is not PostgreSQL-backed. |
## Current system versus production target
| Concern | Current, verified | Required production target |
|---|---|---|
| Runtime packaging | Spring Boot JAR; no container or deployment manifest | One immutable platform-neutral application artifact plus signed platform-specific runtime/model/image bundles, each promoted without rebuild for the same target |
| REST metadata | Memory by default; `jpa` profile uses H2 in-memory | PostgreSQL with validated migrations, backup/restore, credentials, TLS, readiness, and persistent queue |
| REST media | Memory adapter emits example URLs and stub bytes; S3 adapter exists | Approved object storage with real multipart lifecycle, least privilege, encryption, retention, and integration evidence |
| Edit/highlight state | Local filesystem | Durable shared storage or an explicitly justified single-writer volume with recovery evidence |
| Highlight intelligence | Noncompliant loopback CV endpoint plus manual/external director-plan creation | All required director, vision, voice, music, and SFX models pre-provisioned; inference uses in-process or non-network IPC and fails closed |
| Asset failure | Non-strict readiness can log and continue, but synthesis now rejects missing/unlicensed models and inaudible output; requested render assets require license sidecars | Production startup must fail closed; add checksum, origin, allowed-use, and creative-quality gates beyond the current sidecar/audibility checks |
| Security | No Spring Security; Actuator exposes health, info, metrics, Prometheus | Authentication (AuthN), authorization (AuthZ), secured Actuator, restrictive network policy, audit trail, redaction, and threat-model gates |
There is no production profile, Open Container Initiative (OCI) image, Compose file, Kubernetes/Helm definition, or cloud deployment contract in this repository. Do not invent a deployment command. A Linux/VPS/cloud run is a **target**, not a certified procedure.
| Profile | Verified effect and operational status |
|---|---|
| no profile | Unsafe all-on file schedulers/editing defaults from `application.yml`; API uses memory/stub adapters. |
| `cinematic-editing-local` | Disables folder segmentation but enables local director auto-render with an approval gate; it does not disable the highlight scheduler inherited from base configuration. Do not use as a safe baseline. |
| `folder-scheduler-local` | Enables folder segmentation but does not disable inherited editing/highlight defaults. Do not use as an isolated folder lane. |
| `jpa` | Selects JPA and DB queue with H2 in PostgreSQL compatibility mode and stub processing. It is not a PostgreSQL production profile. |
## Preflight every run
Run read-only checks before approval. These commands neither start the service nor modify repository files:
```bash
java -version
mvn -version
ffmpeg -version
ffprobe -version
test -f pom.xml
test -f target/video-editing-1.0-SNAPSHOT.jar
```
The project requires Java 21. An offline Maven command fails instead of downloading missing dependencies:
```bash
mvn -o -DskipTests package
```
Do not replace `-o` with an online build. Provision Maven artifacts through an approved, audited build process. Do not run either `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh`: their `auto` modes can create virtual environments, install packages, and load/download a YOLO model.
Before enabling any media workflow, check directories and binaries without starting it:
```bash
test -x "$(command -v ffmpeg)"
test -x "$(command -v ffprobe)"
find input -maxdepth 4 -type f -print 2>/dev/null
find output -maxdepth 4 -type f -print 2>/dev/null
```
Treat every listed input as work that a scheduler may claim. Confirm ownership, license, retention, and approval before continuing.
## Disabled-first baseline
Use this exact baseline for inspection-only startup. It binds to loopback, disables all file schedulers, editing, local worker bootstrap, rendering, fallbacks, and cleanup. `-o` prevents Maven dependency downloads.
```bash
env \
SERVER_ADDRESS=127.0.0.1 \
FOLDER_SCHEDULER_ENABLED=false \
VIDEO_EDITING_ENABLED=false \
VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED=false \
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED=false \
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED=false \
VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=false \
VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=true \
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL=true \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=false \
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
mvn -o spring-boot:run
```
For a prebuilt artifact, replace the last line with:
```bash
java -jar target/video-editing-1.0-SNAPSHOT.jar
```
Environment variables precede the command and override packaged values. Do not add `cinematic-editing-local` or `folder-scheduler-local`; those profiles enable work. Startup success only proves Spring context creation, not dependency connectivity: the repository, queue, and object-storage health indicators always report `UP` with adapter names and do not perform a real probe.
## Workflow 1: REST clipping API
### Current behavior
The API implements multipart-session metadata, clip jobs, cancellation, events, and signed URL responses. It is not a production upload service with default adapters: memory storage returns `storage.example` URLs, materializes stub content, and memory/stub state disappears on restart.
| Object | States |
|---|---|
| Asset | `PENDING_UPLOAD`, `UPLOADED`, `PROCESSING`, `READY`, `DELETED` |
| Upload | `OPEN`, `COMPLETED`, `ABORTED`, `EXPIRED` |
| Clip job | `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `CANCEL_REQUESTED`, `CANCELLED` |
| DB queue message | `PENDING`, `PROCESSING`, `COMPLETED`, `DLQ` |
### Start and exercise safely
Use the disabled-first baseline. It leaves API defaults at memory repository, memory storage, and stub processing. These are contract-demonstration adapters only; label all results non-production. Send `X-Tenant-Id` and stable `Idempotency-Key` values on create calls. Relevant routes are:
```text
POST /v1/video-assets
POST /v1/video-assets/{assetId}/uploads:complete
GET /v1/video-assets/{assetId}
GET /v1/video-assets/{assetId}/upload-session
POST /v1/video-assets/{assetId}/clip-jobs
GET /v1/video-assets/{assetId}/clip-jobs
GET /v1/video-assets/{assetId}/events
DELETE /v1/video-assets/{assetId}
GET /v1/clip-jobs/{jobId}
GET /v1/clip-jobs/{jobId}/clips
GET /v1/clip-jobs/{jobId}/events
POST /v1/clip-jobs/{jobId}:cancel
POST /v1/clips/{clipId}/download-url
```
A job publish on the memory queue invokes processing asynchronously in the same JVM. The database queue polls, claims with a visibility timeout, retries, and moves exhausted messages to `DLQ`. Cancellation is cooperative only before clip generation; do not claim that it interrupts FFmpeg.
### PostgreSQL and S3 gate
`application-jpa.properties` selects JPA plus the DB queue but configures H2, not PostgreSQL. PostgreSQL migrations `V1` through `V6` exist, but there is no production datasource profile or Testcontainer certification. The S3 adapter is present, but no approved environment, credentials contract, or end-to-end multipart validation is checked in. Treat both as candidates. Do not set `video-clipping.storage=s3`, `video-clipping.repository=jpa`, or `video-clipping.queue=db` in production until change control approves integration, recovery, security, and fault-injection evidence.
Scratch files appear under `tmp/ffmpeg-input`, `tmp/ffmpeg-output`, `tmp/in-memory-storage`, and `tmp/stub-output`. Successful processing deletes staged/generated worker files when `cleanup-local-files=true`; the scheduled cleanup also removes aged files when global cleanup is enabled.
## Workflow 2: folder segmentation
This scheduler turns one source video into numbered fixed-duration clips. It is independent of the REST API and edit project stores.
### Approved start
Obtain explicit approval for the exact source and output directories. Start only with all editing lanes off:
```bash
env \
SERVER_ADDRESS=127.0.0.1 \
FOLDER_SCHEDULER_ENABLED=true \
VIDEO_EDITING_ENABLED=false \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
mvn -o spring-boot:run
```
Defaults are `input/source`, `input/working`, `input/processed`, `input/rejected`, and `output/clips`, polled every five seconds with eight-second segments. Override paths through the documented `FOLDER_SCHEDULER_*` variables before approval when isolation is required.
### Ingest, claim, publish, quarantine
1. Copy a complete source as `name.mp4.part` in `input/source`.
2. Verify size and checksum outside the service.
3. Atomically rename it to a supported final extension only after approval. Accepted extensions are `mp4`, `mov`, `m4v`, `mkv`, `webm`, and `avi`; hidden files and `.tmp`, `.part`, `.download`, `.failed` are ignored.
4. Expect an atomic move to `input/working`.
5. Expect clips in `output/clips/<stem>[-N]/` and then the source in `input/processed`. The first output uses `<stem>`; if that output directory already exists, the clipper chooses `<stem>-1`, `<stem>-2`, and so on.
6. On validation/FFmpeg failure, expect the source in `input/rejected`. If quarantine movement fails, expect a `.failed` name in working.
The scheduler handles one naturally sorted candidate per poll. Output-directory collisions cause suffixing as described above. Source lifecycle collisions are different: moving a source to `working`, `processed`, or `rejected` refuses to overwrite an existing same-name file and fails/quarantines the attempt. `preserve-input-quality=true` uses stream copy where possible; false transcodes with the configured preset. Neither mode alone proves cinematic or delivery quality.
Restart resumes only files still visible as new candidates. A file stranded in `working` is not automatically reclaimed. Diagnose it, preserve evidence, and move it back to source only after proving that no process is active and that output collision handling is understood.
## Workflow 3: multi-clip cinematic edit
This lane accepts a folder of source clips, analyzes them, creates a filesystem project, imports a director plan, and can render `final.mp4`.
### Approved analysis start
Keep automatic rendering and asset/CV bootstrap disabled:
```bash
env \
SERVER_ADDRESS=127.0.0.1 \
FOLDER_SCHEDULER_ENABLED=false \
VIDEO_EDITING_ENABLED=true \
VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED=true \
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED=false \
VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=false \
VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=true \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=false \
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
mvn -o spring-boot:run
```
Stage a complete project folder as `input/editing/source/<project>.tmp`, then atomically rename it to `<project>`. The scanner also groups loose root clips into `source-clips`, so do not leave unrelated media there. Temporary suffixes `.tmp`, `.part`, `.download`, and `.processing` are ignored.
Expected movement is `source/<project>` to `working/<project>` to `processed/<project>`. Failure moves it to `rejected/<project>` or creates a `.failed` marker/name when possible. The project tree is:
```text
output/edit-projects/<project>/
project.json analysis.json
ai-director-prompt.md director-readme.md
thumbnails/ contact-sheets/
proxies/ audio/
assets/requests/ inbox/
rendered-clips/ render-work/
edit-plan.json render-manifest.json
qa-report.json final.mp4
```
Project states are `CREATED`, `ANALYZING`, `ANALYZED`, `WAITING_FOR_DIRECTOR`, `PLANNING`, `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`. Inspect state without modifying files:
```bash
curl --fail-with-body --show-error --silent --connect-timeout 2 --max-time 10 \
http://127.0.0.1:8080/v1/edit-projects/PROJECT_ID
```
Expected: HTTP 200 with the requested project. Exit 22 means an HTTP error such as unknown project; exit 7 means connection failure; exit 28 means a bound timeout. Preserve the response and branch to `video-editing-debugging-playbook`; do not retry without classifying it.
Place strict JSON at `inbox/edit-plan.json`. The inbox scanner validates and archives imports; rejected plans receive a rejected name and do not authorize render. Review source bounds, timeline, transitions, asset provenance, license, voice audibility, and all generated requests. **Do not call the unauthenticated `POST /v1/edit-projects/{projectId}:render` endpoint.** It has no approval check. It remains prohibited until authenticated authorization and auditable approval bound to source, plan, configuration, model, and asset digests are implemented and tested. Automatic render is also non-production until the same binding exists; a bare `inbox/approved.flag` is insufficient.
Do not render when any required asset is missing or was created by a placeholder path. A render writes segment files, `final.mp4`, `render-manifest.json`, and `qa-report.json`; the QA report is evidence to validate, not automatic acceptance.
## Workflow 4: single-source cinematic highlights
This is the strategically primary lane, but it is **not production-ready**. Current analysis calls a loopback local-CV HTTP endpoint, while director-plan authorship remains manual/external. No in-service local director model exists. Strict asset readiness now fails startup when the pre-provisioned Piper, MusicGen, and AudioGen runtime is incomplete, and placeholder success paths have been removed. No complete three-model bundle is checked in, so no current start command satisfies the production requirement of fully local, runtime-resident intelligence with fail-closed cinematic assets.
### No approved operation yet
Do not start this workflow as a certified or approved path. Its visual analysis requires the current loopback HTTP worker, and loopback RPC violates the no-network model/media rule. First replace that boundary with an in-process call or approved non-network IPC through `video-editing-change-control`; then validate the replacement through `video-editing-cinematic-highlights-campaign`. Use the read-only artifact inspection below for existing projects.
Stage one complete file as `.part`, then atomically rename it in `input/highlights/source`. Claim and quarantine rules match folder ingestion: source to working, then processed on success or rejected on failure. The source is also copied into its project.
```text
output/highlight-projects/<project>/
project.json manifest.json
source/ analysis/
director/director-brief.md director/director-prompt.md
director/edit-plan.json director/approved.flag
assets/ highlights/<highlight-id>/
render-manifest.json final.mp4
```
Analysis writes `analysis/source-analysis.json`, `ffprobe.json`, `scene-segments.json`, `audio-analysis.json`, `visual-analysis.json`, plus frames, a contact sheet, proxy, and waveform. The scheduler then calls `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Its scores are deterministic ranking hints from coarse source/shot/audio evidence, not proof of semantic importance. States are `CREATED`, `ANALYZING`, `WAITING_FOR_DIRECTOR`, `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`; a validated plan remains `PLANNED` while required local assets are pending.
Do not enable rendering until a change-controlled local director implementation, immutable local model inventory, license manifest, and fail-closed asset generation are verified. If running a non-production experiment after explicit approval, require `director/approved.flag`; the scanner selects only `WAITING_FOR_DIRECTOR` or `PLANNED` projects with `director/edit-plan.json` and no project-root `final.mp4`. It writes each `highlights/<id>/final.mp4`, then concatenates/copies them to project-root `final.mp4` and writes the root manifest. Plan rejection or render failure marks the project `FAILED`, which the scanner skips; investigate and repair through an explicit recovery procedure rather than changing the status by hand.
## Health, metrics, and logs
For a deliberately started inspection-only service, the following bounded API diagnostics use the service's loopback operator boundary; they do not call model workers:
```bash
for path in health info metrics prometheus; do
curl --fail-with-body --show-error --silent --connect-timeout 2 --max-time 10 \
"http://127.0.0.1:8080/actuator/$path" || exit $?
done
```
Expected: all four requests return HTTP 200. Exit 22 means an endpoint is unavailable/denied, exit 7 means no listener, and exit 28 means a timeout. Stop and route the symptom to `video-editing-debugging-playbook`; do not broaden Actuator exposure to make the probe pass.
The DB queue exports `video.clipping.queue.pending`, `.processing`, `.dlq`, and `.oldest.pending.age.seconds` only when `video-clipping.queue=db`. Editing observability is primarily structured-looking `event=...` log text, not metrics or traces. Search for workflow-specific events:
```bash
rg 'event=(folder_|candidate_|processing_|local_director_|edit_|highlight_|local_asset_|local_cv_)' SERVICE_LOG
```
Do not expose Actuator beyond loopback in the current service. Health `UP` is not proof that storage, repository, queue, models, or FFmpeg work.
## Stop, restart, resume, and cleanup
1. Stop ingestion first by terminating the single application process; flags are not dynamically reloadable.
2. Allow active FFmpeg work to finish when possible. The application has no verified drain/readiness protocol.
3. Record process logs, `project.json`, plans, manifests, QA reports, and working/rejected listings before changing files.
4. Classify every `working` item as active, safely resumable, or quarantined. The file schedulers do not generally reclaim working input after restart.
5. For REST memory adapters, expect all metadata/queue state to disappear. For DB queue, expired `PROCESSING` claims become claimable after the visibility timeout.
6. Never delete outputs, caches, or failed inputs during incident analysis.
Shared edit caches are `input/highlights/assets/{music,sfx,fonts,luts}` and `output/highlight-projects/_voiceover-cache`. Treat them as controlled artifact repositories: verify hash, provenance, license, model/version, and owner before use. The scheduled cleanup does not clean these project trees or shared caches. Global cleanup can delete REST source/clip records and scratch files; keep it disabled until retention and recovery are approved.
Use a new project ID or source filename for a clean retry. Reusing names can collide with processed, rejected, project, or output paths. Never resolve a collision by deleting evidence or enabling overwrite without change control.
## Production promotion checklist
- [ ] One workflow is selected; every other scheduler and renderer is explicitly disabled.
- [ ] Artifact was built offline/reproducibly and traced to a reviewed commit.
- [ ] All local models and dependencies are immutable, pre-provisioned, hashed, licensed, and network-independent.
- [ ] Missing/unhealthy models fail startup or the use case; no heuristic, silence, or tone fallback can publish.
- [ ] Director plan and render approval are authenticated, authorized, auditable, and tested.
- [ ] PostgreSQL/S3 or replacement adapters pass real integration, retry, concurrency, backup, restore, and retention tests.
- [ ] Filesystem state is durable and single-writer/shared-storage semantics are explicit.
- [ ] Health probes test real dependencies; logs, metrics, traces, SLOs, dashboards, and alerts have runbooks.
- [ ] Graceful shutdown, restart, duplicate claim, stuck working item, and DLQ recovery are exercised.
- [ ] Security, vulnerability, secret, image, and license gates pass with no unresolved critical/high findings.
- [ ] `video-editing-validation-and-qa` accepts objective media and human-review evidence.
- [ ] `video-editing-change-control` approves promotion. No checklist item may be waived silently.
## Provenance and maintenance
Ground truth: `application.yml`, application profiles, `VideoClippingProperties`, controllers, schedulers, project stores, queue/storage adapters, cleanup jobs, local worker scripts, renderers, health indicators, and Maven configuration, inspected 2026-07-21.
Re-verify drift with one-line commands:
```bash
rg -n 'enabled:|auto-start:|auto-render|require-.*approval|fallback-to-heuristic|directory:' src/main/resources/application*.yml
rg -n '@(Get|Post|Put|Delete|Patch)Mapping|@RequestMapping' src/main/java/org/example/videoclips/api
rg -n 'ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
rg -n 'Files\.(move|copy)|StandardCopyOption|\.failed|rejected|processed|working' src/main/java/org/example/videoclips/{folder,editing}
rg -n 'pip install|download|from_pretrained|YOLO|silence|fallbackTone|write_fallback_tone' tools src/main/java/org/example/videoclips/editing
rg -n 'video\.clipping\.queue|Health\.up|management\.endpoints' src/main src/main/resources
rg -n '<java.version>|spring-boot-starter-parent|<artifactId>|<version>' pom.xml
```

View File

@ -0,0 +1,305 @@
---
name: video-editing-validation-and-qa
description: Load this skill when defining acceptance criteria, adding or reviewing tests, validating API/database/local-model/media behavior, interpreting Maven or render QA results, creating fixtures or goldens, or deciding whether a video-editing change is fit for production promotion.
---
# Video Editing Validation and QA
Use evidence to decide whether this service is correct, cinematic, and operable. Treat a green test run as one input, never as a release certificate.
## When not to use this skill
- For symptom-first incident triage, use `video-editing-debugging-playbook`.
- For commands that inspect a running system or media artifact, use `video-editing-diagnostics-and-tooling`.
- For change classification, approval, rollout, or rollback, use `video-editing-change-control`.
- For environment recreation and dependency provisioning, use `video-editing-build-and-env`.
- For runtime profiles and operator workflows, use `video-editing-run-and-operate`.
- For media theory and FFmpeg interpretation, use `cinematic-media-engineering-reference`.
- For the executable highlight-quality program, use `video-editing-cinematic-highlights-campaign`.
- For first-principles experiments and derivations, use `video-editing-proof-and-analysis-toolkit`.
## Non-negotiable release rule
As of 2026-07-21, do **not** describe this repository as production-ready, Fortune 500 reference-quality, or creatively certified. Evidence required for those claims is absent.
Fail a gate when evidence is missing, skipped, stale, environment-dependent, or only asserted. Never convert `unknown` to `pass`.
`video-editing-change-control` is authoritative for the seven no-waiver prohibitions; this table defines their validation evidence. `video-editing-config-and-flags` remains authoritative for defaults.
| Prohibited behavior | Required proof |
|---|---|
| Automatic dependency or model download | Run with network egress denied and pre-provisioned, checksum-pinned dependencies/models. Retain the egress policy and process logs. |
| External AI service access | Deny egress; verify no external endpoint is configured or called. |
| Unapproved network access | Prove model/media inference and acquisition use no network, including loopback; verify in-process or approved non-network IPC. Separately test every approved API/database/storage/telemetry connection for security, timeouts, and failure behavior. Maven/process offline flags alone are not egress denial. |
| Unlicensed music, SFX, fonts, LUTs, footage, or models | Require an asset/model manifest with source, license, version, checksum, and approved usage. Missing provenance blocks use. |
| Silence, tones, or synthetic stand-ins presented as finished assets | Detect and reject placeholders. Do not accept file existence as asset quality. |
| Unapproved rendering | Render only approved fixtures in isolated temporary directories until the change-control gate authorizes broader footage or an environment. |
| Production-facing default changes | Route through `video-editing-change-control`; test both old and proposed behavior and provide rollback evidence. |
The 2026-07-21 working tree removes silence/tone/host-speech fallbacks, restricts AudioCraft to existing local paths with offline flags, and makes strict readiness fail. A zero exit code or output WAV still does not prove semantic fit, license, model identity, loudness, or creative quality; retain final-media and provenance gates.
## Evidence tiers
Classify every claim with the highest tier actually earned.
| Tier | Name | Minimum evidence | Claims it permits |
|---|---|---|---|
| E0 | Assertion | Plan, prompt, comment, checkbox, or unexecuted command | Intent only |
| E1 | Static | Compiles; schema/config/command inspected; deterministic static check passes | Structural property only |
| E2 | Isolated | Deterministic unit, architecture, or adapter test with controlled collaborators | Tested component behavior |
| E3 | Integrated | Real local dependency or executable, representative fixture, machine-read output, no mock at the boundary under test | Integration behavior on the recorded platform |
| E4 | Clean system | Clean checkout, offline/reproducible build, real local models and media tools, source-to-output critical journey | System behavior for the recorded build and environment |
| E5 | Production-like | Security, load, failure injection, observability, container/deploy, and recovery evidence in a production-like environment | Readiness against declared SLOs |
| E6 | Certified | E5 plus independent review, signed evidence, licensed golden inventory, repeatability, and controlled promotion | Reference-quality claim for the tested scope |
Do not promote evidence across scopes. A macOS E3 FFmpeg result does not certify Linux/VPS, container, cloud storage, PostgreSQL, or a different FFmpeg/model build.
## Current evidence inventory
The following inventory was verified on 2026-07-21.
| Area | Current evidence | Honest status |
|---|---|---|
| Maven tests | A 2026-07-21 offline `mvn -q -o verify` run records 245 tests in 62 test classes, 0 failures, 0 errors, 0 skipped in the working tree | E2/E3 mix; not clean-checkout or no-egress certification |
| Coverage | `pom.xml` enforces 100% instruction, line, and branch coverage only for `org.example.videoclips.folder` during `verify` | No repository-wide threshold; no mutation testing |
| Real media tools | `FolderFfmpegIntegrationTest` and `CinematicEditingIntegrationTest` are conditional on local `ffmpeg` and `ffprobe` | Real FFmpeg coverage when available; conditional tests can otherwise skip |
| Clean build | A clean archived checkout has failed `mvn verify` at `LocalAssetGenerationStageTest` because success depends on untracked/runtime-local asset tooling or an audible speech fallback | Release blocker; source-alone reproducibility is unproven |
| API | 18 Spring Boot/MockMvc tests cover selected success, validation, and Problem Details paths | No published/validated OpenAPI, compatibility gate, authentication, or authorization evidence |
| Database | Flyway migrations and PostgreSQL runtime driver exist; normal tests use local/in-memory behavior and H2 can be selected | No real PostgreSQL Testcontainer repository/migration/concurrency evidence |
| Architecture | Package separation exists | No ArchUnit or Spring Modulith boundary test; no automated cycle/core-dependency gate |
| Legacy edit render | Real integration renders generated clips. `FfmpegEditRenderer` runs black-frame, long-silence, and audio-peak probes; unit tests cover their interpretation | Useful technical E3 evidence; the integration does not assert `qaReport.passed()` and creative quality is untested |
| Highlight render | Unit tests cover pieces; `HighlightDirectorFlowServiceTest` uses a mocked renderer and a dummy file | No real source-to-final highlight E2E |
| Highlight QA | `HighlightFfmpegRenderer.buildQaReport` probes output duration, black ranges, long silence, and sample peaks; it also checks assets, plan-level overlay bounds, and the configured mastering command. `HighlightDirectorFlowService` blocks project-final publication on failed `ERROR` checks | A fail-closed selected-defect gate, but no integrated-loudness/true-peak, A/V-sync, freeze, raster-safe-area, semantic-fit, or creative acceptance gate |
| Local models | Adapter/process tests exist | No pinned model inventory, offline no-egress proof, model quality baseline, license proof, or macOS/Linux/container execution matrix |
| Creative quality | Prompts and plans describe cinematic intent | No certified footage set, annotations, baseline, blinded rubric result, or accepted numeric threshold |
| Security | Input/error tests exist | No Spring Security, threat-model gate, auth tests, dependency/static/container/secret scans, or release vulnerability report |
| Performance | Benchmark harnesses and planning baselines exist under `src/test/java/.../perf` and `docs/` | No checked-in load generator or executed production-like load signoff |
| Resilience | Queue/cleanup unit tests and runbooks exist | No fault-injection, shutdown, retry, saturation, or dependency-outage certification |
| Container/deploy/CI | Operational planning documents exist | No CI definition, OCI image definition, deployment manifest, image scan, SBOM/signing gate, or automated promotion evidence |
Do not infer certification from tracked or generated artifacts. `docs/*baseline*.md`, dashboards, manifests, prompts, `qa-report.json`, and files under runtime input/output trees become evidence only when their production method, commit, environment, raw measurements, and reviewer are recorded.
The single-source highlight flow now uses `RenderQaReport.passed()` and failed `ERROR` checks to block project-final publication and mark the project failed. The legacy multi-clip renderer still treats its QA report as diagnostic output. In both paths, inspect warnings and remember that passing the implemented technical checks does not certify creative quality.
## Run the current checks safely
Do not allow Maven, Python, Hugging Face, Torch, FFmpeg inputs, or tests to use the network. Provision Java 21, Maven, FFmpeg/ffprobe, Python dependencies, and all models through an approved offline mechanism first.
Create a disposable copy of the committed tree, then run Maven offline. This command does not test uncommitted changes:
```bash
export MAVEN_REPO="${MAVEN_REPO:?set MAVEN_REPO to an approved pre-provisioned Maven repository}"
qa_root="$(mktemp -d)"
git archive --format=tar HEAD | tar -xf - -C "$qa_root"
(
cd "$qa_root"
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
mvn -o -Dmaven.repo.local="$MAVEN_REPO" verify
)
```
Run this only inside an execution environment whose egress is denied; environment variables alone are not a network control. Expect the current clean checkout to fail until local-asset tests stop depending on workstation state. Record that failure rather than relaxing the test.
After Maven completes, reject skipped tests and summarize Surefire results:
```bash
! rg -n 'skipped="[1-9][0-9]*"' "$qa_root"/target/surefire-reports/TEST-*.xml
perl -ne 'if (/<testsuite\b([^>]*)>/) {$a=$1; ($t)=$a=~/\btests="(\d+)"/; ($e)=$a=~/\berrors="(\d+)"/; ($f)=$a=~/\bfailures="(\d+)"/; ($s)=$a=~/\bskipped="(\d+)"/; $T+=$t; $E+=$e; $F+=$f; $S+=$s} END {print "tests=$T errors=$E failures=$F skipped=$S\n"}' "$qa_root"/target/surefire-reports/TEST-*.xml
```
Do not make `216` a permanent threshold. Test count can change; require all intended suites, zero failures/errors/skips, and explicit evidence for each risk.
## Apply the test portfolio
Use the smallest test that can falsify the claim, then add boundary evidence proportional to blast radius.
| Layer | Add it when | Required characteristics | Current gap |
|---|---|---|---|
| Domain unit | A rule, score, state transition, interval, or value object changes | No Spring context; fixed clock/seed; boundary and negative cases; invalid states rejected | Coverage exists but is not mapped to every invariant |
| Architecture | Packages/modules or dependency direction change | Fail build on controller/domain, persistence/API, adapter/core, cycle, and cross-module violations | Entire layer absent |
| Web slice | Controller, DTO, validation, status, or Problem Details changes | Mock only application port; cover success, malformed input, validation, error mapping, bounded collections | Existing tests start full Spring context rather than slices |
| Persistence slice | Mapping, query, constraint, migration, transaction, or concurrency changes | Real PostgreSQL Testcontainer; migrate empty DB; test indexes/constraints, locking, pagination, UTC | Entire layer absent |
| Integration | A real adapter/tool contract changes | Exercise actual FFmpeg, ffprobe, local model, filesystem, PostgreSQL, storage, or queue boundary; parse result | FFmpeg only; other real boundaries absent |
| Contract | Public API, model-worker protocol, object-storage, or queue schema changes | Versioned producer/consumer contract; backward-compatibility gate; failure mapping | Entire layer absent |
| E2E | A critical journey or cross-stage contract changes | Start from approved input; use real intended adapters; assert persisted state, output, telemetry, and failure cleanup | Real highlight source-to-final path absent |
Use the user's suggested minimums as target gates until change control approves evidence-based replacements: 90% line coverage for domain/application logic, 80% overall line coverage, and 80% mutation coverage for critical paths. Report branch coverage as well. Never use coverage to waive missing boundary, failure, concurrency, security, or creative-quality tests.
Never use arbitrary sleeps for asynchronous acceptance. The current `VideoAssetControllerTest` contains bounded `Thread.sleep(50L)` polling; treat replacement with a deterministic executor or condition-based bounded wait as test debt, not a pattern to copy.
## Gate each behavior surface
### Build and dependency gate
Require all of the following before E4:
- Build from a clean checkout with one offline command on macOS and the target Linux runtime.
- Pin and centrally manage dependency/plugin versions; prove no snapshot, milestone, RC, deprecated, or unapproved override.
- Lock or otherwise make dependency resolution reproducible. Record JDK, Maven, OS, architecture, and artifact checksums.
- Produce dependency and Software Bill of Materials (SBOM) evidence. Resolve all critical/high vulnerabilities before release.
- Keep dependency/model acquisition out of application startup and tests.
### API gate
- Validate the OpenAPI document during the build and detect breaking changes.
- Cover every endpoint's success, validation, malformed input, authentication, authorization, not-found/conflict, and dependency-failure responses.
- Assert RFC 9457 media type and stable public fields, including a correlation/trace identifier for diagnosable failures.
- Prove persistence/internal exceptions, paths, keys, stack traces, and secrets never enter responses.
- Bound and test growing collections; test idempotency for retryable writes.
### PostgreSQL gate
- Start empty PostgreSQL and apply every Flyway migration in order.
- Test not-null, unique, foreign-key, and check constraints as database behavior.
- Test repository queries against PostgreSQL, including pagination, indexes, UTC timestamps, isolation, optimistic locking, and concurrent updates.
- Test rolling-deployment migration compatibility and the documented recovery path.
- Never certify production persistence from H2.
### Local-model gate
- Inventory each exact model/runtime with version, checksum, source, license, memory/disk needs, supported platform, and owner.
- Load and infer with egress denied. Fail if a model is absent or its checksum differs; do not download or silently select another model.
- Run representative deterministic or tolerance-bounded fixtures on macOS development and Linux/VPS/container targets.
- Measure task quality, inference latency distribution, warm-up, peak memory, CPU/GPU use, concurrency, timeout, crash recovery, and output determinism.
- Prove output is audible/non-placeholder and semantically matches the request. File existence and nonzero PCM samples are insufficient.
- Block final rendering when requested production music, SFX, or voiceover cannot be produced by an approved local model or licensed asset.
### Media technical gate
Parse actual `ffprobe`/FFmpeg output; never infer output properties from the plan or command string.
| Check | Current repository value or behavior | Promotion discipline |
|---|---|---|
| Container/codecs | Intended render profile names use MP4/H.264/AAC | Assert streams and decode the full output without errors |
| Geometry/rate | Defaults are 1920x1080 at 30 fps | Assert actual width, height, pixel format, frame rate, duration, and frame count against the approved profile |
| Audio | Defaults are 48 kHz and 192 kbit/s | Assert stream presence when required, sample rate, channels, duration, and A/V synchronization |
| Loudness | Mix command configures `I=-16`, `TP=-1.5`, `LRA=11` | Measure the completed file with loudness analysis; command presence is not evidence |
| Black frames | Legacy probe flags any `blackdetect=d=0.5:pic_th=0.98` range as an error | Preserve or revise only with baseline evidence and change control; distinguish intentional fades |
| Long silence | Legacy probe flags `silencedetect=noise=-45dB:d=2` as a warning | Require explicit creative disposition; placeholder or accidental silence is a failure |
| Clipping | Legacy probe flags peak at or above -0.1 dBFS as a warning | Require explicit disposition and reconcile with the -1.5 dB true-peak target |
| Timeline | Legacy in-memory tolerance is 0.05 seconds | Measure final output duration and per-cut alignment; do not compare two plan-derived values |
Establish baselines before setting thresholds for A/V sync, frozen frames, dropped/duplicate frames, encoding quality, scene-boundary accuracy, intelligibility, music/voice balance, and platform transcode survival. Label these thresholds `candidate` until approved.
### Highlight identification and cinematic gate
Build a licensed, diverse corpus spanning the supported categories, motion levels, lighting, speech/no-speech, shot lengths, codecs, resolutions, and negative footage with no worthwhile highlight.
Annotate event boundaries, must-include/must-exclude moments, narrative role, technical defects, and reviewer rationale. Pre-register and compute:
- Candidate recall and precision at the configured maximum highlight count.
- Temporal intersection-over-union and boundary error against annotations.
- Weak/repetitive selected duration and missed high-value duration.
- Diversity/redundancy across selected highlights.
- Category accuracy and confidence calibration where category drives the edit.
- Voiceover factual-grounding errors, text-overlay errors, and inappropriate music/SFX events.
Define a cinematic human-review rubric before evaluating a new approach. Blind and randomize baseline/candidate ordering; use multiple qualified reviewers; retain per-reviewer scores and disagreements. Cover story/hook, shot selection, pacing, continuity, visual treatment, sound design, music fit, voiceover quality and grounding, overlay legibility, emotional coherence, and publishability.
The repository provides no accepted media-specific score thresholds. Establish the baseline, predict target numbers before running, approve thresholds through change control, and then freeze a holdout set. Do not approve by watching only successful examples.
### Security gate
- Test authentication and authorization for every public/sensitive operation, including correct 401 versus 403 behavior.
- Run dependency, static, secret, container, and license scans in CI; release with no unresolved critical/high findings unless formally risk-accepted.
- Verify restrictive CORS, security headers, actuator isolation, input/path validation, archive/media parser limits, and least privilege.
- Test that logs, errors, manifests, prompts, and telemetry redact credentials, tokens, personal data, filesystem internals, and protected media content.
- Exercise the threat model's trust boundaries, especially uploaded media, FFmpeg/model subprocesses, filesystem paths, model/asset supply chain, database, storage, and operator-controlled plans.
### Performance and resilience gate
Treat the user's synchronous API baseline as a candidate until the actual workload is approved: at least 200 requests/second per instance, p95 below 200 ms, p99 below 500 ms, error rate below 0.1%, no continuous memory growth for 60 minutes, and graceful behavior at 2x expected peak. Flag regressions over 10% against the approved baseline.
Do not apply those latency numbers to asynchronous model inference or rendering. Define media-specific SLOs from source minutes, resolution, codec, model/hardware class, concurrency, queue delay, render factor, and output count.
- Record p50/p95/p99 latency, throughput, errors, CPU, memory, GC, threads, disk, database pool, queue age/depth, model memory, and FFmpeg process concurrency.
- Inject slow/unavailable PostgreSQL, storage, queue, model process, disk exhaustion, corrupt media, timeout, worker crash, and termination.
- Prove bounded timeouts/retries/concurrency, idempotency, cleanup, no duplicate/corrupt outputs, graceful shutdown, and recovery.
- Retain raw load input/output and telemetry. Planning baseline documents are not executed load evidence.
### Container, deployment, and operations gate
- Build one platform-neutral application artifact and signed Open Container Initiative (OCI)/runtime/model bundles per supported target. Promote each target bundle unchanged between environments; run non-root with least privilege and a read-only filesystem where practical.
- Scan the image and generate an SBOM/provenance record tied to the commit and model/asset checksums.
- Test liveness, readiness, startup/warm-up, graceful shutdown, resource limits, signals, temporary storage, and no hidden local persistent-state dependency.
- Validate deployment definitions and rolling upgrade/rollback with migrations and in-flight work.
- Prove logs, metrics, traces, health, dashboards, alerts, and runbooks correlate one critical journey without leaking protected data.
- Run smoke/acceptance on Linux/VPS and the intended cloud infrastructure. macOS evidence remains development-only.
## Certified and golden inventory
As of 2026-07-21:
- Certified creative goldens: **none**.
- Certified source-to-final highlight fixtures: **none**.
- Certified local-model artifacts: **none**.
- Certified PostgreSQL/container/deployment/load/security evidence: **none**.
- Technical generated fixtures: real FFmpeg tests generate synthetic video/audio under JUnit temporary directories; these are test inputs, not cinematic goldens.
- Runtime input/output trees and existing rendered files: disposable observations unless separately versioned, licensed, hashed, and approved.
Never silently bless an existing artifact as golden. Golden status requires an approved change with provenance, expected measurements, reviewer, scope, and invalidation conditions.
## Add a fixture or golden
Use this checklist:
1. State the requirement, failure mode, layer, platform scope, and exact claim the fixture can falsify.
2. Obtain legal approval; record source, license, consent/privacy classification, allowed distribution/use, and retention.
3. Minimize the fixture without removing the behavior. Keep secrets, customer media, and personal data out of the repository.
4. Record SHA-256, media/model metadata, generation recipe, deterministic seed/clock, and tool versions.
5. Store small approved fixtures under a purpose-specific `src/test/resources` path. Store large models/media only in an approved versioned artifact system; never auto-download them.
6. Store semantic expectations as structured data. Do not compare transcoded media byte-for-byte when decoded measurements are the contract.
7. Add positive, boundary, negative, malformed, missing-dependency, and cleanup assertions. Write only to JUnit temporary directories.
8. Run with egress denied on macOS and target Linux. Add container/cloud evidence when the claim includes them.
9. Route threshold, fixture, license, and behavior changes through `video-editing-change-control`.
10. Document owner, review date, expiry/invalidation trigger, and the command that reproduces the result.
## Record acceptance evidence
For every promoted change, retain:
```text
Requirement and risk:
Commit and artifact digest:
Evidence tier and scope:
Exact offline command:
OS/architecture/JDK/Maven/FFmpeg/ffprobe versions:
Model and asset versions, checksums, and licenses:
Fixture/golden checksums:
Expected thresholds declared before execution:
Observed raw measurements and artifact locations:
Negative and failure-injection results:
Known limitations:
Independent reviewer and date:
Decision, rollout, rollback, and revisit condition:
```
Do not accept screenshots, prose summaries, or a lone `passed=true` as the only evidence when raw machine-readable output exists.
## Fortune 500 promotion gate
Score each category from 0 (missing) to 4 (reference-quality, verified, reusable): architectural clarity, domain modeling, maintainability, security, test quality, API design, data integrity, resilience, observability, performance, cloud-native operation, CI/CD, developer experience, documentation, and operational readiness.
Require all of the following:
- No category below 3.
- Overall average at least 3.5.
- Security, data integrity, testing, and operational readiness each at least 3.
- Every score cites concrete code, tests, documentation, or automated evidence.
- Clean offline build, complete required checks, API contract, empty-database migrations, telemetry, load/resilience SLOs, secure container, deployment/rollback, ADRs, and absence of undocumented critical assumptions are all proven.
Every listed category is required for project qualification. An unscored category is 0; there is no category-exclusion or “not applicable” route. Route the final promotion through `video-editing-change-control`.
## Provenance and maintenance
This skill was verified against the repository on 2026-07-21. Re-run these one-line, read-only checks when the code or build changes:
- Test files: `find src/test/java -type f -name '*Test.java' | sort`
- Maven plugins/dependencies: `sed -n '1,240p' pom.xml`
- Surefire totals: `perl -ne 'if (/<testsuite\b([^>]*)>/) {$a=$1; ($t)=$a=~/\btests="(\d+)"/; ($e)=$a=~/\berrors="(\d+)"/; ($f)=$a=~/\bfailures="(\d+)"/; ($s)=$a=~/\bskipped="(\d+)"/; $T+=$t; $E+=$e; $F+=$f; $S+=$s} END {print "tests=$T errors=$E failures=$F skipped=$S\n"}' target/surefire-reports/TEST-*.xml`
- Conditional integration tests: `rg -n '@EnabledIf|ffmpeg|ffprobe' src/test/java/org/example/videoclips`
- Coverage scope: `rg -n 'jacoco|folder-coverage-check|COVEREDRATIO|minimum' pom.xml`
- Highlight versus legacy QA: `rg -n 'buildQaReport|runQaProbe|new RenderQaCheck' src/main/java/org/example/videoclips/editing/*Renderer.java`
- Placeholder/model-download paths: `rg -n 'fallbackTone|writeSilence|get_pretrained|strictRuntime|strict-runtime' src/main tools src/test`
- Enterprise gate artifacts: `find . -maxdepth 4 -type f \( -path './.github/*' -o -iname '*openapi*' -o -iname 'Dockerfile*' -o -iname 'compose*.yml' -o -iname '*testcontainer*' -o -iname '*archunit*' -o -iname '*helm*' -o -iname '*k8s*' \) -print`
- Current configuration values: `rg -n 'output-width|output-height|output-frame-rate|audio-sample-rate|audio-bitrate|loudness|strict-runtime|auto-start' src/main/resources src/test/resources`

18
.dockerignore Normal file
View File

@ -0,0 +1,18 @@
# Keep the build context small and never ship generated media, models, or local state.
target/
output/
input/
tmp/
models/
.venv-local-asset/
.git/
.github/
.claude/
*.log
*.mp4
*.wav
*.onnx
*.pt
*.bin
*.safetensors
docs/

51
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,51 @@
name: CI
on:
push:
branches: ["**"]
pull_request:
branches: ["**"]
# Cancel superseded runs on the same ref.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
name: Build & test (Java 21)
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up JDK 21
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"
cache: maven
# Several integration tests shell out to ffmpeg/ffprobe; the local AI models are NOT needed for tests
# (asset workers are mocked), so no Python venv or model download is required in CI.
- name: Install ffmpeg
run: |
sudo apt-get update
sudo apt-get install -y ffmpeg
ffmpeg -version | head -n 1
ffprobe -version | head -n 1
# Use the committed Maven Wrapper so the build uses a pinned Maven version (deterministic across
# environments) rather than whatever the runner ships.
- name: Build and test
run: ./mvnw -B -ntp verify
- name: Upload test reports
if: always()
uses: actions/upload-artifact@v4
with:
name: surefire-reports
path: target/surefire-reports/
if-no-files-found: ignore
retention-days: 7

5
.gitignore vendored
View File

@ -42,3 +42,8 @@ build/
__pycache__/
*.pyc
yolov*.pt
# Local model bundle for the cinematic highlight PoC (large, provisioned out-of-band)
/models/
.claude/settings.local.json
yolov*.pt.license.txt

19
.mvn/wrapper/maven-wrapper.properties vendored Normal file
View File

@ -0,0 +1,19 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
wrapperVersion=3.3.2
distributionType=only-script
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.zip

62
AGENTS.md Normal file
View File

@ -0,0 +1,62 @@
# AGENTS.md — onboarding for any AI assistant (Codex, Claude, etc.)
This file is the portable entry point for any AI or human working on this repo. Everything below is plain
Markdown/Python in the repo — nothing depends on a specific assistant. (Claude Code additionally auto-loads
`.claude/skills` + `.claude/memory`, but those are just Markdown you can read directly.)
## What this project is
A **local, offline** service that turns one source video into a cinematic highlight: it selects the moment,
auto-directs a story-structured montage, generates music/SFX/voiceover with local models, renders with a
cinematic grade, and masters the audio. No external AI services, no runtime downloads. Java 21 / Spring Boot.
## Read these first (source of truth, in order)
1. [`README.md`](README.md) — overview, quality rules, models/licenses, limitations.
2. [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules.md) — the R1R14 source-adaptive rendering/director rules (all implemented; R13/R14 opt-in).
3. [`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md) — exact commands: source → rendered `final.mp4`.
4. [`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md) — local model runtime + version traps (Intel-Mac reference).
5. [`docs/cinematic-highlight-poc-plan.md`](docs/cinematic-highlight-poc-plan.md) — milestone log / roadmap.
6. `.claude/memory/ai-handover.md` — latest state, full commit list, honest status. **Always `git log`/`git status` first** — the repo is the live truth.
## Build / test / run
```bash
./mvnw -B verify # build + all tests + coverage (needs JDK 21 and ffmpeg/ffprobe on PATH)
```
Run the highlight pipeline via the `localpoc` profile — see the runbook (step-by-step). Key entry code:
`src/main/java/org/example/videoclips/editing/` (`HighlightSourceScheduler`, `HighlightMontageDirector` =
Tier-1 director, `HighlightVisionDirector` = Tier-2 VLM director, `HighlightFfmpegRenderer`). Local model
workers: `tools/local_asset_worker.py`, `tools/vision_caption.py`.
## Hard constraints (do not violate)
- No automatic dependency/model downloads at runtime; models load offline from `models/` + the local cache.
- No external AI services in the media path.
- No unlicensed assets; no placeholder silence/tones passed off as generated audio (fail closed instead).
- No rendering without an explicit approval flag.
- Do not run `tools/run_local_*_worker.sh` in a certified environment (their `auto` modes install/download).
## Honest status (do not overclaim)
- Cinematic quality ruleset **R1R14 complete** (R10 24fps, R11 filmic grade + LUT hook, R12 motion blur,
R13 beat-synced cuts, R14 subject-tracking reframe; R13/R14 opt-in). `mvn verify` green (288 tests). Also
fixed a 444→420p playback-compat defect. Production hardening **started**
(CI, README, render-approval gate, Dockerfile, Maven Wrapper).
- The video output is a **technically-clean cinematic draft, NOT certified production-ready.** Blockers: no
blinded human creative review (Gate B) has passed; source is 576p; music is generic `musicgen-small`; audio
models are **CC-BY-NC** (non-commercial). A passing test / valid MP4 is **not** proof of cinematic quality.
## Environment gotcha (2026-07-24)
If the app dies with `Operation not permitted` on a socket bind or `/var/folders/.../T`, the sandbox
restricted the default `$TMPDIR`/network: run the app JVM with
`-Dspring-boot.run.jvmArguments="-Djava.io.tmpdir=<writable>"` and Maven with
`-DargLine="-Djava.io.tmpdir=<writable>"`. HF downloads: `export HF_HUB_DISABLE_XET=1`.
## Conventions
Small, reviewable, test-backed commits. Follow existing patterns. Don't change production-facing defaults
without noting old→new value and rationale. Don't commit generated media/models (`.gitignore` / `.dockerignore`
already exclude `output/`, `input/`, `models/`, `.venv-*`).

38
Dockerfile Normal file
View File

@ -0,0 +1,38 @@
# Multi-stage build for the video-editing service.
#
# The image contains the application + ffmpeg only. The local AI models (Piper, MusicGen, AudioLDM2,
# moondream2) and their Python runtime are large and non-commercially licensed, so they are NOT baked in:
# mount them read-only at runtime (e.g. -v $PWD/models:/app/models -v $PWD/.venv-local-asset:/app/.venv-local-asset)
# and point the localpoc config at them. The REST clipping and folder workflows need no models.
#
# NOTE: this is a starting deployment artifact, not a certified production image — no-egress operation,
# non-root hardening beyond the below, and vulnerability scanning are still to be validated.
# --- build ---------------------------------------------------------------------------------------------
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /build
# Cache dependencies first for faster rebuilds.
COPY pom.xml .
RUN mvn -B -ntp -q dependency:go-offline
COPY src ./src
# Tests run in CI (they need ffmpeg); skip them here to keep the image build fast and hermetic.
RUN mvn -B -ntp -q -DskipTests package \
&& cp target/video-editing-*.jar /build/app.jar
# --- runtime -------------------------------------------------------------------------------------------
FROM eclipse-temurin:21-jre
# ffmpeg/ffprobe are required by the media pipeline.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Run as a non-root user.
RUN useradd --system --create-home --uid 10001 appuser
WORKDIR /app
COPY --from=build /build/app.jar /app/app.jar
USER appuser
EXPOSE 8080
# Bind to loopback by default; override SERVER_ADDRESS for a real deployment behind a proxy.
ENV SERVER_ADDRESS=0.0.0.0
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

105
README.md Normal file
View File

@ -0,0 +1,105 @@
# Video Editing Service — local cinematic highlight generator
Turn a single source video into a cinematic highlight **entirely with local, offline models**: it selects the
moment, cuts a story-structured montage, generates the music/SFX/voiceover, applies a cinematic grade, and
masters the audio — no external AI services, no runtime downloads.
> **Status:** a working, source-adaptive **proof of concept**. The cinematic quality ruleset (R1R9, below) is
> complete and content-agnostic. It is **not yet production-hardened** (no auth on the render endpoint,
> no containers/PostgreSQL/no-egress certification, and the local models are non-commercially licensed — see
> [Limitations](#limitations)). Do not deploy as-is.
## What it does
For one source clip the single-source highlight pipeline runs, fully offline:
```
ingest ─► analyze (ffprobe, scenes, audio, frames)
─► candidates + category
─► DIRECTOR (auto):
Tier 1 measure motion (YDIF) + audio (RMS) ─► story-structured shot list
Tier 2 local VLM (moondream2) captions beats ─► semantic payoff selection + overlay + music mood
─► generate assets (Piper voice · MusicGen music · AudioLDM2 SFX)
─► render (portrait/landscape-aware, exposure-normalized, push-in, crossfades, slow-mo ramp,
bold overlay, ducked source under a swelling score)
─► master loudness ─► QA probes ─► final.mp4
```
Rendering is gated: it stays off by default and requires an explicit approval flag per project.
## Cinematic quality rules (R1R9)
Every rule is **source-adaptive** — it measures the source and adapts, rather than hard-coding constants.
Full detail in [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules.md).
| # | Rule | Measure → adapt |
|---|---|---|
| R1 | Exposure | frame luma → normalize; grade never crushes the subject |
| R2 | Orientation | source rotation → portrait/landscape output, no distortion |
| R3 | Audio balance + loudness | score leads, source ducked; measured loudness corrected to 16 LUFS |
| R4 | Duration | any length, story-driven |
| R5 | Motion push-in | per-shot motion (YDIF) → adaptive in-shot `zoompan` |
| R6 | Transitions | cross-dissolves between beats + ease into slow-motion |
| R7 | Overlays | bold, outlined, animated entrance |
| R8 | Music dynamics | volume swell builds into the payoff |
| R9 | Show the action | Tier-1 measured **+ Tier-2 VLM caption-driven** selection |
| R10 | Cinematic cadence | 24 fps film-standard output (was 30) |
| R11 | Filmic grade | filmic tone S-curve (lifted toe + highlight roll-off) + optional licensed `lut3d` |
| R12 | Motion blur | shutter-angle frame blend on styled shots (toggle) |
| R13 | Beat-synced cuts | snap cut boundaries onto the score's beat grid (librosa; toggle) |
| R14 | Subject-tracking reframe | follow the detected subject vs. static crop (YOLO/AGPL; toggle) |
## Build & test
Requires **JDK 21** and **ffmpeg/ffprobe** on the PATH.
```bash
mvn -B verify # compile, run all tests, JaCoCo gate
```
CI (`.github/workflows/ci.yml`) runs this on every push.
## Run the highlight pipeline (local PoC)
The `localpoc` Spring profile wires the pipeline to pre-provisioned local model paths, isolates its
input/output trees, keeps rendering disabled + approval-required, and never starts a network-capable
bootstrap. The end-to-end command sequence (stage source → analyze → auto-direct → approve → render) is in
**[`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md)**. Outputs land under
`output/localpoc/highlight-projects/<project>/final.mp4`.
Local models used (provisioning + version traps in **[`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md)**; each
needs a provenance sidecar under `models/`):
| Model | Role | License |
|---|---|---|
| Piper (`en_US-lessac-medium`) | voiceover | MIT / Blizzard dataset |
| MusicGen small | music | **CC-BY-NC** |
| AudioLDM2 | SFX | **CC-BY-NC-SA** |
| moondream2 | Tier-2 vision director | Apache-2.0 |
| YOLOv8n (optional CV) | visual analysis | **AGPL-3.0** |
## Non-negotiable constraints
- No automatic dependency/model downloads at runtime; models load offline from `models/` + the local cache.
- No external AI services in the media path.
- No unlicensed assets; no placeholder silence/tones passed off as generated audio (the pipeline fails closed).
- No rendering without an explicit approval flag.
## Limitations
- **Licensing:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are **non-commercial/copyleft**.
Commercial use requires swapping in commercially-licensed models/assets.
- **Not production-hardened:** no Spring Security/authN, no container/K8s/deployment manifests, REST persistence
defaults to in-memory, and no-egress operation is not yet certified. `POST /v1/edit-projects/{projectId}:render`
now requires an `approved.flag` in the project directory, but that is a basic presence gate — not yet an
authenticated, digest-bound authorization.
- **VLM quality:** on distant/small subjects the small local VLM is only weakly discriminative; a stronger
model or closer framing improves Tier-2 selection.
- A director can only cut what was filmed — it cannot show a moment the camera never captured.
## Repository map
- `src/main/java/org/example/videoclips/editing/` — highlight analysis, two-tier director, renderer, QA.
- `tools/` — local model workers (`local_asset_worker.py`, `vision_caption.py`).
- `src/main/resources/application-localpoc.yml` — the opt-in PoC profile.
- `docs/` — the PoC plan, the cinematic quality rules, acceptance review.

View File

@ -0,0 +1,3 @@
{
"cpu_info": "Intel Core(TM) i9-9980HK 2.40GHz"
}

20
Ultralytics/settings.json Normal file
View File

@ -0,0 +1,20 @@
{
"settings_version": "0.0.6",
"datasets_dir": "/Users/jadenseanlee/dev/spring/video-editing/datasets",
"weights_dir": "/Users/jadenseanlee/dev/spring/video-editing/weights",
"runs_dir": "/Users/jadenseanlee/dev/spring/video-editing/runs",
"uuid": "17a267e7f84ff0aef6ad7e01e89f5c46977d1538367efbf019a5c7910b1f7dbf",
"sync": true,
"api_key": "",
"openai_api_key": "",
"clearml": true,
"comet": true,
"dvc": true,
"hub": true,
"mlflow": true,
"neptune": true,
"raytune": true,
"tensorboard": true,
"wandb": false,
"vscode_msg": true
}

100
docs/LOCAL-MODELS.md Normal file
View File

@ -0,0 +1,100 @@
# Local model runtime & provisioning
The pipeline runs entirely on local models. This documents what works and the version traps, so the runtime
can be reproduced (reference environment: **x86_64 macOS, no GPU**; a Linux/GPU host is easier). Python venv:
`./.venv-local-asset/bin/python` (py3.12). Nothing here downloads at service runtime — models load offline.
## Models (each needs a provenance sidecar under `models/`)
| Model | Role | Runtime | Notes | License |
|---|---|---|---|---|
| Piper `en_US-lessac-medium` | voiceover | `piper-tts` 1.5.0 + onnxruntime | real speech; CLI takes `--model`/`--output_file` | MIT / Blizzard dataset |
| MusicGen small | music | `transformers` 4.44.2 | `facebook/musicgen-small`, CPU ~9× realtime, 32 kHz | **CC-BY-NC** |
| AudioLDM2 | SFX | `diffusers==0.30.3` | `cvssp/audioldm2`, CPU ~7× realtime, **16 kHz → resample** | **CC-BY-NC-SA** |
| moondream2 | Tier-2 vision director | `transformers` + `torchvision==0.17.2` | `vikhyatk/moondream2` rev `2024-08-26`, `trust_remote_code`, offline; ~25 s/frame CPU | Apache-2.0 |
| YOLOv8n (optional CV) | visual analysis | `.venv-local-cv` (ultralytics) | `./yolov8n.pt` loads offline; runs behind loopback HTTP `:8091` | **AGPL-3.0** |
## Version traps (x86_64 macOS — all real)
- **PyTorch caps at `torch==2.2.2` / `torchaudio==2.2.2`** (last x86_64 macOS wheels). numpy must be **< 2**
(pinned `numpy==1.26.4`, `numba==0.60.0`, `llvmlite==0.43.0`, `scipy==1.13.1`).
- **transformers must be 4.x** (pinned `4.44.2`). transformers 5.x silently disables PyTorch (needs torch ≥ 2.4)
→ models unavailable.
- **audiocraft does NOT work here** (hard top-level `from xformers import ops`; xformers has no cp312 x86_64
wheel/sdist). AudioGen is audiocraft-only → **SFX uses AudioLDM2 (diffusers)** instead. MusicGen runs via
`transformers`, not audiocraft.
- **HF downloads:** set `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks). HF may **429** after
many pulls — retry resumes from cache.
## Wiring notes
- `tools/local_asset_worker.py` synthesizes voice/music/SFX; `tools/vision_caption.py` runs moondream (batch,
load-once) for the Tier-2 director.
- `LocalAssetSynthesizer` requires each model path to be a licensed **regular file** (adjacent `.license.txt`,
non-blank, not `UNTRACKED`) — `AssetLicensePolicy`.
- Do **not** run `tools/run_local_cv_worker.sh` / `tools/run_local_asset_worker.sh` in a certified environment:
their `auto` modes `pip install` and can fetch a named YOLO model.
- **Licensing blocker for commercial use:** MusicGen (CC-BY-NC), AudioLDM2 (CC-BY-NC-SA) and YOLOv8 (AGPL) are
non-commercial/copyleft. Swap in commercially-licensed models/assets before any commercial release. moondream2
(Apache-2.0) and Piper are fine.
## Optional stronger Tier-2 VLM: Qwen2.5-VL via llama.cpp (recommended upgrade)
moondream2 is small and, on hard footage (distant, portrait), cannot reliably tell a highlight from an
aftermath — which caps the director's vision **judge** (see `docs/cinematic-quality-rules.md` R15). A stronger
local VLM is a **drop-in**: the director calls a captioner script with a fixed manifest→JSON contract, so only
the configured script path changes. **`tools/vision_caption_llamacpp.py`** implements that contract against
**llama.cpp** (GGUF) — which runs on this **x86 CPU via AVX SIMD** and **bypasses the torch==2.2.2 /
transformers 4.x / no-xformers trap entirely** (no PyTorch involved).
**Recommended model:** `Qwen2.5-VL-3B-Instruct` (Apache-2.0 — commercial-friendly; verify the model card),
`Q4_K_M` GGUF + its `mmproj` vision projector. `Qwen3-VL-2B/4B` (official GGUF) or `Gemma 3 4B` are alternatives.
Provision **offline** (on a networked machine, then copy the files over — nothing downloads at service runtime).
The recipe below is the **verified** one for this reference machine (x86_64 macOS, Command Line Tools, no usable
GPU):
```bash
# 1) Build llama.cpp with the multimodal CLI (one-time; needs cmake + a C++ compiler; NOT in a certified env).
# GOTCHA (Command Line Tools, not full Xcode): clang can't find libc++ headers (<array> not found) because
# they live only under the SDK. Point CMAKE at them, or the ggml-base compile fails:
CXXV1=$(xcrun --show-sdk-path)/usr/include/c++/v1
git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF \
-DCMAKE_CXX_FLAGS="-isystem $CXXV1" -DCMAKE_C_FLAGS="-isystem $CXXV1"
cmake --build build --config Release -j --target llama-mtmd-cli # -> build/bin/llama-mtmd-cli
# 2) Fetch the GGUF weights + mmproj (verified: unsloth/Qwen2.5-VL-3B-Instruct-GGUF)
# Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf (~1.9GB) and mmproj-F16.gguf (~1.3GB)
# Record license/SHA-256 next to each file, per the asset-provenance policy.
```
Enable it (no code change — config + env). **localpoc already defaults `vision-caption-script` to the
llama.cpp worker**; if the binary env or weights are missing it **falls back to moondream with a loud WARN**
(`event=vision_backend_not_ready`), so the default never silently degrades. The model/mmproj default to the
repo paths below, so only the (machine-specific) **binary env** is mandatory:
```bash
# FAST (recommended): resident server — model loads ONCE, ~3x faster per clip.
export LLAMACPP_SERVER_BIN=/abs/llama.cpp/build/bin/llama-server
# or SIMPLE: per-frame CLI (reloads the model each frame)
# export LLAMACPP_MTMD_BIN=/abs/llama.cpp/build/bin/llama-mtmd-cli
# Optional — default to ./models/qwen2.5-vl-3b/ if unset:
# export LLAMACPP_VLM_MODEL=/abs/.../Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf
# export LLAMACPP_VLM_MMPROJ=/abs/.../mmproj-F16.gguf
# export LLAMACPP_VLM_NTOKENS=64 LLAMACPP_SERVER_PORT=8123
```
To force moondream instead: set `vision-caption-script: ./tools/vision_caption.py`.
**Runtime GOTCHA — force CPU on an Intel Mac.** The build enables Metal by default, but this machine's
integrated GPU times out on the vision encoder (`ggml_metal_synchronize: command buffer failed … GPU Timeout`).
`tools/vision_caption_llamacpp.py` therefore always passes `-ngl 0 --no-mmproj-offload` (pure CPU/AVX). Expect
~13 min per frame on CPU; ~7 frames per clip (candidates + overlay) ≈ 1020 min, fine for offline batch.
A persistent `llama-server` backend would avoid per-frame reloads — a future optimisation.
**Verified:** Qwen2.5-VL-3B **does** break the moondream ceiling. On the bowling celebration frame moondream
said "standing in a bowling alley"; Qwen2.5-VL says *"The person is raising their arms in a celebratory
gesture"* (highlight-worthiness 1.0), and correctly rates the turn-around/anticipation frames low (0.2 / 0.15)
— so the director's vision judge picks the celebration. This is the recommended production Tier-2 model.

View File

@ -0,0 +1,66 @@
# Runbook — source video → rendered cinematic highlight (local)
Exact steps to run one source clip end-to-end with the `localpoc` profile. Everything is local/offline.
Run from the repo root. Models must be provisioned first — see [`LOCAL-MODELS.md`](LOCAL-MODELS.md).
> The plan is now generated **automatically** by the two-tier director (Tier-1 measured + Tier-2 vision),
> so you no longer author `montage.json` by hand — but you can override it (see "Manual override").
## 1. Stage the source
```bash
cp <video> input/localpoc/highlights/source/<name>.mp4
```
## 2. Analyze + auto-direct (render stays OFF)
```bash
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc > /tmp/app.log 2>&1 &
```
The scheduler ingests the file, analyzes it, and (localpoc has `auto-director-enabled` + `vision-director-enabled`)
writes `director/montage.json` + a minimal `director/edit-plan.json` automatically. Wait for
`event=highlight_auto_director_completed`, then `pkill -f VideoClippingApplication`. The Tier-2 vision pass
(moondream captioning several frames, ~25 s/frame CPU) makes this take a few minutes. Artifacts land in
`output/localpoc/highlight-projects/<projectId>/`.
## 3. Approve (rendering is gated)
```bash
echo approved > output/localpoc/highlight-projects/<projectId>/director/approved.flag
```
## 4. Render — render-enabled MUST be a command-line arg
The `localpoc` profile hard-codes `render-enabled: false`, so the `VIDEO_EDITING_*` env var is ignored;
override with a Spring program arg (highest precedence):
```bash
SERVER_ADDRESS=127.0.0.1 mvn -o spring-boot:run -Dspring-boot.run.profiles=localpoc \
-Dspring-boot.run.arguments="--video-clipping.editing.highlight-scheduler.render-enabled=true" \
> /tmp/render.log 2>&1 &
```
MusicGen generates the score (~95 s CPU) → FFmpeg render. Wait for `final.mp4` at the project root, then
`pkill -f VideoClippingApplication`. Output: `output/localpoc/highlight-projects/<projectId>/final.mp4`.
## 5. Re-render after editing the plan/code
The render scanner skips projects with a root `final.mp4` / status `RENDERED`. Reset:
```bash
BP=output/localpoc/highlight-projects/<projectId>
rm -f $BP/final.mp4 $BP/final-preview.mp4 $BP/render-manifest.json; rm -rf $BP/highlights/* $BP/project-render-work
python3 -c "import json;p='$BP/project.json';d=json.load(open(p));d['status']='WAITING_FOR_DIRECTOR';json.dump(d,open(p,'w'),indent=2)"
```
Then repeat step 4.
## Manual override (optional)
To hand-author the cut instead of the auto-director, write `director/montage.json` yourself (it takes
precedence over `edit-plan.json` at render — `HighlightDirectorFlowService`; the render scanner still needs
`edit-plan.json` to exist — `HighlightDirectorPlanScanner`). Schema:
`{projectId, sourceVideoFileName, grade("hero"), musicDirection, voiceover[],
overlays[{text,timelineStartSeconds,timelineEndSeconds,placement}],
shots[{sourceStartSeconds, durationSeconds, zoom, speed}]}`. Shot `durationSeconds` = TIMELINE seconds;
source consumed = `durationSeconds * speed` (speed < 1 = slow-mo).
## Gotchas
- **HF downloads:** export `HF_HUB_DISABLE_XET=1` (the xet CDN times out on some networks; classic HTTPS works).
- **Sandbox / temp dir:** if the app dies with `Operation not permitted` on a socket bind or `/var/folders/.../T`
(Tomcat/`@TempDir`), the environment restricted the default `$TMPDIR`/network. Run the app JVM with
`-Dspring-boot.run.jvmArguments="-Djava.io.tmpdir=<writable dir>"` and Maven tests with
`-DargLine="-Djava.io.tmpdir=<writable dir>"` (JaCoCo still attaches).
- **Measure output:** `ffprobe` geometry + `ffmpeg -i final.mp4 -filter_complex ebur128=peak=true -f null -`
(target ≈ 16 LUFS, TP ≤ 1.5) + `blackdetect`/`silencedetect`. A valid MP4 is **not** proof of cinematic
quality — that needs a human creative review (see [`cinematic-highlight-acceptance-review.md`](cinematic-highlight-acceptance-review.md)).

View File

@ -0,0 +1,91 @@
# Cinematic Highlight PoC — Acceptance Review (P3.6)
Purpose: freeze the acceptance thresholds BEFORE judging, then record the technical measurements and a
blinded human creative review. The PoC is "done" only when representative output passes BOTH gates.
Frozen: 2026-07-22. Pipeline at commit `6600ded` (feature/imrovements). Sample under review:
`output/localpoc/highlight-projects/dji_21230510013241_0091_d/final.mp4` (canonical render, committed code).
> Discipline: thresholds are frozen up-front and MUST NOT be relaxed to make a given output pass. Preserve
> failed predictions. Creative quality is judged by a human watching/listening — not by file existence,
> a passing test, or objective metrics alone.
---
## Gate A — Technical acceptance (objective, measured)
Measured 2026-07-22 on the canonical render (commit `6600ded`):
| Check | Threshold | Measured | Pass |
|---|---|---|---|
| Container / video codec | MP4 / H.264 | mp4 / h264 | ✅ |
| Audio codec / sample rate | AAC / 48 kHz | aac / 48000 (mono) | ✅ |
| Resolution | 1920x1080 | 1920x1080 | ✅ |
| Frame rate | 30 fps | 30/1 | ✅ |
| Duration vs plan | within ±0.25 s | 27.32 s vs 27.0 plan (+0.32 s aggregate) | △ |
| Integrated loudness | 16 LUFS ±1 | 15.9 LUFS | ✅ |
| True peak (per highlight) | ≤ 1.5 dBTP | 2.4 / 2.5 / 2.7 dBFS | ✅ |
| Black frames (unintended) | none | none detected | ✅ |
| Long silence (unintended) | none | none detected | ✅ |
| Requested assets present | 100% (VO+music+SFX) | 3/3 highlights: VO+music+SFX all present | ✅ |
△ Duration: each highlight renders 9.1 s (the pipeline's per-highlight check uses ±0.1 s and PASSES);
the aggregate 27.32 vs 27.0 is +0.32 s, marginally over the stricter aggregate ±0.25 s I froze here.
Honest call: technically a marginal miss on the aggregate line only; not a defect (deliberate fade tails),
but flagged rather than silently passed. Everything else in Gate A passes.
Gate A: **PASS** (with the noted marginal duration nuance).
---
## Gate B — Creative acceptance (blinded human rubric)
Reviewer(s): _____ Date: _____ (≥3 reviewers recommended; randomize A/B order; do not reveal which is baseline.)
Score each dimension 04 (0 unusable, 2 acceptable, 3 good, 4 excellent).
| # | Dimension | What to judge | Score (04) | Notes |
|---|---|---|---|---|
| 1 | Highlight selection | Are these the strongest moments? Boundaries right? | | |
| 2 | Story / structure | Clear opening hook → build → payoff arc? | | |
| 3 | Pacing / transitions | Rhythm; cuts/fades feel deliberate? | | |
| 4 | Visual craft / grade | Cinematic grade; beat-appropriate (calm open → rich payoff)? | | |
| 5 | Overlays | Legible, tasteful, well-timed, on-brand? | | |
| 6 | Voiceover | Intelligible, well-paced, grounded, sits above music? | | |
| 7 | Music / SFX fit | Scene-appropriate; ducking balance; not distracting? | | |
| 8 | Overall preference vs baseline | Which is better, by how much? | | |
**Pass criteria (frozen):** average ≥ 3.5; no single dimension < 3; factual errors in VO = 0; and, in the
A/B, the current pipeline is preferred over the pre-Phase-3 baseline.
Known-weak areas to watch (from build evidence): voice↔music balance (P3.5 — needs your ears); Haar face
detector false-positives (cosmetic, not in output); heuristic-vs-CV category only matters if the plan uses
category-specific direction.
---
## Baseline for the A/B
- **Deterministic/pre-Phase-3 baseline** = pipeline at commit `a95d1fa` (fixed single grade, plain white
overlays, no true-peak limiter, filename-based category). Regenerate with `git worktree` + a render if a
side-by-side is wanted — ask and I will produce it.
- **Current** = commit `6600ded` (beat-specific grade, premium overlays, true-peak limiter, real YOLO CV
category). Qualitative deltas verified during the build:
| Aspect | Baseline | Current |
|---|---|---|
| Grade | one weak fixed `eq` | beat-specific filmic (cool open → rich payoff) |
| Overlays | plain hard-cut white | refined + soft shadow + alpha fade |
| Audio peaks | clipped at 0.0 dBFS | limited ≤ 1.5 dBTP |
| Category/labels | filename `GENERIC_VLOG` | YOLO `CAR_VLOG` @0.95, measured blur/exposure |
---
## Decision
- [ ] Gate A (technical) PASS
- [ ] Gate B (creative) PASS
- [ ] PoC ACCEPTED as production-quality cinematic output (local + manual flow)
If accepted, production-flow hardening (deferred list in `cinematic-highlight-poc-plan.md`) may begin.
If not, route findings back: selection→P2/P3.4, story/VO→director plan, craft/audio→renderer/P3.5.

View File

@ -0,0 +1,98 @@
# Cinematic Highlight Operator Checklist
Use this when you want one source video turned into a cinematic highlight project.
## Start Here
1. Start the service with the cinematic local profile:
```bash
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
```
2. Put one valid source video in:
```text
input/highlights/source/
```
3. Watch the logs for:
- `event=highlight_scan_started`
- `event=highlight_candidate_selected`
- `event=highlight_project_created`
- `event=highlight_director_prompt_generated`
- `event=highlight_scan_completed`
## Exact Project Folders
Project root:
```text
output/highlight-projects/<project-id>/
```
Files and folders the service writes:
- `project.json`
- `source/`
- `analysis/source-analysis.json`
- `analysis/category.json`
- `analysis/highlight-candidates.json`
- `director/director-brief.md`
- `director/director-prompt.md`
- `director/edit-plan.json`
- `highlights/<highlight-id>/storyboard.md`
- `highlights/<highlight-id>/visual-effects.json`
- `highlights/<highlight-id>/assets/requests/`
- `highlights/<highlight-id>/assets/music/`
- `highlights/<highlight-id>/assets/sfx/`
- `highlights/<highlight-id>/assets/voiceover/`
- `highlights/<highlight-id>/rendered-clips/`
- `highlights/<highlight-id>/preview.mp4`
- `highlights/<highlight-id>/final.mp4`
- `highlights/<highlight-id>/render-manifest.json`
- `highlights/<highlight-id>/qa-report.json`
## Exact Operator Flow
1. Wait for the service to create the highlight project.
2. Open:
```text
output/highlight-projects/<project-id>/director/director-prompt.md
```
3. Run Codex, Claude, or another filesystem-capable AI instance on that prompt.
4. Make the AI write the finished plan to:
```text
output/highlight-projects/<project-id>/director/edit-plan.json
```
5. Confirm `project.json` moved to `WAITING_FOR_DIRECTOR` after prompt generation.
6. The service now turns the director plan into `highlights/<highlight-id>/visual-effects.json`.
7. The asset bridge writes request files and the local asset worker resolves what it can into `highlights/<highlight-id>/assets/`.
8. The renderer publishes `rendered-clips/clip_*.mp4`, `preview.mp4`, `final.mp4`, `render-manifest.json`, and `qa-report.json`.
## What Is Automatic Today
- The scheduler analyzes the source video.
- The service writes the director prompt and brief.
- Existing analyzed projects are backfilled on startup if the prompt is missing.
- The visual-effects stage writes explicit renderer instructions.
- The local asset worker tries to resolve or generate reusable music, SFX, and voiceover assets.
## What Is Still Manual
- Running the AI director.
- Triggering any downstream render step that your current flow requires.
## If Output Is Missing
- No file in `input/highlights/source/`
- Unsupported video file
- The scheduler has not run yet
- `output/highlight-projects/<project-id>/director/director-prompt.md` is missing
- `director/edit-plan.json` was not written by the AI
- Rendering is not wired for the current highlight flow yet

View File

@ -0,0 +1,206 @@
# Local Cinematic Highlight PoC — Plan & Milestones
Owner: (Jaden) · Started: 2026-07-21 · Machine: Intel Mac (x86_64, 16 CPU, 32 GB, no GPU)
## Objective
Produce ONE genuinely cinematic highlight from a local source video, end-to-end through the existing
pipeline, using only local models resident in the service runtime:
Piper (voiceover) + MusicGen (music) + AudioLDM2 (SFX), with deliberate visual treatment and pacing,
and explicit human approval before any render.
A valid MP4 or a passing test is NOT success. Success = representative output passes measured media QA
and a human creative review.
## Non-negotiables (still in force during the PoC)
No external AI services in the media path · no placeholder silence/tones/OS `say` as a finished asset ·
no unlicensed assets · no rendering without explicit approval for the specific project · no inference-time
network (models load from local dirs, `HF_HUB_OFFLINE=1`). NOTE: the user authorized model DOWNLOADS on
2026-07-21 (one-time provisioning), reversing the earlier no-download stance; inference stays offline.
## Proven capability baseline (2026-07-21)
| Model | Runtime | Status | Evidence |
|---|---|---|---|
| Piper `en_US-lessac-medium` | piper-tts 1.5.0 | ✅ works | real speech, 22.05 kHz |
| MusicGen `musicgen-small` | transformers 4.44.2 + torch 2.2.2 | ✅ works | 5 s / 44.7 s CPU, mean 18 dB, 32 kHz |
| AudioLDM2 `cvssp/audioldm2` | diffusers 0.30.3 | ✅ works | 3 s / 22.3 s CPU, mean 21.8 dB, 16 kHz |
Environment traps (see memory `local-model-runtime-intel-mac`): torch capped at 2.2.2 (Intel-Mac),
numpy<2, transformers must be 4.x, audiocraft/xformers unusable here, `HF_HUB_DISABLE_XET=1` for downloads.
## Plan
### Phase 1 — Generate all three assets THROUGH the pipeline
- [x] 1.1 Rewrite `tools/local_asset_worker.py`: music→transformers MusicGen, sfx→diffusers AudioLDM2
(resample to 48 kHz mono), voiceover→Piper (unchanged). audiocraft path removed. CLI + exit codes preserved.
- [x] 1.2 Materialize models into stable `models/` dirs via `save_pretrained` (`tools/provision_local_models.py`):
`models/musicgen-small` (2.2 G), `models/audioldm2` (4.2 G); `models/piper` (60 M) already present.
- [x] 1.3 License/provenance `.license.txt` sidecars for each model marker file (config.json / model_index.json /
voice .onnx). BOTH audio models are NON-COMMERCIAL (MusicGen CC-BY-NC-4.0, AudioLDM2 CC-BY-NC-SA-4.0) —
flagged for production review. Honors the `AssetLicensePolicy` regular-file gate; no Java change.
- [x] 1.4 Smoke test: voiceover 3.67 s/15.8 dB, music 4.94 s/12.0 dB, sfx 3.00 s/20.6 dB — all 48 kHz mono,
real signal, exit 0.
- [x] 1.5 `mvn -o verify` → 245 tests / 62 classes / 0 failures / 0 errors / 0 skips (unchanged from baseline).
**Phase 1 COMPLETE (2026-07-21).**
### Phase 2 — One approved end-to-end highlight (STOP before render for explicit approval)
- [x] 2.1 Opt-in `localpoc` Spring profile added (`application-localpoc.yml`): venv python, model paths,
offline, no bootstrap auto-start, heuristic visual, isolated PoC I/O dirs, render OFF, approval REQUIRED.
- [x] 2.2 DJI source processed under `localpoc` (app started, ~4 min 4K analysis): wrote `category.json`
(GENERIC_VLOG, conf 0.25) + `highlight-candidates.json` (3 windows: 27-39 s, 36-48 s, 81-93 s).
Actual footage = blue Porsche Taycan aerial orbit (EV car reveal).
- [x] 2.3 Hand-authored `director/edit-plan.json`: 3 x 9 s highlights (opening_hook / rising_energy /
hero_payoff), grounded in the visible car; passes ALL validator constraints (containment, speed 1.0,
duration bounds, VO budget, category match). Manual + local-only (no external AI).
- [x] 2.4 **RENDERED (operator-approved 2026-07-21).** render-enabled override + approved.flag → local models
generated 3 VO (Piper) + 3 music (MusicGen) + SFX (AudioLDM2) → 3 highlights rendered → project
`final.mp4` assembled (27.3 s, 51.7 MB). Flow ~8.6 min. All real local-model assets, no placeholders.
- [~] 2.5 QA measured (honest):
**PASS** — 1920x1080 / H.264 / 30 fps / AAC 48 kHz; duration 27.3 s; integrated loudness -15.8 LUFS
(target -16 ±1); no black frames; no long silence; overlays render & are grounded ("First light.",
"Silent power."); VO intelligible.
**FAIL/weak** — true peak: highlight_001 = 0.0 dBFS (clips; target ≤ -1.5 dBTP); h2/h3 = -1.4 dBFS
(loudnorm TP limiting not enforced tightly). Visual "treatment" is essentially PASSTHROUGH — no visible
cinematic grade/reframe (renderer maps any treatment to a generic look); overlays are plain white
captions. LRA 16.9 (high). Pipeline's own qa-report passed because its grade/clip checks are structural.
**Verdict: genuine end-to-end local PoC, NOT yet "genuinely cinematic."** Visual grade + true-peak
limiting are the top Phase 3 items.
### Phase 3 — Iterate to quality
Ranked from the first render's evidence:
- [x] P3.1 Filmic grade in `HighlightFfmpegRenderer.cinematicVisualFilter`: S-curve + teal-orange
colorbalance + eq + unsharp + vignette (replaces the weak fixed eq). Only the highlight renderer
changed; multi-clip FfmpegEditRenderer untouched.
- [x] P3.1b **Beat-specific grading** (a0b6023): grade now varies by story beat (read from the EditPlan
style key, no EditDecision/serialization change) — opening=cool/soft (linear_contrast), rising=balanced
(medium_contrast), hero=rich/warm (strong_contrast). Verified visually (calm→dramatic arc) + unit test.
- [x] P3.2 True-peak safety: added `alimiter=level=disabled:limit=0.72` after loudnorm in the audio mix.
VERIFIED by re-render: project true peak 0.0 dBFS (clipping) → **-1.7 dBFS**; per-highlight -1.7/-2.7/
-2.8 (all <= -1.5 dBTP gate); integrated loudness -16.3 LUFS. (First pass at limit=0.79 gave -1.3 on
h1, 0.2 dB over — tightened to 0.72 for inter-sample margin.) Test added.
- [x] P3.3 Overlay styling: refined 48px caption, soft drop shadow, thin subtle border, smooth alpha
fade in/out (0.4s ramps). Verified rendering; committed 24bd1d7. (Placement/safe-area unchanged.)
- [x] P3.4 **Real CV visual analysis via YOLOv8** (resident model, NO bootstrap script). Deps installed into
an isolated `.venv-local-cv`; uvicorn worker run directly against the existing `yolov8n.pt` offline
(`YOLO_OFFLINE=True`, telemetry sync off). Fixed a latent bug: the Java CV client negotiated HTTP/2
(h2c) which the HTTP/1.1-only worker mishandled (422/empty body) — pinned HTTP/1.1. Result on the DJI
source: category GENERIC_VLOG@0.25 -> **CAR_VLOG@0.95** (real `car` detection), measured blur 0.191 /
exposure 0.95 (OpenCV), method local_cv_worker_opencv_yolo. Provider stays OPT-IN via runtime override
(committed localpoc profile keeps the safe heuristic default; worker started manually).
CAVEATS: yolov8n.pt is **AGPL-3.0** (production blocker on this license alone; see yolov8n.pt.license.txt);
loopback HTTP is flagged non-compliant for a certified path; Haar face detector gives false positives.
- [~] P3.5 Music/SFX + ducking review (measurement-based). FINDINGS: sidechain ducking IS implemented
(`[music_raw][voice]sidechaincompress` with the configured threshold/ratio/attack/release; FFmpeg
auto-splits `[voice]` so it both keys the duck and stays in the mix) and it fires (music drops during
voice). Source audio is near-silent (-54 dB RMS) -> negligible. Loudness/true-peak stay in spec.
OPEN: voice-vs-music balance cannot be validated or tuned by measurement alone — MusicGen produces
different audio each run, so cross-render A/B is confounded, and this needs LISTENING. A blind +3.5 dB
voice boost was tried and reverted (could not verify it helped; measurement suggested it did not).
Per the "do not tune audio blindly" discipline, mix calibration is deferred to P3.6 (human review).
- [~] P3.6 Acceptance gate set up in `docs/cinematic-highlight-acceptance-review.md`: FROZEN technical
thresholds + creative rubric (04, avg≥3.5 / no dim<3). **Gate A (technical) measured = PASS** on the
canonical render (1080p/H.264/AAC 48k, 15.9 LUFS, TP 2.4/2.5/2.7 dBTP, no black/silence, all assets
present; one marginal note on aggregate duration +0.32 s). **Gate B (creative) requires the human
review** — cannot be self-assessed; awaiting reviewer scores.
## Milestone log
- 2026-07-21: Models provisioned & individually proven (Piper, MusicGen, AudioLDM2). Plan approved. Phase 1 started.
- 2026-07-21: **Phase 1 complete.** Worker rewritten (audiocraft→transformers MusicGen + diffusers AudioLDM2),
models materialized to `models/` with license sidecars, all 3 asset kinds generate 48 kHz mono real audio
through the worker, `mvn -o verify` green (245/0/0/0). Paused for review before Phase 2.
- 2026-07-21: Phase 1 committed (a95d1fa). **Phase 2 up to approval gate:** localpoc profile added, DJI source
processed → candidates + category, grounded 3-highlight director plan authored & pre-validated.
STOPPED — awaiting explicit render approval for dji_21230510013241_0091_d.
- 2026-07-21: **Phase 2 rendered** (operator-approved) — first full local source→final highlight reel.
- 2026-07-22: **Phase 3.1 + 3.2 done & verified.** Filmic grade + true-peak limiter. Re-render measured TP
-1.7/-2.7/-2.8 dBFS (was 0.0 clipping), I -16.3 LUFS; mvn verify 245/0/0/0. Committed 588c652 (profile),
0fcfca9 (render quality).
- 2026-07-22: **Phase 3.3 done & verified.** Premium overlay styling (soft shadow, alpha fade). Committed
24bd1d7. Combined re-render (grade+limiter+overlays) verified: TP -2.5/-2.6/-2.8 dBFS, I -16.3 LUFS,
1080p/27.3s; mvn verify 246/0/0/0.
- 2026-07-22: **Beat-specific grading done & verified** (a0b6023). Grade varies per story beat; final
re-render TP -2.3/-2.5/-2.8 dBFS, I -16.3 LUFS, 1080p/27.3s; mvn verify 247/0/0/0. Phase 3 visual/audio
craft items complete.
- 2026-07-22: **P3.4 real CV (YOLOv8) done & verified.** Offline worker vs existing yolov8n.pt (no bootstrap).
Fixed latent HTTP/2 client bug. DJI source now classifies CAR_VLOG@0.95 with measured blur/exposure and a
real `car` label (was GENERIC_VLOG@0.25/filename heuristic). mvn verify 247/0/0/0. AGPL + loopback caveats
recorded. Remaining: P3.5 music/SFX fit, P3.6 frozen thresholds + blinded review.
### Phase 4 — Cinematic pass (after human review: "not genuinely cinematic")
Human Gate-B verdict on the Phase-3 output: functional, not cinematic. Direction chosen: dynamic editing,
better music, NO voiceover, bolder finish. Done so far (commit 842f2a7):
- [x] **No voiceover** — validator made VO optional; new plan carries none (music + SFX driven).
- [x] **2.39:1 letterbox + film grain** in the renderer; overlays raised above the bar.
- [x] **Slow-motion hero** (plan: hero source span < target 0.67x) + beat-specific bold grade.
- [x] **Stronger music direction** — driving orchestral/hybrid prompts (still `musicgen-small`).
- [ ] **Push-in / Ken Burns** — prototyped via time-based `crop` zoom; INVALID (FFmpeg crop can't use `t`
for w/h). Reverted. Redo with `zoompan` as a follow-up.
- [ ] **Bigger music model**`musicgen-medium` download stalled + is very slow on CPU; deferred. Consider
medium, one continuous score across the piece, or licensed production music.
- [ ] Re-run human Gate B on the cinematic cut.
Verified: mvn 247/0; final 16.5 LUFS, TP 2.7 dBTP, 1080p, no VO. Deliverable:
`output/localpoc/highlight-projects/dji_21230510013241_0091_d/final.mp4`.
### 2026-07-23 — Second source (bowling strike) + orientation/audio/duration fixes
- New source `bowling_strike.mp4` (portrait phone footage) rendered e2e via montage; confirmed the DJI
Phase-5 finding that **source footage is the ceiling**.
- **Service-wide fixes applied** (not just PoC), all with tests, `mvn -o verify` 254/0:
- **Orientation-aware rendering.** `FfmpegClipInspector` now reads display rotation (side-data + `rotate`
tag) and stores effective width/height; `HighlightFfmpegRenderer.outputGeometry()` renders portrait
sources to a portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait footage
being stretched into landscape.
- **Audio mix.** Source audio (`[0:a]`) ducked to 20 dB (24 with narration) so the generated score leads
instead of being buried; montage music cue raised to 9 dB.
- **Highlight duration constraint removed.** Validator no longer enforces min/max (only positive/finite +
sane speed + candidate containment); config bounds relaxed to soft candidate hints (min 2 / max 3600).
A highlight can be as long as the story needs.
- E2e render flow captured in memory `highlight-e2e-render-runbook` (token-saver).
### Phase 5 — Next steps (roadmap as of 2026-07-22, after montage mode)
To reach genuinely cinematic output (Gate B):
- [ ] P5.1 **Better source footage** (biggest lever; source is the real ceiling — parked car / industrial lot).
- [x] P5.2 **Automate shot selection — Tier 1 (DONE 2026-07-23).** `HighlightMontageDirector` composes
`director/montage.json` from measured motion (YDIF) + audio-energy (RMS) curves: setup → continuous
action/tension → slow-mo payoff on the audio climax → resolution button, camera-whip tail trimmed. Wired
into the scheduler behind `auto-director-enabled` (localpoc on). Verified: auto cut ≈ hand cut on bowling.
- [ ] P5.3 **Better music** — musicgen-medium/large (slow) or licensed production track; one produced score.
- [ ] P5.4 Polish: proper `zoompan` in-shot push-in; beat-synced cut lengths; better/optional voice.
To make it a real system:
- [x] P5.5 **Resident local director (DONE 2026-07-23).** Two tiers, both offline + auto: Tier-1
`HighlightMontageDirector` (measured motion+audio → shot list) and Tier-2 `HighlightVisionDirector` +
`tools/vision_caption.py` (moondream2 captions the payoff → semantic overlay "STRIKE" + scene music).
Wired into the scheduler (`auto-director-enabled`, `vision-director-enabled`; localpoc on). Plans are no
longer manual. See `docs/cinematic-quality-rules.md` R9.
To reach production (deferred until quality passes):
- [ ] P5.6 Formal Gate B human review (rubric in cinematic-highlight-acceptance-review.md).
- [ ] P5.7 **Licensing blocker**: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL are non-commercial/
copyleft -> production needs commercially-licensed models/assets.
- [ ] P5.8 The hardening backlog below (security, CI/CD, containers, PostgreSQL, no-egress cert, repro build).
### Director review of the bowling portrait cut (2026-07-23) — general craft gaps
Frame-by-frame + audio review. Verdict: technically clean (correct portrait, -15.5 LUFS, TP -2.8, no defects,
score leads) but NOT yet top-tier cinematic. Craft gaps, all fixable as GENERAL pipeline features:
- [x] P5.9 **Adaptive exposure (DONE 2026-07-23).** `probeSourceLuma` measures source frames (signalstats
YAVG); `exposureNormalizationFilter` maps the measured mean to a target band (source-driven gamma); the
grade is now exposure-preserving (black-lift, gamma≥1, brightness≥0, soft vignette). Bowling final went
63→~100 mean luma with a measured gamma of 1.098. General rules captured in `docs/cinematic-quality-rules.md`.
- [x] P5.10 **Life in held frames (DONE 2026-07-23).** `probeSegmentMotion` (YDIF) measures per-shot motion;
`pushInAmount` maps low motion → stronger in-shot `zoompan` push-in, high motion → gentle (floor keeps a
little life). Applied before `setpts` so slow-mo survives. Bowling: static opening got 0.117 push, active
celebration 0.084. See `docs/cinematic-quality-rules.md` R5.
- [ ] P5.11 **Speed-ramp + crossfades.** The cut into the slow-mo hero is an abrupt speed change and beats are
hard cuts. Ease into slow-mo and add short crossfades on beat boundaries (renderer transition upgrade).
- [ ] P5.12 **Bolder, animated overlay synced to the music hit.** Current overlay is small plain white text;
make it larger/bolder with a scale/fade punch timed to the payoff (general overlay upgrade).
- [ ] P5.13 **Music climax sync.** Align the generated score's peak to the hero beat's timeline position
(pass the payoff timestamp into music prompting/trimming). Ties to P5.3.
- [ ] P5.14 **Show the actual action.** The edit never shows the pins fall (source is on the person). General:
shot-selection (P5.2) should locate the true impact via audio-transient + motion, and source coverage
should include it.
## Deferred (until output-quality gate passes)
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
ops, digest-bound authenticated approval. Recorded, not deleted.

View File

@ -0,0 +1,180 @@
# Cinematic Highlight Quality Rules (source-adaptive)
Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle:
**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant
tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next.
Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x).
## R1 — Exposure: normalize to a target, never crush ✅
- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma.
`HighlightFfmpegRenderer.probeSourceLuma`.
- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band
(~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1,
brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098.
## R2 — Orientation: respect the source, never distort ✅
- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H.
- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39
letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.
## R3 — Audio balance: the story audio must lead ✅
- **Rule:** the generated score is the bed and leads; source audio is ducked under it (20 dB, 24 with
narration); with narration, voice leads via side-chain ducking. No element is silently buried.
- **Accurate mastering:** single-pass `loudnorm` is only ~±2 LUFS accurate, so the finished file is measured
(`probeIntegratedLoudness`) and a corrective gain (`loudnessGainDb` → `masterLoudness`) is applied to hit the
target, with a brickwall limiter for true peak. No-ops when already on target (e.g. a render that lands at
15.5 needs no fix; one at 18.7 is boosted). Handles MusicGen loudness variance.
- **Verify:** integrated loudness ≈ 16 LUFS, true peak ≤ 1.5 dBTP; confirm the score is audible, not just present.
## R4 — Duration: length follows the story ✅
- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive
duration + sane playback speed + candidate containment).
## R5 — No dead air: motion in held shots ✅
- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source
range. `HighlightFfmpegRenderer.probeSegmentMotion`.
- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a
gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives.
- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active
celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).
## R6 — Transitions: ease, don't jerk ✅
- **Crossfades:** video xfade + audio acrossfade dissolve montage beats instead of hard-cutting. Run as
SEPARATE passes then muxed (`xfadeVideoCommand` + `acrossfadeAudioCommand` + `muxCommand`) — one combined
filtergraph starves/truncates the audio (that cut the music off early). Timeline compresses by (n-1)*xf;
`shiftOverlayForCrossfade` re-times overlays and the reported duration is adjusted so overlays/loudness/QA
stay aligned. Opt-in via `editing.crossfade-seconds` (0 = hard cuts default; localpoc 0.25).
- **Speed-ramp into slow-mo:** `speedRampSetpts` gives a slow-mo shot a log-integrated `setpts` that eases the
playback speed from normal (1.0) down to below the target across the shot, so the payoff decelerates
smoothly instead of snapping. Stays a SINGLE segment (R5 push-in preserved); the `-t` pin keeps the planned
duration. Verified on bowling (payoff segment carries the ramp expression; output valid).
## R7 — Overlays: bold, animated, synced ✅
- **Rule:** captions are large (fontsize 84), thick-outlined + drop-shadowed for legibility on any background,
with a snappy entrance (0.18 s alpha punch + 34 px rise-up over 0.22 s) and a soft ease-out. `drawTextFilter`.
- **Ties to Tier 2 (R9):** the vision director produces the caption *text* ("STRIKE"); R7 makes it *land*. The
overlay is placed on the payoff beat, so it's timed to the musical/edit accent. Verified on bowling.
## R8 — Music dynamics: build to the payoff ✅
- **Rule:** because MusicGen's internal structure is uncontrolled, the mix applies a deterministic swell
envelope to the score (`volume='min(1,0.5+0.5*t/peak)':eval=frame`) so the music amplitude rises from 0.5x to
full over the run-up to the payoff, then holds. Driven only by the payoff beat's (crossfade-compressed)
timeline position — generic for any source. In `audioMixCommand`; verified on bowling (peak at 6.83s).
- **Still open:** aligning MusicGen's *own* melodic climax (vs a volume swell) needs a controllable music
model or a produced track (P5.3).
## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳
- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve
(YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup →
continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax →
resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled`
(on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut.
- **Tier 2 (semantic VLM director, DONE 2026-07-23/24):** `HighlightVisionDirector` + `tools/vision_caption.py`
run a local vision-language model (moondream2, offline). It now captions *several* beat frames (two questions
each in one call: a discriminative description + a punchy label) and:
1. **guides selection**`semanticScore`/`semanticCurve` turn the descriptions into a per-window
highlight-worthiness signal that the montage director blends with audio to place the payoff on the
semantically-strongest moment (verified: on bowling the payoff moved onto moondream's detected
celebration);
2. **decorates** — the payoff label becomes the bold overlay and the description flavors the music.
All generic: the scoring uses generic emotion/action/idle keywords (no content-specific terms), and it
fails soft to the measured cut. ~25s/frame CPU; enabled by `vision-director-enabled` (localpoc on).
**Honest limit:** a small VLM on distant subjects is only weakly discriminative — a terse question collapses
to a constant answer (use descriptive questions); a stronger VLM or clearer framing would help.
moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can.
## R10 — Cinematic cadence: 24 fps ✅
- **Why:** 24 fps (with a ~180° shutter) is the film-standard motion cadence used by the overwhelming majority
of theatrical productions; 30 fps reads as "video". Output frame rate default moved **30 → 24**
(`editing.output-frame-rate`, `VideoClippingProperties`). Generic, content-agnostic — one global knob.
- Verified: a styled segment renders `r_frame_rate=24/1`, `nb_frames = 24 × seconds`.
## R11 — Filmic grade: S-curve tone + optional licensed LUT ✅
- **Built-in grade (no external asset):** each beat now applies a **filmic tone curve** — a lifted toe
(shadows never crushed) plus a rolled-off highlight **shoulder** (highlights compressed, not clipped) — which
is the core of the "film look", on top of the teal-shadow / warm-highlight ("teal-orange") colour language.
Replaces the old linear `curves=all='0/x 1/1'` ramp. Still **exposure-preserving** (mids ≈ 0.5, gamma ≥ 1):
measured on bowling, source mean luma 112 → graded 121 (toward the 120 target), never crushed.
- **Optional film-emulation LUT (`editing.film-lut-path`):** when it points to a readable `.cube` that has an
adjacent, non-blank `<name>.license.txt` sidecar (same provenance discipline as generated assets), the
renderer applies it via `lut3d` and **suppresses** the built-in colour moves (no double-grading), keeping only
sharpen + vignette. Empty by default → built-in grade; **fails closed** on a missing/unlicensed LUT, so no
unlicensed asset is ever applied. This is the upgrade path to real film color science (e.g. Kodak 2383) once a
commercially-licensed LUT is provisioned.
## R12 — Motion blur: shutter-angle emulation ✅
- **Why:** film's ~180° shutter produces natural motion blur; sharp digital frames strobe at 24 fps. Styled
shots now blend each frame with its predecessor (`tmix=frames=2`, normalised so static frames are unchanged;
only moving pixels gain a short trail). Toggle `editing.cinematic-motion-blur` (default on).
- **Honest limit:** on already-soft / low-resolution source (e.g. 576p phone footage) motion blur can further
reduce perceived sharpness — it most helps crisp, high-shutter footage. Disable it for soft source.
## R13 — Beat-synced cuts: cut to the music ✅
- **Why:** cinematic edits cut on the beat; consumer AI editors (CapCut, DJI LightCut) beat-sync automatically.
After the score is generated, `tools/beat_detect.py` (librosa) extracts its beat grid and `HighlightBeatSync`
snaps each internal cut boundary onto the nearest beat (default ±0.18 s), never reordering shots, never
changing the total length (the score still fits), and never shortening a shot below 0.5 s. Toggle
`editing.beat-sync-enabled`. Runs in `HighlightDirectorFlowService` after asset generation, before render.
- **Fails soft:** no beats / unreadable track / missing interpreter → original cuts kept. Verified on bowling:
194 BPM detected, boundaries {1.7, 5.6, 6.4, 9.26 s} all snapped onto beats.
- **Honest limit:** MusicGen tempo is loose, so beats can be irregular; the tolerance + fail-soft keep it safe.
## R14 — Subject-tracking reframe: follow the subject ✅
- **Why:** DJI/Insta360 "AI reframe" keeps the moving subject framed; a static centre crop lets them drift off.
`tools/subject_track.py` (YOLO, CV venv) samples the dominant subject's normalised centre across each shot;
`HighlightSubjectTracker` smooths it and encodes a `pan=` path into the shot treatment; the renderer
(`subjectFollowFilter`) drives a `zoompan` crop that follows the path (interpolated, clamped inside frame),
co-existing with the R5 push-in. Toggle `editing.subject-reframe-enabled`.
- **Fails soft:** sparse/no detections → centred crop. Verified on bowling: the bowler's centre tracked
cx 0.54→0.44→0.68 across the frame.
- **Honest limits:** **YOLOv8 is AGPL-3.0 → non-commercial** (matches the repo's CV stance); per-frame detection
is CPU-slow; on tiny/distant subjects detection is unreliable.
## R15 — Decisive moment: measurement proposes, the vision model judges ✅ (with a hard VLM ceiling)
A highlight is an **action unit** — a start, a decisive peak, and an outcome/reaction — and choosing *which*
moment is the highlight is a question of **meaning**, not of motion or loudness. **There is no generic rule in
measurement alone:** a camera turn-away or a loud aftermath routinely out-scores a quiet celebration on both
motion and audio (measured on the real bowling clip: the turn-around has the clip's highest motion *and* is
louder than the celebration). So the design separates the two responsibilities — no positional bands, no
per-video thresholds, ever:
- **Measurement PROPOSES (`HighlightMontageDirector.candidatePeaks`):** the local maxima of intensity
(normalised motion + audio), strongest first, min-separated. Every real event becomes a candidate — a strike,
a celebration, a turn-away, a goal, an anticipation — with **no opinion** about which is the highlight.
- **The vision model JUDGES (`HighlightVisionDirector.rankDecisiveMoment`):** it captions each candidate and
scores **highlight-worthiness by meaning** — a celebration or a scored goal outranks a loud turn-away or an
"about to…" build (anticipation is explicitly *not* the payoff). The highest-scoring candidate wins.
- **Fallback:** if the model declines/fails, the strongest-intensity peak is used (a measured last resort).
- **Segment (`composeMontageAt`):** around the chosen peak, the **onset** (measured motion rising into it,
build-capped) and the **resolution** (measured motion settling after it, sweeping in the outcome + reaction —
the pins falling *and* the celebration; the ball crossing the line *and* settling in the net) are built.
- **Honest labelling (R9/C):** a dense read of the *shown* segment names it and never asserts an action the
segment doesn't contain — an anticipatory cut becomes a teaser question, not a false "KICK".
**Honest ceiling (verified, not theoretical):** the judge is only as good as its eyes. The local **moondream2**
model reliably perceives some actions (soccer: *"kicking a soccer ball"* → the goal is chosen correctly) but
**cannot** perceive others — on the distant, portrait bowling clip it describes every frame as *"standing"* /
*"walking"* / *"a bowling alley"* and never sees the arms-raised celebration, regardless of prompt (a
posture-focused prompt collapsed to a constant *"Standing still"*). When the model can't discriminate, all
candidates tie and it falls back to the loudest peak (the turn-away). This is a **model-capability limit, not a
design flaw** — the fix is a stronger local VLM, which is a **drop-in**: the judge is a clean interface
(`MomentChooser` / `rankDecisiveMoment`) with no director changes required. Forcing the weak case with more
heuristics is prohibited — that is the hack this rule exists to avoid.
---
Rules R1R15 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightMontageDirector`
/ `HighlightVisionDirector` / `HighlightBeatSync` / `HighlightSubjectTracker` and apply to **every** project
automatically (R13/R14 behind opt-in flags). Each is driven by a source measurement or a global cinematic
standard, never a per-video constant.
## Still missing for "cinematic" (researched gap — not yet implemented)
Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work:
- **Real film-emulation LUT** — R11 ships the `lut3d` hook but no licensed `.cube` is provisioned; the built-in
filmic curve is the fallback. Dropping in a commercial LUT (e.g. Kodak 2383) is the biggest colour upgrade.
- **Shallow depth of field** — subject/background separation; hard to fake post-hoc without segmentation/depth.
- **Non-code ceilings:** higher-resolution source (576p upscales soft) and a produced/licensed music track are
the two largest quality levers and are outside the renderer.

66
docs/current_state.md Normal file
View File

@ -0,0 +1,66 @@
- Highlight rendering defaults off and director approval defaults on.
- Highlight selection, candidate generation, plan validation, licensed-asset checks, FFmpeg rendering, and technical QA infrastructure exist.
What Is Not Ready
- No approved, resident Piper/MusicGen/AudioGen model bundle is configured.
- Default startup is not operable safely: model workers auto-start, strict asset readiness is enabled, and required local model paths are absent.
- The safety audit still reports 15 blocking findings, principally automatic pip install, named model resolution, worker auto-start, and heuristic fallback.
- Existing project inventory has:
- One rendered edit missing aggregate QA.
- One highlight still CREATED.
- One highlight WAITING_FOR_DIRECTOR.
- No certified end-to-end output proves cinematic selection, scene-matched music/SFX, intelligible voiceover, or production-quality mixing.
- Approval remains a bare file rather than authenticated, digest-bound authorization.
- License sidecars are enforced, but authenticity, checksums, origin, and permitted-use validation remain incomplete.
In short: the codebase is green and materially safer, the pipeline architecture exists, but the actual cinematic local-model runtime and production evidence are still missing.
A commit now exists despite the earlier “do not commit” instruction. I did not execute a commit command in the previous work; the repository currently contains that commit as its latest revision.
Next steps:
1. Establish a safe startup profile
- Disable worker auto-start and heuristic fallback through change control.
- Prevent runtime pip install and named-model acquisition.
- Keep rendering disabled and approval required.
- Gate: safety audit has no blocking findings for the certified profile.
2. Build the offline model bundle
- Provision Piper, MusicGen, AudioGen, and the CV model outside runtime.
- Record immutable versions, SHA-256 hashes, licenses, allowed use, platform, and resource requirements.
- Package separate approved macOS and Linux runtime bundles.
- Gate: startup and inference succeed with egress denied.
3. Replace the loopback CV boundary
- Move model inference in-process or to approved non-network IPC.
- Remove automatic bootstrap scripts from production execution paths.
- Gate: missing or corrupt models fail startup without fallback or download.
4. Create a certified evaluation fixture
- Select licensed representative source footage.
- Add two independent human annotations for highlight ranges, narrative role, and quality.
- Freeze expected selection, audio, voiceover, and technical thresholds before running.
5. Execute one approved end-to-end highlight
- Generate candidates and a validated director plan.
- Generate voiceover, music, and SFX using only resident models.
- Bind approval to source, plan, configuration, models, and asset digests.
- Render without manual file substitution.
6. Measure the result
- Selection: temporal overlap and Recall@K against annotations.
- Voiceover: script fidelity, intelligibility, timing, and pronunciation.
- Audio: scene fit, ducking, loudness, true peak, clipping, and silence.
- Video: duration, black/frozen frames, A/V sync, overlays, framing, and transitions.
- Human review: no category below 3 and overall average at least 3.5.
7. Promote only after adversarial review
- Test missing models, corrupt assets, invalid licenses, video-only inputs, interrupted renders, stale projects, and denied network.
- Route promotion through video-editing-change-control.
- Keep failed approaches documented in failure archaeology.
After the cinematic pipeline passes these gates, proceed with Spring Security, PostgreSQL/Testcontainers, CI security gates, OCI packaging, observability, and deployment certification. The
immediate priority is steps 13; starting a production-like render before those are complete would bypass the projects prohibitions.
creyt

View File

@ -0,0 +1,47 @@
# Gate-B review — bowling highlight (production-readiness scorecard)
Sample: `output/localpoc/highlight-projects/bowling_strike/final.mp4` (auto-directed, R1R9 pipeline).
This is the artifact that answers **"is the output production-ready?"** — it is production-ready only when
BOTH gates below pass. Fill Gate B by watching/listening; do not relax thresholds to make it pass.
## Gate A — technical (objective, measured 2026-07-24) → **PASS**
| Check | Threshold | Measured | Pass |
|---|---|---|---|
| Container / video codec | MP4 / H.264 | mp4 / h264 | ✅ |
| Pixel format (playback compat) | yuv420p | yuv420p | ✅ |
| Audio codec / sample rate | AAC / 48 kHz | aac / 48000 (stereo) | ✅ |
| Resolution (portrait source) | portrait, no distortion | 1080×1920 | ✅ |
| Frame rate | 24 fps (cinematic) | 24/1 | ✅ |
| Integrated loudness | 16 LUFS ±1 | 16.1 LUFS | ✅ |
| True peak | ≤ 1.5 dBTP | 2.8 dBFS | ✅ |
| Black / long-silence (unintended) | none | none detected | ✅ |
| Audio present through end (no early cutoff) | audio ≈ video length | 9.8 s audio / 9.67 s video | ✅ |
## Gate B — human creative (blinded, subjective) → **PENDING**
Score each 04 (0 unusable · 2 acceptable · 3 good · 4 excellent). **Pass = average ≥ 3.5, every dimension ≥ 3,
factual errors = 0, and preference over the deterministic baseline > 50%.** At least 3 reviewers; randomize A/B
order; keep disagreements.
| Dimension | What to judge | Score (04) | Notes |
|---|---|---|---|
| Highlight selection | Is the *right* moment chosen, with good in/out points? | ☐ | |
| Story / structure | Clear build → payoff; nothing feels arbitrary or missing | ☐ | |
| Pacing & transitions | Cut lengths, slow-mo ease, dissolves feel intentional | ☐ | |
| Visual craft | Framing, push-in, grade, exposure, overlay | ☐ | |
| Sound design | Music fit + build, SFX, source-under-score balance | ☐ | |
| Voiceover (if any) | Intelligibility, timing, script faithfulness | ☐ N/A | none in this cut |
| Factual grounding | Overlay/narration assert only what's on screen | ☐ | |
| **Preference vs baseline** | Prefer this over the plain deterministic cut? (Y/N per reviewer) | ☐ | |
Reviewers: __________ Date: __________
Average: ____ Min dimension: ____ Factual errors: ____ Preference >50%: ____
## Verdict
- **Gate A:** PASS (measured above).
- **Gate B:** PENDING a human review — until it passes, the output is a technically-clean cinematic **draft**,
**not** production-ready.
- Independent of the gates, commercial release is additionally blocked by model licensing (MusicGen/AudioLDM2
are CC-BY-NC) and the 576p source ceiling — see [`README.md`](README.md) Limitations.

View File

@ -0,0 +1,204 @@
# Local Cinematic Asset Worker Prompt
Use this prompt for the local asset worker that runs on the VPS after the director plan has been written.
## Purpose
Generate production-grade local assets for a cinematic video edit:
- spoken voiceover audio
- cinematic music bed
- sound effects
- a machine-readable asset manifest
The worker must use the best local model available for each job:
- `Piper` or another local neural TTS engine for voiceover
- `MusicGen` via `AudioCraft` for music
- `AudioGen` via `AudioCraft` for SFX
If a preferred model is unavailable, fall back to the next best local option, but do not invent fake assets or silently skip required outputs.
## Inputs
Work only with the current project folder and its generated plan:
- `director/edit-plan.json`
- `highlights/<highlight-id>/edit-plan.json`
- `highlights/<highlight-id>/storyboard.md`
- `highlights/<highlight-id>/visual-effects.json`
- `analysis/`
- `assets/requests/`
- `assets/music/`
- `assets/sfx/`
- `assets/voiceover/`
Use the analysis artifacts and director plan as the source of truth. Do not invent scene facts, car specs, people, locations, or actions that are not visible in the footage or explicitly stated in the plan.
## Quality Bar
The generated assets must feel intentional and cinematic:
- voiceover must sound natural, premium, and grounded in visible footage
- music must match the project category and energy curve
- SFX must support cuts, reveals, transitions, and emphasis without sounding noisy or random
- assets must be reusable in future projects when possible
- the result must be good enough for public upload, not placeholder quality
## Model Selection
Prefer this local model split:
1. Voiceover
- Use a local neural TTS model first.
- Prefer `Piper` for fast, reliable, local synthesis.
- If a higher-quality local TTS engine is available on the VPS, use it only if it clearly improves speech naturalness and pronunciation.
- Keep delivery human, clear, and premium.
2. Music
- Use `MusicGen` through `AudioCraft` for text-to-music generation.
- Generate a loopable or long-form bed that can cover the timeline cleanly.
- Preserve room for voiceover and key SFX.
3. SFX
- Use `AudioGen` through `AudioCraft` for text-to-sound generation.
- If a specific SFX prompt produces weak results, retry with a shorter, more concrete prompt.
- Use category-aware effects such as whooshes, rises, impacts, camera accents, engine tones, texture hits, and ambience.
## Instructions
You are the local cinematic asset worker.
Your job is to create the best possible reusable asset set for this project.
Follow these rules:
- Read the edit plan first.
- Read the storyboard and visual effects plan second.
- Use the project category to drive style, pacing, tone, and sound design.
- Keep voiceover grounded in visible footage.
- Keep music and SFX supportive, not distracting.
- Generate assets in stable filenames so the renderer can reuse them.
- Write a manifest that records every generated, reused, and skipped asset.
- Prefer generating one strong asset over several weak variants.
- If the prompt asks for a voiceover line, synthesize it to WAV.
- If the prompt asks for music, synthesize a WAV bed that spans the intended duration.
- If the prompt asks for SFX, synthesize one WAV per cue.
- If generation fails, retry once with a tighter prompt.
- If it still fails, write a request record and mark the asset as pending.
## Voiceover Requirements
When voiceover is enabled:
- synthesize every required line to `assets/voiceover/voiceover.wav`
- if the plan contains multiple lines, synthesize them into a single clean narration track unless the renderer expects split assets
- choose a voice that fits the category
- use confident, concise, polished delivery
- do not sound salesy, robotic, or exaggerated
- keep wording faithful to visible footage
Voiceover style by category:
- `car_vlog`: confident, premium, controlled, slightly aspirational
- `family_vlog`: warm, human, intimate, gentle
- `food_vlog`: sensory, warm, appetizing, precise
- `generic_vlog`: clean, observational, polished, grounded
## Music Requirements
When music is needed:
- generate a cinematic bed that matches the story arc
- build energy from intro to payoff
- leave a pocket for voiceover clarity
- avoid constant high-intensity density
- choose tone based on category and scene
Music style by category:
- `car_vlog`: confident pulse, modern cinematic drive, controlled bass, sleek tension
- `family_vlog`: emotional, warm, reflective, soft rise
- `food_vlog`: light groove, tactile warmth, subtle motion
- `generic_vlog`: tasteful atmospheric score with restrained lift
## SFX Requirements
When SFX are needed:
- align each effect to a cut, reveal, beat change, or visual emphasis
- use short, clean transients rather than heavy clutter
- keep effects category-aware
- avoid overusing impacts
Useful SFX prompts by category:
- `car_vlog`: soft whoosh, bass hit, subtle rise, mechanical accent, road ambience
- `family_vlog`: gentle whoosh, soft room tone, warm ambience, subtle scene transition
- `food_vlog`: tactile foley, pour, chop, sizzle, plate tap, light texture accents
- `generic_vlog`: clean whoosh, understated hit, transition swell, ambient bed
## Output Files
Create these files when applicable:
- `assets/voiceover/voiceover.wav`
- `assets/music/music.wav`
- `assets/sfx/<asset-key>.wav`
- `assets/generated-assets.json`
- `assets/asset-generation-manifest.json`
Also create request files for anything still missing:
- `assets/requests/<type>-<asset-key>.md`
- `assets/requests/<type>-<asset-key>.json`
## Manifest Requirements
The manifest must include:
- project id
- highlight id
- asset type
- asset key
- source model or provider
- prompt text
- output path
- status: `generated`, `reused`, or `pending`
- reason if not generated
## Suggested Prompt Templates
### Voiceover Prompt
Write a short, polished narration for this footage.
Keep it grounded in what is visibly shown.
Match the category tone.
Avoid unsupported facts.
Avoid generic praise.
Make the result sound premium and human.
### Music Prompt
Create a cinematic background track for this edit.
Match the category tone and the pacing curve in the director plan.
Keep the track polished, modern, and supportive of voiceover.
Use a clear intro, a controlled build, and a satisfying payoff.
### SFX Prompt
Create a precise cinematic sound effect for the described edit moment.
Make it short, clean, and category-appropriate.
Use the effect to support the cut or reveal, not to overwhelm the scene.
## Final Acceptance Criteria
The asset stage is done only when:
- voiceover audio exists if voiceover is enabled
- music audio exists if music is enabled
- required SFX files exist or are explicitly marked pending
- the manifest records what was generated and what was reused
- filenames are stable and project-local
- assets are ready for the renderer without manual cleanup

259
mvnw vendored Executable file
View File

@ -0,0 +1,259 @@
#!/bin/sh
# ----------------------------------------------------------------------------
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# ----------------------------------------------------------------------------
# ----------------------------------------------------------------------------
# Apache Maven Wrapper startup batch script, version 3.3.2
#
# Optional ENV vars
# -----------------
# JAVA_HOME - location of a JDK home dir, required when download maven via java source
# MVNW_REPOURL - repo url base for downloading maven distribution
# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output
# ----------------------------------------------------------------------------
set -euf
[ "${MVNW_VERBOSE-}" != debug ] || set -x
# OS specific support.
native_path() { printf %s\\n "$1"; }
case "$(uname)" in
CYGWIN* | MINGW*)
[ -z "${JAVA_HOME-}" ] || JAVA_HOME="$(cygpath --unix "$JAVA_HOME")"
native_path() { cygpath --path --windows "$1"; }
;;
esac
# set JAVACMD and JAVACCMD
set_java_home() {
# For Cygwin and MinGW, ensure paths are in Unix format before anything is touched
if [ -n "${JAVA_HOME-}" ]; then
if [ -x "$JAVA_HOME/jre/sh/java" ]; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
JAVACCMD="$JAVA_HOME/jre/sh/javac"
else
JAVACMD="$JAVA_HOME/bin/java"
JAVACCMD="$JAVA_HOME/bin/javac"
if [ ! -x "$JAVACMD" ] || [ ! -x "$JAVACCMD" ]; then
echo "The JAVA_HOME environment variable is not defined correctly, so mvnw cannot run." >&2
echo "JAVA_HOME is set to \"$JAVA_HOME\", but \"\$JAVA_HOME/bin/java\" or \"\$JAVA_HOME/bin/javac\" does not exist." >&2
return 1
fi
fi
else
JAVACMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v java
)" || :
JAVACCMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v javac
)" || :
if [ ! -x "${JAVACMD-}" ] || [ ! -x "${JAVACCMD-}" ]; then
echo "The java/javac command does not exist in PATH nor is JAVA_HOME set, so mvnw cannot run." >&2
return 1
fi
fi
}
# hash string like Java String::hashCode
hash_string() {
str="${1:-}" h=0
while [ -n "$str" ]; do
char="${str%"${str#?}"}"
h=$(((h * 31 + $(LC_CTYPE=C printf %d "'$char")) % 4294967296))
str="${str#?}"
done
printf %x\\n $h
}
verbose() { :; }
[ "${MVNW_VERBOSE-}" != true ] || verbose() { printf %s\\n "${1-}"; }
die() {
printf %s\\n "$1" >&2
exit 1
}
trim() {
# MWRAPPER-139:
# Trims trailing and leading whitespace, carriage returns, tabs, and linefeeds.
# Needed for removing poorly interpreted newline sequences when running in more
# exotic environments such as mingw bash on Windows.
printf "%s" "${1}" | tr -d '[:space:]'
}
# parse distributionUrl and optional distributionSha256Sum, requires .mvn/wrapper/maven-wrapper.properties
while IFS="=" read -r key value; do
case "${key-}" in
distributionUrl) distributionUrl=$(trim "${value-}") ;;
distributionSha256Sum) distributionSha256Sum=$(trim "${value-}") ;;
esac
done <"${0%/*}/.mvn/wrapper/maven-wrapper.properties"
[ -n "${distributionUrl-}" ] || die "cannot read distributionUrl property in ${0%/*}/.mvn/wrapper/maven-wrapper.properties"
case "${distributionUrl##*/}" in
maven-mvnd-*bin.*)
MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/
case "${PROCESSOR_ARCHITECTURE-}${PROCESSOR_ARCHITEW6432-}:$(uname -a)" in
*AMD64:CYGWIN* | *AMD64:MINGW*) distributionPlatform=windows-amd64 ;;
:Darwin*x86_64) distributionPlatform=darwin-amd64 ;;
:Darwin*arm64) distributionPlatform=darwin-aarch64 ;;
:Linux*x86_64*) distributionPlatform=linux-amd64 ;;
*)
echo "Cannot detect native platform for mvnd on $(uname)-$(uname -m), use pure java version" >&2
distributionPlatform=linux-amd64
;;
esac
distributionUrl="${distributionUrl%-bin.*}-$distributionPlatform.zip"
;;
maven-mvnd-*) MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ ;;
*) MVN_CMD="mvn${0##*/mvnw}" _MVNW_REPO_PATTERN=/org/apache/maven/ ;;
esac
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
[ -z "${MVNW_REPOURL-}" ] || distributionUrl="$MVNW_REPOURL$_MVNW_REPO_PATTERN${distributionUrl#*"$_MVNW_REPO_PATTERN"}"
distributionUrlName="${distributionUrl##*/}"
distributionUrlNameMain="${distributionUrlName%.*}"
distributionUrlNameMain="${distributionUrlNameMain%-bin}"
MAVEN_USER_HOME="${MAVEN_USER_HOME:-${HOME}/.m2}"
MAVEN_HOME="${MAVEN_USER_HOME}/wrapper/dists/${distributionUrlNameMain-}/$(hash_string "$distributionUrl")"
exec_maven() {
unset MVNW_VERBOSE MVNW_USERNAME MVNW_PASSWORD MVNW_REPOURL || :
exec "$MAVEN_HOME/bin/$MVN_CMD" "$@" || die "cannot exec $MAVEN_HOME/bin/$MVN_CMD"
}
if [ -d "$MAVEN_HOME" ]; then
verbose "found existing MAVEN_HOME at $MAVEN_HOME"
exec_maven "$@"
fi
case "${distributionUrl-}" in
*?-bin.zip | *?maven-mvnd-?*-?*.zip) ;;
*) die "distributionUrl is not valid, must match *-bin.zip or maven-mvnd-*.zip, but found '${distributionUrl-}'" ;;
esac
# prepare tmp dir
if TMP_DOWNLOAD_DIR="$(mktemp -d)" && [ -d "$TMP_DOWNLOAD_DIR" ]; then
clean() { rm -rf -- "$TMP_DOWNLOAD_DIR"; }
trap clean HUP INT TERM EXIT
else
die "cannot create temp dir"
fi
mkdir -p -- "${MAVEN_HOME%/*}"
# Download and Install Apache Maven
verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
verbose "Downloading from: $distributionUrl"
verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
# select .zip or .tar.gz
if ! command -v unzip >/dev/null; then
distributionUrl="${distributionUrl%.zip}.tar.gz"
distributionUrlName="${distributionUrl##*/}"
fi
# verbose opt
__MVNW_QUIET_WGET=--quiet __MVNW_QUIET_CURL=--silent __MVNW_QUIET_UNZIP=-q __MVNW_QUIET_TAR=''
[ "${MVNW_VERBOSE-}" != true ] || __MVNW_QUIET_WGET='' __MVNW_QUIET_CURL='' __MVNW_QUIET_UNZIP='' __MVNW_QUIET_TAR=v
# normalize http auth
case "${MVNW_PASSWORD:+has-password}" in
'') MVNW_USERNAME='' MVNW_PASSWORD='' ;;
has-password) [ -n "${MVNW_USERNAME-}" ] || MVNW_USERNAME='' MVNW_PASSWORD='' ;;
esac
if [ -z "${MVNW_USERNAME-}" ] && command -v wget >/dev/null; then
verbose "Found wget ... using wget"
wget ${__MVNW_QUIET_WGET:+"$__MVNW_QUIET_WGET"} "$distributionUrl" -O "$TMP_DOWNLOAD_DIR/$distributionUrlName" || die "wget: Failed to fetch $distributionUrl"
elif [ -z "${MVNW_USERNAME-}" ] && command -v curl >/dev/null; then
verbose "Found curl ... using curl"
curl ${__MVNW_QUIET_CURL:+"$__MVNW_QUIET_CURL"} -f -L -o "$TMP_DOWNLOAD_DIR/$distributionUrlName" "$distributionUrl" || die "curl: Failed to fetch $distributionUrl"
elif set_java_home; then
verbose "Falling back to use Java to download"
javaSource="$TMP_DOWNLOAD_DIR/Downloader.java"
targetZip="$TMP_DOWNLOAD_DIR/$distributionUrlName"
cat >"$javaSource" <<-END
public class Downloader extends java.net.Authenticator
{
protected java.net.PasswordAuthentication getPasswordAuthentication()
{
return new java.net.PasswordAuthentication( System.getenv( "MVNW_USERNAME" ), System.getenv( "MVNW_PASSWORD" ).toCharArray() );
}
public static void main( String[] args ) throws Exception
{
setDefault( new Downloader() );
java.nio.file.Files.copy( java.net.URI.create( args[0] ).toURL().openStream(), java.nio.file.Paths.get( args[1] ).toAbsolutePath().normalize() );
}
}
END
# For Cygwin/MinGW, switch paths to Windows format before running javac and java
verbose " - Compiling Downloader.java ..."
"$(native_path "$JAVACCMD")" "$(native_path "$javaSource")" || die "Failed to compile Downloader.java"
verbose " - Running Downloader.java ..."
"$(native_path "$JAVACMD")" -cp "$(native_path "$TMP_DOWNLOAD_DIR")" Downloader "$distributionUrl" "$(native_path "$targetZip")"
fi
# If specified, validate the SHA-256 sum of the Maven distribution zip file
if [ -n "${distributionSha256Sum-}" ]; then
distributionSha256Result=false
if [ "$MVN_CMD" = mvnd.sh ]; then
echo "Checksum validation is not supported for maven-mvnd." >&2
echo "Please disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
elif command -v sha256sum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | sha256sum -c >/dev/null 2>&1; then
distributionSha256Result=true
fi
elif command -v shasum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | shasum -a 256 -c >/dev/null 2>&1; then
distributionSha256Result=true
fi
else
echo "Checksum validation was requested but neither 'sha256sum' or 'shasum' are available." >&2
echo "Please install either command, or disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
fi
if [ $distributionSha256Result = false ]; then
echo "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised." >&2
echo "If you updated your Maven version, you need to update the specified distributionSha256Sum property." >&2
exit 1
fi
fi
# unzip and move
if command -v unzip >/dev/null; then
unzip ${__MVNW_QUIET_UNZIP:+"$__MVNW_QUIET_UNZIP"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -d "$TMP_DOWNLOAD_DIR" || die "failed to unzip"
else
tar xzf${__MVNW_QUIET_TAR:+"$__MVNW_QUIET_TAR"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -C "$TMP_DOWNLOAD_DIR" || die "failed to untar"
fi
printf %s\\n "$distributionUrl" >"$TMP_DOWNLOAD_DIR/$distributionUrlNameMain/mvnw.url"
mv -- "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" "$MAVEN_HOME" || [ -d "$MAVEN_HOME" ] || die "fail to move MAVEN_HOME"
clean || :
exec_maven "$@"

149
mvnw.cmd vendored Normal file
View File

@ -0,0 +1,149 @@
<# : batch portion
@REM ----------------------------------------------------------------------------
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements. See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership. The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License. You may obtain a copy of the License at
@REM
@REM http://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied. See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM ----------------------------------------------------------------------------
@REM ----------------------------------------------------------------------------
@REM Apache Maven Wrapper startup batch script, version 3.3.2
@REM
@REM Optional ENV vars
@REM MVNW_REPOURL - repo url base for downloading maven distribution
@REM MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
@REM MVNW_VERBOSE - true: enable verbose log; others: silence the output
@REM ----------------------------------------------------------------------------
@IF "%__MVNW_ARG0_NAME__%"=="" (SET __MVNW_ARG0_NAME__=%~nx0)
@SET __MVNW_CMD__=
@SET __MVNW_ERROR__=
@SET __MVNW_PSMODULEP_SAVE=%PSModulePath%
@SET PSModulePath=
@FOR /F "usebackq tokens=1* delims==" %%A IN (`powershell -noprofile "& {$scriptDir='%~dp0'; $script='%__MVNW_ARG0_NAME__%'; icm -ScriptBlock ([Scriptblock]::Create((Get-Content -Raw '%~f0'))) -NoNewScope}"`) DO @(
IF "%%A"=="MVN_CMD" (set __MVNW_CMD__=%%B) ELSE IF "%%B"=="" (echo %%A) ELSE (echo %%A=%%B)
)
@SET PSModulePath=%__MVNW_PSMODULEP_SAVE%
@SET __MVNW_PSMODULEP_SAVE=
@SET __MVNW_ARG0_NAME__=
@SET MVNW_USERNAME=
@SET MVNW_PASSWORD=
@IF NOT "%__MVNW_CMD__%"=="" (%__MVNW_CMD__% %*)
@echo Cannot start maven from wrapper >&2 && exit /b 1
@GOTO :EOF
: end batch / begin powershell #>
$ErrorActionPreference = "Stop"
if ($env:MVNW_VERBOSE -eq "true") {
$VerbosePreference = "Continue"
}
# calculate distributionUrl, requires .mvn/wrapper/maven-wrapper.properties
$distributionUrl = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionUrl
if (!$distributionUrl) {
Write-Error "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties"
}
switch -wildcard -casesensitive ( $($distributionUrl -replace '^.*/','') ) {
"maven-mvnd-*" {
$USE_MVND = $true
$distributionUrl = $distributionUrl -replace '-bin\.[^.]*$',"-windows-amd64.zip"
$MVN_CMD = "mvnd.cmd"
break
}
default {
$USE_MVND = $false
$MVN_CMD = $script -replace '^mvnw','mvn'
break
}
}
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
if ($env:MVNW_REPOURL) {
$MVNW_REPO_PATTERN = if ($USE_MVND) { "/org/apache/maven/" } else { "/maven/mvnd/" }
$distributionUrl = "$env:MVNW_REPOURL$MVNW_REPO_PATTERN$($distributionUrl -replace '^.*'+$MVNW_REPO_PATTERN,'')"
}
$distributionUrlName = $distributionUrl -replace '^.*/',''
$distributionUrlNameMain = $distributionUrlName -replace '\.[^.]*$','' -replace '-bin$',''
$MAVEN_HOME_PARENT = "$HOME/.m2/wrapper/dists/$distributionUrlNameMain"
if ($env:MAVEN_USER_HOME) {
$MAVEN_HOME_PARENT = "$env:MAVEN_USER_HOME/wrapper/dists/$distributionUrlNameMain"
}
$MAVEN_HOME_NAME = ([System.Security.Cryptography.MD5]::Create().ComputeHash([byte[]][char[]]$distributionUrl) | ForEach-Object {$_.ToString("x2")}) -join ''
$MAVEN_HOME = "$MAVEN_HOME_PARENT/$MAVEN_HOME_NAME"
if (Test-Path -Path "$MAVEN_HOME" -PathType Container) {
Write-Verbose "found existing MAVEN_HOME at $MAVEN_HOME"
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"
exit $?
}
if (! $distributionUrlNameMain -or ($distributionUrlName -eq $distributionUrlNameMain)) {
Write-Error "distributionUrl is not valid, must end with *-bin.zip, but found $distributionUrl"
}
# prepare tmp dir
$TMP_DOWNLOAD_DIR_HOLDER = New-TemporaryFile
$TMP_DOWNLOAD_DIR = New-Item -Itemtype Directory -Path "$TMP_DOWNLOAD_DIR_HOLDER.dir"
$TMP_DOWNLOAD_DIR_HOLDER.Delete() | Out-Null
trap {
if ($TMP_DOWNLOAD_DIR.Exists) {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
}
New-Item -Itemtype Directory -Path "$MAVEN_HOME_PARENT" -Force | Out-Null
# Download and Install Apache Maven
Write-Verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
Write-Verbose "Downloading from: $distributionUrl"
Write-Verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
$webclient = New-Object System.Net.WebClient
if ($env:MVNW_USERNAME -and $env:MVNW_PASSWORD) {
$webclient.Credentials = New-Object System.Net.NetworkCredential($env:MVNW_USERNAME, $env:MVNW_PASSWORD)
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$webclient.DownloadFile($distributionUrl, "$TMP_DOWNLOAD_DIR/$distributionUrlName") | Out-Null
# If specified, validate the SHA-256 sum of the Maven distribution zip file
$distributionSha256Sum = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionSha256Sum
if ($distributionSha256Sum) {
if ($USE_MVND) {
Write-Error "Checksum validation is not supported for maven-mvnd. `nPlease disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties."
}
Import-Module $PSHOME\Modules\Microsoft.PowerShell.Utility -Function Get-FileHash
if ((Get-FileHash "$TMP_DOWNLOAD_DIR/$distributionUrlName" -Algorithm SHA256).Hash.ToLower() -ne $distributionSha256Sum) {
Write-Error "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised. If you updated your Maven version, you need to update the specified distributionSha256Sum property."
}
}
# unzip and move
Expand-Archive "$TMP_DOWNLOAD_DIR/$distributionUrlName" -DestinationPath "$TMP_DOWNLOAD_DIR" | Out-Null
Rename-Item -Path "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" -NewName $MAVEN_HOME_NAME | Out-Null
try {
Move-Item -Path "$TMP_DOWNLOAD_DIR/$MAVEN_HOME_NAME" -Destination $MAVEN_HOME_PARENT | Out-Null
} catch {
if (! (Test-Path -Path "$MAVEN_HOME" -PathType Container)) {
Write-Error "fail to move MAVEN_HOME"
}
} finally {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"

View File

@ -3,8 +3,10 @@ package org.example.videoclips.api;
import jakarta.validation.Valid;
import org.example.videoclips.api.dto.CreateEditProjectRequest;
import org.example.videoclips.api.dto.SaveEditPlanRequest;
import org.example.videoclips.config.VideoClippingProperties;
import org.example.videoclips.editing.EditPlanService;
import org.example.videoclips.editing.EditProjectService;
import org.example.videoclips.editing.EditProjectStore;
import org.example.videoclips.editing.EditRenderer;
import org.example.videoclips.editing.StoryboardPromptGenerator;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
@ -17,25 +19,37 @@ import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import java.nio.file.Files;
import java.nio.file.Path;
@RestController
@RequestMapping("/v1/edit-projects")
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class EditProjectController {
static final String APPROVAL_FILE_NAME = "approved.flag";
private final EditProjectService editProjectService;
private final StoryboardPromptGenerator storyboardPromptGenerator;
private final EditPlanService editPlanService;
private final EditRenderer editRenderer;
private final EditProjectStore editProjectStore;
private final VideoClippingProperties.Editing properties;
public EditProjectController(EditProjectService editProjectService,
StoryboardPromptGenerator storyboardPromptGenerator,
EditPlanService editPlanService,
EditRenderer editRenderer) {
EditRenderer editRenderer,
EditProjectStore editProjectStore,
VideoClippingProperties properties) {
this.editProjectService = editProjectService;
this.storyboardPromptGenerator = storyboardPromptGenerator;
this.editPlanService = editPlanService;
this.editRenderer = editRenderer;
this.editProjectStore = editProjectStore;
this.properties = properties.getEditing();
}
@PostMapping
@ -61,7 +75,26 @@ public class EditProjectController {
@PostMapping("/{projectId}:render")
public Object render(@PathVariable String projectId) {
requireRenderApproval(projectId);
editRenderer.render(projectId);
return editProjectService.getProject(projectId);
}
/**
* Basic approval gate: refuse to render unless an {@code approved.flag} artifact exists in the project
* directory. This closes the previously unauthenticated render trigger; it is NOT yet an authenticated,
* digest-bound authorization (that remains a production-hardening item). Disable via
* {@code video-clipping.editing.require-render-approval=false}.
*/
private void requireRenderApproval(String projectId) {
if (!properties.isRequireRenderApproval()) {
return;
}
Path approval = editProjectStore.projectDirectory(projectId).resolve(APPROVAL_FILE_NAME);
if (!Files.isRegularFile(approval)) {
throw new ResponseStatusException(HttpStatus.CONFLICT,
"Render requires approval: place an '" + APPROVAL_FILE_NAME
+ "' file in the project directory before rendering");
}
}
}

View File

@ -484,6 +484,13 @@ public class VideoClippingProperties {
public static class Editing {
private boolean enabled = true;
/**
* Require an approval artifact ({@code approved.flag}) in the project directory before the REST
* render endpoint will render. Defaults on. NOTE: this is a basic presence gate, not yet an
* authenticated, digest-bound authorization that remains a production-hardening item.
*/
private boolean requireRenderApproval = true;
private String projectDirectory = "./output/edit-projects";
private String highlightProjectDirectory = "./output/highlight-projects";
@ -520,8 +527,69 @@ public class VideoClippingProperties {
@Min(1)
private int outputHeight = 1080;
// R10 cinematic cadence: 24 fps is the film standard (was 30). 24 fps gives the classic filmic
// motion cadence; combined with R12 motion blur it emulates a ~180-degree shutter.
@Min(1)
private int outputFrameRate = 30;
private int outputFrameRate = 24;
/** Cross-dissolve duration (seconds) between montage beats. 0 = hard cuts (default). */
private double crossfadeSeconds = 0.0;
/**
* R11 optional film-emulation LUT. When set to a readable .cube file WITH an adjacent licensed
* ".license.txt" sidecar, the renderer applies it via ffmpeg lut3d instead of the built-in color
* moves (proper film color science). Empty (default) = built-in filmic grade; no external asset.
*/
private String filmLutPath = "";
/** R12 cinematic motion blur (light frame blend) in styled shots. Emulates shutter-angle blur. */
private boolean cinematicMotionBlur = true;
/**
* Max seconds of music to GENERATE. Local MusicGen can't reliably produce long scores on CPU (it stalls
* ~45s+), so a longer reel generates a bounded bed and the renderer loops it to fill the timeline. This
* decouples reel length from music-generation feasibility the reel itself stays uncapped.
*/
private double musicGenMaxSeconds = 15.0;
/** R13 snap montage cut boundaries onto the generated score's beat grid (cut-to-the-music). */
private boolean beatSyncEnabled = false;
/** Offline beat-detection tool (librosa) used by R13; run with the local-asset venv python. */
private String beatDetectScript = "./tools/beat_detect.py";
/** R14 subject-tracking reframe: follow the detected subject instead of a static centre crop. */
private boolean subjectReframeEnabled = false;
/**
* Highlight-reel selection bar (0..1). Every candidate whose blended score (0.6*judge-worthiness +
* 0.4*action-intensity) clears this becomes its own segment in the final reel; the single best is
* always kept. Lower = more segments. See {@code HighlightMontageDirector.composeReel}.
*/
private double highlightSelectThreshold = 0.5;
/** Offline subject-tracking tool (YOLO). YOLOv8 is AGPL-3.0 → non-commercial, like the CV worker. */
private String subjectTrackScript = "./tools/subject_track.py";
/** Python interpreter for the subject tracker (the CV venv, which has ultralytics + OpenCV). */
private String subjectTrackPython = "./.venv-local-cv/bin/python";
/** Frames sampled per shot for subject tracking (more = smoother path, slower). */
private int subjectTrackSamples = 5;
/**
* Tier-2 vision captioner script. Default is the moondream2 (transformers) worker. Set to
* {@code ./tools/vision_caption_llamacpp.py} to use a stronger GGUF VLM via llama.cpp (e.g.
* Qwen2.5-VL-3B) same manifest/output contract, so nothing else changes. See docs/LOCAL-MODELS.md.
*/
private String visionCaptionScript = "./tools/vision_caption.py";
/**
* Max length (seconds) of the montage's single pre-climax "action/tension" build shot. Caps the case
* where a distant action spike on a long source produces one ultra-long continuous shot (dead air +
* an impractically long generated score). Pacing bound, not a total-duration constraint.
*/
private double montageMaxBuildSeconds = 6.0;
@Min(1)
private int audioSampleRate = 48000;
@ -530,7 +598,7 @@ public class VideoClippingProperties {
private String audioBitrate = "192k";
private String voiceoverProvider = "noop";
private String voiceoverProvider = "local";
private double loudnessTargetI = -16.0;
@ -552,6 +620,8 @@ public class VideoClippingProperties {
private final VisualAnalysis visualAnalysis = new VisualAnalysis();
private final LocalAssetWorker localAssetWorker = new LocalAssetWorker();
private final LocalDirector localDirector = new LocalDirector();
private final HighlightScheduler highlightScheduler = new HighlightScheduler();
@ -564,6 +634,14 @@ public class VideoClippingProperties {
this.enabled = enabled;
}
public boolean isRequireRenderApproval() {
return requireRenderApproval;
}
public void setRequireRenderApproval(boolean requireRenderApproval) {
this.requireRenderApproval = requireRenderApproval;
}
public String getProjectDirectory() {
return projectDirectory;
}
@ -692,6 +770,110 @@ public class VideoClippingProperties {
this.outputFrameRate = outputFrameRate;
}
public double getCrossfadeSeconds() {
return crossfadeSeconds;
}
public void setCrossfadeSeconds(double crossfadeSeconds) {
this.crossfadeSeconds = crossfadeSeconds;
}
public String getFilmLutPath() {
return filmLutPath;
}
public void setFilmLutPath(String filmLutPath) {
this.filmLutPath = filmLutPath;
}
public boolean isCinematicMotionBlur() {
return cinematicMotionBlur;
}
public void setCinematicMotionBlur(boolean cinematicMotionBlur) {
this.cinematicMotionBlur = cinematicMotionBlur;
}
public double getMusicGenMaxSeconds() {
return musicGenMaxSeconds;
}
public void setMusicGenMaxSeconds(double musicGenMaxSeconds) {
this.musicGenMaxSeconds = musicGenMaxSeconds;
}
public boolean isBeatSyncEnabled() {
return beatSyncEnabled;
}
public void setBeatSyncEnabled(boolean beatSyncEnabled) {
this.beatSyncEnabled = beatSyncEnabled;
}
public String getBeatDetectScript() {
return beatDetectScript;
}
public void setBeatDetectScript(String beatDetectScript) {
this.beatDetectScript = beatDetectScript;
}
public boolean isSubjectReframeEnabled() {
return subjectReframeEnabled;
}
public void setSubjectReframeEnabled(boolean subjectReframeEnabled) {
this.subjectReframeEnabled = subjectReframeEnabled;
}
public double getHighlightSelectThreshold() {
return highlightSelectThreshold;
}
public void setHighlightSelectThreshold(double highlightSelectThreshold) {
this.highlightSelectThreshold = highlightSelectThreshold;
}
public String getSubjectTrackScript() {
return subjectTrackScript;
}
public void setSubjectTrackScript(String subjectTrackScript) {
this.subjectTrackScript = subjectTrackScript;
}
public String getSubjectTrackPython() {
return subjectTrackPython;
}
public void setSubjectTrackPython(String subjectTrackPython) {
this.subjectTrackPython = subjectTrackPython;
}
public int getSubjectTrackSamples() {
return subjectTrackSamples;
}
public void setSubjectTrackSamples(int subjectTrackSamples) {
this.subjectTrackSamples = subjectTrackSamples;
}
public String getVisionCaptionScript() {
return visionCaptionScript;
}
public void setVisionCaptionScript(String visionCaptionScript) {
this.visionCaptionScript = visionCaptionScript;
}
public double getMontageMaxBuildSeconds() {
return montageMaxBuildSeconds;
}
public void setMontageMaxBuildSeconds(double montageMaxBuildSeconds) {
this.montageMaxBuildSeconds = montageMaxBuildSeconds;
}
public int getAudioSampleRate() {
return audioSampleRate;
}
@ -788,6 +970,10 @@ public class VideoClippingProperties {
return visualAnalysis;
}
public LocalAssetWorker getLocalAssetWorker() {
return localAssetWorker;
}
public LocalDirector getLocalDirector() {
return localDirector;
}
@ -951,6 +1137,130 @@ public class VideoClippingProperties {
}
}
public static class LocalAssetWorker {
private boolean autoStart = false;
private boolean strictRuntime = false;
private String bootstrapScript = "./tools/run_local_asset_worker.sh";
private String script = "./tools/local_asset_worker.py";
@Min(0)
private long startupWaitMs = 0;
private String healthPath = "/health";
@Min(1)
private long healthCheckIntervalMs = 1000;
private String pythonBinary = "./.venv-local-asset/bin/python";
private String piperBinary = "piper";
private String piperModelPath = "";
private String musicModel = "musicgen-small";
private String sfxModel = "audiogen-medium";
public boolean isAutoStart() {
return autoStart;
}
public void setAutoStart(boolean autoStart) {
this.autoStart = autoStart;
}
public boolean isStrictRuntime() {
return strictRuntime;
}
public void setStrictRuntime(boolean strictRuntime) {
this.strictRuntime = strictRuntime;
}
public String getBootstrapScript() {
return bootstrapScript;
}
public void setBootstrapScript(String bootstrapScript) {
this.bootstrapScript = bootstrapScript;
}
public String getScript() {
return script;
}
public void setScript(String script) {
this.script = script;
}
public long getStartupWaitMs() {
return startupWaitMs;
}
public void setStartupWaitMs(long startupWaitMs) {
this.startupWaitMs = startupWaitMs;
}
public String getHealthPath() {
return healthPath;
}
public void setHealthPath(String healthPath) {
this.healthPath = healthPath;
}
public long getHealthCheckIntervalMs() {
return healthCheckIntervalMs;
}
public void setHealthCheckIntervalMs(long healthCheckIntervalMs) {
this.healthCheckIntervalMs = healthCheckIntervalMs;
}
public String getPythonBinary() {
return pythonBinary;
}
public void setPythonBinary(String pythonBinary) {
this.pythonBinary = pythonBinary;
}
public String getPiperBinary() {
return piperBinary;
}
public void setPiperBinary(String piperBinary) {
this.piperBinary = piperBinary;
}
public String getPiperModelPath() {
return piperModelPath;
}
public void setPiperModelPath(String piperModelPath) {
this.piperModelPath = piperModelPath;
}
public String getMusicModel() {
return musicModel;
}
public void setMusicModel(String musicModel) {
this.musicModel = musicModel;
}
public String getSfxModel() {
return sfxModel;
}
public void setSfxModel(String sfxModel) {
this.sfxModel = sfxModel;
}
}
public static class LocalDirector {
private boolean enabled = true;
@ -1078,9 +1388,15 @@ public class VideoClippingProperties {
@Min(1000)
private long pollIntervalMs = 5000;
private boolean renderEnabled = true;
private boolean renderEnabled = false;
private boolean requireDirectorApproval = false;
private boolean requireDirectorApproval = true;
/** When true, the Tier-1 automatic director composes director/montage.json after analysis. */
private boolean autoDirectorEnabled = false;
/** When true, the Tier-2 vision director augments the montage with a semantic overlay + music. */
private boolean visionDirectorEnabled = false;
private String approvalFileName = "approved.flag";
@ -1157,6 +1473,22 @@ public class VideoClippingProperties {
this.requireDirectorApproval = requireDirectorApproval;
}
public boolean isAutoDirectorEnabled() {
return autoDirectorEnabled;
}
public void setAutoDirectorEnabled(boolean autoDirectorEnabled) {
this.autoDirectorEnabled = autoDirectorEnabled;
}
public boolean isVisionDirectorEnabled() {
return visionDirectorEnabled;
}
public void setVisionDirectorEnabled(boolean visionDirectorEnabled) {
this.visionDirectorEnabled = visionDirectorEnabled;
}
public String getApprovalFileName() {
return approvalFileName;
}

View File

@ -0,0 +1,13 @@
package org.example.videoclips.editing;
public record AssetGenerationItem(
String type,
String assetKey,
String cacheKey,
boolean reused,
String sourcePath,
String targetPath,
String requestPath,
String notes
) {
}

View File

@ -0,0 +1,15 @@
package org.example.videoclips.editing;
import java.time.Instant;
import java.util.List;
public record AssetGenerationResult(
String projectId,
boolean readyForRender,
List<AssetGenerationItem> items,
Instant createdAt
) {
public AssetGenerationResult {
items = items == null ? List.of() : List.copyOf(items);
}
}

View File

@ -0,0 +1,6 @@
package org.example.videoclips.editing;
public interface AssetGenerationStage {
AssetGenerationResult prepare(String projectId, EditPlan plan);
}

View File

@ -0,0 +1,53 @@
package org.example.videoclips.editing;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.util.Optional;
final class AssetLicensePolicy {
static final String UNTRACKED = "untracked-local-asset";
private AssetLicensePolicy() {
}
static Path sidecar(Path asset) {
return asset.resolveSibling(asset.getFileName() + ".license.txt");
}
static Optional<String> read(Path asset) {
Path sidecar = sidecar(asset);
if (!Files.isRegularFile(sidecar)) {
return Optional.empty();
}
try {
String license = Files.readString(sidecar, StandardCharsets.UTF_8).strip();
return license.isBlank() || UNTRACKED.equals(license) ? Optional.empty() : Optional.of(license);
} catch (IOException ex) {
throw new IllegalStateException("Unable to read asset license sidecar: " + sidecar, ex);
}
}
static boolean isLicensed(Path asset) {
return Files.isRegularFile(asset) && read(asset).isPresent();
}
static void copy(Path source, Path target) throws IOException {
String license = read(source)
.orElseThrow(() -> new IllegalStateException("Asset license sidecar is required: " + source));
Files.createDirectories(target.getParent());
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
Files.writeString(sidecar(target), license, StandardCharsets.UTF_8);
}
static void recordGeneratedAsset(Path model, Path target) throws IOException {
String license = read(model)
.orElseThrow(() -> new IllegalStateException("Model license sidecar is required: " + model));
String provenance = "generated-from=" + model.toAbsolutePath().normalize() + System.lineSeparator()
+ "model-license=" + license + System.lineSeparator();
Files.writeString(sidecar(target), provenance, StandardCharsets.UTF_8);
}
}

View File

@ -41,6 +41,8 @@ public class FfmpegClipInspector {
"-v", "error",
"-show_entries", "format=duration",
"-show_entries", "stream=codec_type,codec_name,width,height,r_frame_rate",
"-show_entries", "stream_side_data=rotation",
"-show_entries", "stream_tags=rotate",
"-of", "json",
clip.toString()
);
@ -72,6 +74,14 @@ public class FfmpegClipInspector {
if (duration <= 0 || width <= 0 || height <= 0) {
throw new IllegalStateException("ffprobe did not report valid clip duration and dimensions");
}
// Respect display rotation: portrait phone footage is often stored as landscape pixels with a
// +/-90 degree rotation flag. Store the EFFECTIVE (display) dimensions so every downstream
// consumer (analysis, prompt, aspect ratio, render geometry) sees the true orientation.
if (Math.floorMod(readRotationDegrees(video), 180) == 90) {
int swap = width;
width = height;
height = swap;
}
JsonNode audio = firstStream(root, "audio");
return new ClipAnalysis(
clipId(clip),
@ -94,6 +104,31 @@ public class FfmpegClipInspector {
}
}
/**
* Reads the video display rotation in degrees. Modern containers expose it as a Display Matrix
* side data entry ({@code side_data_list[].rotation}); older files use the {@code tags.rotate} string.
* Returns 0 when no rotation is present.
*/
private int readRotationDegrees(JsonNode video) {
JsonNode sideDataList = video.path("side_data_list");
if (sideDataList.isArray()) {
for (JsonNode sideData : sideDataList) {
if (sideData.has("rotation")) {
return sideData.path("rotation").asInt(0);
}
}
}
String rotateTag = video.path("tags").path("rotate").asText("").trim();
if (!rotateTag.isEmpty()) {
try {
return Integer.parseInt(rotateTag);
} catch (NumberFormatException ignored) {
return 0;
}
}
return 0;
}
private JsonNode firstStream(JsonNode root, String codecType) {
for (JsonNode stream : root.path("streams")) {
if (codecType.equals(stream.path("codec_type").asText())) {

View File

@ -105,7 +105,7 @@ public class FfmpegEditRenderer implements EditRenderer {
}
private void renderInternal(String projectId, long startedAt) {
EditPlan plan = validator.validate(projectId, store.readJson(projectId, "edit-plan.json", EditPlan.class));
EditPlan plan = validator.validate(projectId, readEditPlan(projectId));
if (observability != null) {
observability.renderStarted(projectId, plan.decisions().size());
}
@ -114,6 +114,8 @@ public class FfmpegEditRenderer implements EditRenderer {
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
}
Path audioDirectory = store.projectDirectory(projectId).resolve("audio");
requireRequestedAssets(plan, audioDirectory);
projectService.updateProject(projectId, EditProjectStatus.RENDERING, Path.of(project.inputDirectory()), null);
Path work = store.projectDirectory(projectId).resolve("render-work");
createDirectory(work);
@ -138,7 +140,6 @@ public class FfmpegEditRenderer implements EditRenderer {
runAndRecord(overlayCommand(timeline, plan.overlays(), videoTimeline), commands);
}
Path output = store.projectDirectory(projectId).resolve("final.mp4");
Path audioDirectory = store.projectDirectory(projectId).resolve("audio");
Path music = audioDirectory.resolve("music.wav");
Path voiceover = audioDirectory.resolve("voiceover.wav");
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
@ -146,9 +147,13 @@ public class FfmpegEditRenderer implements EditRenderer {
.filter(cue -> "sfx".equals(cue.type()))
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
.toList();
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !soundEffects.isEmpty()) {
runAndRecord(audioMixCommand(videoTimeline, Files.isRegularFile(music) ? music : null,
Files.isRegularFile(voiceover) ? voiceover : null, soundEffects, output), commands);
AudioCue musicCue = plan.audioCues().stream()
.filter(cue -> "music".equals(cue.type()))
.findFirst()
.orElse(null);
if (musicCue != null || !plan.voiceover().isEmpty() || !soundEffects.isEmpty()) {
runAndRecord(audioMixCommand(videoTimeline, musicCue == null ? null : music, musicCue,
plan.voiceover().isEmpty() ? null : voiceover, soundEffects, output), commands);
} else {
copy(videoTimeline, output);
}
@ -167,6 +172,23 @@ public class FfmpegEditRenderer implements EditRenderer {
}
}
private EditPlan readEditPlan(String projectId) {
Path editPlan = store.editPlanFile(projectId);
if (Files.isRegularFile(editPlan)) {
return store.readJson(projectId, "edit-plan.json", EditPlan.class);
}
Path inboxPlan = store.inboxDirectory(projectId).resolve("edit-plan.json");
if (Files.isRegularFile(inboxPlan)) {
try {
return new com.fasterxml.jackson.databind.ObjectMapper().findAndRegisterModules()
.readValue(inboxPlan.toFile(), EditPlan.class);
} catch (IOException ex) {
throw new IllegalStateException("Unable to read edit plan from inbox: " + inboxPlan, ex);
}
}
throw new IllegalStateException("Unable to locate edit plan for project: " + projectId);
}
List<String> segmentCommand(String source, EditDecision decision, Path output) {
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
StringBuilder filter = new StringBuilder();
@ -272,10 +294,15 @@ public class FfmpegEditRenderer implements EditRenderer {
}
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, Path output) {
return audioMixCommand(timeline, music, voiceover, List.of(), output);
return audioMixCommand(timeline, music, null, voiceover, List.of(), output);
}
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
return audioMixCommand(timeline, music, null, voiceover, soundEffects, output);
}
List<String> audioMixCommand(Path timeline, Path music, AudioCue musicCue, Path voiceover,
List<SfxInput> soundEffects, Path output) {
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
"-i", timeline.toString()));
List<String> labels = new ArrayList<>(List.of("[0:a]"));
@ -285,7 +312,16 @@ public class FfmpegEditRenderer implements EditRenderer {
boolean hasVoiceover = voiceover != null;
if (music != null) {
command.addAll(List.of("-i", music.toString()));
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
filters.append("[").append(input).append(":a]");
if (musicCue != null) {
double duration = musicCue.timelineEndSeconds() - musicCue.timelineStartSeconds();
long delayMillis = Math.round(musicCue.timelineStartSeconds() * 1000);
filters.append("atrim=duration=").append(duration)
.append(",asetpts=PTS-STARTPTS,volume=").append(musicCue.gainDb()).append("dB,adelay=")
.append(delayMillis).append("|").append(delayMillis).append("[music_raw];");
} else {
filters.append("volume=-12dB[music_raw];");
}
input++;
}
if (voiceover != null) {
@ -338,12 +374,12 @@ public class FfmpegEditRenderer implements EditRenderer {
Path music = audioDirectory.resolve("music.wav");
if (Files.isRegularFile(music)) {
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
"project-local", "project-audio"));
assetLicense(music), "project-audio"));
}
Path voiceover = audioDirectory.resolve("voiceover.wav");
if (Files.isRegularFile(voiceover)) {
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
"project-local", "project-audio"));
assetLicense(voiceover), "project-audio"));
}
for (AudioCue cue : plan.audioCues()) {
if (assetProvider != null) {
@ -355,13 +391,38 @@ public class FfmpegEditRenderer implements EditRenderer {
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
if (Files.isRegularFile(sfx)) {
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
"project-local", "project-audio"));
assetLicense(sfx), "project-audio"));
}
}
}
return List.copyOf(assets);
}
void requireRequestedAssets(EditPlan plan, Path audioDirectory) {
List<String> missing = new ArrayList<>();
if (plan.audioCues().stream().anyMatch(cue -> "music".equals(cue.type()))
&& !AssetLicensePolicy.isLicensed(audioDirectory.resolve("music.wav"))) {
missing.add("music");
}
if (!plan.voiceover().isEmpty() && !AssetLicensePolicy.isLicensed(audioDirectory.resolve("voiceover.wav"))) {
missing.add("voiceover");
}
for (AudioCue cue : plan.audioCues()) {
if ("sfx".equals(cue.type())
&& !AssetLicensePolicy.isLicensed(
audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"))) {
missing.add("sfx:" + cue.assetKey());
}
}
if (!missing.isEmpty()) {
throw new IllegalStateException("Requested render assets are missing: " + missing);
}
}
private String assetLicense(Path asset) {
return AssetLicensePolicy.read(asset).orElse(AssetLicensePolicy.UNTRACKED);
}
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
List<Path> renderedClips,
List<List<String>> commands, List<ResolvedEditAsset> assets) {
@ -411,6 +472,10 @@ public class FfmpegEditRenderer implements EditRenderer {
private RenderQaCheck requiredAssetsResolvedCheck(EditPlan plan, List<ResolvedEditAsset> assets) {
List<String> missing = new ArrayList<>();
for (AudioCue cue : plan.audioCues()) {
if ("music".equals(cue.type()) && assets.stream()
.noneMatch(asset -> asset.type() == EditAssetType.MUSIC)) {
missing.add("music:" + cue.assetKey());
}
if ("sfx".equals(cue.type()) && assets.stream()
.noneMatch(asset -> asset.type() == EditAssetType.SFX && cue.assetKey().equals(asset.assetKey()))) {
missing.add("sfx:" + cue.assetKey());

View File

@ -0,0 +1,228 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightAssetPreparationService {
private static final Logger log = LoggerFactory.getLogger(HighlightAssetPreparationService.class);
private final HighlightProjectStore store;
private final EditAssetProvider assetProvider;
private final EditAssetLibrary assetLibrary;
private final VideoClippingProperties.Editing.Assets assets;
private final double musicGenMaxSeconds;
private final ObjectMapper objectMapper;
public HighlightAssetPreparationService(VideoClippingProperties properties, HighlightProjectStore store,
EditAssetProvider assetProvider, EditAssetLibrary assetLibrary,
ObjectMapper objectMapper) {
this.store = store;
this.assetProvider = assetProvider;
this.assetLibrary = assetLibrary;
this.assets = properties.getEditing().getAssets();
this.musicGenMaxSeconds = properties.getEditing().getMusicGenMaxSeconds();
this.objectMapper = objectMapper;
}
public HighlightAssetPreparationResult prepare(String projectId, HighlightProject project,
HighlightDirectorPlan.HighlightItem highlight,
ContentCategory category) {
log.info("event=highlight_asset_preparation_started project_id={} highlight_id={} category={} target_duration_seconds={}",
projectId, highlight.highlightId(), category, highlight.targetDurationSeconds());
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
Path assetsDirectory = highlightDirectory.resolve("assets");
Path musicDirectory = assetsDirectory.resolve("music");
Path sfxDirectory = assetsDirectory.resolve("sfx");
Path voiceoverDirectory = assetsDirectory.resolve("voiceover");
Path requestDirectory = assetsDirectory.resolve("requests");
createDirectories(musicDirectory, sfxDirectory, voiceoverDirectory, requestDirectory);
List<String> requestFiles = new ArrayList<>();
List<String> resolvedAssets = new ArrayList<>();
Optional<ResolvedEditAsset> music = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.MUSIC,
category, highlight.musicDirection(), highlight.targetDurationSeconds()));
if (music.isPresent()) {
Path target = musicDirectory.resolve("music.wav");
copy(Path.of(music.get().path()), target);
resolvedAssets.add(target.toString());
} else {
// Generate a BOUNDED music bed (MusicGen stalls on long scores); the renderer loops it to fill the
// reel. The reel length itself is unaffected.
double musicSeconds = Math.min(highlight.targetDurationSeconds(), musicGenMaxSeconds);
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "music",
"music_bed", highlight.musicDirection(), musicDirectory.resolve("music.wav"),
musicSeconds, false));
}
if (!highlight.voiceover().isEmpty()) {
Path script = voiceoverDirectory.resolve("voiceover-script.txt");
writeString(script, String.join(System.lineSeparator(), highlight.voiceover()));
List<VoiceoverLine> lines = plannedVoiceoverLines(highlight);
for (int index = 0; index < lines.size(); index++) {
VoiceoverLine line = lines.get(index);
String assetKey = "voiceover_%03d".formatted(index + 1);
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "voiceover",
assetKey, line.text(), voiceoverDirectory.resolve(assetKey + ".wav"),
line.timelineEndSeconds() - line.timelineStartSeconds(), true));
}
}
List<AudioCue> sfxCues = plannedSfxCues(highlight);
for (AudioCue cue : sfxCues) {
Optional<ResolvedEditAsset> resolved = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
cue.assetKey(), category, cue.timelineEndSeconds() - cue.timelineStartSeconds(), cue.notes()));
Path target = sfxDirectory.resolve(cue.assetKey() + ".wav");
if (resolved.isPresent()) {
copy(Path.of(resolved.get().path()), target);
resolvedAssets.add(target.toString());
} else {
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "sfx",
cue.assetKey(), cue.notes(), target, cue.timelineEndSeconds() - cue.timelineStartSeconds(),
true));
}
}
log.info("event=highlight_asset_preparation_completed project_id={} highlight_id={} resolved={} requests={}",
projectId, highlight.highlightId(), resolvedAssets.size(), requestFiles.size());
return new HighlightAssetPreparationResult(highlightDirectory, assetsDirectory, requestFiles, resolvedAssets);
}
static List<AudioCue> plannedSfxCues(HighlightDirectorPlan.HighlightItem highlight) {
if (highlight.sfxDirection() == null || highlight.sfxDirection().isBlank()) {
return List.of();
}
double duration = Math.max(1.0, highlight.targetDurationSeconds());
List<AudioCue> cues = new ArrayList<>();
if (highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
cues.add(new AudioCue("sfx", "whoosh_soft", 0.0, Math.min(0.8, duration), -6.0,
highlight.sfxDirection()));
}
cues.add(new AudioCue("sfx", "impact_hit", Math.max(0.0, duration - 0.8), duration, -3.0,
highlight.sfxDirection()));
return cues;
}
static List<VoiceoverLine> plannedVoiceoverLines(HighlightDirectorPlan.HighlightItem highlight) {
if (highlight.voiceover() == null || highlight.voiceover().isEmpty()) {
return List.of();
}
double targetDuration = Math.max(1.0, highlight.targetDurationSeconds());
List<Double> durations = highlight.voiceover().stream()
.map(HighlightAssetPreparationService::estimatedVoiceoverDuration)
.toList();
double speechDuration = durations.stream().mapToDouble(Double::doubleValue).sum();
if (speechDuration > targetDuration + 0.001) {
throw new IllegalArgumentException("Voiceover script exceeds the highlight timing budget");
}
double gap = (targetDuration - speechDuration) / (durations.size() + 1);
double cursor = gap;
List<VoiceoverLine> lines = new ArrayList<>();
for (int index = 0; index < durations.size(); index++) {
double duration = durations.get(index);
double start = roundSeconds(cursor);
double end = roundSeconds(Math.min(targetDuration, cursor + duration));
lines.add(new VoiceoverLine(highlight.voiceover().get(index), start, end, "cinematic_narration"));
cursor += duration + gap;
}
return List.copyOf(lines);
}
static double estimatedVoiceoverDuration(String text) {
long words = text == null || text.isBlank() ? 0 : text.trim().split("\\s+").length;
return Math.max(1.2, words / 2.5 + 0.4);
}
private static double roundSeconds(double value) {
return Math.round(value * 1000.0) / 1000.0;
}
private String writeRequest(Path requestDirectory, String projectId, String highlightId, String type,
String assetKey, String notes, Path target, double durationSeconds, boolean blocking) {
Path requestFile = requestDirectory.resolve(type + "-" + assetKey + ".md");
Path requestJson = requestDirectory.resolve(type + "-" + assetKey + ".json");
String content = """
# Highlight Asset Request
Project: `%s`
Highlight: `%s`
Type: `%s`
Asset key: `%s`
Project target: `%s`
Blocking for render: `%s`
Duration seconds: %.3f
Notes:
%s
""".formatted(projectId, highlightId, type, assetKey, target, blocking, durationSeconds,
notes == null ? "" : notes);
writeString(requestFile, content);
writeRequestJson(requestJson, new HighlightAssetRequest(projectId, highlightId, type, assetKey,
target.toString(), requestFile.toString(), notes, durationSeconds, blocking));
return requestFile.toString();
}
private void writeRequestJson(Path path, HighlightAssetRequest request) {
try {
Files.createDirectories(path.getParent());
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), request);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight asset request JSON: " + path, ex);
}
}
private void writeString(Path path, String value) {
try {
Files.createDirectories(path.getParent());
Files.writeString(path, value, StandardCharsets.UTF_8);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight asset file: " + path, ex);
}
}
private void copy(Path source, Path target) {
try {
AssetLicensePolicy.copy(source, target);
} catch (IOException ex) {
throw new IllegalStateException("Unable to copy highlight asset: " + source, ex);
}
}
private void createDirectories(Path... directories) {
for (Path directory : directories) {
try {
Files.createDirectories(directory);
} catch (IOException ex) {
throw new IllegalStateException("Unable to create highlight asset directory: " + directory, ex);
}
}
}
public record HighlightAssetPreparationResult(
Path highlightDirectory,
Path assetsDirectory,
List<String> requestFiles,
List<String> resolvedAssets
) {
public HighlightAssetPreparationResult {
requestFiles = requestFiles == null ? List.of() : List.copyOf(requestFiles);
resolvedAssets = resolvedAssets == null ? List.of() : List.copyOf(resolvedAssets);
}
}
}

View File

@ -0,0 +1,179 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
/**
* R13 beat-synced cuts. After the score is generated the montage cut boundaries are snapped onto the
* music's beat grid so the edit feels cut *to the music* (what consumer AI editors CapCut, DJI LightCut
* do). This is a small, deterministic re-time: it never reorders shots, never changes the total timeline
* length (so the score still fits), and only moves an internal boundary when a beat is within tolerance AND
* both adjacent shots stay above the minimum length. Fails soft no beats, an unreadable track, or a
* missing interpreter all leave the original cuts untouched, so beat-sync can never break a render.
*/
@Component
public class HighlightBeatSync {
private static final Logger log = LoggerFactory.getLogger(HighlightBeatSync.class);
/** Max distance (seconds) a boundary may move to land on a beat. Beyond this the original cut is kept. */
static final double DEFAULT_TOLERANCE_SECONDS = 0.18;
/** A shot must stay at least this long after snapping. */
static final double MIN_SHOT_SECONDS = 0.5;
private final VideoClippingProperties properties;
private final ObjectMapper objectMapper;
private final CommandRunner commandRunner;
@org.springframework.beans.factory.annotation.Autowired
public HighlightBeatSync(VideoClippingProperties properties, ObjectMapper objectMapper) {
this(properties, objectMapper, command -> {
Process process = new ProcessBuilder(command).redirectErrorStream(false).start();
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
process.waitFor();
return out;
});
}
HighlightBeatSync(VideoClippingProperties properties, ObjectMapper objectMapper, CommandRunner commandRunner) {
this.properties = properties;
this.objectMapper = objectMapper;
this.commandRunner = commandRunner;
}
/**
* Runs the offline beat-detection tool on an audio file and returns the beat timestamps (seconds).
* Returns an empty list on any failure so the caller keeps the original, un-synced cuts.
*/
List<Double> detectBeats(Path audio) {
if (audio == null || !Files.isRegularFile(audio)) {
return List.of();
}
String python = properties.getEditing().getLocalAssetWorker().getPythonBinary();
String script = properties.getEditing().getBeatDetectScript();
try {
String out = commandRunner.run(List.of(python, script, audio.toAbsolutePath().toString()));
String json = lastJsonObject(out);
if (json == null) {
return List.of();
}
BeatResult result = objectMapper.readValue(json, BeatResult.class);
if (result.beats() == null) {
return List.of();
}
List<Double> beats = new ArrayList<>(result.beats());
beats.removeIf(b -> b == null || !Double.isFinite(b));
beats.sort(Double::compareTo);
return List.copyOf(beats);
} catch (Exception ex) { // fail soft: never break the render on a beat-detection problem
log.warn("event=beat_detect_failed audio={} error={}", audio, ex.toString());
return List.of();
}
}
/** Convenience overload using the default tolerance / minimum-shot length. */
List<EditDecision> align(List<EditDecision> decisions, List<Double> beats, double sourceDuration) {
return align(decisions, beats, sourceDuration, DEFAULT_TOLERANCE_SECONDS, MIN_SHOT_SECONDS);
}
/**
* Snaps the internal timeline boundaries of an ordered decision list onto the nearest beats. The first
* start (0) and the final end (total) are anchors and never move, so the timeline length is preserved.
* A boundary only moves when the nearest beat is within {@code tolerance} and both adjacent shots stay
* >= {@code minShot}. Each shot's source window is re-derived from its new timeline duration.
*/
List<EditDecision> align(List<EditDecision> decisions, List<Double> beats, double sourceDuration,
double tolerance, double minShot) {
int n = decisions == null ? 0 : decisions.size();
if (n < 2 || beats == null || beats.isEmpty()) {
return decisions;
}
// boundaries[0..n]: original cut points on the timeline.
double[] boundaries = new double[n + 1];
boundaries[0] = decisions.get(0).timelineStartSeconds();
for (int i = 0; i < n; i++) {
boundaries[i + 1] = decisions.get(i).timelineEndSeconds();
}
double[] snapped = boundaries.clone();
int moved = 0;
for (int i = 1; i < n; i++) { // internal boundaries only; endpoints are anchors
double original = boundaries[i];
double beat = nearest(beats, original);
if (Math.abs(beat - original) <= tolerance
&& beat - snapped[i - 1] >= minShot
&& boundaries[i + 1] - beat >= minShot) {
snapped[i] = beat;
moved++;
}
}
if (moved == 0) {
return decisions;
}
List<EditDecision> aligned = new ArrayList<>(n);
for (int i = 0; i < n; i++) {
EditDecision d = decisions.get(i);
double newStart = snapped[i];
double newEnd = snapped[i + 1];
double newDur = newEnd - newStart;
double speed = d.playbackSpeed() <= 0 ? 1.0 : d.playbackSpeed();
double span = newDur * speed;
double srcStart = d.sourceStartSeconds();
if (srcStart + span > sourceDuration) {
srcStart = Math.max(0.0, sourceDuration - span); // keep the intended duration; slide the window
}
double srcEnd = Math.min(sourceDuration, srcStart + span);
aligned.add(new EditDecision(d.clipId(), round(srcStart), round(srcEnd),
round(newStart), round(newEnd), d.transitionIn(), d.transitionOut(),
d.playbackSpeed(), d.visualTreatment(), d.reason()));
}
log.info("event=beat_sync_applied shots={} boundaries_moved={} beats={}", n, moved, beats.size());
return aligned;
}
private static double nearest(List<Double> beats, double t) {
double best = beats.get(0);
double bestDist = Math.abs(best - t);
for (double b : beats) {
double dist = Math.abs(b - t);
if (dist < bestDist) {
bestDist = dist;
best = b;
}
}
return best;
}
private static double round(double v) {
return Math.round(v * 1000.0) / 1000.0;
}
/** Extracts the last {...} block from tool output so stray stderr/log lines don't break parsing. */
private static String lastJsonObject(String out) {
if (out == null) {
return null;
}
int end = out.lastIndexOf('}');
int start = out.lastIndexOf('{', end);
if (start < 0 || end < 0 || end < start) {
return null;
}
return out.substring(start, end + 1);
}
@FunctionalInterface
interface CommandRunner {
String run(List<String> command) throws Exception;
}
private record BeatResult(double tempo, List<Double> beats, double duration) {
}
}

View File

@ -0,0 +1,345 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.time.Clock;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.LinkedHashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Locale;
import java.util.Set;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightCandidateGenerator {
private static final Logger log = LoggerFactory.getLogger(HighlightCandidateGenerator.class);
private static final double DEFAULT_WINDOW_SECONDS = 12.0;
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final HighlightProjectStore store;
private final Clock clock;
@Autowired
public HighlightCandidateGenerator(VideoClippingProperties properties, HighlightProjectStore store) {
this(properties, store, Clock.systemUTC());
}
HighlightCandidateGenerator(
VideoClippingProperties properties,
HighlightProjectStore store,
Clock clock
) {
this.properties = properties.getEditing().getHighlightScheduler();
this.store = store;
this.clock = clock;
}
public CinematicHighlightAnalysis generate(String projectId, HighlightSourceAnalysis analysis) {
if (!projectId.equals(analysis.projectId())) {
throw new IllegalArgumentException("Highlight source analysis belongs to a different project");
}
if (analysis.source() == null || analysis.source().durationSeconds() <= 0) {
throw new IllegalArgumentException("Highlight source duration must be greater than zero");
}
long startedAt = System.nanoTime();
log.info("event=highlight_candidates_started project_id={} duration_seconds={} shot_segments={}",
projectId, analysis.source().durationSeconds(), safeShots(analysis).size());
CategoryDecision category = classify(analysis.visualAnalysis());
Map<RangeKey, ScoredWindow> uniqueRanges = new LinkedHashMap<>();
windows(analysis).stream().map(window -> score(window, analysis)).forEach(scored ->
uniqueRanges.merge(new RangeKey(scored.window().startSeconds(), scored.window().endSeconds()),
scored, (left, right) -> right.score() > left.score() ? right : left));
List<ScoredWindow> ranked = uniqueRanges.values().stream()
.sorted(Comparator.comparingDouble(ScoredWindow::score).reversed()
.thenComparingDouble(scored -> scored.window().startSeconds())
.thenComparingDouble(scored -> scored.window().endSeconds()))
.limit(properties.getMaxHighlightsPerSource())
.toList();
List<HighlightCandidate> candidates = new ArrayList<>();
for (int index = 0; index < ranked.size(); index++) {
ScoredWindow scored = ranked.get(index);
Window window = scored.window();
HighlightCandidate candidate = new HighlightCandidate(
"candidate_%03d".formatted(index + 1),
analysis.source().clipId(),
round(window.startSeconds()),
round(window.endSeconds()),
round(scored.score()),
role(window, analysis.source().durationSeconds()),
scored.reasons()
);
candidates.add(candidate);
log.info("event=highlight_candidate_scored project_id={} candidate_id={} start={} end={} score={} "
+ "reasons={}",
projectId, candidate.id(), candidate.sourceStartSeconds(), candidate.sourceEndSeconds(),
candidate.score(), candidate.reasons());
}
CinematicHighlightAnalysis result = new CinematicHighlightAnalysis(
projectId,
category.category(),
category.confidence(),
category.reasons(),
List.copyOf(candidates),
Instant.now(clock)
);
store.writeJson(projectId, "analysis/category.json", result);
store.writeJson(projectId, "analysis/highlight-candidates.json", candidates.toArray(HighlightCandidate[]::new));
log.info("event=highlight_candidates_completed project_id={} count={} category={} "
+ "category_confidence={} elapsed_ms={}",
projectId, candidates.size(), result.category(), result.categoryConfidence(),
elapsedMillis(startedAt));
return result;
}
private CategoryDecision classify(SourceVisualAnalysis visual) {
if (visual == null || !isIndependentVisualEvidence(visual)) {
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.25, List.of(
"Category-specific visual evidence is unavailable or came from metadata fallback.",
"Review frames and contact sheets before assigning category-specific creative direction."
));
}
List<VisualObjectLabel> labels = visual.objectLabels() == null ? List.of() : visual.objectLabels();
CategoryDecision best = null;
for (VisualObjectLabel label : labels) {
if (label == null || label.label() == null || label.confidence() < 0.55
|| "metadata_heuristic".equalsIgnoreCase(label.source())) {
continue;
}
ContentCategory category = categoryFor(label.label());
if (category == ContentCategory.GENERIC_VLOG) {
continue;
}
double confidence = round(Math.min(0.95, label.confidence()));
if (best == null || confidence > best.confidence()) {
best = new CategoryDecision(category, confidence, List.of(
"Local visual model label '%s' supports %s at confidence %.3f."
.formatted(label.label(), category.name().toLowerCase(Locale.ROOT), confidence),
"Confirm the category against representative frames before approving the edit plan."
));
}
}
if (best != null) {
return best;
}
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.35, List.of(
"Local visual analysis completed but produced no category-discriminating label above 0.55.",
"Use generic direction until a reviewer confirms the visible subject."
));
}
private boolean isIndependentVisualEvidence(SourceVisualAnalysis visual) {
String method = visual.analysisMethod() == null ? "" : visual.analysisMethod().toLowerCase(Locale.ROOT);
return !method.isBlank() && !method.contains("fallback") && !method.contains("metadata");
}
private ContentCategory categoryFor(String rawLabel) {
String label = rawLabel.toLowerCase(Locale.ROOT).replace('_', ' ').trim();
if (containsAny(label, "car", "vehicle", "automobile", "truck", "motorcycle", "porsche")) {
return ContentCategory.CAR_VLOG;
}
if (containsAny(label, "food", "meal", "dish", "pizza", "sandwich", "cake", "kitchen")) {
return ContentCategory.FOOD_VLOG;
}
if (containsAny(label, "family", "child", "children", "baby", "birthday party")) {
return ContentCategory.FAMILY_VLOG;
}
return ContentCategory.GENERIC_VLOG;
}
private boolean containsAny(String value, String... candidates) {
for (String candidate : candidates) {
if (value.contains(candidate)) {
return true;
}
}
return false;
}
private List<Window> windows(HighlightSourceAnalysis analysis) {
double duration = analysis.source().durationSeconds();
List<ShotSegment> shots = safeShots(analysis).stream()
.filter(shot -> validShot(shot, duration))
.sorted(Comparator.comparingDouble(ShotSegment::startSeconds))
.toList();
if (shots.size() <= 1) {
String shotId = shots.isEmpty() ? "source_coverage" : shots.get(0).shotId();
return splitRange(0, duration, shotId, shots.isEmpty() ? 0 : shots.get(0).sceneScore(), true);
}
Set<Window> windows = new LinkedHashSet<>();
for (ShotSegment shot : shots) {
if (shot.durationSeconds() > properties.getHighlightMaxDurationSeconds()) {
windows.addAll(splitRange(shot.startSeconds(), shot.endSeconds(), shot.shotId(),
shot.sceneScore(), false));
} else {
windows.add(expandToMinimum(shot, duration));
}
}
return List.copyOf(windows);
}
private List<ShotSegment> safeShots(HighlightSourceAnalysis analysis) {
return analysis.shotSegments() == null ? List.of() : analysis.shotSegments();
}
private boolean validShot(ShotSegment shot, double sourceDuration) {
return shot != null && shot.startSeconds() >= 0 && shot.endSeconds() > shot.startSeconds()
&& shot.endSeconds() <= sourceDuration + 0.001;
}
private Window expandToMinimum(ShotSegment shot, double sourceDuration) {
double minimum = Math.min(properties.getHighlightMinDurationSeconds(), sourceDuration);
double start = shot.startSeconds();
double end = shot.endSeconds();
if (end - start < minimum) {
double center = clamp(shot.representativeTimestampSeconds(), start, end);
start = clamp(center - minimum / 2.0, 0, Math.max(0, sourceDuration - minimum));
end = Math.min(sourceDuration, start + minimum);
}
return new Window(round(start), round(end), shot.shotId(), shot.sceneScore(), false);
}
private List<Window> splitRange(
double rangeStart,
double rangeEnd,
String shotId,
double sceneScore,
boolean coverageFallback
) {
double rangeDuration = rangeEnd - rangeStart;
double minimum = Math.min(properties.getHighlightMinDurationSeconds(), rangeDuration);
double target = Math.min(properties.getHighlightMaxDurationSeconds(),
Math.max(minimum, Math.min(DEFAULT_WINDOW_SECONDS, rangeDuration)));
if (rangeDuration <= target + 0.001) {
return List.of(new Window(round(rangeStart), round(rangeEnd), shotId, sceneScore, coverageFallback));
}
double step = Math.max(1.0, target * 0.75);
Set<Window> windows = new LinkedHashSet<>();
for (double start = rangeStart; start + target <= rangeEnd + 0.001; start += step) {
double boundedStart = Math.min(start, rangeEnd - target);
windows.add(new Window(round(boundedStart), round(boundedStart + target), shotId, sceneScore,
coverageFallback));
}
double finalStart = rangeEnd - target;
windows.add(new Window(round(finalStart), round(rangeEnd), shotId, sceneScore, coverageFallback));
return List.copyOf(windows);
}
private ScoredWindow score(Window window, HighlightSourceAnalysis analysis) {
SourceVisualAnalysis visual = analysis.visualAnalysis();
double visualQuality = visual == null ? 0
: clamp(visual.blurScore()) * 0.30
+ clamp(visual.exposureScore()) * 0.20
+ clamp(visual.motionScore()) * 0.25
+ clamp(visual.compositionScore()) * 0.25;
double nonSilence = nonSilenceFraction(window, analysis.audioAnalysis());
double durationFit = durationFit(window.durationSeconds());
double sceneInterest = clamp(window.sceneScore());
double raw = 0.10 + visualQuality * 0.35 + nonSilence * 0.25 + sceneInterest * 0.20
+ durationFit * 0.10;
boolean weakVisualEvidence = visual == null || !isIndependentVisualEvidence(visual);
double score = clamp(raw * (weakVisualEvidence ? 0.72 : 1.0));
List<String> reasons = new ArrayList<>();
reasons.add(window.coverageFallback()
? "Coverage window created because analysis produced fewer than two valid shot boundaries."
: "Window is aligned to analyzed shot " + window.shotId() + ".");
reasons.add("Source-level visual quality score is %.3f; it is ranking context, not temporal proof."
.formatted(round(visualQuality)));
reasons.add("Audio is %.1f%% non-silent in this window according to FFmpeg silence detection."
.formatted(nonSilence * 100.0));
if (weakVisualEvidence) {
reasons.add("Visual evidence is heuristic or fallback-derived; inspect frames before selection.");
}
return new ScoredWindow(window, score, List.copyOf(reasons));
}
private double nonSilenceFraction(Window window, SourceAudioAnalysis audio) {
if (audio == null || !audio.audioPresent() || audio.sections() == null || audio.sections().isEmpty()) {
return 0;
}
double usable = 0;
for (AudioSection section : audio.sections()) {
if (section == null || "silence".equalsIgnoreCase(section.kind())
|| "missing_audio".equalsIgnoreCase(section.kind())) {
continue;
}
usable += overlap(window.startSeconds(), window.endSeconds(), section.startSeconds(), section.endSeconds());
}
return clamp(usable / window.durationSeconds());
}
private double overlap(double startA, double endA, double startB, double endB) {
return Math.max(0, Math.min(endA, endB) - Math.max(startA, startB));
}
private double durationFit(double duration) {
double target = Math.min(properties.getHighlightMaxDurationSeconds(),
Math.max(properties.getHighlightMinDurationSeconds(), DEFAULT_WINDOW_SECONDS));
return clamp(1.0 - Math.abs(duration - target) / Math.max(target, 1.0));
}
private String role(Window window, double sourceDuration) {
double midpoint = (window.startSeconds() + window.endSeconds()) / 2.0;
if (midpoint <= sourceDuration * 0.25) {
return "opening_hook_candidate";
}
if (midpoint >= sourceDuration * 0.75) {
return "final_payoff_candidate";
}
return "story_build_candidate";
}
private double clamp(double value) {
return Math.max(0, Math.min(1, value));
}
private double clamp(double value, double minimum, double maximum) {
return Math.max(minimum, Math.min(maximum, value));
}
private double round(double value) {
return Math.round(value * 1000.0) / 1000.0;
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
private record Window(
double startSeconds,
double endSeconds,
String shotId,
double sceneScore,
boolean coverageFallback
) {
double durationSeconds() {
return endSeconds - startSeconds;
}
}
private record ScoredWindow(Window window, double score, List<String> reasons) {
}
private record RangeKey(double startSeconds, double endSeconds) {
}
private record CategoryDecision(ContentCategory category, double confidence, List<String> reasons) {
}
}

View File

@ -26,24 +26,35 @@ public class HighlightDirectorFlowService {
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final ObjectMapper objectMapper;
private final HighlightProjectStore store;
private final HighlightDirectorPlanValidator planValidator;
private final HighlightVisualEffectsStage visualEffectsStage;
private final HighlightAssetPreparationService assetPreparationService;
private final HighlightLocalAssetWorker assetWorker;
private final HighlightFfmpegRenderer renderer;
private final HighlightBeatSync beatSync;
private final HighlightSubjectTracker subjectTracker;
private final VideoClippingProperties.Editing editing;
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
HighlightProjectStore store,
HighlightDirectorPlanValidator planValidator,
HighlightVisualEffectsStage visualEffectsStage,
HighlightAssetPreparationService assetPreparationService,
HighlightLocalAssetWorker assetWorker,
HighlightFfmpegRenderer renderer) {
HighlightFfmpegRenderer renderer,
HighlightBeatSync beatSync,
HighlightSubjectTracker subjectTracker) {
this.properties = properties.getEditing().getHighlightScheduler();
this.editing = properties.getEditing();
this.objectMapper = objectMapper;
this.store = store;
this.planValidator = planValidator;
this.visualEffectsStage = visualEffectsStage;
this.assetPreparationService = assetPreparationService;
this.assetWorker = assetWorker;
this.renderer = renderer;
this.beatSync = beatSync;
this.subjectTracker = subjectTracker;
}
public HighlightFlowResult process(String projectId, long scanId) {
@ -60,18 +71,24 @@ public class HighlightDirectorFlowService {
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
}
HighlightDirectorPlan plan = readPlan(planFile);
if (plan.highlights().isEmpty()) {
markFailed(project, "No highlights were provided by the director plan");
throw new IllegalStateException("Director plan contains no highlights");
Path montageFile = store.directorDirectory(projectId).resolve("montage.json");
if (Files.isRegularFile(montageFile)) {
return processMontage(projectId, flowId, scanId, project, montageFile, startedAt);
}
HighlightDirectorPlan plan;
try {
plan = planValidator.validate(projectId, readPlan(planFile));
} catch (RuntimeException ex) {
markFailed(project, "Director plan rejected: " + ex.getMessage());
throw ex;
}
markStatus(project, HighlightProjectStatus.RENDERING, null);
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
markStatus(project, HighlightProjectStatus.PLANNED, null);
List<Path> finalOutputs = new ArrayList<>();
List<String> renderedHighlightIds = new ArrayList<>();
ContentCategory category = contentCategory(plan.contentCategory());
int limit = Math.min(properties.getMaxHighlightsPerSource(), plan.highlights().size());
int limit = plan.highlights().size();
for (int index = 0; index < limit; index++) {
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
@ -89,7 +106,20 @@ public class HighlightDirectorFlowService {
log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}",
flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(),
assetWorkerResult.pendingRequests().size());
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
if (!assetWorkerResult.pendingRequests().isEmpty()) {
log.warn("event=highlight_flow_waiting_for_assets flow_id={} project_id={} highlight_id={} pending_requests={}",
flowId, projectId, highlight.highlightId(), assetWorkerResult.pendingRequests());
return HighlightFlowResult.skipped(projectId, flowId, "assets_pending");
}
markStatus(project, HighlightProjectStatus.RENDERING, null);
HighlightFfmpegRenderer.HighlightRenderResult result;
try {
result = renderer.render(projectId, highlight, editPlan);
requireQaPassed(result);
} catch (RuntimeException ex) {
markFailed(project, "Highlight render rejected: " + ex.getMessage());
throw ex;
}
finalOutputs.add(result.finalOutput());
renderedHighlightIds.add(result.highlightId());
}
@ -105,6 +135,144 @@ public class HighlightDirectorFlowService {
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
}
private void requireQaPassed(HighlightFfmpegRenderer.HighlightRenderResult result) {
if (result.qaReport() == null) {
throw new IllegalStateException("Highlight render did not produce a QA report");
}
List<String> blockingFailures = result.qaReport().checks().stream()
.filter(check -> !check.passed() && "ERROR".equals(check.severity()))
.map(RenderQaCheck::name)
.toList();
if (!result.qaReport().passed() || !blockingFailures.isEmpty()) {
throw new IllegalStateException("Highlight render failed blocking QA checks: " + blockingFailures);
}
}
private HighlightFlowResult processMontage(String projectId, String flowId, long scanId,
HighlightProject project, Path montageFile, long startedAt) {
MontagePlan montage;
try {
montage = objectMapper.readValue(montageFile.toFile(), MontagePlan.class);
} catch (IOException ex) {
markFailed(project, "Montage plan unreadable: " + ex.getMessage());
throw new IllegalStateException("Unable to read montage plan: " + montageFile, ex);
}
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
String clipId = analysis.source().clipId();
double sourceDuration = analysis.source().durationSeconds();
boolean reframe = editing.isSubjectReframeEnabled();
Path sourceVideo = store.sourceDirectory(projectId).resolve(project.sourceVideoFileName());
List<EditDecision> decisions = new ArrayList<>();
double timeline = 0.0;
List<MontagePlan.Shot> shots = montage.shots();
for (int i = 0; i < shots.size(); i++) {
MontagePlan.Shot shot = shots.get(i);
double speed = shot.speed() <= 0 ? 1.0 : shot.speed();
double dur = Math.max(0.2, shot.durationSeconds());
double srcSpan = dur * speed;
double srcStart = Math.max(0.0, Math.min(shot.sourceStartSeconds(), sourceDuration - srcSpan));
double srcEnd = Math.min(sourceDuration, srcStart + srcSpan);
String transitionIn = (i == 0) ? "fade-in" : "cut";
String transitionOut = (i == shots.size() - 1) ? "fade-out" : "cut";
String treatment = "zoom=%.3f cinematic".formatted(shot.zoom() <= 0 ? 1.2 : shot.zoom());
if (reframe) {
// R14: track the subject over this shot's source range; append the path as a "pan=" token so
// the renderer follows it. Fails soft an empty path leaves the token off (centred crop).
String panToken = HighlightSubjectTracker.panToken(
subjectTracker.track(sourceVideo, srcStart, srcEnd));
if (!panToken.isEmpty()) {
treatment = treatment + " " + panToken;
}
}
decisions.add(new EditDecision(clipId, srcStart, srcEnd, timeline, timeline + dur,
transitionIn, transitionOut, speed, treatment, "montage"));
timeline += dur;
}
double total = timeline;
List<AudioCue> audioCues = new ArrayList<>();
if (montage.musicDirection() != null && !montage.musicDirection().isBlank()) {
audioCues.add(new AudioCue("music", safeKey("music", montage.musicDirection()), 0.0, total, -9.0,
montage.musicDirection()));
}
List<TextOverlay> overlays = new ArrayList<>();
if (montage.overlays() != null) {
for (MontagePlan.Overlay o : montage.overlays()) {
overlays.add(new TextOverlay(o.text(), o.timelineStartSeconds(), o.timelineEndSeconds(),
o.placement() == null || o.placement().isBlank() ? "lower_center_safe" : o.placement(),
"fade", "montage"));
}
}
String gradeKeyword = montage.grade() == null || montage.grade().isBlank() ? "hero" : montage.grade();
List<String> voiceoverLines = montage.voiceover() == null ? List.of() : montage.voiceover();
HighlightDirectorPlan.HighlightItem montageHighlight = new HighlightDirectorPlan.HighlightItem(
"montage", "montage", "Cinematic Montage", 0.0, total, total,
gradeKeyword.contains("hero") ? "hero_payoff" : gradeKeyword,
"cinematic montage", montage.musicDirection(), "", voiceoverLines,
overlays.stream().map(TextOverlay::text).toList(), "montage");
List<VoiceoverLine> voiceover = HighlightAssetPreparationService.plannedVoiceoverLines(montageHighlight);
String style = safeKey("style", "montage", gradeKeyword);
EditPlan editPlan = new EditPlan(projectId, style, total, decisions, audioCues, voiceover, overlays,
"mp4-h264-aac-1080p", "cinematic montage");
store.writeJson(projectId, "highlights/montage/edit-plan.json", editPlan);
log.info("event=highlight_montage_started flow_id={} scan_id={} project_id={} shots={} duration={}",
flowId, scanId, projectId, decisions.size(), total);
markStatus(project, HighlightProjectStatus.PLANNED, null);
ContentCategory category;
try {
category = store.readJson(projectId, "analysis/category.json", CinematicHighlightAnalysis.class).category();
} catch (RuntimeException ex) {
category = ContentCategory.GENERIC_VLOG;
}
assetPreparationService.prepare(projectId, project, montageHighlight, category);
HighlightLocalAssetWorker.HighlightAssetWorkerResult assetResult =
assetWorker.process(projectId, montageHighlight, category);
if (!assetResult.pendingRequests().isEmpty()) {
markFailed(project, "Montage assets pending: " + assetResult.pendingRequests());
return HighlightFlowResult.skipped(projectId, flowId, "assets_pending");
}
// R13 beat-synced cuts: the score now exists, so snap the cut boundaries onto its beat grid before
// rendering. Fails soft align() returns the original decisions when no beat is close enough.
if (editing.isBeatSyncEnabled()) {
Path music = store.highlightsDirectory(projectId)
.resolve("montage").resolve("assets").resolve("music").resolve("music.wav");
List<EditDecision> synced = beatSync.align(decisions, beatSync.detectBeats(music), sourceDuration);
if (synced != decisions) {
decisions = synced;
editPlan = new EditPlan(projectId, style, total, decisions, audioCues, voiceover, overlays,
"mp4-h264-aac-1080p", "cinematic montage");
store.writeJson(projectId, "highlights/montage/edit-plan.json", editPlan);
log.info("event=highlight_montage_beat_synced flow_id={} project_id={} shots={}",
flowId, projectId, decisions.size());
}
}
markStatus(project, HighlightProjectStatus.RENDERING, null);
HighlightFfmpegRenderer.HighlightRenderResult result;
try {
result = renderer.render(projectId, montageHighlight, editPlan);
requireQaPassed(result);
} catch (RuntimeException ex) {
markFailed(project, "Montage render rejected: " + ex.getMessage());
throw ex;
}
Path projectOutput = store.projectDirectory(projectId).resolve("final.mp4");
concatFinalOutputs(projectOutput, List.of(result.finalOutput()));
RenderManifest manifest = new RenderManifest(projectId, List.of(result.highlightId()),
List.of(result.finalOutput().toString()), projectOutput.toString(), total, List.of(), List.of(),
Instant.now());
store.writeJson(projectId, "render-manifest.json", manifest);
markStatus(project, HighlightProjectStatus.RENDERED, null);
log.info("event=highlight_montage_completed flow_id={} project_id={} shots={} duration={} elapsed_ms={}",
flowId, projectId, decisions.size(), total, (System.nanoTime() - startedAt) / 1_000_000);
return HighlightFlowResult.rendered(projectId, flowId, List.of(result.finalOutput()), projectOutput);
}
private HighlightDirectorPlan readPlan(Path planFile) {
try {
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
@ -118,42 +286,22 @@ public class HighlightDirectorFlowService {
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
double requestedDuration = highlight.targetDurationSeconds() > 0 ? highlight.targetDurationSeconds()
: sourceDuration;
double targetDuration = Math.max(properties.getHighlightMinDurationSeconds(),
Math.min(properties.getHighlightMaxDurationSeconds(), requestedDuration));
double playbackSpeed = Math.max(0.25, Math.min(4.0, sourceDuration / targetDuration));
double targetDuration = highlight.targetDurationSeconds();
double playbackSpeed = sourceDuration / targetDuration;
String clipId = analysis.source().clipId();
EditDecision decision = new EditDecision(
clipId,
highlight.sourceStartSeconds(),
highlight.sourceEndSeconds(),
0.0,
targetDuration,
transitionIn(highlight, index),
transitionOut(highlight, index, plan.highlights().size()),
playbackSpeed,
highlight.visualTreatment(),
highlight.renderNotes()
);
List<EditDecision> decisions = buildCuts(highlight, index, plan.highlights().size(), clipId,
targetDuration, playbackSpeed);
List<AudioCue> audioCues = new ArrayList<>();
if (highlight.musicDirection() != null && !highlight.musicDirection().isBlank()) {
// Music sits as a bed under the narration (-14 dB); side-chain ducking drops it further during
// voiceover lines so the narration always leads.
audioCues.add(new AudioCue("music", safeKey("music", highlight.musicDirection()), 0.0, targetDuration,
-10.0, highlight.musicDirection()));
-14.0, highlight.musicDirection()));
}
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
audioCues.add(new AudioCue("sfx", safeKey("sfx", highlight.storyPurpose(), highlight.highlightId()),
Math.max(0.0, targetDuration - 0.8), targetDuration, -6.0, highlight.sfxDirection()));
}
List<VoiceoverLine> voiceover = new ArrayList<>();
if (!highlight.voiceover().isEmpty()) {
double step = targetDuration / highlight.voiceover().size();
for (int i = 0; i < highlight.voiceover().size(); i++) {
double start = Math.min(targetDuration, i * step);
double end = Math.min(targetDuration, start + Math.max(1.0, step));
voiceover.add(new VoiceoverLine(highlight.voiceover().get(i), start, end, "cinematic_narration"));
}
audioCues.addAll(HighlightAssetPreparationService.plannedSfxCues(highlight));
}
List<VoiceoverLine> voiceover = HighlightAssetPreparationService.plannedVoiceoverLines(highlight);
List<TextOverlay> overlays = new ArrayList<>();
if (!highlight.overlays().isEmpty()) {
double step = targetDuration / highlight.overlays().size();
@ -168,7 +316,7 @@ public class HighlightDirectorFlowService {
return new EditPlan(project.id(),
safeKey("style", plan.contentCategory(), highlight.storyPurpose()),
targetDuration,
List.of(decision),
decisions,
audioCues,
voiceover,
overlays,
@ -176,18 +324,41 @@ public class HighlightDirectorFlowService {
highlight.title() + " / " + highlight.storyPurpose());
}
private String transitionIn(HighlightDirectorPlan.HighlightItem highlight, int index) {
if (index == 0 || highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
return "fade-in";
// Split a beat into several contiguous cuts so the render is an edit (with a progressive punch-in per
// cut, applied in the renderer) rather than one long pan. Cuts hard-cut between one another; only the
// very first cut of the piece fades in and the very last fades out. Playback speed is preserved so a
// slow-motion hero beat stays slow across its cuts.
private List<EditDecision> buildCuts(HighlightDirectorPlan.HighlightItem highlight, int index, int total,
String clipId, double targetDuration, double playbackSpeed) {
double start = highlight.sourceStartSeconds();
double span = Math.max(0.01, highlight.sourceEndSeconds() - start);
int cuts = cutsForBeat(highlight.storyPurpose());
List<EditDecision> decisions = new ArrayList<>();
double cutTarget = targetDuration / cuts;
for (int c = 0; c < cuts; c++) {
double cutStart = start + span * c / cuts;
double cutEnd = start + span * (c + 1) / cuts;
// Sequential timeline positions so the concatenated beat spans 0..targetDuration (the QA
// duration/overlay checks read the last cut's timeline end as the total beat duration).
double timelineStart = cutTarget * c;
double timelineEnd = cutTarget * (c + 1);
String transitionIn = (index == 0 && c == 0) ? "fade-in" : "cut";
String transitionOut = (index == total - 1 && c == cuts - 1) ? "fade-out" : "cut";
decisions.add(new EditDecision(clipId, cutStart, cutEnd, timelineStart, timelineEnd,
transitionIn, transitionOut, playbackSpeed, highlight.visualTreatment(),
highlight.renderNotes()));
}
return "cut";
return decisions;
}
private String transitionOut(HighlightDirectorPlan.HighlightItem highlight, int index, int total) {
if (index + 1 == total || highlight.storyPurpose() != null && highlight.storyPurpose().contains("hero")) {
return "fade-out";
private int cutsForBeat(String storyPurpose) {
if (storyPurpose == null) {
return 2;
}
return "cut";
if (storyPurpose.contains("rising")) {
return 3; // rising energy: faster, more cuts
}
return 2; // opening / hero: establishing and held (hero cuts stay slow-motion)
}
private void writeStoryboard(Path highlightDirectory, HighlightDirectorPlan plan,

View File

@ -27,13 +27,16 @@ public class HighlightDirectorPlanScanner {
private final VideoClippingProperties properties;
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
private final HighlightProjectStore store;
private final HighlightDirectorFlowService flowService;
private final AtomicBoolean scanning = new AtomicBoolean(false);
private final AtomicLong scanSequence = new AtomicLong();
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightDirectorFlowService flowService) {
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightProjectStore store,
HighlightDirectorFlowService flowService) {
this.properties = properties;
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
this.store = store;
this.flowService = flowService;
}
@ -78,10 +81,36 @@ public class HighlightDirectorPlanScanner {
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
.filter(this::isRetryableProject)
.filter(this::hasRequiredApproval)
.map(path -> path.getFileName().toString())
.findFirst();
} catch (IOException ex) {
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
}
}
private boolean isRetryableProject(Path projectDirectory) {
String projectId = projectDirectory.getFileName().toString();
if (!Files.isRegularFile(projectDirectory.resolve("project.json"))) {
log.warn("event=highlight_render_project_skipped project_id={} reason=missing_project_metadata",
projectId);
return false;
}
try {
HighlightProjectStatus status = store.readJson(projectId, "project.json", HighlightProject.class).status();
return status == HighlightProjectStatus.WAITING_FOR_DIRECTOR
|| status == HighlightProjectStatus.PLANNED;
} catch (RuntimeException ex) {
log.warn("event=highlight_render_project_skipped project_id={} reason=invalid_project_metadata "
+ "error_type={}", projectId, ex.getClass().getSimpleName());
return false;
}
}
private boolean hasRequiredApproval(Path projectDirectory) {
return !highlightScheduler.isRequireDirectorApproval()
|| Files.isRegularFile(projectDirectory.resolve("director")
.resolve(highlightScheduler.getApprovalFileName()));
}
}

View File

@ -0,0 +1,200 @@
package org.example.videoclips.editing;
import org.example.videoclips.application.BadRequestException;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.function.Function;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightDirectorPlanValidator {
private static final Pattern SAFE_ID = Pattern.compile("[A-Za-z0-9][A-Za-z0-9._-]{0,127}");
private static final Set<String> STORY_PURPOSES = Set.of("opening_hook", "rising_energy", "hero_payoff");
private static final double EPSILON = 0.001;
private static final int MAX_TITLE_LENGTH = 120;
private static final int MAX_DIRECTION_LENGTH = 1_000;
private static final int MAX_VOICEOVER_LINE_LENGTH = 240;
private static final int MAX_OVERLAY_LENGTH = 80;
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final HighlightProjectStore store;
public HighlightDirectorPlanValidator(VideoClippingProperties properties, HighlightProjectStore store) {
this.properties = properties.getEditing().getHighlightScheduler();
this.store = store;
}
public HighlightDirectorPlan validate(String projectId, HighlightDirectorPlan plan) {
if (plan == null) {
reject("Highlight director plan is required");
}
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
if (!projectId.equals(plan.projectId())) {
reject("Director plan projectId must match the project");
}
if (!project.sourceVideoFileName().equals(plan.sourceVideoFileName())) {
reject("Director plan sourceVideoFileName must match the project source");
}
ContentCategory requestedCategory = parseCategory(plan.contentCategory());
Path categoryFile = store.analysisDirectory(projectId).resolve("category.json");
if (!Files.isRegularFile(categoryFile)) {
reject("Persisted category analysis is required before accepting a director plan");
}
CinematicHighlightAnalysis category = store.readJson(projectId, "analysis/category.json",
CinematicHighlightAnalysis.class);
if (requestedCategory != category.category()) {
reject("Director plan contentCategory must match the reviewed category analysis");
}
Path candidateFile = store.analysisDirectory(projectId).resolve("highlight-candidates.json");
if (!Files.isRegularFile(candidateFile)) {
reject("Persisted highlight candidates are required before accepting a director plan");
}
HighlightCandidate[] persisted = store.readJson(projectId, "analysis/highlight-candidates.json",
HighlightCandidate[].class);
Map<String, HighlightCandidate> candidates = Arrays.stream(persisted)
.collect(Collectors.toMap(HighlightCandidate::id, Function.identity(), (left, right) -> {
reject("Persisted highlight candidate IDs must be unique");
return left;
}));
if (plan.highlights() == null || plan.highlights().isEmpty()) {
reject("Director plan must contain at least one highlight");
}
if (plan.highlights().size() > properties.getMaxHighlightsPerSource()) {
reject("Director plan exceeds maxHighlightsPerSource=" + properties.getMaxHighlightsPerSource());
}
Set<String> highlightIds = new HashSet<>();
Set<String> candidateIds = new HashSet<>();
for (HighlightDirectorPlan.HighlightItem highlight : plan.highlights()) {
validateHighlight(highlight, analysis, candidates, highlightIds, candidateIds);
}
if (plan.summary() == null || plan.summary().isBlank() || plan.summary().length() > MAX_DIRECTION_LENGTH) {
reject("Director plan summary is required and must be 1000 characters or fewer");
}
return plan;
}
private void validateHighlight(
HighlightDirectorPlan.HighlightItem highlight,
HighlightSourceAnalysis analysis,
Map<String, HighlightCandidate> candidates,
Set<String> highlightIds,
Set<String> candidateIds
) {
if (highlight == null) {
reject("Director plan highlight item cannot be null");
}
if (!safeId(highlight.highlightId()) || !highlightIds.add(highlight.highlightId())) {
reject("Highlight IDs must be unique safe identifiers");
}
if (!safeId(highlight.candidateId()) || !candidateIds.add(highlight.candidateId())) {
reject("Candidate IDs must be unique safe identifiers in the director plan");
}
HighlightCandidate candidate = candidates.get(highlight.candidateId());
if (candidate == null) {
reject("Unknown highlight candidate: " + highlight.candidateId());
}
if (!analysis.source().clipId().equals(candidate.clipId())) {
reject("Highlight candidate does not belong to the analyzed source clip: " + candidate.id());
}
if (!finite(highlight.sourceStartSeconds()) || !finite(highlight.sourceEndSeconds())
|| highlight.sourceStartSeconds() < candidate.sourceStartSeconds() - EPSILON
|| highlight.sourceEndSeconds() > candidate.sourceEndSeconds() + EPSILON
|| highlight.sourceEndSeconds() <= highlight.sourceStartSeconds()
|| highlight.sourceEndSeconds() > analysis.source().durationSeconds() + EPSILON) {
reject("Highlight source range must be contained by candidate: " + candidate.id());
}
// A highlight may be as long as the story needs there is no fixed min/max duration. The only
// hard requirements are that the target duration is a positive, finite number (checked here), that
// the source range is contained by the candidate and the source (checked above), and that the
// resulting playback speed stays in a sane range (checked below).
if (!finite(highlight.targetDurationSeconds()) || highlight.targetDurationSeconds() <= EPSILON) {
reject("Highlight target duration must be a positive number of seconds");
}
double requiredPlaybackSpeed = (highlight.sourceEndSeconds() - highlight.sourceStartSeconds())
/ highlight.targetDurationSeconds();
if (requiredPlaybackSpeed < 0.25 - EPSILON || requiredPlaybackSpeed > 4.0 + EPSILON) {
reject("Highlight source range and target duration require playback speed outside 0.25..4.0");
}
requiredText(highlight.title(), MAX_TITLE_LENGTH, "Highlight title");
if (!STORY_PURPOSES.contains(highlight.storyPurpose())) {
reject("Unsupported storyPurpose: " + highlight.storyPurpose());
}
requiredText(highlight.visualTreatment(), MAX_DIRECTION_LENGTH, "Visual treatment");
requiredText(highlight.musicDirection(), MAX_DIRECTION_LENGTH, "Music direction");
// SFX direction is optional: a clean music+narration edit may use no sound effects. Validate
// length only when a direction is provided.
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()
&& highlight.sfxDirection().length() > MAX_DIRECTION_LENGTH) {
reject("SFX direction must be 1000 characters or fewer");
}
requiredText(highlight.renderNotes(), MAX_DIRECTION_LENGTH, "Render notes");
// Voiceover is optional: a music-driven cinematic edit may carry no narration. When present,
// every line must be non-blank and the script must fit the highlight timing budget.
if (highlight.voiceover() != null && !highlight.voiceover().isEmpty()) {
for (String line : highlight.voiceover()) {
requiredText(line, MAX_VOICEOVER_LINE_LENGTH, "Voiceover line");
}
double voiceoverDuration = highlight.voiceover().stream()
.mapToDouble(HighlightAssetPreparationService::estimatedVoiceoverDuration)
.sum();
if (voiceoverDuration > highlight.targetDurationSeconds() + EPSILON) {
reject("Voiceover script exceeds the highlight timing budget");
}
}
if (highlight.overlays() == null) {
reject("Highlight overlays list is required");
}
for (String overlay : highlight.overlays()) {
requiredText(overlay, MAX_OVERLAY_LENGTH, "Overlay text");
}
}
private ContentCategory parseCategory(String value) {
if (value == null || value.isBlank()) {
reject("Director plan contentCategory is required");
}
try {
return ContentCategory.valueOf(value.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException ex) {
reject("Unsupported director plan contentCategory: " + value);
return ContentCategory.GENERIC_VLOG;
}
}
private boolean safeId(String value) {
return value != null && SAFE_ID.matcher(value).matches() && !value.contains("..");
}
private boolean finite(double value) {
return Double.isFinite(value);
}
private void requiredText(String value, int maximumLength, String field) {
if (value == null || value.isBlank() || value.length() > maximumLength) {
reject(field + " is required and must be " + maximumLength + " characters or fewer");
}
}
private void reject(String message) {
throw new BadRequestException(message);
}
}

View File

@ -66,6 +66,12 @@ public class HighlightDirectorPromptGenerator {
) {
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
String evidenceStatus = cinematic == null || cinematic.categoryConfidence() < 0.55
? "review_required"
: "model_supported_review_required";
String categoryEvidence = cinematic == null
? "No persisted category analysis is available."
: "confidence=%s reasons=%s".formatted(cinematic.categoryConfidence(), cinematic.categoryReasons());
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
@ -97,10 +103,14 @@ public class HighlightDirectorPromptGenerator {
- Project ID: `%s`
- Source file: `%s`
- Detected content category: `%s`
- Category evidence status: `%s`
- Category evidence: `%s`
- Final highlight duration must be content-driven, not fixed.
- Keep the edit cinematic, premium, and grounded in the visible footage.
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
- Do not invent clips, timestamps, or facts.
- Treat candidate scores as ranking hints, not proof of cinematic quality.
- Inspect the referenced media before selecting any candidate or confirming the category.
- Do not render video.
- Do not modify source media or analysis artifacts.
- Return strict JSON only. No Markdown fences.
@ -146,7 +156,7 @@ public class HighlightDirectorPromptGenerator {
"summary": "short rationale for the selected highlights"
}
""".formatted(
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category,
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category, evidenceStatus, categoryEvidence,
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),

View File

@ -16,23 +16,30 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.regex.Pattern;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightLocalAssetWorker {
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
private static final Pattern SAFE_KEY = Pattern.compile("[A-Za-z0-9][A-Za-z0-9._-]{0,127}");
private static final Set<String> SUPPORTED_TYPES = Set.of("music", "sfx", "voiceover");
private final HighlightProjectStore store;
private final EditAssetProvider assetProvider;
private final EditAssetLibrary assetLibrary;
private final LocalAssetSynthesizer synthesizer;
private final ObjectMapper objectMapper;
public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider,
EditAssetLibrary assetLibrary, ObjectMapper objectMapper) {
EditAssetLibrary assetLibrary, LocalAssetSynthesizer synthesizer,
ObjectMapper objectMapper) {
this.store = store;
this.assetProvider = assetProvider;
this.assetLibrary = assetLibrary;
this.synthesizer = synthesizer;
this.objectMapper = objectMapper;
}
@ -44,9 +51,15 @@ public class HighlightLocalAssetWorker {
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of());
}
List<HighlightAssetRequest> requests = readRequests(requestsDirectory);
log.info("event=highlight_asset_worker_started project_id={} highlight_id={} request_count={} request_directory={} category={}",
projectId, highlight.highlightId(), requests.size(), requestsDirectory, category);
List<String> resolved = new ArrayList<>();
List<String> pending = new ArrayList<>();
for (HighlightAssetRequest request : requests) {
validateRequest(projectId, highlight, request, requestsDirectory);
log.info("event=highlight_asset_generation_requested project_id={} highlight_id={} type={} asset_key={} target={} blocking={} duration_seconds={} request={}",
projectId, highlight.highlightId(), request.type(), request.assetKey(), request.targetPath(),
request.blocking(), request.durationSeconds(), request.requestPath());
Optional<Path> materialized = materialize(request, category);
if (materialized.isPresent()) {
resolved.add(materialized.get().toString());
@ -58,9 +71,59 @@ public class HighlightLocalAssetWorker {
projectId, highlight.highlightId(), request.type(), request.requestPath());
}
}
log.info("event=highlight_asset_worker_completed project_id={} highlight_id={} resolved={} pending={}",
projectId, highlight.highlightId(), resolved.size(), pending.size());
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
}
private void validateRequest(String projectId, HighlightDirectorPlan.HighlightItem highlight,
HighlightAssetRequest request, Path requestsDirectory) {
if (request == null || !projectId.equals(request.projectId())
|| !highlight.highlightId().equals(request.highlightId())) {
throw new IllegalArgumentException("Highlight asset request identity does not match the active project");
}
if (!SUPPORTED_TYPES.contains(request.type()) || request.assetKey() == null
|| !SAFE_KEY.matcher(request.assetKey()).matches() || request.assetKey().contains("..")) {
throw new IllegalArgumentException("Highlight asset request type or asset key is invalid");
}
if (!Double.isFinite(request.durationSeconds()) || request.durationSeconds() <= 0
|| request.durationSeconds() > Math.max(1.0, highlight.targetDurationSeconds()) + 0.001) {
throw new IllegalArgumentException("Highlight asset request duration is outside the highlight bounds");
}
if (request.notes() == null || request.notes().isBlank()) {
throw new IllegalArgumentException("Highlight asset request notes are required");
}
Path assetsDirectory = requestsDirectory.getParent().toAbsolutePath().normalize();
Path expectedTarget = switch (request.type()) {
case "music" -> assetsDirectory.resolve("music/music.wav");
case "voiceover" -> assetsDirectory.resolve("voiceover").resolve(request.assetKey() + ".wav");
case "sfx" -> assetsDirectory.resolve("sfx").resolve(request.assetKey() + ".wav");
default -> throw new IllegalArgumentException("Unsupported highlight asset request type");
};
Path actualTarget = Path.of(request.targetPath()).toAbsolutePath().normalize();
Path expectedRequest = requestsDirectory.resolve(request.type() + "-" + request.assetKey() + ".md")
.toAbsolutePath().normalize();
Path actualRequest = Path.of(request.requestPath()).toAbsolutePath().normalize();
if (!actualTarget.equals(expectedTarget) || !actualRequest.equals(expectedRequest)) {
throw new IllegalArgumentException("Highlight asset request paths do not match the project contract");
}
rejectSymlinkPath(store.projectDirectory(projectId).toAbsolutePath().normalize(), expectedTarget);
}
private void rejectSymlinkPath(Path projectDirectory, Path target) {
if (!target.startsWith(projectDirectory)) {
throw new IllegalArgumentException("Highlight asset target escapes the project directory");
}
Path cursor = projectDirectory;
for (Path part : projectDirectory.relativize(target)) {
cursor = cursor.resolve(part);
if (Files.isSymbolicLink(cursor)) {
throw new IllegalArgumentException("Highlight asset target contains a symbolic link");
}
}
}
private List<HighlightAssetRequest> readRequests(Path directory) {
try {
try (var files = Files.list(directory)) {
@ -101,7 +164,7 @@ public class HighlightLocalAssetWorker {
copy(Path.of(asset.get().path()), target);
return Optional.of(target);
}
return generateToneBed(target, request.durationSeconds());
return synthesizer.synthesizeMusic(request.projectId(), request.notes(), request.durationSeconds(), target);
}
private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) {
@ -118,7 +181,7 @@ public class HighlightLocalAssetWorker {
copy(Path.of(fallback.get().path()), target);
return Optional.of(target);
}
return generateImpactTone(target, request.durationSeconds());
return synthesizer.synthesizeSfx(request.projectId(), request.notes(), request.durationSeconds(), target);
}
private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) {
@ -127,76 +190,13 @@ public class HighlightLocalAssetWorker {
return Optional.empty();
}
Path target = Path.of(request.targetPath());
if (runSay(text, target)) {
return Optional.of(target);
}
if (runEspeak(text, target)) {
return Optional.of(target);
}
return Optional.empty();
}
private Optional<Path> generateToneBed(Path target, double durationSeconds) {
return generateAudio(target, durationSeconds, "sine=frequency=110:sample_rate=48000", 0.02);
}
private Optional<Path> generateImpactTone(Path target, double durationSeconds) {
return generateAudio(target, Math.max(0.5, Math.min(1.0, durationSeconds)),
"sine=frequency=880:sample_rate=48000", 0.12);
}
private Optional<Path> generateAudio(Path target, double durationSeconds, String source, double volume) {
try {
Files.createDirectories(target.getParent());
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", source, "-t", Double.toString(durationSeconds),
"-af", "volume=" + volume, "-c:a", "aac", target.toString());
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0) {
return Optional.of(target);
}
log.warn("event=highlight_asset_generation_failed target={} message={}", target, output);
return Optional.empty();
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_asset_generation_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
}
}
private boolean runSay(String text, Path target) {
return runSpeechCommand(List.of("say", "-o", target.toString(), text), target);
}
private boolean runEspeak(String text, Path target) {
return runSpeechCommand(List.of("espeak", "-w", target.toString(), text), target);
}
private boolean runSpeechCommand(List<String> command, Path target) {
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
return true;
}
if (!output.isBlank()) {
log.warn("event=highlight_voiceover_generation_failed target={} output={}", target, output);
}
return false;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_voiceover_generation_failed target={} error_type={} message={}", target,
ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
return synthesizer.synthesizeVoiceover(request.projectId(), List.of(new VoiceoverLine(text, 0.0,
Math.max(1.0, request.durationSeconds()), "cinematic_narration")), target);
}
private void copy(Path source, Path target) {
try {
Files.createDirectories(target.getParent());
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
AssetLicensePolicy.copy(source, target);
} catch (IOException ex) {
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
}

View File

@ -0,0 +1,452 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
/**
* Tier-1 automatic director. It inspects MEASURED signals from the source per-window motion (temporal luma
* difference) and audio energy (RMS) and composes a story-structured montage from them:
*
* <ol>
* <li>setup (before the action);</li>
* <li>a continuous action + tension shot (release/roll/watch never chopped);</li>
* <li>a short realization shot;</li>
* <li>a slow-motion payoff centered on the audio climax;</li>
* <li>a resolution button after the climax;</li>
* </ol>
*
* <p>while trimming a high-motion tail (e.g. a camera whip). Every decision is derived from a measurement of
* this source nothing is hard-coded to one video or one kind of content. Semantic captions/overlays and
* emotional nuance are deliberately NOT attempted here; that is the Tier-2 vision-language director's job.
* This tier gives a strong, deterministic, offline baseline cut for any source.
*/
@Component
public class HighlightMontageDirector {
static final double WINDOW_SECONDS = 0.5;
/** Minimum spacing (seconds) between proposed candidates, so they are distinct moments. Kept short so a
* reaction close on the heels of the action (a celebration right after the strike) stays a separate
* candidate the vision judge can score, rather than being merged into the louder neighbouring peak. */
static final double MIN_CANDIDATE_SEPARATION = 1.0;
/** Selection-score blend: how the judge's meaning-rating and the measured action intensity combine. */
static final double SELECT_SEMANTIC_WEIGHT = 0.6;
static final double SELECT_INTENSITY_WEIGHT = 0.4;
private final VideoClippingProperties.Editing properties;
public HighlightMontageDirector(VideoClippingProperties properties) {
this.properties = properties.getEditing();
}
/** Measure the source and compose a reel, choosing decisive moments by measured intensity alone. */
public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds) {
return direct(projectId, sourceFileName, source, durationSeconds, null);
}
/**
* Compose a multi-segment HIGHLIGHT REEL. Measurement proposes candidate moments across the WHOLE clip
* (intensity peaks); the Tier-2 vision {@code judge} rates each by meaning and supplies an overlay; the
* director keeps EVERY moment worth showing (blending the judge's rating with measured action intensity,
* so a dynamic section is kept even when the model can't name it), builds an action segment
* (entry -> peak -> exit) around each, and concatenates them all into one video. No cap on how many
* segments or how long. When no judge is supplied, selection is by measured intensity alone.
*/
public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds,
MomentJudge judge) {
double[] motion = probeCurve(source.toString(), true, durationSeconds);
double[] audio = probeCurve(source.toString(), false, durationSeconds);
List<Double> candidates = candidatePeaks(motion, audio, WINDOW_SECONDS, durationSeconds);
List<Judgement> judgements = (judge != null && !candidates.isEmpty())
? judge.judge(List.copyOf(candidates)) : null;
return composeReel(projectId, sourceFileName, motion, audio, candidates, judgements,
WINDOW_SECONDS, durationSeconds);
}
/** Per-candidate verdict from the Tier-2 vision judge: how highlight-worthy (0..1) and a ready overlay. */
public record Judgement(double worthiness, String overlayText) {
}
/** Rates each candidate moment; returns one {@link Judgement} per candidate (same order). */
@FunctionalInterface
public interface MomentJudge {
List<Judgement> judge(List<Double> candidateTimeSeconds);
}
/**
* Proposes candidate decisive moments as the local maxima of measured intensity (motion + audio),
* min-separated, returned in TIME order. GENERIC: no positional band, no per-video threshold and NO cap on
* how many every real event becomes a candidate, and the judge + intensity decide which make the reel.
*/
List<Double> candidatePeaks(double[] motion, double[] audio, double window, double duration) {
int n = Math.min(motion.length, audio.length);
if (n < 4 || duration <= 0) {
return List.of();
}
double[] mN = normalize(smooth(motion, n), n);
double[] aN = normalize(smooth(audio, n), n);
double[] intensity = new double[n];
double globalMax = 0;
for (int i = 0; i < n; i++) {
intensity[i] = aN[i] + mN[i];
globalMax = Math.max(globalMax, intensity[i]);
}
if (globalMax <= 1e-9) {
return List.of();
}
int neigh = Math.max(1, (int) Math.round(1.2 / window)); // local-max over +/-1.2s
double threshold = 0.35 * globalMax; // ignore minor bumps
List<Integer> peaks = new ArrayList<>();
for (int i = 1; i < n - 1; i++) { // skip degenerate first/last window
if (intensity[i] < threshold) {
continue;
}
boolean isMax = true;
for (int j = Math.max(0, i - neigh); j <= Math.min(n - 1, i + neigh); j++) {
if (intensity[j] > intensity[i]) {
isMax = false;
break;
}
}
if (isMax) {
peaks.add(i);
}
}
peaks.sort((x, y) -> Double.compare(intensity[y], intensity[x])); // strongest first for dedup
List<Double> chosen = new ArrayList<>();
for (int idx : peaks) {
double t = idx * window;
boolean tooClose = chosen.stream().anyMatch(c -> Math.abs(c - t) < MIN_CANDIDATE_SEPARATION);
if (!tooClose) {
chosen.add(t); // no cap on how many candidates
}
}
chosen.sort(Double::compareTo); // return in time order
return chosen;
}
/**
* Builds the multi-segment reel from the candidates. Each candidate's SELECTION SCORE blends the judge's
* highlight-worthiness with the candidate's measured action intensity: {@code 0.6*worthiness +
* 0.4*intensity}. Every candidate whose score clears the threshold becomes a segment (at least the best
* one always does); each segment is an action unit (entry -> slow-mo peak -> exit) that never overlaps the
* previous one. Pure and deterministic; package-visible for unit testing without ffmpeg.
*/
MontagePlan composeReel(String projectId, String sourceFileName, double[] motion, double[] audio,
List<Double> candidates, List<Judgement> judgements, double window, double duration) {
int n = Math.min(motion.length, audio.length);
if (n < 4 || duration <= 0 || candidates == null || candidates.isEmpty()) {
return straightCut(projectId, sourceFileName, duration);
}
double[] m = smooth(motion, n);
double[] mN = normalize(m, n);
double[] aN = normalize(smooth(audio, n), n);
// Score every candidate = 0.6 * (judge worthiness) + 0.4 * (measured intensity). When the model can't
// discriminate (all "riding" -> neutral), intensity decides which sections are the real highlights.
int k = candidates.size();
double[] score = new double[k];
for (int i = 0; i < k; i++) {
int idx = Math.max(0, Math.min(n - 1, (int) Math.round(candidates.get(i) / window)));
double intensity = (aN[idx] + mN[idx]) / 2.0; // 0..1
double worthiness = (judgements != null && i < judgements.size())
? judgements.get(i).worthiness() : 0.4; // neutral when no judge
score[i] = SELECT_SEMANTIC_WEIGHT * worthiness + SELECT_INTENSITY_WEIGHT * intensity;
}
double bar = properties.getHighlightSelectThreshold();
int best = 0;
for (int i = 1; i < k; i++) {
if (score[i] > score[best]) {
best = i;
}
}
List<Integer> selected = new ArrayList<>();
for (int i = 0; i < k; i++) {
if (score[i] >= bar) {
selected.add(i); // candidates are time-ordered
}
}
if (selected.isEmpty()) {
selected.add(best); // always at least the single best
}
List<MontagePlan.Shot> allShots = new ArrayList<>();
List<MontagePlan.Overlay> overlays = new ArrayList<>();
double timeline = 0.0;
double prevExit = Double.NEGATIVE_INFINITY;
for (int i : selected) {
Segment seg = buildSegment(m, candidates.get(i), window, duration, prevExit);
if (seg == null) {
continue; // overlapped or degenerate -> merged away
}
double preDur = 0;
for (int s = 0; s < seg.payoffShotIndex(); s++) {
preDur += seg.shots().get(s).durationSeconds();
}
double payoffTlStart = timeline + preDur;
double payoffTlDur = seg.shots().get(seg.payoffShotIndex()).durationSeconds();
allShots.addAll(seg.shots());
for (MontagePlan.Shot sh : seg.shots()) {
timeline += sh.durationSeconds();
}
String overlayText = (judgements != null && i < judgements.size()) ? judgements.get(i).overlayText() : "";
if (overlayText != null && !overlayText.isBlank()) {
overlays.add(new MontagePlan.Overlay(overlayText, round(payoffTlStart + 0.2),
round(Math.max(payoffTlStart + 0.8, payoffTlStart + payoffTlDur - 0.2)), "lower_center_safe"));
}
prevExit = seg.exitSrc();
}
if (allShots.isEmpty()) {
return straightCut(projectId, sourceFileName, duration);
}
return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), overlays, allShots);
}
private record Segment(List<MontagePlan.Shot> shots, double exitSrc, int payoffShotIndex) {
}
/**
* One action segment around a peak: MEASURED entry (build into the peak, capped) -> slow-mo payoff on the
* peak -> MEASURED exit (motion settles). Returns null when the peak is already covered by the previous
* segment (so segments never overlap) or the span is degenerate.
*/
private Segment buildSegment(double[] m, double climaxTime, double window, double duration, double prevExit) {
int n = m.length;
int climaxIdx = Math.max(1, Math.min(n - 1, (int) Math.round(climaxTime / window)));
climaxTime = climaxIdx * window;
if (climaxTime <= prevExit + 0.5) {
return null; // this peak is inside the previous segment
}
// Exit: extend past the peak while motion stays elevated, stop when it settles / at a whip / at a cap.
double whipLevel = Math.max(1e-6, percentile(m, n, 0.4) * 4.0);
int whipCutoff = n;
for (int i = climaxIdx + 2; i < n; i++) {
if (m[i] > whipLevel) {
whipCutoff = i;
break;
}
}
double baseMotion = mean(m, 0, Math.max(1, climaxIdx));
double settleLevel = baseMotion * 1.4;
double maxResolution = climaxTime + 6.0;
double resolution = climaxTime + 1.0;
int settledRun = 0;
for (int i = climaxIdx + 1; i < Math.min(n, whipCutoff); i++) {
double t = i * window;
if (t > maxResolution) {
break;
}
resolution = t;
if (m[i] <= settleLevel) {
if (++settledRun >= 3) {
break;
}
} else {
settledRun = 0;
}
}
double exit = Math.min(duration, Math.max(climaxTime + 1.5, resolution + 0.3));
// Entry: build INTO the peak, bounded by the cap and never earlier than the previous segment's exit.
int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)),
Math.max(2, (int) Math.floor((climaxTime - 1.0) / window)));
double actionTime = actionIdx * window;
double payoffStart = climaxTime - 0.3;
double desiredBuild = payoffStart - (actionTime - 1.0);
double build = Math.min(Math.max(0.6, desiredBuild), Math.max(1.0, properties.getMontageMaxBuildSeconds()));
double buildStart = Math.max(Math.max(0.0, prevExit), payoffStart - build);
double payoffEnd = Math.min(climaxTime + 2.0, exit - 0.4);
if (exit - payoffStart < 0.6) {
return null; // too short to be a segment
}
if (payoffEnd < payoffStart + 0.5) {
payoffEnd = Math.min(exit, payoffStart + 0.8);
}
List<MontagePlan.Shot> shots = new ArrayList<>();
if (payoffStart - buildStart >= 0.4) {
addShot(shots, buildStart, payoffStart, 1.04, 1.0, duration); // entry / build into the peak
}
int payoffShotIndex = shots.size();
addShot(shots, payoffStart, payoffEnd, 1.05, 0.7, duration); // slow-mo payoff on the peak
if (shots.size() <= payoffShotIndex) {
return null; // payoff shot was degenerate
}
addShot(shots, payoffEnd, exit, 1.04, 0.9, duration); // exit / resolution
return new Segment(shots, exit, payoffShotIndex);
}
private MontagePlan straightCut(String projectId, String sourceFileName, double duration) {
double d = Math.max(0.5, Math.min(duration <= 0 ? 6.0 : duration, 8.0));
return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), List.of(),
List.of(new MontagePlan.Shot(0.0, d, 1.03, 1.0)));
}
private void addShot(List<MontagePlan.Shot> shots, double srcStart, double srcEnd,
double zoom, double speed, double duration) {
double s = clamp(srcStart, 0.0, duration);
double e = clamp(srcEnd, 0.0, duration);
double span = e - s;
if (span < 0.4) {
return; // skip degenerate shots
}
shots.add(new MontagePlan.Shot(round(s), round(span / speed), zoom, speed));
}
private String genericMusic() {
return "cinematic build, quiet tense strings rising to a triumphant orchestral and percussion hit at "
+ "the climax, then a short warm resolve, modern trailer film score, no vocals";
}
// --- measurement ---------------------------------------------------------------------------------------
/**
* Measures a per-window curve over the source: motion (signalstats YDIF, temporal luma diff) when
* {@code motion} is true, else audio energy (astats RMS in dB). Bins samples into {@code WINDOW_SECONDS}
* buckets. Missing buckets get a floor value. Returns an empty array when nothing could be measured.
*/
double[] probeCurve(String source, boolean motion, double duration) {
int size = Math.max(1, (int) Math.ceil(duration / WINDOW_SECONDS));
List<String> command = motion
? List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-an",
"-vf", "signalstats,metadata=print:key=lavfi.signalstats.YDIF", "-f", "null", "-")
: List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-vn",
"-af", "astats=metadata=1:reset=1,ametadata=print:key=lavfi.astats.Overall.RMS_level",
"-f", "null", "-");
String token = motion ? "YDIF" : "RMS_level";
double floor = motion ? 0.0 : -90.0;
double[] sum = new double[size];
int[] count = new int[size];
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
process.waitFor();
double lastPts = -1;
for (String line : out.split("\\R")) {
int pts = line.indexOf("pts_time:");
if (pts >= 0) {
lastPts = parseTrailingNumber(line.substring(pts + "pts_time:".length()));
continue;
}
int tok = line.indexOf(token + "=");
if (tok >= 0 && lastPts >= 0) {
double value = parseTrailingNumber(line.substring(tok + token.length() + 1));
int idx = Math.min(size - 1, (int) Math.floor(lastPts / WINDOW_SECONDS));
if (idx >= 0 && !Double.isNaN(value)) {
sum[idx] += value;
count[idx]++;
}
}
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
return new double[0];
} catch (IOException | RuntimeException ex) {
return new double[0];
}
double[] curve = new double[size];
boolean any = false;
for (int i = 0; i < size; i++) {
curve[i] = count[i] == 0 ? floor : sum[i] / count[i];
any |= count[i] > 0;
}
return any ? curve : new double[0];
}
private static double parseTrailingNumber(String text) {
int i = 0;
StringBuilder sb = new StringBuilder();
String trimmed = text.trim();
while (i < trimmed.length()) {
char c = trimmed.charAt(i);
if (Character.isDigit(c) || c == '.' || c == '-' || c == '+' || c == 'e' || c == 'E') {
sb.append(c);
i++;
} else {
break;
}
}
try {
return sb.length() == 0 ? Double.NaN : Double.parseDouble(sb.toString());
} catch (NumberFormatException ex) {
return Double.NaN;
}
}
// --- small numeric helpers ----------------------------------------------------------------------------
private static double[] smooth(double[] values, int n) {
double[] out = new double[n];
for (int i = 0; i < n; i++) {
double s = 0;
int c = 0;
for (int j = Math.max(0, i - 1); j <= Math.min(n - 1, i + 1); j++) {
s += values[j];
c++;
}
out[i] = s / c;
}
return out;
}
private static int argMax(double[] values, int from, int to) {
int lo = Math.max(0, from);
int hi = Math.min(values.length, Math.max(lo + 1, to));
int best = lo;
for (int i = lo; i < hi; i++) {
if (values[i] > values[best]) {
best = i;
}
}
return best;
}
/** The {@code p}-quantile (0..1) of the first {@code n} values — a robust, spike-resistant level. */
private static double percentile(double[] values, int n, double p) {
double[] copy = java.util.Arrays.copyOf(values, n);
java.util.Arrays.sort(copy);
int idx = Math.min(n - 1, Math.max(0, (int) (p * n)));
return copy[idx];
}
/** Min-max normalizes the first {@code n} values to 0..1; returns zeros when the range is flat. */
private static double[] normalize(double[] values, int n) {
double min = Double.POSITIVE_INFINITY;
double max = Double.NEGATIVE_INFINITY;
for (int i = 0; i < n; i++) {
min = Math.min(min, values[i]);
max = Math.max(max, values[i]);
}
double[] out = new double[n];
double range = max - min;
if (range <= 1e-9) {
return out;
}
for (int i = 0; i < n; i++) {
out[i] = (values[i] - min) / range;
}
return out;
}
private static double mean(double[] values, int from, int to) {
int lo = Math.max(0, from);
int hi = Math.min(values.length, Math.max(lo + 1, to));
double s = 0;
for (int i = lo; i < hi; i++) {
s += values[i];
}
return s / (hi - lo);
}
private static double clamp(double v, double lo, double hi) {
return Math.max(lo, Math.min(hi, v));
}
private static double round(double v) {
return Math.round(v * 1000.0) / 1000.0;
}
}

View File

@ -18,6 +18,7 @@ import java.nio.file.StandardCopyOption;
import java.time.Clock;
import java.time.Instant;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
import java.util.Set;
@ -39,7 +40,10 @@ public class HighlightSourceScheduler {
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final HighlightProjectStore store;
private final HighlightSourceAnalyzer analyzer;
private final HighlightCandidateGenerator candidateGenerator;
private final HighlightDirectorPromptGenerator directorPromptGenerator;
private final HighlightMontageDirector montageDirector;
private final HighlightVisionDirector visionDirector;
private final Clock clock;
private final AtomicBoolean scanning = new AtomicBoolean(false);
private final AtomicLong scanSequence = new AtomicLong();
@ -49,22 +53,32 @@ public class HighlightSourceScheduler {
VideoClippingProperties properties,
HighlightProjectStore store,
HighlightSourceAnalyzer analyzer,
HighlightDirectorPromptGenerator directorPromptGenerator
HighlightCandidateGenerator candidateGenerator,
HighlightDirectorPromptGenerator directorPromptGenerator,
HighlightMontageDirector montageDirector,
HighlightVisionDirector visionDirector
) {
this(properties, store, analyzer, directorPromptGenerator, Clock.systemUTC());
this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, montageDirector,
visionDirector, Clock.systemUTC());
}
HighlightSourceScheduler(
VideoClippingProperties properties,
HighlightProjectStore store,
HighlightSourceAnalyzer analyzer,
HighlightCandidateGenerator candidateGenerator,
HighlightDirectorPromptGenerator directorPromptGenerator,
HighlightMontageDirector montageDirector,
HighlightVisionDirector visionDirector,
Clock clock
) {
this.properties = properties.getEditing().getHighlightScheduler();
this.store = store;
this.analyzer = analyzer;
this.candidateGenerator = candidateGenerator;
this.directorPromptGenerator = directorPromptGenerator;
this.montageDirector = montageDirector;
this.visionDirector = visionDirector;
this.clock = clock;
}
@ -162,6 +176,11 @@ public class HighlightSourceScheduler {
+ "project_source_directory={}",
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
CinematicHighlightAnalysis cinematic = candidateGenerator.generate(projectId, analysis);
log.info("event=highlight_flow_candidates_completed scan_id={} project_id={} count={} category={} "
+ "category_confidence={}",
scanId, projectId, cinematic.candidates().size(), cinematic.category(),
cinematic.categoryConfidence());
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
+ "processed_directory={}",
@ -170,6 +189,11 @@ public class HighlightSourceScheduler {
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
store.directorDirectory(projectId).resolve("director-brief.md"));
if (properties.isAutoDirectorEnabled()) {
autoDirect(projectId, project.sourceVideoFileName(),
store.sourceDirectory(projectId).resolve(workingFile.getFileName()),
analysis, cinematic, scanId);
}
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
+ "analysis_file={} elapsed_ms={}",
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
@ -200,6 +224,57 @@ public class HighlightSourceScheduler {
}
}
private void autoDirect(String projectId, String sourceFileName, Path sourcePath,
HighlightSourceAnalysis analysis, CinematicHighlightAnalysis cinematic, long scanId) {
try {
double duration = analysis.source() == null ? 0.0 : analysis.source().durationSeconds();
Path visionWork = store.directorDirectory(projectId).resolve("vision-work");
// GENERIC rule: measurement PROPOSES candidate moments across the WHOLE clip; the vision judge rates
// each by meaning and supplies an honest overlay; the director keeps EVERY worthy moment (blended
// with action intensity) and builds a multi-segment reel. No band, no per-video threshold, no cap.
HighlightMontageDirector.MomentJudge judge = properties.isVisionDirectorEnabled()
? candidates -> visionDirector.judgeMoments(sourcePath, candidates, visionWork)
: null;
MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration, judge);
store.writeJson(projectId, "director/montage.json", montage);
// Minimal edit-plan.json so the render scanner selects the project; montage.json takes precedence.
store.writeJson(projectId, "director/edit-plan.json",
fallbackEditPlan(projectId, sourceFileName, cinematic, duration));
log.info("event=highlight_auto_director_completed scan_id={} project_id={} shots={} source_duration={}",
scanId, projectId, montage.shots().size(), duration);
} catch (RuntimeException ex) {
log.warn("event=highlight_auto_director_failed scan_id={} project_id={} error_type={} message={}",
scanId, projectId, ex.getClass().getSimpleName(), ex.getMessage());
}
}
private HighlightDirectorPlan fallbackEditPlan(String projectId, String sourceFileName,
CinematicHighlightAnalysis cinematic, double duration) {
String category = cinematic.category() == null ? "generic_vlog"
: cinematic.category().name().toLowerCase(java.util.Locale.ROOT);
String candidateId;
double start;
double end;
if (cinematic.candidates() != null && !cinematic.candidates().isEmpty()) {
HighlightCandidate candidate = cinematic.candidates().get(0);
candidateId = candidate.id();
start = candidate.sourceStartSeconds();
end = candidate.sourceEndSeconds();
} else {
candidateId = "candidate_001";
start = 0.0;
end = Math.max(1.0, Math.min(duration <= 0 ? 8.0 : duration, 8.0));
}
double span = Math.max(1.0, end - start);
HighlightDirectorPlan.HighlightItem item = new HighlightDirectorPlan.HighlightItem(
"highlight_001", candidateId, "Highlight", start, end, span,
"hero_payoff", "cinematic hero grade", "cinematic score", "",
java.util.List.of(), java.util.List.of(),
"Auto-generated fallback; the adjacent montage.json takes precedence.");
return new HighlightDirectorPlan(projectId, sourceFileName, category, java.util.List.of(item),
"Auto-director fallback edit-plan; montage.json drives the render.");
}
private void copySourceToProject(Path source, Path target) {
try {
Files.copy(source, target, StandardCopyOption.COPY_ATTRIBUTES);
@ -209,18 +284,33 @@ public class HighlightSourceScheduler {
}
private Path moveToDirectory(Path source, Path targetDirectory) {
Path target = targetDirectory.resolve(source.getFileName());
if (Files.exists(target)) {
throw new IllegalStateException("Refusing to overwrite highlight source file: " + target);
}
try {
Files.createDirectories(targetDirectory);
// Never overwrite an existing file, but also never fail on a name collision: re-processing a source
// whose name already exists here (a recurring filename) picks a unique "<name>-<n>.<ext>" instead.
Path target = uniqueTarget(targetDirectory, source.getFileName().toString());
return Files.move(source, target, StandardCopyOption.ATOMIC_MOVE);
} catch (IOException ex) {
throw new IllegalStateException("Unable to move highlight source file to: " + targetDirectory, ex);
}
}
Path uniqueTarget(Path directory, String fileName) {
Path candidate = directory.resolve(fileName);
if (!Files.exists(candidate)) {
return candidate;
}
String base = stripExtension(fileName);
String ext = extension(fileName);
String suffixExt = ext.isEmpty() ? "" : "." + ext;
int n = 1;
do {
candidate = directory.resolve(base + "-" + n + suffixExt);
n++;
} while (Files.exists(candidate));
return candidate;
}
private boolean isCandidateFile(Path path) {
if (!Files.isRegularFile(path)) {
return false;

View File

@ -0,0 +1,145 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
/**
* R14 subject-tracking reframe. Runs a local YOLO tracker over a shot's source range and returns a smoothed,
* normalised subject-centre path so the renderer can follow the subject instead of using a static centre crop
* (the "AI reframe" consumer editors like DJI/Insta360 do). Fails soft everywhere a missing interpreter, no
* detections, or a parse error yield an empty path and the renderer keeps its centred crop.
*
* <p>YOLOv8 is AGPL-3.0, so this path is non-commercial matching the repo's existing CV stance.</p>
*/
@Component
public class HighlightSubjectTracker {
private static final Logger log = LoggerFactory.getLogger(HighlightSubjectTracker.class);
private final VideoClippingProperties.Editing editing;
private final ObjectMapper objectMapper;
private final HighlightBeatSync.CommandRunner commandRunner;
@org.springframework.beans.factory.annotation.Autowired
public HighlightSubjectTracker(VideoClippingProperties properties, ObjectMapper objectMapper) {
this(properties, objectMapper, command -> {
Process process = new ProcessBuilder(command).redirectErrorStream(false).start();
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
process.waitFor();
return out;
});
}
HighlightSubjectTracker(VideoClippingProperties properties, ObjectMapper objectMapper,
HighlightBeatSync.CommandRunner commandRunner) {
this.editing = properties.getEditing();
this.objectMapper = objectMapper;
this.commandRunner = commandRunner;
}
/**
* Tracks the dominant subject across {@code [startSeconds, endSeconds)} and returns a smoothed list of
* normalised centres {@code [cx, cy]} (each 0..1), ordered by time. Empty on any failure.
*/
List<double[]> track(Path source, double startSeconds, double endSeconds) {
if (source == null || endSeconds <= startSeconds) {
return List.of();
}
int samples = Math.max(1, editing.getSubjectTrackSamples());
List<String> command = List.of(
editing.getSubjectTrackPython(), editing.getSubjectTrackScript(),
source.toAbsolutePath().toString(),
String.format(Locale.ROOT, "%.3f", Math.max(0.0, startSeconds)),
String.format(Locale.ROOT, "%.3f", endSeconds),
Integer.toString(samples));
try {
String out = commandRunner.run(command);
String json = lastJsonObject(out);
if (json == null) {
return List.of();
}
JsonNode path = objectMapper.readTree(json).path("path");
if (!path.isArray() || path.isEmpty()) {
return List.of();
}
List<double[]> centers = new ArrayList<>();
for (JsonNode point : path) {
double cx = point.path("cx").asDouble(Double.NaN);
double cy = point.path("cy").asDouble(Double.NaN);
if (Double.isFinite(cx) && Double.isFinite(cy)) {
centers.add(new double[]{clamp01(cx), clamp01(cy)});
}
}
return smooth(centers);
} catch (Exception ex) { // fail soft: keep the centred crop
log.warn("event=subject_track_failed source={} error={}", source, ex.toString());
return List.of();
}
}
/**
* Encodes a centre path as a compact treatment token {@code pan=cx0:cy0;cx1:cy1;...} the renderer parses.
* Returns "" for an empty path so callers can append it unconditionally.
*/
static String panToken(List<double[]> centers) {
if (centers == null || centers.isEmpty()) {
return "";
}
StringBuilder token = new StringBuilder("pan=");
for (int i = 0; i < centers.size(); i++) {
if (i > 0) {
token.append(';');
}
token.append(String.format(Locale.ROOT, "%.4f:%.4f", centers.get(i)[0], centers.get(i)[1]));
}
return token.toString();
}
/** 3-tap moving average to take the jitter out of frame-by-frame detections. */
private static List<double[]> smooth(List<double[]> centers) {
int n = centers.size();
if (n <= 2) {
return List.copyOf(centers);
}
List<double[]> out = new ArrayList<>(n);
for (int i = 0; i < n; i++) {
int lo = Math.max(0, i - 1);
int hi = Math.min(n - 1, i + 1);
double sx = 0;
double sy = 0;
for (int j = lo; j <= hi; j++) {
sx += centers.get(j)[0];
sy += centers.get(j)[1];
}
int count = hi - lo + 1;
out.add(new double[]{sx / count, sy / count});
}
return out;
}
private static double clamp01(double v) {
return v < 0 ? 0 : (v > 1 ? 1 : v);
}
private static String lastJsonObject(String out) {
if (out == null) {
return null;
}
int end = out.lastIndexOf('}');
int start = out.indexOf('{');
if (start < 0 || end < 0 || end < start) {
return null;
}
return out.substring(start, end + 1);
}
}

View File

@ -0,0 +1,379 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
/**
* Tier-2 semantic director. It samples the payoff frame of a Tier-1 {@link MontagePlan}, asks a local
* vision-language model (moondream2, via {@code tools/vision_caption.py}) what the moment is and what its
* mood is, and AUGMENTS the plan with a semantic overlay caption and a scene-informed music direction the
* nuance measurement alone cannot reach.
*
* <p>It never fails the pipeline: on any error (model missing, worker failure, timeout) it returns the input
* plan unchanged, so the Tier-1 cut always stands.
*/
@Component
public class HighlightVisionDirector {
private static final Logger log = LoggerFactory.getLogger(HighlightVisionDirector.class);
private static final long TIMEOUT_SECONDS = 600;
/**
* A descriptive, open-ended question. Small VLMs give far more discriminative answers to this than to a
* terse posture question (an example-laden "arms raised / mid-throw / standing" prompt collapses to a
* constant "Standing still" on distant footage). The description feeds both the highlight-worthiness score
* and the honest-overlay anticipation check.
*/
private static final String DESCRIPTIVE_QUESTION =
"In one sentence, describe what the person is doing with their body right now.";
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
private final String ffmpegBinary;
private final String captionScript;
private final ObjectMapper objectMapper;
private static final String MOONDREAM_SCRIPT = "./tools/vision_caption.py";
private static final String LLAMACPP_DEFAULT_MODEL = "./models/qwen2.5-vl-3b/Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf";
private static final String LLAMACPP_DEFAULT_MMPROJ = "./models/qwen2.5-vl-3b/mmproj-F16.gguf";
public HighlightVisionDirector(VideoClippingProperties properties, ObjectMapper objectMapper) {
this.worker = properties.getEditing().getLocalAssetWorker();
this.ffmpegBinary = properties.getEditing().getFfmpegBinary();
this.captionScript = resolveCaptionScript(properties.getEditing().getVisionCaptionScript());
this.objectMapper = objectMapper;
}
/**
* Resolve the effective captioner. When the llama.cpp backend is configured but not fully provisioned (no
* binary env var, or missing weights), fall back to moondream and WARN loudly so a default of the stronger
* model never silently degrades into a wrong pick on an unprovisioned machine.
*/
static String resolveCaptionScript(String configured) {
if (configured == null || !configured.contains("vision_caption_llamacpp")) {
return configured; // moondream or a custom backend use as-is
}
boolean scriptOk = isFile(configured);
boolean binOk = isFile(System.getenv("LLAMACPP_SERVER_BIN")) || isFile(System.getenv("LLAMACPP_MTMD_BIN"));
boolean modelOk = isFile(envOr("LLAMACPP_VLM_MODEL", LLAMACPP_DEFAULT_MODEL));
boolean mmprojOk = isFile(envOr("LLAMACPP_VLM_MMPROJ", LLAMACPP_DEFAULT_MMPROJ));
if (scriptOk && binOk && modelOk && mmprojOk) {
log.info("event=vision_backend backend=llamacpp script={}", configured);
return configured;
}
log.warn("event=vision_backend_not_ready backend=llamacpp fallback=moondream "
+ "script_ok={} binary_env_ok={} model_ok={} mmproj_ok={} "
+ "(set LLAMACPP_SERVER_BIN or LLAMACPP_MTMD_BIN and provision the weights to use Qwen)",
scriptOk, binOk, modelOk, mmprojOk);
return MOONDREAM_SCRIPT;
}
private static boolean isFile(String path) {
return path != null && !path.isBlank() && Files.isRegularFile(Path.of(path.trim()));
}
private static String envOr(String name, String fallback) {
String v = System.getenv(name);
return v == null || v.isBlank() ? fallback : v;
}
/**
* Captions {@code samples} frames evenly across the source with the local VLM (one worker call, model
* loaded once). Returns timed captions describing the key action/achievement at each moment, or an empty
* list on any failure (the caller then proceeds with measurement only).
*/
public List<TimedCaption> captionTimeline(Path source, double duration, Path workDir, int samples) {
try {
Files.createDirectories(workDir);
List<Manifest> manifest = new ArrayList<>();
List<Double> times = new ArrayList<>();
for (int i = 0; i < samples; i++) {
double t = duration * (i + 0.5) / samples;
Path frame = workDir.resolve("beat-" + i + ".jpg");
if (extractFrame(source, t, frame)) {
// Two questions per frame in one worker call: a DESCRIPTIVE one (small VLMs give far more
// discriminative descriptions than terse verb phrases, which collapse to a constant answer
// on distant subjects) to score the moment, and a PUNCHY one for the overlay caption.
manifest.add(new Manifest("d" + i, frame.toAbsolutePath().toString(),
"In one sentence, describe what the person is doing with their body right now."));
manifest.add(new Manifest("l" + i, frame.toAbsolutePath().toString(),
"In one to three words, what is the exciting achievement or action in this moment?"));
times.add(t);
}
}
if (manifest.isEmpty()) {
return List.of();
}
List<Caption> captions = runCaptioner(workDir, manifest);
List<TimedCaption> result = new ArrayList<>();
for (int i = 0; i < times.size(); i++) {
result.add(new TimedCaption(times.get(i), answerFor(captions, "d" + i),
answerFor(captions, "l" + i)));
}
log.info("event=highlight_vision_timeline_captioned frames={}", result.size());
return result;
} catch (RuntimeException | IOException | InterruptedException ex) {
if (ex instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
log.warn("event=highlight_vision_director_failed stage=timeline error_type={} message={}",
ex.getClass().getSimpleName(), ex.getMessage());
return List.of();
}
}
/**
* The Tier-2 vision JUDGE for a highlight reel. Captions EVERY candidate moment in one batched pass (a
* description to rate it, plus a punchy label + a teaser question for its overlay) and returns, per
* candidate, its highlight-worthiness and a ready, honest overlay. A celebration or a scored goal rates
* high; a loud turn-away or an "about to…" build rates low meaning, not motion. The director blends these
* ratings with measured intensity to decide which moments make the reel and how to label each. Fail-soft:
* returns an empty list on any error, so the director selects by measured intensity alone.
*/
public List<HighlightMontageDirector.Judgement> judgeMoments(Path source, List<Double> candidateTimes,
Path workDir) {
if (candidateTimes == null || candidateTimes.isEmpty()) {
return List.of();
}
try {
Files.createDirectories(workDir);
List<Manifest> manifest = new ArrayList<>();
List<Integer> ids = new ArrayList<>();
for (int i = 0; i < candidateTimes.size(); i++) {
Path frame = workDir.resolve("cand-" + i + ".jpg");
if (extractFrame(source, candidateTimes.get(i), frame)) {
String img = frame.toAbsolutePath().toString();
manifest.add(new Manifest("d" + i, img, DESCRIPTIVE_QUESTION));
manifest.add(new Manifest("l" + i, img,
"In one to three words, what is the exciting action in this moment?"));
manifest.add(new Manifest("q" + i, img,
"In three to five words, ask a suspenseful question about what happens next."));
ids.add(i);
}
}
if (ids.isEmpty()) {
return List.of();
}
List<Caption> caps = runCaptioner(workDir, manifest);
List<HighlightMontageDirector.Judgement> out = new ArrayList<>();
for (int i = 0; i < candidateTimes.size(); i++) {
if (!ids.contains(i)) {
out.add(new HighlightMontageDirector.Judgement(0.0, "")); // unreadable frame
continue;
}
String description = answerFor(caps, "d" + i);
double worthiness = highlightWorthiness(description);
String overlay = honestOverlayText(description, answerFor(caps, "l" + i), answerFor(caps, "q" + i));
out.add(new HighlightMontageDirector.Judgement(worthiness, overlay));
}
log.info("event=highlight_vision_moments_judged candidates={} rated={}", candidateTimes.size(), ids.size());
return out;
} catch (RuntimeException | IOException | InterruptedException ex) {
if (ex instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
log.warn("event=highlight_vision_director_failed stage=judge error_type={} message={}",
ex.getClass().getSimpleName(), ex.getMessage());
return List.of();
}
}
/** Highlight-worthiness of a described moment: anticipation is NOT the payoff, so it scores low. */
static double highlightWorthiness(String description) {
if (isAnticipatory(description)) {
return 0.15; // "about to kick" / "walking up to the ball" is the build-up, never the highlight
}
return semanticScore(description);
}
/** True when a description reads as anticipation ("about to…") rather than a shown/ongoing action. */
static boolean isAnticipatory(String description) {
if (description == null || description.isBlank()) {
return false;
}
String d = description.toLowerCase(Locale.ROOT);
String[] anticipation = {"about to", "preparing", "getting ready", "ready to", "going to",
"approaching", "approaches", "walking up", "walking toward", "walking towards", "lining up",
"waiting", "prepares to", "poised", "is going to", "sets up", "setting up", "before"};
return containsAny(d, anticipation);
}
/**
* The honest overlay: assert the action label only when the frame actually SHOWS the action; when it only
* shows anticipation, pose the grounded teaser question instead never claim an action that is not on
* screen (a "KICK" overlay over a cut that stops before the kick). Falls back to a generic teaser when the
* model gave no usable question.
*/
static String honestOverlayText(String description, String label, String teaser) {
if (isAnticipatory(description)) {
String q = toOverlayText(teaser);
if (q.isBlank()) {
q = "WHAT HAPPENS NEXT";
}
return q.endsWith("?") ? q : q + "?";
}
return toOverlayText(label);
}
/**
* A per-window "highlight-worthiness" curve from timeline captions (each window takes the nearest
* caption's semantic score). Feeds the montage director's payoff selection.
*/
public static double[] semanticCurve(List<TimedCaption> captions, double windowSeconds, double duration) {
int size = Math.max(1, (int) Math.ceil(duration / windowSeconds));
double[] curve = new double[size];
if (captions == null || captions.isEmpty()) {
return curve;
}
for (int i = 0; i < size; i++) {
double t = (i + 0.5) * windowSeconds;
TimedCaption nearest = captions.stream()
.min(java.util.Comparator.comparingDouble(c -> Math.abs(c.timeSeconds() - t)))
.orElse(null);
curve[i] = nearest == null ? 0.0 : semanticScore(nearest.description());
}
return curve;
}
/**
* Scores a caption for how much it looks like a highlight payoff. Emotion/celebration ranks highest, then
* action, then idle/setup. Keyword heuristic (deliberately simple and offline).
*/
static double semanticScore(String caption) {
if (caption == null || caption.isBlank()) {
return 0.4;
}
String c = caption.toLowerCase(Locale.ROOT);
// Keep keywords specific enough not to false-match (e.g. bare "win" hides inside "throwing").
String[] payoff = {"celebrat", "cheer", "victor", "triumph", "raising", "arms up", "arms rais",
"excit", "happy", "applau", "fist", "champion", "thumbs", "dancing", "jumping"};
String[] action = {"throw", "roll", "release", "swing", "kick", "shoot", "spin", "sliding"};
String[] idle = {"stand", "wait", "empty", "prepare", "background", "looking", "watching", "sitting",
"walk", "turning", "turns", "turned", "away", "leaving", "adjust"};
if (containsAny(c, payoff)) {
return 1.0;
}
if (containsAny(c, action)) {
return 0.65;
}
if (containsAny(c, idle)) {
return 0.2;
}
return 0.4;
}
private static boolean containsAny(String haystack, String[] needles) {
for (String needle : needles) {
if (haystack.contains(needle)) {
return true;
}
}
return false;
}
/** The strongest slow-motion shot is the payoff; -1 if there is none. */
static int payoffShotIndex(MontagePlan plan) {
int best = -1;
double slowest = 1.0;
List<MontagePlan.Shot> shots = plan.shots();
for (int i = 0; i < shots.size(); i++) {
if (shots.get(i).speed() < slowest) {
slowest = shots.get(i).speed();
best = i;
}
}
return best;
}
static double[] payoffTimeline(MontagePlan plan, int index) {
double start = 0;
for (int i = 0; i < index; i++) {
start += plan.shots().get(i).durationSeconds();
}
return new double[]{start, start + plan.shots().get(index).durationSeconds()};
}
/**
* Cleans a model answer into a short, bold overlay: strips punctuation, keeps up to three words, upper-cases.
* Returns "" when the answer is empty or unusably long/gibberish.
*/
static String toOverlayText(String answer) {
if (answer == null) {
return "";
}
String cleaned = answer.replaceAll("[\"'.!?,:;]", " ").trim();
if (cleaned.isBlank()) {
return "";
}
String[] words = cleaned.split("\\s+");
StringBuilder sb = new StringBuilder();
for (int i = 0; i < Math.min(3, words.length); i++) {
if (words[i].length() > 18) {
continue;
}
if (sb.length() > 0) {
sb.append(' ');
}
sb.append(words[i]);
}
return sb.toString().toUpperCase(Locale.ROOT).trim();
}
private boolean extractFrame(Path source, double atSeconds, Path output) throws IOException, InterruptedException {
List<String> command = List.of(ffmpegBinary, "-hide_banner", "-loglevel", "error",
"-y", "-ss", Double.toString(Math.max(0.0, atSeconds)), "-i", source.toString(),
"-frames:v", "1", output.toString());
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
process.getInputStream().readAllBytes();
return process.waitFor() == 0 && Files.isRegularFile(output);
}
private List<Caption> runCaptioner(Path workDir, List<Manifest> manifest)
throws IOException, InterruptedException {
Path manifestFile = workDir.resolve("vision-manifest.json");
Path outputFile = workDir.resolve("vision-captions.json");
objectMapper.writeValue(manifestFile.toFile(), manifest);
List<String> command = List.of(worker.getPythonBinary(), captionScript,
"--manifest", manifestFile.toString(), "--output", outputFile.toString());
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
boolean finished = process.waitFor(TIMEOUT_SECONDS, java.util.concurrent.TimeUnit.SECONDS);
if (!finished) {
process.destroyForcibly();
throw new IllegalStateException("vision caption worker timed out");
}
if (process.exitValue() != 0 || !Files.isRegularFile(outputFile)) {
throw new IllegalStateException("vision caption worker failed: " + out.strip());
}
Caption[] captions = objectMapper.readValue(outputFile.toFile(), Caption[].class);
return List.of(captions);
}
record Manifest(String id, String image, String question) {
}
record Caption(String id, String answer) {
}
private static String answerFor(List<Caption> captions, String id) {
return captions.stream().filter(c -> id.equals(c.id())).map(Caption::answer)
.filter(java.util.Objects::nonNull).findFirst().orElse("");
}
/**
* Captions for a specific moment (seconds): a full {@code description} (used to score the moment and flavor
* the music) and a punchy {@code label} (used for the overlay text).
*/
public record TimedCaption(double timeSeconds, String description, String label) {
}
}

View File

@ -28,15 +28,17 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
private final EditProjectStore store;
private final EditAssetProvider provider;
private final EditAssetLibrary library;
private final LocalAssetSynthesizer synthesizer;
private final Path voiceoverCacheRoot;
private final ObjectMapper objectMapper;
public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store,
EditAssetProvider provider, EditAssetLibrary library,
ObjectMapper objectMapper) {
LocalAssetSynthesizer synthesizer, ObjectMapper objectMapper) {
this.store = store;
this.provider = provider;
this.library = library;
this.synthesizer = synthesizer;
this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize();
this.objectMapper = objectMapper;
}
@ -59,20 +61,24 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
.filter(cue -> "music".equals(cue.type()))
.findFirst();
if (musicCue.isPresent()) {
items.add(materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory, requestsDirectory));
AssetGenerationItem item = materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory,
requestsDirectory);
items.add(item);
readyForRender &= item.reused();
}
if (!plan.voiceover().isEmpty()) {
items.add(materializeVoiceover(projectId, plan, projectAudioDirectory, requestsDirectory));
AssetGenerationItem item = materializeVoiceover(projectId, plan, projectAudioDirectory,
requestsDirectory);
items.add(item);
readyForRender &= item.reused();
}
for (AudioCue cue : plan.audioCues()) {
if ("sfx".equals(cue.type())) {
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
items.add(item);
if (!item.reused()) {
readyForRender = false;
}
readyForRender &= item.reused();
}
}
@ -92,7 +98,8 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
"music", cacheKey + ".wav");
Path projectTarget = projectAudioDirectory.resolve("music.wav");
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true,
List.of());
}
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
@ -103,7 +110,7 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
"voiceover lines=" + plan.voiceover().size(), requestsDirectory, plan.targetDurationSeconds(), false);
voiceoverText, requestsDirectory, plan.targetDurationSeconds(), true, plan.voiceover());
}
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
@ -115,15 +122,17 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
"sfx", cacheKey + ".wav");
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true,
List.of());
}
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
Path cachePath, Path projectTarget, String notes,
Path requestsDirectory, double durationSeconds, boolean blocking) {
Path requestsDirectory, double durationSeconds, boolean blocking,
List<VoiceoverLine> voiceoverLines) {
try {
createDirectory(cachePath.getParent());
if (Files.isRegularFile(cachePath)) {
if (AssetLicensePolicy.isLicensed(cachePath)) {
copy(cachePath, projectTarget);
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
@ -135,8 +144,6 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
if (selected.isPresent()) {
copy(Path.of(selected.get().path()), cachePath);
copy(Path.of(selected.get().path()), projectTarget);
copyIfMissing(Path.of(selected.get().path()).resolveSibling(Path.of(selected.get().path()).getFileName()
+ ".license.txt"), projectTarget.resolveSibling(projectTarget.getFileName() + ".license.txt"));
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
@ -151,6 +158,14 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
requestFile.toString(), notes));
Optional<Path> generated = generate(type, projectId, notes, durationSeconds, projectTarget,
voiceoverLines);
if (generated.isPresent()) {
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={} strategy=local-worker",
projectId, type, assetKey, cacheKey, generated.get(), projectTarget);
return new AssetGenerationItem(type, assetKey, cacheKey, true, generated.get().toString(),
projectTarget.toString(), requestFile.toString(), notes);
}
log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}",
projectId, type, assetKey, cacheKey, requestFile);
return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
@ -166,7 +181,8 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
}
if ("voiceover".equals(type)) {
return provider.list(EditAssetType.VOICEOVER, null).stream().findFirst();
return provider.resolve(new EditAssetRequest(EditAssetType.VOICEOVER, assetKey, null,
durationSeconds, notes));
}
if ("sfx".equals(type)) {
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
@ -179,6 +195,20 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
return Optional.empty();
}
private Optional<Path> generate(String type, String projectId, String notes, double durationSeconds,
Path target, List<VoiceoverLine> voiceoverLines) {
if ("music".equals(type)) {
return synthesizer.synthesizeMusic(projectId, notes, durationSeconds, target);
}
if ("voiceover".equals(type)) {
return synthesizer.synthesizeVoiceover(projectId, voiceoverLines, target);
}
if ("sfx".equals(type)) {
return synthesizer.synthesizeSfx(projectId, notes, durationSeconds, target);
}
return Optional.empty();
}
private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath,
Path projectTarget, String notes, double durationSeconds, boolean blocking) {
return """
@ -230,14 +260,7 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
if (source.normalize().equals(target.normalize())) {
return;
}
createDirectory(target.getParent());
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
}
private void copyIfMissing(Path source, Path target) throws IOException {
if (Files.isRegularFile(source) && !Files.exists(target)) {
copy(source, target);
}
AssetLicensePolicy.copy(source, target);
}
private void writeJson(Path path, Object value) {

View File

@ -0,0 +1,175 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalAssetRuntimeVerifier implements ApplicationRunner {
private static final Logger log = LoggerFactory.getLogger(LocalAssetRuntimeVerifier.class);
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
public LocalAssetRuntimeVerifier(VideoClippingProperties properties) {
this.worker = properties.getEditing().getLocalAssetWorker();
}
@Override
public void run(ApplicationArguments args) {
log.info("event=local_asset_runtime_check_started auto_start={} strict_runtime={} script={} python={} "
+ "bootstrap_script={} piper_binary={} piper_model_path={} music_model={} sfx_model={}",
worker.isAutoStart(), worker.isStrictRuntime(), worker.getScript(), worker.getPythonBinary(),
worker.getBootstrapScript(),
worker.getPiperBinary(), worker.getPiperModelPath(), worker.getMusicModel(), worker.getSfxModel());
if (!worker.isAutoStart()) {
log.info("event=local_asset_runtime_check_skipped reason=auto_start_disabled");
return;
}
RuntimeReadiness readiness = verifyRuntime();
if (!readiness.ready()) {
log.error("event=local_asset_runtime_check_failed missing={} voiceover_ready={} music_ready={} sfx_ready={}",
readiness.missing(), readiness.voiceoverReady(), readiness.musicReady(),
readiness.sfxReady());
if (worker.isStrictRuntime()) {
throw new IllegalStateException("Strict local asset runtime is not ready: " + readiness.missing());
}
return;
}
log.info("event=local_asset_runtime_check_completed status=ready voiceover_ready=true music_ready=true "
+ "sfx_ready=true python={} piper_binary={} piper_model_path={}",
worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
}
private RuntimeReadiness verifyRuntime() {
List<String> missing = new ArrayList<>();
boolean pythonPresent = Files.isRegularFile(Path.of(worker.getPythonBinary()));
boolean audioStackReady = pythonPresent && isPythonAudioStackReady();
if (!pythonPresent) {
missing.add("python_binary:" + worker.getPythonBinary());
}
if (!Files.isRegularFile(Path.of(worker.getScript()))) {
missing.add("worker_script:" + worker.getScript());
}
if (!isExecutablePresent(worker.getPiperBinary())) {
missing.add("piper_binary:" + worker.getPiperBinary());
}
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
missing.add("piper_model_path:missing");
} else if (!Files.isRegularFile(Path.of(worker.getPiperModelPath()))) {
missing.add("piper_model_path:" + worker.getPiperModelPath());
} else if (AssetLicensePolicy.read(Path.of(worker.getPiperModelPath())).isEmpty()) {
missing.add("piper_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getPiperModelPath())));
}
boolean musicModelReady = isLocalModelPath(worker.getMusicModel());
if (!musicModelReady) {
missing.add("music_model_path:" + worker.getMusicModel());
} else if (AssetLicensePolicy.read(Path.of(worker.getMusicModel())).isEmpty()) {
missing.add("music_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getMusicModel())));
musicModelReady = false;
}
boolean sfxModelReady = isLocalModelPath(worker.getSfxModel());
if (!sfxModelReady) {
missing.add("sfx_model_path:" + worker.getSfxModel());
} else if (AssetLicensePolicy.read(Path.of(worker.getSfxModel())).isEmpty()) {
missing.add("sfx_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getSfxModel())));
sfxModelReady = false;
}
if (!audioStackReady) {
missing.add("python_audio_stack:torch,audiocraft,soundfile,numpy");
}
boolean voiceoverReady = isVoiceoverReady(pythonPresent);
boolean musicReady = audioStackReady && musicModelReady;
boolean sfxReady = audioStackReady && sfxModelReady;
return new RuntimeReadiness(
voiceoverReady && musicReady && sfxReady,
voiceoverReady,
musicReady,
sfxReady,
List.copyOf(missing)
);
}
private boolean isVoiceoverReady(boolean pythonReady) {
if (!pythonReady) {
return false;
}
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
return false;
}
return Files.isRegularFile(Path.of(worker.getPiperModelPath()))
&& AssetLicensePolicy.read(Path.of(worker.getPiperModelPath())).isPresent()
&& isExecutablePresent(worker.getPiperBinary());
}
private boolean isLocalModelPath(String value) {
if (value == null || value.isBlank()) {
return false;
}
Path path = Path.of(value).normalize();
return Files.isRegularFile(path) || Files.isDirectory(path);
}
private boolean isPythonAudioStackReady() {
List<String> command = List.of(worker.getPythonBinary(), "-c",
"import torch, audiocraft, soundfile, numpy");
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
int exitCode = process.waitFor();
if (exitCode != 0 && !output.isBlank()) {
log.warn("event=local_asset_python_stack_check_failed python={} exit_code={} output={}",
worker.getPythonBinary(), exitCode, output);
}
return exitCode == 0;
} catch (IOException ex) {
log.warn("event=local_asset_python_stack_check_failed python={} error_type={} message={}",
worker.getPythonBinary(), ex.getClass().getSimpleName(), ex.getMessage());
return false;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_python_stack_check_failed python={} error_type={} message={}",
worker.getPythonBinary(), ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private boolean isExecutablePresent(String executable) {
try {
Process process = new ProcessBuilder(executable, "--help").redirectErrorStream(true).start();
process.waitFor();
return true;
} catch (IOException ex) {
return false;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
return false;
}
}
private record RuntimeReadiness(
boolean ready,
boolean voiceoverReady,
boolean musicReady,
boolean sfxReady,
List<String> missing
) {
private RuntimeReadiness {
missing = missing == null ? List.of() : List.copyOf(missing);
}
}
}

View File

@ -0,0 +1,202 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.UnsupportedAudioFileException;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalAssetSynthesizer {
private static final Logger log = LoggerFactory.getLogger(LocalAssetSynthesizer.class);
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
private final ProcessExecutor processExecutor;
@Autowired
public LocalAssetSynthesizer(VideoClippingProperties properties) {
this(properties, command -> new ProcessBuilder(command).redirectErrorStream(true).start());
}
LocalAssetSynthesizer(VideoClippingProperties properties, ProcessExecutor processExecutor) {
this.worker = properties.getEditing().getLocalAssetWorker();
this.processExecutor = processExecutor;
}
public Optional<Path> synthesizeVoiceover(String projectId, List<VoiceoverLine> lines, Path target) {
if (lines == null || lines.isEmpty()) {
return Optional.empty();
}
String prompt = String.join(System.lineSeparator(),
lines.stream().map(VoiceoverLine::text).toList());
return synthesize(projectId, "voiceover", prompt, target, Math.max(1.0, lines.size() * 2.5));
}
public Optional<Path> synthesizeMusic(String projectId, String prompt, double durationSeconds, Path target) {
return synthesize(projectId, "music", prompt, target, durationSeconds);
}
public Optional<Path> synthesizeSfx(String projectId, String prompt, double durationSeconds, Path target) {
return synthesize(projectId, "sfx", prompt, target, durationSeconds);
}
private Optional<Path> synthesize(String projectId, String kind, String prompt, Path target,
double durationSeconds) {
if (prompt == null || prompt.isBlank()) {
return Optional.empty();
}
Path promptFile = null;
try {
String model = selectedModel(kind);
if (model == null || model.isBlank() || !AssetLicensePolicy.isLicensed(Path.of(model))) {
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} "
+ "reason=model_license_missing",
projectId, kind, model, target);
return Optional.empty();
}
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=local-worker",
projectId, kind, model, target, durationSeconds);
Files.createDirectories(target.getParent());
promptFile = Files.createTempFile(target.getParent(),
sanitize(kind) + "-prompt-", ".txt");
Files.writeString(promptFile, prompt, StandardCharsets.UTF_8);
List<String> command = new ArrayList<>();
command.add(worker.getPythonBinary());
command.add(Path.of(worker.getScript()).toAbsolutePath().normalize().toString());
command.add(kind);
command.add("--prompt-file");
command.add(promptFile.toString());
command.add("--output");
command.add(target.toString());
command.add("--duration");
command.add(Double.toString(durationSeconds));
if ("music".equals(kind)) {
command.add("--model");
command.add(worker.getMusicModel());
} else if ("sfx".equals(kind)) {
command.add("--model");
command.add(worker.getSfxModel());
} else if ("voiceover".equals(kind) && !worker.getPiperModelPath().isBlank()) {
command.add("--model");
command.add(worker.getPiperModelPath());
command.add("--piper-binary");
command.add(worker.getPiperBinary());
}
if (runWorker(command, target) && hasAudibleSignal(target)) {
AssetLicensePolicy.recordGeneratedAsset(Path.of(model), target);
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=local-worker",
projectId, kind, model, target);
return Optional.of(target);
}
deleteInvalidOutput(target);
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=local_model_failed_or_inaudible",
projectId, kind, model, target);
return Optional.empty();
} catch (IOException ex) {
deleteInvalidOutput(target);
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
} finally {
if (promptFile != null) {
try {
Files.deleteIfExists(promptFile);
} catch (IOException ex) {
log.warn("event=local_asset_prompt_cleanup_failed path={} message={}", promptFile,
ex.getMessage());
}
}
}
}
private boolean runWorker(List<String> command, Path target) {
try {
log.info("event=local_asset_worker_invoked script={} target={}", worker.getScript(), target);
Process process = processExecutor.execute(command);
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
int exitCode = process.waitFor();
if (exitCode == 0 && Files.isRegularFile(target)) {
log.info("event=local_asset_worker_completed script={} target={} exit_code={}",
worker.getScript(), target, exitCode);
return true;
}
if (!output.isBlank()) {
log.warn("event=local_asset_worker_failed target={} output={}", target, output);
}
return false;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_worker_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private void deleteInvalidOutput(Path target) {
try {
Files.deleteIfExists(target);
} catch (IOException ex) {
log.warn("event=local_asset_invalid_output_cleanup_failed target={} message={}", target, ex.getMessage());
}
}
private String selectedModel(String kind) {
if ("music".equals(kind)) {
return worker.getMusicModel();
}
if ("sfx".equals(kind)) {
return worker.getSfxModel();
}
if ("voiceover".equals(kind)) {
return worker.getPiperModelPath();
}
return "unknown";
}
private boolean hasAudibleSignal(Path audioFile) {
if (!Files.isRegularFile(audioFile)) {
return false;
}
try (AudioInputStream input = AudioSystem.getAudioInputStream(audioFile.toFile())) {
byte[] buffer = new byte[8192];
int read;
while ((read = input.read(buffer)) != -1) {
for (int index = 0; index < read; index++) {
if (buffer[index] != 0) {
return true;
}
}
}
return false;
} catch (UnsupportedAudioFileException | IOException ex) {
log.warn("event=local_asset_voiceover_audibility_check_failed target={} error_type={} message={}",
audioFile, ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private String sanitize(String value) {
return value == null ? "asset" : value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9._-]+", "-");
}
@FunctionalInterface
interface ProcessExecutor {
Process execute(List<String> command) throws IOException;
}
}

View File

@ -143,7 +143,12 @@ public class LocalCvVisualAnalysisProvider implements VisualAnalysisProvider {
}
private static class JdkHttpExecutor implements HttpExecutor {
private final HttpClient httpClient = HttpClient.newHttpClient();
// Force HTTP/1.1: the default client negotiates HTTP/2 (h2c cleartext upgrade) for http://, which
// the HTTP/1.1-only local CV worker (uvicorn/h11) mishandles by dropping the request body,
// producing a 422. Pinning HTTP/1.1 makes the loopback call reliable.
private final HttpClient httpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.build();
@Override
public HttpResult execute(HttpCall call) throws IOException, InterruptedException {

View File

@ -37,7 +37,7 @@ public class LocalEditAssetLibrary implements EditAssetLibrary {
if (request.category() != null && request.category().equals(asset.category())) {
score += 100;
}
if (!"untracked-local-asset".equals(asset.license())) {
if (!AssetLicensePolicy.UNTRACKED.equals(asset.license())) {
score += 25;
}
Set<String> assetTags = asset.tags().stream()

View File

@ -36,6 +36,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
}
return candidateRoots(request.type(), request.category()).stream()
.flatMap(root -> resolveInRoot(root, request).stream())
.filter(asset -> AssetLicensePolicy.isLicensed(Path.of(asset.path())))
.findFirst();
}
@ -47,6 +48,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
return candidateRoots(type, category).stream()
.filter(Files::isDirectory)
.flatMap(root -> listRoot(type, root).stream())
.filter(asset -> AssetLicensePolicy.isLicensed(Path.of(asset.path())))
.sorted(Comparator.comparing(ResolvedEditAsset::assetKey))
.toList();
}
@ -86,16 +88,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
}
private String license(Path path) {
Path sidecar = path.resolveSibling(path.getFileName() + ".license.txt");
if (!Files.isRegularFile(sidecar)) {
return "untracked-local-asset";
}
try {
String license = Files.readString(sidecar).strip();
return license.isBlank() ? "untracked-local-asset" : license;
} catch (IOException ex) {
throw new IllegalStateException("Unable to read edit asset license sidecar: " + sidecar, ex);
}
return AssetLicensePolicy.read(path).orElse(AssetLicensePolicy.UNTRACKED);
}
private List<String> tags(Path path) {

View File

@ -0,0 +1,43 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.stereotype.Component;
import java.nio.file.Path;
import java.util.List;
@Component
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
+ "'${video-clipping.editing.voiceover-provider:local}' == 'local'")
public class LocalVoiceoverGenerator implements VoiceoverGenerator {
private final EditProjectStore store;
private final LocalAssetSynthesizer synthesizer;
public LocalVoiceoverGenerator(EditProjectStore store, LocalAssetSynthesizer synthesizer) {
this.store = store;
this.synthesizer = synthesizer;
}
@Override
public Path generateVoiceover(String projectId, List<VoiceoverLine> lines) {
Path script = store.projectDirectory(projectId).resolve("voiceover-script.txt");
Path target = store.projectDirectory(projectId).resolve("audio").resolve("voiceover.wav");
StringBuilder content = new StringBuilder("Voiceover script for project ").append(projectId).append('\n');
for (VoiceoverLine line : lines) {
content.append("[%.3f-%.3f] %s%nDelivery: %s%n%n".formatted(
line.timelineStartSeconds(), line.timelineEndSeconds(), line.text(), line.delivery()));
}
try {
java.nio.file.Files.createDirectories(script.getParent());
java.nio.file.Files.writeString(script, content.toString());
} catch (java.io.IOException ex) {
throw new IllegalStateException("Unable to write voiceover script for project: " + projectId, ex);
}
synthesizer.synthesizeVoiceover(projectId, lines, target)
.orElseThrow(() -> new IllegalStateException("Unable to synthesize voiceover for project: "
+ projectId));
return target;
}
}

View File

@ -0,0 +1,29 @@
package org.example.videoclips.editing;
import java.util.List;
/**
* A cinematic montage: many short shots pulled from anywhere in the source, sequenced and cut to a
* rhythm, rather than a few fixed 8-35s highlight windows. Rendered as a single continuous edit with one
* music bed. Present as {@code director/montage.json} to select the montage path.
*/
public record MontagePlan(
String projectId,
String sourceVideoFileName,
String grade, // grade keyword: opening | rising | hero (drives the look)
String musicDirection,
List<String> voiceover, // optional narration lines, distributed across the montage
List<Overlay> overlays, // optional titles
List<Shot> shots
) {
/**
* One shot. {@code durationSeconds} is how long it plays on the timeline; {@code speed} is the playback
* speed (0.5 = slow motion). The source span consumed is durationSeconds * speed. {@code zoom} is the
* punch-in framing (1.0 = full frame, 1.4 = tight).
*/
public record Shot(double sourceStartSeconds, double durationSeconds, double zoom, double speed) {
}
public record Overlay(String text, double timelineStartSeconds, double timelineEndSeconds, String placement) {
}
}

View File

@ -0,0 +1,87 @@
# Opt-in profile for the LOCAL cinematic-highlight Proof of Concept.
# Activate with: --spring.profiles.active=localpoc (or SPRING_PROFILES_ACTIVE=localpoc)
#
# This profile does NOT change base/production defaults. It points only at concrete, pre-provisioned
# local model paths, keeps render disabled + approval required, isolates PoC input/output, and never
# starts a network-capable bootstrap script.
#
# Prohibitions still in force: no rendering without an explicit approved.flag for the specific project,
# no external AI, no placeholder audio. Models load offline from models/ (the worker sets HF_HUB_OFFLINE).
video-clipping:
# Keep the other schedulers out of the PoC run.
folder-scheduler:
enabled: false
editing:
enabled: true
# Isolated PoC output tree (base output/ is left untouched).
highlight-project-directory: ./output/localpoc/highlight-projects
# Soft cross-dissolves between montage beats (0 = hard cuts). Softens the abrupt cut into the slow-mo payoff.
crossfade-seconds: 0.25
# R13 cut-to-the-music: snap cut boundaries onto the generated score's beat grid (needs the asset venv).
beat-sync-enabled: true
# R14 subject-tracking reframe: follow the detected subject (YOLO/CV venv) instead of a static centre crop.
subject-reframe-enabled: true
# Tier-2 vision JUDGE: prefer the stronger Qwen2.5-VL (llama.cpp/GGUF) backend. It falls back to moondream
# with a loud WARN if the binary env (LLAMACPP_SERVER_BIN / LLAMACPP_MTMD_BIN) or weights aren't provisioned
# — so this default never silently degrades. See docs/LOCAL-MODELS.md.
vision-caption-script: ./tools/vision_caption_llamacpp.py
assets:
# Empty/absent asset folders -> the pipeline generates assets with local models instead of copying.
music-folder: ./input/localpoc/assets/music
sfx-folder: ./input/localpoc/assets/sfx
fonts-folder: ./input/localpoc/assets/fonts
luts-folder: ./input/localpoc/assets/luts
voiceover-folder: ./output/localpoc/highlight-projects/_voiceover-cache
# Heuristic visual analysis for the first PoC (real CV worker requires the prohibited bootstrap
# script; wiring resident CV is a Phase 3 quality improvement). No network, no fallback ambiguity.
visual-analysis:
provider: heuristic
fallback-to-heuristic: false
local-cv-worker:
auto-start: false
# The other (multi-clip) local director scheduler is not part of the highlight PoC.
local-director:
enabled: false
# Local generative audio models — concrete pre-provisioned paths. Never auto-start the bootstrap.
local-asset-worker:
auto-start: false
strict-runtime: false
python-binary: ./.venv-local-asset/bin/python
script: ./tools/local_asset_worker.py
piper-binary: ./.venv-local-asset/bin/piper
piper-model-path: ./models/piper/en_US-lessac-medium/en_US-lessac-medium.onnx
music-model: ./models/musicgen-small/config.json
sfx-model: ./models/audioldm2/model_index.json
# Single-source highlight flow: ingest + candidate generation ON, render OFF, approval REQUIRED.
highlight-scheduler:
enabled: true
source-directory: ./input/localpoc/highlights/source
working-directory: ./input/localpoc/highlights/working
processed-directory: ./input/localpoc/highlights/processed
rejected-directory: ./input/localpoc/highlights/rejected
render-enabled: false
require-director-approval: true
# Tier-1 automatic director: compose director/montage.json from measured motion + audio after analysis.
auto-director-enabled: true
# Tier-2 vision director: augment the montage with a semantic overlay + scene-informed music (moondream2).
vision-director-enabled: true
max-highlights-per-source: 3
# No fixed highlight length — a highlight can be as long as the story needs (soft candidate hints only).
highlight-min-duration-seconds: 2
highlight-max-duration-seconds: 3600
logging:
level:
org.example.videoclips: INFO

View File

@ -30,11 +30,11 @@ video-clipping:
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:600}
output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920}
output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080}
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30}
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:24}
audio-sample-rate: ${VIDEO_EDITING_AUDIO_SAMPLE_RATE:48000}
video-bitrate: ${VIDEO_EDITING_VIDEO_BITRATE:12000k}
audio-bitrate: ${VIDEO_EDITING_AUDIO_BITRATE:192k}
voiceover-provider: ${VIDEO_EDITING_VOICEOVER_PROVIDER:noop}
voiceover-provider: ${VIDEO_EDITING_VOICEOVER_PROVIDER:local}
loudness-target-i: ${VIDEO_EDITING_LOUDNESS_TARGET_I:-16.0}
loudness-true-peak: ${VIDEO_EDITING_LOUDNESS_TRUE_PEAK:-1.5}
loudness-range: ${VIDEO_EDITING_LOUDNESS_RANGE:11.0}
@ -56,9 +56,22 @@ video-clipping:
local-cv-worker:
auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true}
script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:120000}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:0}
health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health}
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
local-asset-worker:
auto-start: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START:true}
strict-runtime: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_STRICT_RUNTIME:true}
bootstrap-script: ${VIDEO_EDITING_LOCAL_ASSET_BOOTSTRAP_SCRIPT:./tools/run_local_asset_worker.sh}
script: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_SCRIPT:./tools/local_asset_worker.py}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_STARTUP_WAIT_MS:0}
health-path: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_PATH:/health}
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
python-binary: ${VIDEO_EDITING_LOCAL_ASSET_PYTHON_BINARY:./.venv-local-asset/bin/python}
piper-binary: ${VIDEO_EDITING_LOCAL_ASSET_PIPER_BINARY:piper}
piper-model-path: ${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:}
music-model: ${VIDEO_EDITING_LOCAL_ASSET_MUSIC_MODEL:musicgen-small}
sfx-model: ${VIDEO_EDITING_LOCAL_ASSET_SFX_MODEL:audiogen-medium}
local-director:
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}
@ -78,12 +91,15 @@ video-clipping:
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:true}
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:false}
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:false}
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:true}
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35}
# Highlights are not restricted to a fixed length: a highlight may be as long as the story needs.
# These bounds are only soft hints for coverage-window candidate generation; the director plan
# validator no longer rejects a highlight on duration (only positive/finite duration + sane speed).
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:2}
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:3600}
logging:
level:

View File

@ -123,11 +123,15 @@ class EditProjectControllerTest {
mockMvc.perform(put("/v1/edit-projects/{projectId}/plan", projectId)
.contentType(MediaType.APPLICATION_JSON)
.content("""
{"style":"cinematic-porsche-promo","targetDurationSeconds":45,
{"style":"cinematic-porsche-promo","targetDurationSeconds":4,
"decisions":[{"clipId":"clip-1","sourceStartSeconds":0,"sourceEndSeconds":4,
"timelineStartSeconds":0,"timelineEndSeconds":4,"transitionIn":"cut",
"transitionOut":"cut","playbackSpeed":1,"visualTreatment":"grade","reason":"hero"}],
"audioCues":[],"voiceover":[],"renderProfile":"mp4-h264-aac-1080p"}
"audioCues":[{"type":"music","assetKey":"cinematic-bed","timelineStartSeconds":0,
"timelineEndSeconds":4,"gainDb":-12,"notes":"bed"}],"voiceover":[],
"renderProfile":"mp4-h264-aac-1080p","overlays":[{"text":"Hero shot",
"timelineStartSeconds":0,"timelineEndSeconds":4,"placement":"lower_center_safe",
"animation":"fade_in","reason":"title"}]}
"""))
.andExpect(status().isOk())
.andExpect(jsonPath("$.projectId").value(projectId));
@ -176,4 +180,11 @@ class EditProjectControllerTest {
.andExpect(status().isNotFound())
.andExpect(jsonPath("$.type").value("https://api.example.com/problems/not-found"));
}
@Test
void rejectsRenderWithoutApproval() throws Exception {
// No approved.flag exists for this project -> the render endpoint must refuse (409), not render.
mockMvc.perform(post("/v1/edit-projects/unapproved-" + UUID.randomUUID() + ":render"))
.andExpect(status().isConflict());
}
}

View File

@ -1,9 +1,11 @@
package org.example.videoclips.api;
import org.example.videoclips.api.dto.EditProjectResponse;
import org.example.videoclips.config.VideoClippingProperties;
import org.example.videoclips.editing.EditPlanService;
import org.example.videoclips.editing.EditProjectService;
import org.example.videoclips.editing.EditProjectStatus;
import org.example.videoclips.editing.EditProjectStore;
import org.example.videoclips.editing.EditRenderer;
import org.example.videoclips.editing.StoryboardPromptGenerator;
import org.junit.jupiter.api.Test;
@ -25,8 +27,10 @@ class EditProjectRenderControllerTest {
"input", "output", 60, "cinematic-porsche-promo", true, true, true,
Instant.EPOCH, Instant.EPOCH, null);
when(projects.getProject("project")).thenReturn(rendered);
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setRequireRenderApproval(false); // this test verifies delegation, not the gate
EditProjectController controller = new EditProjectController(projects, mock(StoryboardPromptGenerator.class),
mock(EditPlanService.class), renderer);
mock(EditPlanService.class), renderer, mock(EditProjectStore.class), properties);
Object response = controller.render("project");

View File

@ -10,7 +10,8 @@ import org.springframework.test.web.servlet.MvcResult;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.isOneOf;
import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.is;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
@ -106,7 +107,7 @@ class VideoAssetControllerTest {
mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId))
.andExpect(status().isOk())
.andExpect(jsonPath("$.assetId").value(assetId))
.andExpect(jsonPath("$.status", isOneOf("QUEUED", "RUNNING", "SUCCEEDED")));
.andExpect(jsonPath("$.status", anyOf(is("QUEUED"), is("RUNNING"), is("SUCCEEDED"))));
for (int attempt = 0; attempt < 20; attempt++) {
String statusBody = mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId))

View File

@ -34,11 +34,11 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(600);
assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920);
assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080);
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30);
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(24);
assertThat(properties.getEditing().getAudioSampleRate()).isEqualTo(48000);
assertThat(properties.getEditing().getVideoBitrate()).isEqualTo("12000k");
assertThat(properties.getEditing().getAudioBitrate()).isEqualTo("192k");
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("noop");
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("local");
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-16.0);
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-1.5);
assertThat(properties.getEditing().getLoudnessRange()).isEqualTo(11.0);
@ -67,6 +67,21 @@ class VideoClippingPropertiesTest {
.isEqualTo("/health");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(1000);
assertThat(properties.getEditing().getLocalAssetWorker().isAutoStart()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getBootstrapScript())
.isEqualTo("./tools/run_local_asset_worker.sh");
assertThat(properties.getEditing().getLocalAssetWorker().getScript())
.isEqualTo("./tools/local_asset_worker.py");
assertThat(properties.getEditing().getLocalAssetWorker().getStartupWaitMs()).isZero();
assertThat(properties.getEditing().getLocalAssetWorker().getHealthPath()).isEqualTo("/health");
assertThat(properties.getEditing().getLocalAssetWorker().getHealthCheckIntervalMs()).isEqualTo(1000);
assertThat(properties.getEditing().getLocalAssetWorker().isStrictRuntime()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getPythonBinary())
.isEqualTo("./.venv-local-asset/bin/python");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperBinary()).isEqualTo("piper");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperModelPath()).isEmpty();
assertThat(properties.getEditing().getLocalAssetWorker().getMusicModel()).isEqualTo("musicgen-small");
assertThat(properties.getEditing().getLocalAssetWorker().getSfxModel()).isEqualTo("audiogen-medium");
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
.isEqualTo("./input/editing/source");
@ -77,6 +92,8 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getHighlightScheduler().getSourceDirectory())
.isEqualTo("./input/highlights/source");
assertThat(properties.getEditing().getHighlightScheduler().getPollIntervalMs()).isEqualTo(5000);
assertThat(properties.getEditing().getHighlightScheduler().isRenderEnabled()).isFalse();
assertThat(properties.getEditing().getHighlightScheduler().isRequireDirectorApproval()).isTrue();
});
}
@ -114,6 +131,17 @@ class VideoClippingPropertiesTest {
"video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250",
"video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready",
"video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25",
"video-clipping.editing.local-asset-worker.auto-start=true",
"video-clipping.editing.local-asset-worker.bootstrap-script=/tmp/local-asset-bootstrap.sh",
"video-clipping.editing.local-asset-worker.script=/tmp/local-asset.py",
"video-clipping.editing.local-asset-worker.startup-wait-ms=250",
"video-clipping.editing.local-asset-worker.health-path=/asset-health",
"video-clipping.editing.local-asset-worker.health-check-interval-ms=50",
"video-clipping.editing.local-asset-worker.python-binary=/tmp/python",
"video-clipping.editing.local-asset-worker.piper-binary=/tmp/piper",
"video-clipping.editing.local-asset-worker.piper-model-path=/tmp/voice.onnx",
"video-clipping.editing.local-asset-worker.music-model=musicgen-large",
"video-clipping.editing.local-asset-worker.sfx-model=audiogen-large",
"video-clipping.editing.local-director.enabled=false",
"video-clipping.editing.local-director.source-directory=/tmp/source",
"video-clipping.editing.local-director.poll-interval-ms=9000",
@ -162,6 +190,22 @@ class VideoClippingPropertiesTest {
.isEqualTo("/ready");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(25);
assertThat(properties.getEditing().getLocalAssetWorker().isAutoStart()).isTrue();
assertThat(properties.getEditing().getLocalAssetWorker().getBootstrapScript())
.isEqualTo("/tmp/local-asset-bootstrap.sh");
assertThat(properties.getEditing().getLocalAssetWorker().getScript()).isEqualTo("/tmp/local-asset.py");
assertThat(properties.getEditing().getLocalAssetWorker().getStartupWaitMs()).isEqualTo(250);
assertThat(properties.getEditing().getLocalAssetWorker().getHealthPath()).isEqualTo("/asset-health");
assertThat(properties.getEditing().getLocalAssetWorker().getHealthCheckIntervalMs()).isEqualTo(50);
assertThat(properties.getEditing().getLocalAssetWorker().isStrictRuntime()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getPythonBinary()).isEqualTo("/tmp/python");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperBinary()).isEqualTo("/tmp/piper");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperModelPath())
.isEqualTo("/tmp/voice.onnx");
assertThat(properties.getEditing().getLocalAssetWorker().getMusicModel())
.isEqualTo("musicgen-large");
assertThat(properties.getEditing().getLocalAssetWorker().getSfxModel())
.isEqualTo("audiogen-large");
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);

View File

@ -0,0 +1,52 @@
package org.example.videoclips.editing;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class AssetLicensePolicyTest {
@TempDir
Path tempDir;
@Test
void copiesAssetAndPreservesLicenseEvidence() throws Exception {
Path source = tempDir.resolve("source.wav");
Files.writeString(source, "audio");
Files.writeString(AssetLicensePolicy.sidecar(source), "SPDX-License-Identifier: CC0-1.0\n");
Path target = tempDir.resolve("project/assets/music.wav");
AssetLicensePolicy.copy(source, target);
assertThat(Files.readString(target)).isEqualTo("audio");
assertThat(AssetLicensePolicy.read(target))
.contains("SPDX-License-Identifier: CC0-1.0");
}
@Test
void rejectsMissingAndUntrackedLicenseEvidence() throws Exception {
Path source = tempDir.resolve("source.wav");
Files.writeString(source, "audio");
assertThatThrownBy(() -> AssetLicensePolicy.copy(source, tempDir.resolve("target.wav")))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("license sidecar is required");
Files.writeString(AssetLicensePolicy.sidecar(source), AssetLicensePolicy.UNTRACKED);
assertThat(AssetLicensePolicy.isLicensed(source)).isFalse();
}
@Test
void requiresModelFileBeforeRecordingGeneratedAsset() throws Exception {
Path missingModel = tempDir.resolve("missing-model.bin");
Files.writeString(AssetLicensePolicy.sidecar(missingModel), "Apache-2.0");
assertThat(AssetLicensePolicy.isLicensed(missingModel)).isFalse();
}
}

View File

@ -135,8 +135,11 @@ class CinematicEditingIntegrationTest {
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
"cinematic grade", "integration sequence"))
.toList();
List<TextOverlay> overlays = List.of(new TextOverlay("Cinematic highlight", 0.0,
decisions.get(0).timelineEndSeconds(), "lower_center_safe", "fade_in", "integration title"));
return new EditPlan(projectId, "category-aware-cinematic-highlights",
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(),
overlays,
"mp4-h264-aac-320p", "Generated integration edit");
}

View File

@ -34,7 +34,7 @@ class ContactSheetGeneratorTest {
assertCommandPair(command.get(), "-i", "./clips/clip_00001.mp4");
assertCommandPair(command.get(), "-vf", "fps=1/2,scale=320:-1,tile=4x4");
assertCommandPair(command.get(), "-frames:v", "1");
assertThat(command.get().getLast()).endsWith("clip_00001.jpg");
assertThat(command.get().get(command.get().size() - 1)).endsWith("clip_00001.jpg");
}
@Test

View File

@ -32,9 +32,10 @@ class EditPlanServiceTest {
new ClipAnalysis("clip-1", "clip.mp4", 8, "h264", "aac", 1920, 1080, 30,
List.of(), null, null, 0, 0, 0)), List.of(), Instant.now()));
EditPlanService service = new EditPlanService(store, projects, new EditPlanValidator(store));
SaveEditPlanRequest request = new SaveEditPlanRequest("cinematic-porsche-promo", 60,
List<AudioCue> audioCues = List.of(new AudioCue("music", "cinematic-bed", 0, 4, -12, "bed"));
SaveEditPlanRequest request = new SaveEditPlanRequest("cinematic-porsche-promo", 4,
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "grade", "hero")),
List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
audioCues, List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
EditPlan saved = service.save("project", request);

View File

@ -53,9 +53,9 @@ class EditProjectAnalyzerTest {
assertThat(analysis.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z"));
assertThat(analysis.errors()).isEmpty();
assertThat(analysis.clips()).hasSize(1);
assertThat(analysis.clips().getFirst().thumbnails()).containsExactly("thumb-1.jpg");
assertThat(analysis.clips().getFirst().contactSheet()).isEqualTo("contact-sheet.jpg");
assertThat(analysis.clips().getFirst().proxyPath()).isEqualTo("proxy.mp4");
assertThat(analysis.clips().get(0).thumbnails()).containsExactly("thumb-1.jpg");
assertThat(analysis.clips().get(0).contactSheet()).isEqualTo("contact-sheet.jpg");
assertThat(analysis.clips().get(0).proxyPath()).isEqualTo("proxy.mp4");
assertThat(Files.exists(tempDir.resolve("projects/project-1/analysis.json"))).isTrue();
assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis);
}

View File

@ -121,6 +121,67 @@ class FfmpegClipInspectorTest {
}
}
@Test
void swapsDimensionsForRotatedPortraitSideData() {
AtomicReference<List<String>> command = new AtomicReference<>();
FfmpegClipInspector inspector = inspector(arguments -> {
command.set(arguments);
return new FfmpegClipInspector.ProcessResult(0, """
{
"streams": [
{
"codec_type": "video",
"codec_name": "h264",
"width": 1024,
"height": 576,
"r_frame_rate": "30/1",
"side_data_list": [ { "rotation": -90 } ]
}
],
"format": { "duration": "16.0" }
}
""");
});
ClipAnalysis analysis = inspector.inspect(Path.of("portrait.mp4"));
// Effective (display) orientation is portrait: stored landscape pixels are swapped.
assertThat(analysis.width()).isEqualTo(576);
assertThat(analysis.height()).isEqualTo(1024);
assertThat(command.get()).contains("stream_side_data=rotation", "stream_tags=rotate");
}
@Test
void swapsDimensionsForRotateTagAndIgnoresHalfTurn() {
FfmpegClipInspector inspector = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """
{
"streams": [ {
"codec_type": "video", "codec_name": "h264",
"width": 1080, "height": 1920, "r_frame_rate": "30/1",
"tags": { "rotate": "90" }
} ],
"format": { "duration": "5.0" }
}
"""));
ClipAnalysis rotated = inspector.inspect(Path.of("tag.mp4"));
assertThat(rotated.width()).isEqualTo(1920);
assertThat(rotated.height()).isEqualTo(1080);
FfmpegClipInspector upsideDown = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """
{
"streams": [ {
"codec_type": "video", "codec_name": "h264",
"width": 1920, "height": 1080, "r_frame_rate": "30/1",
"side_data_list": [ { "rotation": 180 } ]
} ],
"format": { "duration": "5.0" }
}
"""));
ClipAnalysis flipped = upsideDown.inspect(Path.of("flip.mp4"));
assertThat(flipped.width()).isEqualTo(1920);
assertThat(flipped.height()).isEqualTo(1080);
}
private FfmpegClipInspector inspector(FfmpegClipInspector.ProcessExecutor executor) {
return new FfmpegClipInspector(new VideoClippingProperties(), new ObjectMapper(), executor);
}

View File

@ -2,15 +2,22 @@ package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.Mockito.mock;
class FfmpegEditRendererTest {
@TempDir
Path tempDir;
@Test
void buildsNormalizedSegmentCommand() {
VideoClippingProperties properties = new VideoClippingProperties();
@ -51,7 +58,7 @@ class FfmpegEditRendererTest {
Path.of("voiceover.wav"), Path.of("final.mp4"));
assertThat(command).containsSubsequence("-i", "timeline.mp4", "-i", "music.wav", "-i", "voiceover.wav");
assertThat(command).contains("[1:a]volume=0.25[music_raw];[2:a]volume=1.0[voice];"
assertThat(command).contains("[1:a]volume=-12dB[music_raw];[2:a]volume=1.0[voice];"
+ "[music_raw][voice]sidechaincompress=threshold=0.045:ratio=8.0:attack=20:release=250[music];"
+ "[0:a][music][voice]amix=inputs=3:duration=first:dropout_transition=2,"
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
@ -67,7 +74,7 @@ class FfmpegEditRendererTest {
var command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), null,
Path.of("final.mp4"));
assertThat(command).contains("[1:a]volume=0.25[music_raw];[music_raw]anull[music];"
assertThat(command).contains("[1:a]volume=-12dB[music_raw];[music_raw]anull[music];"
+ "[0:a][music]amix=inputs=2:duration=first:dropout_transition=2,"
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
assertThat(command).noneMatch(argument -> argument.contains("sidechaincompress"));
@ -163,6 +170,45 @@ class FfmpegEditRendererTest {
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
}
@Test
void appliesMusicCueTimingAndGainToFinalMix() {
VideoClippingProperties properties = new VideoClippingProperties();
FfmpegEditRenderer renderer = new FfmpegEditRenderer(properties, mock(EditProjectStore.class),
mock(EditProjectService.class), mock(EditPlanValidator.class), command -> null);
AudioCue musicCue = new AudioCue("music", "scene-score", 2.0, 12.0, -7.5, "rising tension");
var command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), musicCue,
null, List.of(), Path.of("final.mp4"));
assertThat(command).anySatisfy(argument -> assertThat(argument)
.contains("[1:a]atrim=duration=10.0,asetpts=PTS-STARTPTS,volume=-7.5dB,")
.contains("adelay=2000|2000[music_raw]"));
}
@Test
void rejectsRenderWhenAnyRequestedCreativeAssetIsMissing() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
FfmpegEditRenderer renderer = new FfmpegEditRenderer(properties, mock(EditProjectStore.class),
mock(EditProjectService.class), mock(EditPlanValidator.class), command -> null);
EditPlan plan = planWithRequestedAudio();
assertThatThrownBy(() -> renderer.requireRequestedAssets(plan, tempDir))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("music")
.hasMessageContaining("voiceover")
.hasMessageContaining("sfx:impact");
Files.createDirectories(tempDir.resolve("sfx"));
Files.writeString(tempDir.resolve("music.wav"), "music");
Files.writeString(tempDir.resolve("voiceover.wav"), "voice");
Files.writeString(tempDir.resolve("sfx/impact.wav"), "impact");
Files.writeString(tempDir.resolve("music.wav.license.txt"), "approved-music");
Files.writeString(tempDir.resolve("voiceover.wav.license.txt"), "approved-voice");
Files.writeString(tempDir.resolve("sfx/impact.wav.license.txt"), "approved-sfx");
assertThatCode(() -> renderer.requireRequestedAssets(plan, tempDir)).doesNotThrowAnyException();
}
@Test
void buildsOverlayCommandWithSafeDrawTextFilters() {
VideoClippingProperties properties = new VideoClippingProperties();
@ -220,4 +266,15 @@ class FfmpegEditRendererTest {
assertThat(renderer.audioTempoFilter(0.25)).isEqualTo("atempo=0.5,atempo=0.5");
assertThat(renderer.audioTempoFilter(1.5)).isEqualTo("atempo=1.5");
}
private EditPlan planWithRequestedAudio() {
return new EditPlan("project", "cinematic", 10,
List.of(new EditDecision("clip", 0, 10, 0, 10, "cut", "cut", 1, "grade", "hero")),
List.of(
new AudioCue("music", "score", 0, 10, -8, "scene-fit score"),
new AudioCue("sfx", "impact", 2, 3, -6, "reveal")
),
List.of(new VoiceoverLine("Precision.", 1, 3, "controlled")),
"mp4-h264-aac-1080p", "summary");
}
}

View File

@ -46,12 +46,25 @@ class HighlightAssetPreparationServiceTest {
assertThat(result.requestFiles()).isNotEmpty();
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
assertThat(result.requestFiles()).anyMatch(path -> path.endsWith("voiceover-voiceover_001.md"));
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
.resolve("assets/requests"))) {
assertThat(files.count()).isGreaterThan(0);
}
}
@Test
void usesStableSfxCueKeysSharedWithTheRendererPlan() {
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening Reveal", 0.0, 12.0, 12.0,
"opening_hook", "premium cinematic grade", "low cinematic pulse",
"whoosh on reveal", List.of(), List.of(), "hold the opening beat");
assertThat(HighlightAssetPreparationService.plannedSfxCues(highlight))
.extracting(AudioCue::assetKey)
.containsExactly("whoosh_soft", "impact_hit");
}
private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());

View File

@ -0,0 +1,111 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.within;
class HighlightBeatSyncTest {
@TempDir
Path tempDir;
private final ObjectMapper mapper = new ObjectMapper();
private EditDecision shot(double tlStart, double tlEnd) {
return new EditDecision("clip", tlStart, tlEnd, tlStart, tlEnd, "cut", "cut", 1.0, "grade", "montage");
}
@Test
void snapsInternalBoundariesToNearestBeatsAndPreservesTotal() {
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper);
List<EditDecision> decisions = List.of(
shot(0.0, 1.7), shot(1.7, 5.6), shot(5.6, 6.4), shot(6.4, 9.26), shot(9.26, 10.7));
List<Double> beats = List.of(1.856, 3.0, 5.7067, 6.3573, 9.2587);
List<EditDecision> aligned = sync.align(decisions, beats, 12.0);
// internal boundaries moved onto the nearest beat (all within the default 0.18s tolerance)
assertThat(aligned.get(1).timelineStartSeconds()).isCloseTo(1.856, within(1e-3));
assertThat(aligned.get(2).timelineStartSeconds()).isCloseTo(5.7067, within(1e-3));
assertThat(aligned.get(3).timelineStartSeconds()).isCloseTo(6.3573, within(1e-3));
assertThat(aligned.get(4).timelineStartSeconds()).isCloseTo(9.2587, within(1e-3));
// endpoints are anchors: first start and total length are unchanged
assertThat(aligned.get(0).timelineStartSeconds()).isEqualTo(0.0);
assertThat(aligned.get(4).timelineEndSeconds()).isCloseTo(10.7, within(1e-3));
// adjacent shots stay contiguous (end == next start)
for (int i = 0; i < aligned.size() - 1; i++) {
assertThat(aligned.get(i).timelineEndSeconds())
.isCloseTo(aligned.get(i + 1).timelineStartSeconds(), within(1e-6));
}
}
@Test
void keepsOriginalCutWhenNoBeatIsWithinTolerance() {
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper);
List<EditDecision> decisions = List.of(shot(0.0, 3.0), shot(3.0, 6.0));
List<Double> farBeats = List.of(0.9, 5.4); // nearest to 3.0 is >0.18s away
List<EditDecision> aligned = sync.align(decisions, farBeats, 6.0);
assertThat(aligned).isSameAs(decisions); // nothing moved -> original list returned
}
@Test
void doesNotSnapWhenItWouldMakeAShotTooShort() {
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper);
// boundary at 1.0; a beat at 0.9 is within tolerance but would leave shot 0 only 0.9s (< 0.5? no).
// Use a beat that would collapse shot 1 below the 0.5s minimum instead.
List<EditDecision> decisions = List.of(shot(0.0, 1.0), shot(1.0, 1.4));
List<Double> beats = List.of(1.05); // shot 1 would become 0.35s (< 0.5) -> reject
List<EditDecision> aligned = sync.align(decisions, beats, 6.0);
assertThat(aligned).isSameAs(decisions);
}
@Test
void reDerivesSourceWindowFromNewTimelineDurationForSlowMoShots() {
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper);
// slow-mo shot (speed 0.5): 2.0s timeline consumes 1.0s of source.
EditDecision slow = new EditDecision("clip", 4.0, 5.0, 2.0, 4.0, "cut", "cut", 0.5, "grade", "montage");
List<EditDecision> decisions = List.of(shot(0.0, 2.0), slow);
List<Double> beats = List.of(1.9); // move the 2.0 boundary to 1.9
List<EditDecision> aligned = sync.align(decisions, beats, 20.0);
EditDecision alignedSlow = aligned.get(1);
double newDur = alignedSlow.timelineEndSeconds() - alignedSlow.timelineStartSeconds(); // 4.0 - 1.9 = 2.1
double srcSpan = alignedSlow.sourceEndSeconds() - alignedSlow.sourceStartSeconds();
assertThat(srcSpan).isCloseTo(newDur * 0.5, within(1e-3)); // source span tracks timeline * speed
}
@Test
void detectBeatsParsesToolJsonEvenWithLeadingLogLines() throws Exception {
Path audio = tempDir.resolve("music.wav");
Files.writeString(audio, "not really audio");
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper,
command -> "loading model...\n{\"tempo\": 120.0, \"beats\": [0.5, 1.0, 1.5], \"duration\": 2.0}\n");
assertThat(sync.detectBeats(audio)).containsExactly(0.5, 1.0, 1.5);
}
@Test
void detectBeatsFailsSoftToEmptyOnError() throws Exception {
Path audio = tempDir.resolve("music.wav");
Files.writeString(audio, "x");
HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper,
command -> {
throw new IllegalStateException("interpreter missing");
});
assertThat(sync.detectBeats(audio)).isEmpty();
}
}

View File

@ -0,0 +1,162 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightCandidateGeneratorTest {
@TempDir
Path tempDir;
@Test
void persistsRankedCoverageWindowsAndMarksFallbackEvidenceForReview() {
Fixture fixture = fixture();
HighlightSourceAnalysis analysis = analysis(
42,
List.of(new ShotSegment("shot_0001", 0, 42, 42, 0, 21)),
new SourceAudioAnalysis("source", true, -30, -5, -35, 0.5, List.of(
new AudioSection("a1", "silence", 0, 12, 12),
new AudioSection("a2", "unclassified_audio", 12, 42, 30)
), null),
new SourceVisualAnalysis("source", 0.5, 0.5, 0.25, 0.75,
"unknown_without_face_detector",
List.of(new VisualObjectLabel("car", 0.82, "metadata_heuristic")),
List.of(), "local_cv_failed_fallback_metadata_thumbnail_scene_heuristic")
);
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
assertThat(result.category()).isEqualTo(ContentCategory.GENERIC_VLOG);
assertThat(result.categoryConfidence()).isEqualTo(0.25);
assertThat(result.categoryReasons()).anyMatch(reason -> reason.contains("metadata fallback"));
assertThat(result.candidates()).hasSize(3)
.allSatisfy(candidate -> {
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(42);
assertThat(candidate.sourceEndSeconds() - candidate.sourceStartSeconds()).isBetween(8.0, 35.0);
assertThat(candidate.reasons()).anyMatch(reason -> reason.contains("inspect frames"));
});
assertThat(result.candidates().get(0).sourceStartSeconds()).isGreaterThanOrEqualTo(9.0);
assertThat(Files.exists(fixture.store().analysisDirectory("project-1").resolve("category.json"))).isTrue();
assertThat(Files.exists(fixture.store().analysisDirectory("project-1")
.resolve("highlight-candidates.json"))).isTrue();
HighlightCandidate[] persisted = fixture.store().readJson("project-1",
"analysis/highlight-candidates.json", HighlightCandidate[].class);
assertThat(persisted).containsExactlyElementsOf(result.candidates());
}
@Test
void usesIndependentLocalModelLabelButStillRequiresReview() {
Fixture fixture = fixture();
HighlightSourceAnalysis analysis = analysis(
24,
List.of(
new ShotSegment("shot_0001", 0, 12, 12, 0.4, 6),
new ShotSegment("shot_0002", 12, 24, 12, 0.8, 18)
),
fullAudio(24),
new SourceVisualAnalysis("source", 0.8, 0.7, 0.75, 0.8, "none",
List.of(new VisualObjectLabel("car", 0.91, "yolo")),
List.of("frame.jpg"), "local_cv_yolo_clip")
);
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
assertThat(result.category()).isEqualTo(ContentCategory.CAR_VLOG);
assertThat(result.categoryConfidence()).isEqualTo(0.91);
assertThat(result.categoryReasons()).anyMatch(reason -> reason.contains("Confirm the category"));
assertThat(result.candidates()).extracting(HighlightCandidate::sourceStartSeconds)
.containsExactly(12.0, 0.0);
}
@Test
void expandsShortShotsWithoutLeavingSourceBounds() {
Fixture fixture = fixture();
HighlightSourceAnalysis analysis = analysis(
20,
List.of(
new ShotSegment("shot_0001", 0, 2, 2, 0.2, 1),
new ShotSegment("shot_0002", 18, 20, 2, 0.9, 19)
),
fullAudio(20),
new SourceVisualAnalysis("source", 0.7, 0.7, 0.7, 0.7, "unknown",
List.of(), List.of(), "local_cv")
);
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
assertThat(result.candidates()).hasSize(2)
.allSatisfy(candidate -> {
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(20);
assertThat(candidate.sourceEndSeconds() - candidate.sourceStartSeconds()).isEqualTo(8.0);
});
}
@Test
void removesDuplicateRangesCreatedByAdjacentShortShots() {
Fixture fixture = fixture();
HighlightSourceAnalysis analysis = analysis(
12,
List.of(
new ShotSegment("shot_0001", 0, 2, 2, 0.2, 1),
new ShotSegment("shot_0002", 2, 4, 2, 0.9, 3)
),
fullAudio(12),
new SourceVisualAnalysis("source", 0.7, 0.7, 0.7, 0.7, "unknown",
List.of(), List.of(), "local_cv")
);
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
assertThat(result.candidates()).hasSize(1);
assertThat(result.candidates().get(0).sourceStartSeconds()).isZero();
assertThat(result.candidates().get(0).sourceEndSeconds()).isEqualTo(8.0);
}
private Fixture fixture() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
properties, new ObjectMapper().findAndRegisterModules());
Instant now = Instant.parse("2026-07-21T10:00:00Z");
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.ANALYZING,
"source.mp4", store.projectDirectory("project-1").toString(), now, now, null);
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
HighlightFolderContract.standard(), now));
Clock clock = Clock.fixed(now, ZoneOffset.UTC);
return new Fixture(store, new HighlightCandidateGenerator(properties, store, clock));
}
private HighlightSourceAnalysis analysis(
double duration,
List<ShotSegment> shots,
SourceAudioAnalysis audio,
SourceVisualAnalysis visual
) {
ClipAnalysis source = new ClipAnalysis("source", "source/source.mp4", duration, "h264", "aac",
1920, 1080, 30, List.of(), null, null, 0, 0, 0);
return new HighlightSourceAnalysis("project-1", "source.mp4", source, List.of(), null, null, null,
"analysis/scene-segments.json", shots, "analysis/audio-analysis.json", audio,
"analysis/visual-analysis.json", visual, Instant.parse("2026-07-21T10:00:00Z"));
}
private SourceAudioAnalysis fullAudio(double duration) {
return new SourceAudioAnalysis("source", true, -18, -3, -35, 0.5,
List.of(new AudioSection("a1", "unclassified_audio", 0, duration, duration)), null);
}
private record Fixture(FileSystemHighlightProjectStore store, HighlightCandidateGenerator generator) {
}
}

View File

@ -11,10 +11,13 @@ import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.when;
import static org.mockito.Mockito.verify;
class HighlightDirectorFlowServiceTest {
@ -29,6 +32,7 @@ class HighlightDirectorFlowServiceTest {
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
writeDirectorEvidence(store);
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
createDummyFile(source);
@ -54,21 +58,111 @@ class HighlightDirectorFlowServiceTest {
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
visualEffectsStage, assetPreparationService, assetWorker, renderer);
new HighlightDirectorPlanValidator(properties, store),
visualEffectsStage, assetPreparationService, assetWorker, renderer,
new HighlightBeatSync(properties, mapper), new HighlightSubjectTracker(properties, mapper));
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
assertThat(result.finalOutputs()).hasSize(1);
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
EditPlan persistedPlan = store.readJson(project.id(), "highlights/highlight_001/edit-plan.json",
EditPlan.class);
assertThat(persistedPlan.audioCues().stream()
.filter(cue -> "sfx".equals(cue.type()))
.map(AudioCue::assetKey))
.containsExactly("whoosh_soft", "impact_hit");
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.RENDERED);
}
@Test
void waitsForAssetsInsteadOfRenderingWhenLocalWorkerLeavesPendingRequests() throws Exception {
VideoClippingProperties properties = properties();
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
writeDirectorEvidence(store);
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
createDummyFile(source);
HighlightVisualEffectsStage visualEffectsStage = mock(HighlightVisualEffectsStage.class);
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
HighlightAssetPreparationService.HighlightAssetPreparationResult assetResult =
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
store.highlightsDirectory(project.id()).resolve("highlight_001"),
store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("assets"),
List.of("requests/voiceover.json"), List.of());
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(assetResult);
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
when(assetWorker.process(anyString(), any(), any())).thenReturn(
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
List.of(), List.of("requests/voiceover.json")));
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
new HighlightDirectorPlanValidator(properties, store),
visualEffectsStage, assetPreparationService, assetWorker, renderer,
new HighlightBeatSync(properties, mapper), new HighlightSubjectTracker(properties, mapper));
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
assertThat(result.reason()).isEqualTo("assets_pending");
assertThat(result.finalOutputs()).isEmpty();
verify(renderer, never()).render(anyString(), any(), any());
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.PLANNED);
}
@Test
void rejectsRenderWhenBlockingQaCheckFails() throws Exception {
VideoClippingProperties properties = properties();
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
writeDirectorEvidence(store);
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
createDummyFile(store.sourceDirectory(project.id()).resolve("source.mp4"));
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
store.highlightsDirectory(project.id()).resolve("highlight_001"),
store.highlightsDirectory(project.id()).resolve("highlight_001/assets"),
List.of(), List.of()));
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
when(assetWorker.process(anyString(), any(), any())).thenReturn(
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
List.of(), List.of()));
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
Path failedOutput = store.highlightsDirectory(project.id()).resolve("highlight_001/final.mp4");
when(renderer.render(anyString(), any(), any())).thenReturn(
new HighlightFfmpegRenderer.HighlightRenderResult("highlight_001", failedOutput, failedOutput,
List.of(), null, new RenderQaReport(project.id(), false,
List.of(new RenderQaCheck("audio_clipping", false, "ERROR", "peak=0.0")), Instant.now())));
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
new HighlightDirectorPlanValidator(properties, store), mock(HighlightVisualEffectsStage.class),
assetPreparationService, assetWorker, renderer, new HighlightBeatSync(properties, mapper),
new HighlightSubjectTracker(properties, mapper));
assertThatThrownBy(() -> service.process(project.id(), 1))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("audio_clipping");
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.FAILED);
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).doesNotExist();
}
private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
properties.getEditing().getHighlightScheduler().setRequireDirectorApproval(false);
return properties;
}
@ -102,6 +196,16 @@ class HighlightDirectorFlowServiceTest {
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
}
private void writeDirectorEvidence(FileSystemHighlightProjectStore store) {
HighlightCandidate candidate = new HighlightCandidate("candidate_001", "source", 0, 12, 0.7,
"opening_hook_candidate", List.of("reviewed"));
store.writeJson("highlight-project-001", "analysis/highlight-candidates.json",
new HighlightCandidate[] { candidate });
store.writeJson("highlight-project-001", "analysis/category.json", new CinematicHighlightAnalysis(
"highlight-project-001", ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"),
List.of(candidate), Instant.parse("2026-07-11T08:00:00Z")));
}
private void createDummyFile(Path file) throws Exception {
Files.createDirectories(file.getParent());
Files.writeString(file, "dummy");

View File

@ -0,0 +1,77 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
class HighlightDirectorPlanScannerTest {
@TempDir
Path tempDir;
@Test
void skipsTerminalAndInProgressProjectsWhenSelectingWork() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
properties, new ObjectMapper().findAndRegisterModules());
createProject(store, "a-failed", HighlightProjectStatus.FAILED);
createProject(store, "b-rendering", HighlightProjectStatus.RENDERING);
createProject(store, "c-ready", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
approve(store, "c-ready");
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
properties, store, mock(HighlightDirectorFlowService.class));
assertThat(scanner.findNextRenderableProject()).contains("c-ready");
}
@Test
void leavesFailedProjectTerminalInsteadOfRetryingIt() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
properties, new ObjectMapper().findAndRegisterModules());
createProject(store, "failed", HighlightProjectStatus.FAILED);
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
properties, store, mock(HighlightDirectorFlowService.class));
assertThat(scanner.findNextRenderableProject()).isEmpty();
}
@Test
void unapprovedProjectDoesNotStarveLaterApprovedProject() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
properties, new ObjectMapper().findAndRegisterModules());
createProject(store, "a-unapproved", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
createProject(store, "b-approved", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
approve(store, "b-approved");
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
properties, store, mock(HighlightDirectorFlowService.class));
assertThat(scanner.findNextRenderableProject()).contains("b-approved");
}
private void createProject(FileSystemHighlightProjectStore store, String projectId,
HighlightProjectStatus status) throws Exception {
Instant now = Instant.parse("2026-07-21T10:00:00Z");
store.createProject(new HighlightProject(projectId, projectId, status, "source.mp4",
store.projectDirectory(projectId).toString(), now, now,
status == HighlightProjectStatus.FAILED ? "failed" : null),
new HighlightProjectManifest(projectId, "source.mp4", HighlightFolderContract.standard(), now));
Files.writeString(store.directorDirectory(projectId).resolve("edit-plan.json"), "{}");
}
private void approve(FileSystemHighlightProjectStore store, String projectId) throws Exception {
Files.writeString(store.directorDirectory(projectId).resolve("approved.flag"), "approved");
}
}

View File

@ -0,0 +1,149 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.application.BadRequestException;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class HighlightDirectorPlanValidatorTest {
@TempDir
Path tempDir;
private VideoClippingProperties properties;
private FileSystemHighlightProjectStore store;
private HighlightDirectorPlanValidator validator;
@BeforeEach
void setUp() {
properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
Instant now = Instant.parse("2026-07-21T10:00:00Z");
store.createProject(new HighlightProject("project-1", "Project", HighlightProjectStatus.WAITING_FOR_DIRECTOR,
"source.mp4", store.projectDirectory("project-1").toString(), now, now, null),
new HighlightProjectManifest("project-1", "source.mp4", HighlightFolderContract.standard(), now));
ClipAnalysis source = new ClipAnalysis("source", "source.mp4", 30, "h264", "aac", 1920, 1080, 30,
List.of(), null, null, 0, 0, 0);
store.writeJson("project-1", "analysis/source-analysis.json", new HighlightSourceAnalysis(
"project-1", "source.mp4", source, List.of(), null, null, null,
"analysis/scene-segments.json", List.of(), "analysis/audio-analysis.json",
new SourceAudioAnalysis("source", true, -18, -3, -35, 0.5, List.of(), null),
"analysis/visual-analysis.json", new SourceVisualAnalysis("source", 0.5, 0.5, 0.5, 0.5,
"unknown", List.of(), List.of(), "metadata_thumbnail_scene_heuristic"), now));
HighlightCandidate candidate = candidate();
store.writeJson("project-1", "analysis/highlight-candidates.json",
new HighlightCandidate[] { candidate });
store.writeJson("project-1", "analysis/category.json", new CinematicHighlightAnalysis(
"project-1", ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"),
List.of(candidate), now));
validator = new HighlightDirectorPlanValidator(properties, store);
}
@Test
void acceptsPlanGroundedInPersistedProjectCategoryAndCandidate() {
assertThat(validator.validate("project-1", plan(item("candidate_001", 2, 14))))
.isEqualTo(plan(item("candidate_001", 2, 14)));
}
@Test
void rejectsUnknownCandidateAndRangeOutsideCandidate() {
assertThatThrownBy(() -> validator.validate("project-1", plan(item("candidate_999", 2, 14))))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("Unknown highlight candidate");
assertThatThrownBy(() -> validator.validate("project-1", plan(item("candidate_001", 1, 14))))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("contained by candidate");
}
@Test
void rejectsMismatchedCategoryAndMissingProductionDirections() {
HighlightDirectorPlan wrongCategory = new HighlightDirectorPlan("project-1", "source.mp4", "car_vlog",
List.of(item("candidate_001", 2, 14)), "summary");
assertThatThrownBy(() -> validator.validate("project-1", wrongCategory))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("reviewed category analysis");
HighlightDirectorPlan.HighlightItem missingMusic = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening", 2, 14, 12,
"opening_hook", "cinematic grade", "", "whoosh at transition",
List.of("Grounded narration."), List.of(), "reviewed treatment");
assertThatThrownBy(() -> validator.validate("project-1", plan(missingMusic)))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("Music direction");
}
@Test
void rejectsUnsafeHighlightIdBeforeItCanBecomeAPath() {
HighlightDirectorPlan.HighlightItem unsafe = new HighlightDirectorPlan.HighlightItem(
"../outside", "candidate_001", "Opening", 2, 14, 12,
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
List.of("Grounded narration."), List.of(), "reviewed treatment");
assertThatThrownBy(() -> validator.validate("project-1", plan(unsafe)))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("safe identifiers");
}
@Test
void rejectsTargetDurationThatRequiresUnsupportedPlaybackSpeed() {
HighlightDirectorPlan.HighlightItem tooSlow = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening", 2, 4, 12,
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
List.of("Grounded narration."), List.of(), "reviewed treatment");
assertThatThrownBy(() -> validator.validate("project-1", plan(tooSlow)))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("playback speed outside 0.25..4.0");
}
@Test
void acceptsHighlightLongerThanTheFormerFixedMaximum() {
// Source span 2..14 (12s) stretched to a 40s target (0.3x slow-mo) would have been rejected by the
// old 35s cap. Highlights may now be as long as the story needs (speed must still be sane).
HighlightDirectorPlan.HighlightItem longHighlight = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Epic", 2, 14, 40,
"hero_payoff", "cinematic grade", "scene-fit score", "impact",
List.of("Grounded narration."), List.of(), "reviewed treatment");
assertThat(validator.validate("project-1", plan(longHighlight)))
.isEqualTo(plan(longHighlight));
}
@Test
void rejectsNonPositiveTargetDuration() {
HighlightDirectorPlan.HighlightItem zero = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening", 2, 14, 0,
"opening_hook", "cinematic grade", "scene-fit score", "impact",
List.of("Grounded narration."), List.of(), "reviewed treatment");
assertThatThrownBy(() -> validator.validate("project-1", plan(zero)))
.isInstanceOf(BadRequestException.class)
.hasMessageContaining("positive number of seconds");
}
private HighlightCandidate candidate() {
return new HighlightCandidate("candidate_001", "source", 2, 14, 0.7,
"opening_hook_candidate", List.of("reviewed"));
}
private HighlightDirectorPlan plan(HighlightDirectorPlan.HighlightItem item) {
return new HighlightDirectorPlan("project-1", "source.mp4", "generic_vlog", List.of(item), "summary");
}
private HighlightDirectorPlan.HighlightItem item(String candidateId, double start, double end) {
return new HighlightDirectorPlan.HighlightItem(
"highlight_001", candidateId, "Opening", start, end, 12,
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
List.of("Grounded narration."), List.of(), "reviewed treatment");
}
}

View File

@ -37,6 +37,8 @@ class HighlightDirectorPromptGeneratorTest {
.contains("analysis/highlight-candidates.json")
.contains("director/edit-plan.json")
.contains("car_vlog")
.contains("Category evidence status: `model_supported_review_required`")
.contains("Treat candidate scores as ranking hints")
.contains("candidate_001")
.contains("premium, powerful, precise");
assertThat(Files.readString(Path.of(files.briefPath())))

View File

@ -0,0 +1,444 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.within;
import static org.mockito.Mockito.mock;
class HighlightFfmpegRendererTest {
@TempDir
Path tempDir;
@Test
void appliesMusicCueTimingAndGain() {
HighlightFfmpegRenderer renderer = renderer();
AudioCue cue = new AudioCue("music", "local-score", 1.5, 9.5, -7, "match reveal");
List<String> command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), cue,
List.of(), List.of(), Path.of("final.mp4"));
assertThat(command).anySatisfy(argument -> assertThat(argument)
.contains("[1:a]atrim=duration=8.0,asetpts=PTS-STARTPTS,volume=-7.0dB,")
.contains("adelay=1500|1500[music_raw]"));
}
@Test
void rejectsMissingRequestedAssetsBeforeRendering() throws Exception {
HighlightFfmpegRenderer renderer = renderer();
EditPlan plan = planWithRequestedAudio();
assertThatThrownBy(() -> renderer.requireRequestedAssets(plan, tempDir))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("music")
.hasMessageContaining("voiceover")
.hasMessageContaining("sfx:impact");
Files.createDirectories(tempDir.resolve("music"));
Files.createDirectories(tempDir.resolve("voiceover"));
Files.createDirectories(tempDir.resolve("sfx"));
Files.writeString(tempDir.resolve("music/music.wav"), "music");
Files.writeString(tempDir.resolve("voiceover/voiceover_001.wav"), "voice");
Files.writeString(tempDir.resolve("sfx/impact.wav"), "impact");
Files.writeString(tempDir.resolve("music/music.wav.license.txt"), "approved-music");
Files.writeString(tempDir.resolve("voiceover/voiceover_001.wav.license.txt"), "approved-voice");
Files.writeString(tempDir.resolve("sfx/impact.wav.license.txt"), "approved-sfx");
assertThatCode(() -> renderer.requireRequestedAssets(plan, tempDir)).doesNotThrowAnyException();
}
@Test
void alignsEachVoiceoverLineToItsPlannedTimelineRange() {
HighlightFfmpegRenderer renderer = renderer();
List<HighlightFfmpegRenderer.VoiceoverInput> voiceovers = List.of(
new HighlightFfmpegRenderer.VoiceoverInput(Path.of("line-1.wav"),
new VoiceoverLine("First.", 1.25, 2.75, "controlled")),
new HighlightFfmpegRenderer.VoiceoverInput(Path.of("line-2.wav"),
new VoiceoverLine("Second.", 5.0, 7.0, "controlled"))
);
List<String> command = renderer.audioMixCommand(Path.of("timeline.mp4"), null, null,
voiceovers, List.of(), Path.of("final.mp4"));
assertThat(command).anySatisfy(argument -> assertThat(argument)
.contains("[1:a]atrim=duration=1.5")
.contains("adelay=1250|1250[voice_line_0]")
.contains("[2:a]atrim=duration=2.0")
.contains("adelay=5000|5000[voice_line_1]")
.contains("[voice_line_0][voice_line_1]amix=inputs=2:duration=longest"));
}
@Test
void appliesBeatSpecificGradePerStoryPurpose() {
HighlightFfmpegRenderer renderer = renderer();
EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "premium grade", "notes");
String opening = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_opening_hook", 0, 2, Path.of("o.mp4")));
String rising = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_rising_energy", 0, 2, Path.of("r.mp4")));
String hero = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
// R11 filmic tone curve: every beat uses a filmic S-curve (lifted toe + rolled-off highlight
// shoulder) rather than a linear ramp, and the grade differs per story beat.
assertThat(opening).contains("curves=all='0/0.05 0.5/0.5 0.75/0.78 1/0.95'").contains("vignette=PI/10");
assertThat(rising).contains("curves=all='0/0.045 0.25/0.23 0.75/0.80 1/0.96'").contains("saturation=1.12");
assertThat(hero).contains("curves=all='0/0.055 0.25/0.24 0.75/0.82 1/0.97'").contains("saturation=1.16");
// Exposure-preserving: every beat lifts the black point (toe > 0), rolls off highlights (shoulder < 1),
// and never pushes gamma < 1 or brightness < 0.
assertThat(hero).contains("gamma=1.05").contains("brightness=0.016");
assertThat(opening).isNotEqualTo(hero);
}
@Test
void stylesOverlaysWithSoftShadowAndAlphaFade() {
HighlightFfmpegRenderer renderer = renderer();
String filter = renderer.overlayFilter(List.of(
new TextOverlay("First light.", 0.5, 3.0, "center_safe", "fade", "hook")));
assertThat(filter)
.contains("drawtext=text='First light.'")
.contains("fontsize=84") // bold
.contains("borderw=4:bordercolor=black@0.6") // thick outline
.contains("shadowcolor=black@0.7:shadowx=3:shadowy=3") // strong shadow
.contains("alpha='if(lt(t\\,0.5+0.18)") // snappy 0.18s punch-in
.contains("34*(1-min(1\\,(t-0.5)/0.22))") // rise-up entrance animation
.contains("enable='between(t\\,0.5\\,3.0)'");
}
@Test
void buildsMeasuredQaProbeCommands() {
HighlightFfmpegRenderer renderer = renderer();
assertThat(renderer.durationProbeCommand(Path.of("final.mp4")))
.containsSubsequence("-show_entries", "format=duration");
assertThat(renderer.blackDetectCommand(Path.of("final.mp4")))
.containsSubsequence("-vf", "blackdetect=d=0.5:pic_th=0.98");
assertThat(renderer.silenceDetectCommand(Path.of("final.mp4")))
.containsSubsequence("-af", "silencedetect=noise=-45dB:d=2");
assertThat(renderer.clippingDetectCommand(Path.of("final.mp4")))
.containsSubsequence("-af", "astats=metadata=1:reset=1");
}
@Test
void requiresMeasuredAudioPeakAndRejectsClipping() {
HighlightFfmpegRenderer renderer = renderer();
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
"[Parsed_astats] Peak level dB: -1.2"))).isTrue();
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
"[Parsed_astats] Peak level dB: 0.0"))).isFalse();
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
"no peak measurement"))).isFalse();
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(1,
"failed"))).isFalse();
}
@Test
void splitsVoiceSoDuckingDoesNotDropNarrationFromTheMix() {
HighlightFfmpegRenderer renderer = renderer();
AudioCue music = new AudioCue("music", "score", 0, 9, -14, "bed");
List<HighlightFfmpegRenderer.VoiceoverInput> voiceovers = List.of(
new HighlightFfmpegRenderer.VoiceoverInput(Path.of("vo.wav"),
new VoiceoverLine("Grounded line.", 2.7, 6.3, "narration")));
String filter = String.join(" ", renderer.audioMixCommand(Path.of("timeline.mp4"),
Path.of("music.wav"), music, voiceovers, List.of(), false, Path.of("final.mp4")));
// voice must be duplicated: one copy keys the duck, one copy stays in the amix
assertThat(filter)
.contains("[voice]asplit=2[voice_key][voice_mix];")
.contains("[music_raw][voice_key]sidechaincompress=")
.contains("[voice_mix]")
.doesNotContain("[music_raw][voice]sidechaincompress");
}
@Test
void mixesGeneratedAudioWithoutReferencingMissingSourceAudio() {
HighlightFfmpegRenderer renderer = renderer();
AudioCue music = new AudioCue("music", "score", 0, 10, -8, "scene-fit score");
List<String> command = renderer.audioMixCommand(Path.of("video-only.mp4"), Path.of("music.wav"), music,
List.of(), List.of(), false, Path.of("final.mp4"));
assertThat(command).noneMatch(argument -> argument.contains("[0:a]"));
assertThat(command).anySatisfy(argument -> assertThat(argument)
.contains("amix=inputs=1:duration=longest")
.contains("loudnorm=I=")
.contains(",alimiter=level=disabled:limit=0.72")
.contains(",apad[a]"));
assertThat(command).contains("-shortest");
}
@Test
void choosesPortraitGeometryForPortraitSourceAndLandscapeOtherwise() {
HighlightFfmpegRenderer renderer = renderer(); // configured 1920x1080
assertThat(renderer.outputGeometry(analysisWithDimensions(576, 1024)))
.containsExactly(1080, 1920);
assertThat(renderer.outputGeometry(analysisWithDimensions(1920, 1080)))
.containsExactly(1920, 1080);
assertThat(renderer.outputGeometry(null)).containsExactly(1920, 1080);
}
@Test
void portraitSegmentSkipsWidescreenLetterbox() {
HighlightFfmpegRenderer renderer = renderer();
EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "fade-in", "cut", 1.0,
"hero cinematic", "montage");
String landscape = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1920, 1080,
Path.of("l.mp4")));
String portrait = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1080, 1920,
Path.of("p.mp4")));
assertThat(landscape).contains("scale=1920:1080").contains("crop=1920:"); // 2.39 letterbox present
assertThat(portrait).contains("scale=1080:1920").doesNotContain("crop=1080:"); // no letterbox
}
@Test
void easesIntoSlowMotionOnlyForSlowMoShots() {
EditDecision normal = new EditDecision("c", 0, 4, 0, 4, "cut", "cut", 1.0, "hero", "montage");
EditDecision slow = new EditDecision("c", 9.2, 11.2, 0, 2.857, "cut", "cut", 0.7, "hero", "montage");
assertThat(HighlightFfmpegRenderer.speedRampSetpts(normal)).isEqualTo(",setpts=PTS/1.0");
String ramp = HighlightFfmpegRenderer.speedRampSetpts(slow);
assertThat(ramp).contains("setpts=").contains("log(").contains("(PTS-STARTPTS)*TB")
.doesNotContain("PTS/0.7"); // ramped, not a constant-speed setpts
}
@Test
void swellsTheMusicTowardThePayoffWhenAPeakIsGiven() {
HighlightFfmpegRenderer renderer = renderer();
AudioCue music = new AudioCue("music", "score", 0, 10, -9, "bed");
String withSwell = String.join(" ", renderer.audioMixCommand(Path.of("t.mp4"), Path.of("m.wav"), music,
List.of(), List.of(), true, 6.0, Path.of("f.mp4")));
String noSwell = String.join(" ", renderer.audioMixCommand(Path.of("t.mp4"), Path.of("m.wav"), music,
List.of(), List.of(), true, 0.0, Path.of("f.mp4")));
assertThat(withSwell).contains("volume='min(1\\,0.5+0.5*t/6.000)':eval=frame");
assertThat(noSwell).doesNotContain("eval=frame");
}
@Test
void crossfadeChainOverlapsSegmentsAndShiftsOverlaysOntoTheCompressedTimeline() {
HighlightFfmpegRenderer renderer = renderer();
List<Double> durations = List.of(2.0, 2.5, 3.0);
var segs = List.of(Path.of("s0.mp4"), Path.of("s1.mp4"), Path.of("s2.mp4"));
// Video and audio crossfade in SEPARATE passes (a combined graph starves/truncates the audio).
String video = String.join(" ", renderer.xfadeVideoCommand(segs, durations, 0.25, Path.of("v.mp4")));
assertThat(video).contains("xfade=transition=fade:duration=0.250:offset=1.750") // dur0 - xf
.contains("offset=4.000") // dur0+dur1 - 2*xf
.contains("[vout]").contains("-an").doesNotContain("acrossfade");
String audio = String.join(" ", renderer.acrossfadeAudioCommand(segs, 0.25, Path.of("a.m4a")));
assertThat(audio).contains("acrossfade=d=0.250").contains("[aout]").contains("-vn")
.doesNotContain("xfade=transition");
// An overlay on the 3rd beat (starts 4.5s uncompressed) shifts back by 2 transitions * xf.
TextOverlay overlay = new TextOverlay("STRIKE", 4.5, 6.5, "lower_center_safe", "fade", "montage");
TextOverlay shifted = HighlightFfmpegRenderer.shiftOverlayForCrossfade(overlay, durations, 0.25);
assertThat(shifted.timelineStartSeconds()).isCloseTo(4.0, within(0.001));
assertThat(shifted.timelineEndSeconds()).isCloseTo(6.0, within(0.001));
// Disabled by default (crossfade-seconds defaults to 0).
assertThat(renderer.crossfadeDuration(durations)).isZero();
}
@Test
void loudnessMasteringCorrectsTowardTargetOnlyWhenNeeded() {
// Too quiet -> positive boost; too loud -> negative cut; on-target and implausible -> no change.
assertThat(HighlightFfmpegRenderer.loudnessGainDb(-18.7, -16.0)).isCloseTo(2.7, within(0.001));
assertThat(HighlightFfmpegRenderer.loudnessGainDb(-14.0, -16.0)).isCloseTo(-2.0, within(0.001));
assertThat(HighlightFfmpegRenderer.loudnessGainDb(-16.3, -16.0)).isZero(); // within tolerance
assertThat(HighlightFfmpegRenderer.loudnessGainDb(-40.0, -16.0)).isZero(); // implausible -> skip
assertThat(HighlightFfmpegRenderer.loudnessGainDb(Double.NaN, -16.0)).isZero();
}
@Test
void exposureNormalizationAdaptsToMeasuredSourceBrightness() {
HighlightFfmpegRenderer renderer = renderer();
// A dark source is brightened (gamma > 1).
String dark = renderer.exposureNormalizationFilter(60);
assertThat(dark).startsWith(",eq=gamma=");
assertThat(Double.parseDouble(dark.replace(",eq=gamma=", ""))).isGreaterThan(1.0);
// A well-exposed source is left alone.
assertThat(renderer.exposureNormalizationFilter(120)).isEmpty();
// An over-bright source is darkened (gamma < 1).
String bright = renderer.exposureNormalizationFilter(215);
assertThat(Double.parseDouble(bright.replace(",eq=gamma=", ""))).isLessThan(1.0);
// Unmeasurable source -> no adjustment.
assertThat(renderer.exposureNormalizationFilter(-1)).isEmpty();
}
@Test
void pushInAmountIsStrongerForLowMotionShots() {
HighlightFfmpegRenderer renderer = renderer();
double staticShot = renderer.pushInAmount(2.6); // near-static (held)
double activeShot = renderer.pushInAmount(8.0); // lots of motion
assertThat(staticShot).isGreaterThan(activeShot);
assertThat(staticShot).isBetween(0.03, 0.12);
assertThat(activeShot).isGreaterThanOrEqualTo(0.03); // floor: always a little life
assertThat(renderer.pushInAmount(-1)).isGreaterThan(0.0); // unknown -> mild default push
}
@Test
void appliesMotionAdaptivePushInOnlyWhenRequested() {
HighlightFfmpegRenderer renderer = renderer();
EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "cut", "cut", 1.0, "hero cinematic",
"montage");
String withPush = String.join(" ", renderer.segmentCommand("s.mp4", decision,
"style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.10, Path.of("p.mp4")));
String noPush = String.join(" ", renderer.segmentCommand("s.mp4", decision,
"style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.0, Path.of("n.mp4")));
assertThat(withPush).contains("zoompan=z=");
assertThat(noPush).doesNotContain("zoompan").contains("crop=iw/"); // static crop preserved at 0
}
private HighlightSourceAnalysis analysisWithDimensions(int width, int height) {
ClipAnalysis source = new ClipAnalysis("clip", "s.mp4", 16, "h264", "aac", width, height, 30,
List.of(), null, null, 0, 0, 0);
return new HighlightSourceAnalysis("p", "s.mp4", source, List.of(), null, null, null, null,
List.of(), null, null, null, null, null);
}
@Test
void rendersAt24fpsCinematicCadence() {
HighlightFfmpegRenderer renderer = renderer();
EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes");
List<String> command =
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"));
assertThat(command).containsSequence("-r", "24");
// 420p pinned so filter format negotiation can't leave the output as browser-incompatible 444 H.264.
assertThat(command).containsSequence("-pix_fmt", "yuv420p");
}
@Test
void pinsYuv420pOnEveryVideoEncodePass() {
HighlightFfmpegRenderer renderer = renderer();
var segs = List.of(Path.of("s0.mp4"), Path.of("s1.mp4"));
assertThat(renderer.xfadeVideoCommand(segs, List.of(2.0, 2.0), 0.25, Path.of("v.mp4")))
.containsSequence("-pix_fmt", "yuv420p");
assertThat(renderer.overlayCommand(Path.of("in.mp4"), List.of(), Path.of("o.mp4")))
.containsSequence("-pix_fmt", "yuv420p");
assertThat(renderer.previewCommand(Path.of("in.mp4"), Path.of("p.mp4")))
.containsSequence("-pix_fmt", "yuv420p");
}
@Test
void appliesMotionBlurWhenEnabledAndOmitsWhenDisabled() {
VideoClippingProperties on = new VideoClippingProperties();
on.getEditing().setCinematicMotionBlur(true);
VideoClippingProperties off = new VideoClippingProperties();
off.getEditing().setCinematicMotionBlur(false);
EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes");
String withBlur = String.join(" ", renderer(on)
.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
String noBlur = String.join(" ", renderer(off)
.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
assertThat(withBlur).contains("tmix=frames=2:weights='1 1'");
assertThat(noBlur).doesNotContain("tmix");
}
@Test
void filmLutFilterIsEmptyByDefaultAndAppliedOnlyWhenLicensed() throws Exception {
assertThat(renderer().filmLutFilter()).isEmpty();
Path lut = tempDir.resolve("kodak2383.cube");
Files.writeString(lut, "LUT_3D_SIZE 2\n");
VideoClippingProperties props = new VideoClippingProperties();
props.getEditing().setFilmLutPath(lut.toString());
// No licence sidecar yet -> fail closed, built-in grade is used.
assertThat(renderer(props).filmLutFilter()).isEmpty();
Files.writeString(lut.resolveSibling("kodak2383.cube.license.txt"), "Licensed to Acme, commercial use.\n");
assertThat(renderer(props).filmLutFilter())
.startsWith(",lut3d='").contains("kodak2383.cube");
// When a LUT is active the built-in colour moves are suppressed (no double-grading).
EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes");
String cmd = String.join(" ", renderer(props)
.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
assertThat(cmd).contains("lut3d=").doesNotContain("colorbalance").doesNotContain("curves=all");
}
@Test
void parsesPanTokenIntoAnchors() {
HighlightFfmpegRenderer renderer = renderer();
double[][] pan = renderer.parsePan("zoom=1.050 cinematic pan=0.5400:0.6800;0.4444:0.6884");
assertThat(pan.length).isEqualTo(2);
assertThat(pan[0][0]).isCloseTo(0.54, within(1e-6));
assertThat(pan[0][1]).isCloseTo(0.68, within(1e-6));
assertThat(pan[1][0]).isCloseTo(0.4444, within(1e-6));
}
@Test
void subjectReframeFollowsPathInsteadOfCentredCrop() {
HighlightFfmpegRenderer renderer = renderer();
// treatment carries a pan path -> renderer must build a following zoompan, not the centred crop.
EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0,
"zoom=1.050 cinematic pan=0.5400:0.6800;0.4444:0.6884;0.6840:0.6723", "notes");
String cmd = String.join(" ",
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
assertThat(cmd).contains("zoompan").contains("min(iw-iw/zoom"); // clamped follow crop
assertThat(cmd).doesNotContain("iw/2-(iw/zoom/2)"); // not the centred formula
}
@Test
void subjectFollowFilterHoldsZoomConstantWithoutPushIn() {
HighlightFfmpegRenderer renderer = renderer();
String constant = renderer.subjectFollowFilter(new double[][]{{0.5, 0.5}}, 1.2, 0.0, 48, 1080, 1920);
String pushed = renderer.subjectFollowFilter(new double[][]{{0.5, 0.5}}, 1.2, 0.1, 48, 1080, 1920);
assertThat(constant).contains("zoompan=z='1.200'");
assertThat(pushed).contains("zoompan=z='min(1.200"); // animated zoom when pushing in
}
private HighlightFfmpegRenderer renderer() {
return renderer(new VideoClippingProperties());
}
private HighlightFfmpegRenderer renderer(VideoClippingProperties properties) {
return new HighlightFfmpegRenderer(properties, mock(HighlightProjectStore.class),
mock(EditAssetProvider.class), mock(EditObservability.class), command -> null);
}
private EditPlan planWithRequestedAudio() {
return new EditPlan("project", "cinematic", 10,
List.of(new EditDecision("clip", 0, 10, 0, 10, "cut", "cut", 1, "grade", "hero")),
List.of(
new AudioCue("music", "score", 0, 10, -8, "scene-fit score"),
new AudioCue("sfx", "impact", 2, 3, -6, "reveal")
),
List.of(new VoiceoverLine("Precision.", 1, 3, "controlled")),
"mp4-h264-aac-1080p", "summary");
}
}

View File

@ -11,6 +11,7 @@ import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@ -36,6 +37,7 @@ class HighlightLocalAssetWorkerTest {
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
Files.createDirectories(musicSource.getParent());
Files.writeString(musicSource, "music");
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.license.txt"), "approved-music");
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
@ -52,7 +54,8 @@ class HighlightLocalAssetWorkerTest {
"premium cinematic grade", "low pulse", "whoosh",
List.of("A clean first look."), List.of("Pure presence"), "open with presence");
prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG);
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary, mapper);
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary,
new LocalAssetSynthesizer(properties), mapper);
HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight,
ContentCategory.GENERIC_VLOG);
@ -61,4 +64,35 @@ class HighlightLocalAssetWorkerTest {
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
.exists();
}
@Test
void rejectsTamperedAssetRequestTargetOutsideProject() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.PLANNED,
"source.mp4", store.projectDirectory("project-1").toString(),
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
Path requests = store.highlightsDirectory("project-1").resolve("highlight_001/assets/requests");
Files.createDirectories(requests);
Path requestMarkdown = requests.resolve("music-music_bed.md");
Files.writeString(requestMarkdown, "request");
mapper.writeValue(requests.resolve("music-music_bed.json").toFile(), new HighlightAssetRequest(
"project-1", "highlight_001", "music", "music_bed",
tempDir.resolve("outside.wav").toString(), requestMarkdown.toString(),
"scene-fit score", 12, true));
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, mock(EditAssetProvider.class),
mock(EditAssetLibrary.class), mock(LocalAssetSynthesizer.class), mapper);
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening", 0, 12, 12, "opening_hook",
"grade", "score", "whoosh", List.of("Narration."), List.of(), "notes");
assertThatThrownBy(() -> worker.process("project-1", highlight, ContentCategory.GENERIC_VLOG))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("paths do not match");
assertThat(tempDir.resolve("outside.wav")).doesNotExist();
}
}

View File

@ -0,0 +1,161 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.example.videoclips.editing.HighlightMontageDirector.Judgement;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightMontageDirectorTest {
private final HighlightMontageDirector director = new HighlightMontageDirector(new VideoClippingProperties());
@Test
void proposesEveryRealMomentAsACandidateInTimeOrderWithNoCap() {
// Several distinct action bursts across a long clip: measurement must propose them ALL (no cap), in
// time order. It has no opinion on which is the highlight that is the judge's job.
int n = 160;
double[] motion = new double[n];
double[] audio = new double[n];
for (int i = 0; i < n; i++) {
double t = i * 0.5;
motion[i] = 2.0;
audio[i] = -30.0;
for (double peak : new double[]{8, 22, 41, 60}) {
if (Math.abs(t - peak) < 0.6) {
motion[i] = 7.0;
audio[i] = -12.0;
}
}
}
List<Double> candidates = director.candidatePeaks(motion, audio, 0.5, 80.0);
assertThat(candidates).hasSizeGreaterThanOrEqualTo(4);
assertThat(candidates).isSorted(); // time order
for (double peak : new double[]{8, 22, 41, 60}) {
assertThat(candidates).anySatisfy(t -> assertThat(t).isCloseTo(peak, org.assertj.core.api.Assertions.within(1.0)));
}
}
@Test
void buildsAMultiSegmentReelFromEveryWorthyMoment() {
// Two DISTINCT worthy actions, far apart (~10s and ~40s). Both must become segments in ONE reel, each
// with its own slow-mo payoff and its own overlay.
int n = 120;
double[] motion = new double[n];
double[] audio = new double[n];
for (int i = 0; i < n; i++) {
double t = i * 0.5;
motion[i] = 2.0;
audio[i] = -30.0;
if (Math.abs(t - 10.0) < 0.8) { motion[i] = 8.0; audio[i] = -10.0; }
if (Math.abs(t - 40.0) < 0.8) { motion[i] = 8.0; audio[i] = -10.0; }
}
List<Double> candidates = director.candidatePeaks(motion, audio, 0.5, 60.0);
List<Judgement> judgements = judge(candidates, 1.0, "GOAL"); // both worthy
MontagePlan plan = director.composeReel("p", "s.mp4", motion, audio, candidates, judgements, 0.5, 60.0);
long slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).count();
assertThat(slowMo).isEqualTo(2); // two payoffs -> two segments
assertThat(plan.overlays()).hasSize(2); // one overlay per segment
// The two payoffs sit on the two actions (~10s and ~40s of source).
List<Double> payoffs = plan.shots().stream().filter(s -> s.speed() < 0.8)
.map(MontagePlan.Shot::sourceStartSeconds).sorted().toList();
assertThat(payoffs.get(0)).isBetween(8.0, 12.0);
assertThat(payoffs.get(1)).isBetween(38.0, 42.0);
}
@Test
void intensityDecidesWhichSectionsMakeTheReelWhenTheJudgeCannotDiscriminate() {
// Continuous-motion content: the judge rates every moment the same (neutral 0.4, "riding"). A high
// intensity section (~10s) must make the reel; a much weaker one (~40s) must not.
int n = 120;
double[] motion = new double[n];
double[] audio = new double[n];
for (int i = 0; i < n; i++) {
double t = i * 0.5;
motion[i] = 2.0;
audio[i] = -30.0;
if (Math.abs(t - 10.0) < 0.8) { motion[i] = 10.0; audio[i] = -8.0; } // strong
if (Math.abs(t - 40.0) < 0.8) { motion[i] = 3.5; audio[i] = -26.0; } // weak
}
List<Double> candidates = director.candidatePeaks(motion, audio, 0.5, 60.0);
List<Judgement> judgements = judge(candidates, 0.4, "RIDING"); // model can't discriminate
MontagePlan plan = director.composeReel("p", "s.mp4", motion, audio, candidates, judgements, 0.5, 60.0);
double maxSrc = plan.shots().stream()
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
double payoff = plan.shots().stream().filter(s -> s.speed() < 0.8)
.mapToDouble(MontagePlan.Shot::sourceStartSeconds).min().orElse(-1);
assertThat(payoff).isBetween(8.0, 12.0); // the strong section
assertThat(maxSrc).isLessThan(30.0); // the weak ~40s one excluded
}
@Test
void mergesCandidatesThatWouldOverlapIntoOneSegment() {
// Two peaks only ~1.5s apart belong to the same action; the reel must not build two overlapping segments.
int n = 120;
double[] motion = new double[n];
double[] audio = new double[n];
for (int i = 0; i < n; i++) {
double t = i * 0.5;
motion[i] = 2.0;
audio[i] = -30.0;
if (Math.abs(t - 20.0) < 0.6 || Math.abs(t - 21.5) < 0.6) { motion[i] = 8.0; audio[i] = -10.0; }
}
List<Double> candidates = director.candidatePeaks(motion, audio, 0.5, 60.0);
List<Judgement> judgements = judge(candidates, 1.0, "MOMENT");
MontagePlan plan = director.composeReel("p", "s.mp4", motion, audio, candidates, judgements, 0.5, 60.0);
long slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).count();
assertThat(slowMo).isEqualTo(1); // merged into one segment
}
@Test
void eachSegmentIncludesItsOutcomeAndTheBuildIsCapped() {
int n = 128;
double[] motion = new double[n];
double[] audio = new double[n];
for (int i = 0; i < n; i++) {
double t = i * 0.5;
motion[i] = 2.0;
audio[i] = -30.0;
if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0; // early action spike (far from the peak)
if (t >= 44.0 && t <= 47.0) motion[i] = 4.0; // the action + its outcome
if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // the peak ~45s
}
List<Double> candidates = List.of(45.0);
List<Judgement> judgements = List.of(new Judgement(1.0, "MOMENT"));
MontagePlan plan = director.composeReel("p", "s.mp4", motion, audio, candidates, judgements, 0.5, 64.0);
double longestSpan = plan.shots().stream()
.mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0);
assertThat(longestSpan).isLessThanOrEqualTo(6.5); // build capped
double maxSrc = plan.shots().stream()
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
assertThat(maxSrc).isGreaterThan(45.5); // outcome (~46-47s) included
}
@Test
void fallsBackToAStraightCutForVeryShortSources() {
MontagePlan plan = director.composeReel("p", "src.mp4", new double[]{1, 1}, new double[]{-30, -30},
List.of(), null, 0.5, 1.0);
assertThat(plan.shots()).hasSize(1);
assertThat(plan.shots().get(0).sourceStartSeconds()).isEqualTo(0.0);
}
private static List<Judgement> judge(List<Double> candidates, double worthiness, String overlay) {
List<Judgement> out = new ArrayList<>();
for (int i = 0; i < candidates.size(); i++) {
out.add(new Judgement(worthiness, overlay));
}
return out;
}
}

View File

@ -27,6 +27,7 @@ class HighlightSourceSchedulerTest {
private VideoClippingProperties properties;
private FileSystemHighlightProjectStore store;
private HighlightSourceAnalyzer analyzer;
private HighlightCandidateGenerator candidateGenerator;
private HighlightDirectorPromptGenerator promptGenerator;
@BeforeEach
@ -41,6 +42,7 @@ class HighlightSourceSchedulerTest {
new HighlightProjectDirectoryInitializer(properties).initialize();
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
analyzer = mock(HighlightSourceAnalyzer.class);
candidateGenerator = mock(HighlightCandidateGenerator.class);
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
}
@ -75,6 +77,7 @@ class HighlightSourceSchedulerTest {
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
verify(analyzer).analyze("1");
verify(candidateGenerator).generate("1", analysis("1"));
verify(promptGenerator).generate("1");
}
@ -105,13 +108,39 @@ class HighlightSourceSchedulerTest {
assertThat(tempDir.resolve("highlight-projects/porsche-1/project.json")).exists();
}
@Test
void movesToAUniqueNameInsteadOfFailingWhenTheTargetNameIsTaken() throws Exception {
Path dir = tempDir.resolve("processed");
Files.createDirectories(dir);
HighlightSourceScheduler scheduler = scheduler();
// A free name is used as-is.
assertThat(scheduler.uniqueTarget(dir, "soccer.mp4")).isEqualTo(dir.resolve("soccer.mp4"));
// A taken name never overwrites: it gets a "-<n>" suffix before the extension, and increments.
Files.writeString(dir.resolve("soccer.mp4"), "one");
assertThat(scheduler.uniqueTarget(dir, "soccer.mp4")).isEqualTo(dir.resolve("soccer-1.mp4"));
Files.writeString(dir.resolve("soccer-1.mp4"), "two");
assertThat(scheduler.uniqueTarget(dir, "soccer.mp4")).isEqualTo(dir.resolve("soccer-2.mp4"));
}
private HighlightSourceScheduler scheduler() {
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
when(analyzer.analyze("1")).thenReturn(analysis("1"));
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
return new HighlightSourceScheduler(properties, store, analyzer, promptGenerator, clock);
stubAnalysis("1");
stubAnalysis("porsche");
stubAnalysis("porsche-1");
stubAnalysis("porsche-drive");
return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator,
new HighlightMontageDirector(properties),
new HighlightVisionDirector(properties, new com.fasterxml.jackson.databind.ObjectMapper()), clock);
}
private void stubAnalysis(String projectId) {
HighlightSourceAnalysis sourceAnalysis = analysis(projectId);
when(analyzer.analyze(projectId)).thenReturn(sourceAnalysis);
when(candidateGenerator.generate(projectId, sourceAnalysis)).thenReturn(new CinematicHighlightAnalysis(
projectId, ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"), List.of(),
Instant.parse("2026-07-11T08:00:00Z")));
}
private HighlightSourceAnalysis analysis(String projectId) {

View File

@ -0,0 +1,54 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import java.nio.file.Path;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightSubjectTrackerTest {
private final ObjectMapper mapper = new ObjectMapper();
@Test
void parsesAndSmoothsTrackPath() {
HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper,
command -> "detecting...\n{\"path\": [{\"t\":0.1,\"cx\":0.2,\"cy\":0.5,\"area\":0.1},"
+ "{\"t\":0.2,\"cx\":0.8,\"cy\":0.5,\"area\":0.1},"
+ "{\"t\":0.3,\"cx\":0.2,\"cy\":0.5,\"area\":0.1}]}\n");
List<double[]> path = tracker.track(Path.of("s.mp4"), 0.0, 1.0);
assertThat(path).hasSize(3);
// 3-tap smoothing pulls the spiky middle sample toward its neighbours (0.2,0.8,0.2 -> middle = 0.4).
assertThat(path.get(1)[0]).isCloseTo(0.4, org.assertj.core.api.Assertions.within(1e-9));
}
@Test
void trackFailsSoftToEmptyOnError() {
HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper,
command -> {
throw new IllegalStateException("no interpreter");
});
assertThat(tracker.track(Path.of("s.mp4"), 0.0, 1.0)).isEmpty();
}
@Test
void trackReturnsEmptyForEmptyPath() {
HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper,
command -> "{\"path\": []}");
assertThat(tracker.track(Path.of("s.mp4"), 0.0, 1.0)).isEmpty();
}
@Test
void panTokenEncodesAnchorsAndIsEmptyForNoPath() {
assertThat(HighlightSubjectTracker.panToken(List.of())).isEmpty();
String token = HighlightSubjectTracker.panToken(List.of(new double[]{0.5, 0.6}, new double[]{0.4, 0.7}));
assertThat(token).isEqualTo("pan=0.5000:0.6000;0.4000:0.7000");
}
}

View File

@ -0,0 +1,115 @@
package org.example.videoclips.editing;
import org.junit.jupiter.api.Test;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightVisionDirectorTest {
@Test
void cleansModelAnswersIntoShortBoldOverlays() {
assertThat(HighlightVisionDirector.toOverlayText("A strike!")).isEqualTo("A STRIKE");
assertThat(HighlightVisionDirector.toOverlayText("Bowling strike celebration moment"))
.isEqualTo("BOWLING STRIKE CELEBRATION"); // capped at 3 words
assertThat(HighlightVisionDirector.toOverlayText(" ")).isEmpty();
assertThat(HighlightVisionDirector.toOverlayText(null)).isEmpty();
}
@Test
void findsThePayoffShotAndItsTimelineWindow() {
MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(), List.of(
new MontagePlan.Shot(0.0, 1.5, 1.03, 1.0),
new MontagePlan.Shot(2.0, 2.0, 1.04, 1.0),
new MontagePlan.Shot(8.0, 3.0, 1.05, 0.7), // payoff (slowest)
new MontagePlan.Shot(11.0, 1.2, 1.04, 0.9)));
int payoff = HighlightVisionDirector.payoffShotIndex(plan);
assertThat(payoff).isEqualTo(2);
double[] window = HighlightVisionDirector.payoffTimeline(plan, payoff);
assertThat(window[0]).isEqualTo(3.5); // 1.5 + 2.0
assertThat(window[1]).isEqualTo(6.5); // + 3.0
}
@Test
void hasNoPayoffWhenNothingIsSlowMotion() {
MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(),
List.of(new MontagePlan.Shot(0.0, 2.0, 1.03, 1.0)));
assertThat(HighlightVisionDirector.payoffShotIndex(plan)).isEqualTo(-1);
}
@Test
void scoresCaptionsByHighlightWorthiness() {
assertThat(HighlightVisionDirector.semanticScore("A man raising his arms in celebration")).isEqualTo(1.0);
assertThat(HighlightVisionDirector.semanticScore("Throwing the ball")).isEqualTo(0.65);
assertThat(HighlightVisionDirector.semanticScore("Standing and waiting")).isEqualTo(0.2);
assertThat(HighlightVisionDirector.semanticScore("A quiet room")).isEqualTo(0.4); // neutral
assertThat(HighlightVisionDirector.semanticScore("")).isEqualTo(0.4);
// A subject turning/walking away is idle, not a highlight.
assertThat(HighlightVisionDirector.semanticScore("The person is turning away from the lane")).isEqualTo(0.2);
assertThat(HighlightVisionDirector.semanticScore("walking back to the seats")).isEqualTo(0.2);
}
@Test
void theJudgeRanksTheHighlightAboveLoudOrHighMotionNonHighlights() {
// This is the generic rule: a celebration or scored goal outranks a loud turn-away or an anticipatory
// build, because highlight-worthiness is a question of MEANING, not of motion/loudness.
double celebration = HighlightVisionDirector.highlightWorthiness("raising his arms in celebration");
double goal = HighlightVisionDirector.highlightWorthiness("kicking the ball into the net");
double turnAway = HighlightVisionDirector.highlightWorthiness("turning away and walking back");
double anticipation = HighlightVisionDirector.highlightWorthiness("about to kick the ball");
assertThat(celebration).isGreaterThan(turnAway);
assertThat(goal).isGreaterThan(turnAway);
assertThat(celebration).isGreaterThan(anticipation); // anticipation is the build-up, not the payoff
assertThat(goal).isGreaterThan(anticipation);
}
@Test
void buildsSemanticCurveFromNearestCaption() {
List<HighlightVisionDirector.TimedCaption> captions = List.of(
new HighlightVisionDirector.TimedCaption(1.0, "standing still", ""), // idle 0.2
new HighlightVisionDirector.TimedCaption(5.0, "arms raised celebration", "")); // payoff 1.0
double[] curve = HighlightVisionDirector.semanticCurve(captions, 1.0, 6.0); // 6 windows
assertThat(curve[0]).isEqualTo(0.2); // window ~0.5s -> nearest 1.0s "standing still"
assertThat(curve[5]).isEqualTo(1.0); // window ~5.5s -> nearest 5.0s "arms raised celebration"
}
@Test
void llamacppBackendFallsBackToMoondreamWhenNotProvisioned() {
// A non-llama.cpp script is used as-is.
assertThat(HighlightVisionDirector.resolveCaptionScript("./tools/vision_caption.py"))
.isEqualTo("./tools/vision_caption.py");
assertThat(HighlightVisionDirector.resolveCaptionScript(null)).isNull();
// The llama.cpp backend requires a binary env (LLAMACPP_SERVER_BIN / LLAMACPP_MTMD_BIN), which is not
// set in the test JVM -> it must fall back to moondream rather than silently use a broken backend.
assertThat(HighlightVisionDirector.resolveCaptionScript("./tools/vision_caption_llamacpp.py"))
.isEqualTo("./tools/vision_caption.py");
}
@Test
void detectsAnticipatoryDescriptions() {
assertThat(HighlightVisionDirector.isAnticipatory("The person is about to kick the ball")).isTrue();
assertThat(HighlightVisionDirector.isAnticipatory("A boy preparing to throw")).isTrue();
assertThat(HighlightVisionDirector.isAnticipatory("walking up to the ball")).isTrue();
assertThat(HighlightVisionDirector.isAnticipatory("kicking the ball hard")).isFalse();
assertThat(HighlightVisionDirector.isAnticipatory("raising arms in celebration")).isFalse();
assertThat(HighlightVisionDirector.isAnticipatory("")).isFalse();
assertThat(HighlightVisionDirector.isAnticipatory(null)).isFalse();
}
@Test
void overlayAssertsShownActionButPosesAQuestionForAnticipation() {
// The frame SHOWS the action -> assert the punchy label.
assertThat(HighlightVisionDirector.honestOverlayText(
"kicking the ball hard", "Kick", "will he score")).isEqualTo("KICK");
// The frame only shows anticipation -> never assert "KICK"; pose the grounded teaser question instead.
assertThat(HighlightVisionDirector.honestOverlayText(
"the person is about to kick the ball", "Kick", "will he score")).isEqualTo("WILL HE SCORE?");
// Anticipation with no usable teaser -> a generic honest question, still never the false action claim.
assertThat(HighlightVisionDirector.honestOverlayText(
"preparing to shoot", "Shot", " ")).isEqualTo("WHAT HAPPENS NEXT?");
}
}

View File

@ -18,13 +18,17 @@ class LocalAssetGenerationStageTest {
Path tempDir;
@Test
void reusesExistingSharedAssetsAndWritesRequestsForMissingVoiceover() throws Exception {
void blocksRenderWhenRequestedVoiceoverCannotBeGenerated() throws Exception {
VideoClippingProperties properties = properties();
Files.createDirectories(tempDir.resolve("shared/music"));
Files.createDirectories(tempDir.resolve("shared/sfx"));
Files.createDirectories(tempDir.resolve("voiceover-cache"));
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav.license.txt"), "approved-music");
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav.license.txt"), "approved-sfx");
Files.writeString(tempDir.resolve("voiceover-cache/unrelated.wav"), "wrong narration");
Files.writeString(tempDir.resolve("voiceover-cache/unrelated.wav.license.txt"), "approved-voice");
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
@ -36,6 +40,7 @@ class LocalAssetGenerationStageTest {
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new LocalAssetSynthesizer(properties),
new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
@ -52,16 +57,16 @@ class LocalAssetGenerationStageTest {
AssetGenerationResult result = stage.prepare("project", plan);
assertThat(result.readyForRender()).isTrue();
assertThat(result.readyForRender()).isFalse();
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
assertThat(tempDir.resolve("projects/project/audio/voiceover.wav")).doesNotExist();
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/requests/voiceover")).exists();
}
@Test
void writesBlockingRequestsWhenSfxIsMissing() throws Exception {
void blocksRenderWhenRequestedSfxCannotBeGenerated() throws Exception {
VideoClippingProperties properties = properties();
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
@ -73,6 +78,7 @@ class LocalAssetGenerationStageTest {
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new LocalAssetSynthesizer(properties),
new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
@ -83,7 +89,7 @@ class LocalAssetGenerationStageTest {
AssetGenerationResult result = stage.prepare("project", plan);
assertThat(result.readyForRender()).isFalse();
assertThat(tempDir.resolve("projects/project/assets/requests/sfx")).exists();
assertThat(tempDir.resolve("projects/project/audio/sfx/engine-hit.wav")).doesNotExist();
}
private VideoClippingProperties properties() {

View File

@ -0,0 +1,34 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.springframework.boot.DefaultApplicationArguments;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class LocalAssetRuntimeVerifierTest {
@Test
void strictRuntimeFailsClosedWhenResidentModelsAreMissing() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getLocalAssetWorker().setAutoStart(true);
properties.getEditing().getLocalAssetWorker().setStrictRuntime(true);
LocalAssetRuntimeVerifier verifier = new LocalAssetRuntimeVerifier(properties);
assertThatThrownBy(() -> verifier.run(new DefaultApplicationArguments()))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("Strict local asset runtime is not ready")
.hasMessageContaining("music_model_path")
.hasMessageContaining("sfx_model_path");
}
@Test
void disabledRuntimeDoesNotAttemptProvisioning() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getLocalAssetWorker().setAutoStart(false);
LocalAssetRuntimeVerifier verifier = new LocalAssetRuntimeVerifier(properties);
assertThatCode(() -> verifier.run(new DefaultApplicationArguments())).doesNotThrowAnyException();
}
}

View File

@ -80,7 +80,7 @@ class LocalDirectorSchedulerTest {
assertThat(project.status()).isEqualTo(EditProjectStatus.WAITING_FOR_DIRECTOR);
assertThat(project.inputDirectory()).isEqualTo(tempDir.resolve("processed/01-porsche").toString());
EditProjectAnalysis analysis = store.readJson("01-porsche", "analysis.json", EditProjectAnalysis.class);
assertThat(analysis.clips().getFirst().sourcePath())
assertThat(analysis.clips().get(0).sourcePath())
.isEqualTo(tempDir.resolve("processed/01-porsche/clip_00001.mp4").toString());
assertThat(firstClip).doesNotExist();
}

View File

@ -20,6 +20,7 @@ class LocalEditAssetLibraryTest {
Path base = Files.createDirectories(tempDir.resolve("music"));
Path car = Files.createDirectories(base.resolve("car_vlog"));
Files.writeString(base.resolve("ambient.wav"), "base");
Files.writeString(base.resolve("ambient.wav.license.txt"), "approved-ambient");
Files.writeString(base.resolve("ambient.wav.tags.txt"), "soft, calm");
Files.writeString(car.resolve("premium-drive.wav"), "car");
Files.writeString(car.resolve("premium-drive.wav.license.txt"), "licensed-local-track");
@ -42,6 +43,8 @@ class LocalEditAssetLibraryTest {
Path sfx = Files.createDirectories(tempDir.resolve("sfx"));
Files.writeString(sfx.resolve("alpha.wav"), "a");
Files.writeString(sfx.resolve("beta.wav"), "b");
Files.writeString(sfx.resolve("alpha.wav.license.txt"), "approved-alpha");
Files.writeString(sfx.resolve("beta.wav.license.txt"), "approved-beta");
LocalEditAssetLibrary library = new LocalEditAssetLibrary(new LocalEditAssetProvider(properties));
var selected = library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, "", 1));

Some files were not shown because too many files have changed in this diff Show More