forked from jsl/video_editing_poc
Harden cinematic highlight planning and rendering
- require explicit approval and skip ineligible highlight projects - validate timing and deduplicate candidate ranges - enforce licensed local assets and fail-closed generation - preserve video duration when mixing generated audio - refresh skill runbooks and regression coverage
This commit is contained in:
parent
97ba827d50
commit
f6dc6b8a83
|
|
@ -0,0 +1,271 @@
|
||||||
|
---
|
||||||
|
name: cinematic-media-engineering-reference
|
||||||
|
description: Load when designing, debugging, or reviewing this repository's highlight detection, FFmpeg trimming/rendering, scene and visual analysis, audio mixing/mastering, voiceover, music/SFX generation, overlays/color, proxies/thumbnails, or local media-model runtime. Use it to interpret media metrics correctly and to separate current heuristics from production evidence.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Cinematic Media Engineering Reference
|
||||||
|
|
||||||
|
Last verified against the repository: **2026-07-21**.
|
||||||
|
|
||||||
|
## Use this skill correctly
|
||||||
|
|
||||||
|
Use this skill to reason about the media domain and this repository's implementation. Before changing behavior, load `video-editing-change-control`. For commands, artifact locations, and startup safety, load `video-editing-run-and-operate`. For a failing run, load `video-editing-debugging-playbook`. For release evidence and thresholds, load `video-editing-validation-and-qa`. For property defaults, load `video-editing-config-and-flags`. For architecture boundaries, load `video-editing-architecture-contract`.
|
||||||
|
|
||||||
|
Do **not** use this skill as:
|
||||||
|
|
||||||
|
- permission to render, download a model, access a network, or change a production-facing default;
|
||||||
|
- proof that an output is cinematic because a heuristic score or `qa-report.json` says so;
|
||||||
|
- a licensing opinion or permission to use an asset/model;
|
||||||
|
- a replacement for final-media measurement and an approved human creative review.
|
||||||
|
|
||||||
|
## Context-specific operating consequences
|
||||||
|
|
||||||
|
The no-waiver policy is authoritative in `video-editing-change-control`. Apply these media-specific consequences before every media or model experiment:
|
||||||
|
|
||||||
|
1. Work offline. Pre-provision every executable, Python wheel, model weight, voice, font, LUT, music track, and SFX file through an approved artifact path.
|
||||||
|
2. Pin the artifact version and SHA-256; record its license, source, approval, target platform, and expected output contract.
|
||||||
|
3. Fail closed when a required model or asset is missing. Never substitute silence, a sine tone, OS speech, a heuristic provider, or an unrelated library asset.
|
||||||
|
4. Treat rendering as a behavior-changing operation. Require explicit approval; do not rely on current packaged defaults.
|
||||||
|
5. Never use external AI services. Model/media inference must be in-process or use approved non-network IPC; loopback HTTP is network and is not a certified path.
|
||||||
|
6. Route any code, configuration, model, threshold, or render-profile change through `video-editing-change-control`, then through `video-editing-validation-and-qa` before promotion.
|
||||||
|
|
||||||
|
Current code still violates parts of this policy. `tools/run_local_cv_worker.sh` installs packages and may fetch `yolov8n.pt`; `tools/run_local_asset_worker.sh` installs unpinned packages; and `SourceVisualAnalyzer` can fall back to filename heuristics. The 2026-07-21 working tree removed asset silence/tone/host-speech fallbacks, made strict asset readiness fail startup, and requires local model paths, but no complete resident asset-model bundle is present. Do not run the bootstrap launchers in a controlled or production environment.
|
||||||
|
|
||||||
|
## Map the actual media flow
|
||||||
|
|
||||||
|
| Stage | Current implementation | Interpret it as |
|
||||||
|
|---|---|---|
|
||||||
|
| Inspect | `FfmpegClipInspector` reads format duration, codecs, width/height, and `r_frame_rate` | Container-level summary, not a complete timing/color/channel contract |
|
||||||
|
| Sample | `ThumbnailExtractor` takes 5 evenly spaced JPEGs by default; `ProxyGenerator` makes 640-wide H.264 CRF 28/AAC 96k proxies | Review aids; neither is full-source evidence |
|
||||||
|
| Segment | `ShotSceneSegmenter` uses FFmpeg `select='gt(scene,0.35)',metadata=print`, then rejects sub-1-second boundaries | Frame-difference boundary proposals, not semantic scenes or highlights |
|
||||||
|
| Analyze audio | `SourceAudioAnalyzer` chains `silencedetect` and `volumedetect` | dBFS mean/maximum and thresholded silence only; not LUFS, true peak, dialogue, beat, or quality analysis |
|
||||||
|
| Analyze visuals | `LocalCvVisualAnalysisProvider` calls `tools/local_cv_worker.py`; optional fallback is `HeuristicVisualAnalysisProvider` | Sparse-thumbnail technical hints with major blind spots |
|
||||||
|
| Rank | `HighlightCandidateGenerator` makes shot-aligned or overlapping 12-second coverage windows, ranks with source-level visual scores, scene score, duration, and non-silence, and keeps the configured top 3 by default | Deterministic baseline with explicit fallback downweighting; not semantic or temporally resolved multimodal understanding |
|
||||||
|
| Direct | `HighlightDirectorPromptGenerator` asks a filesystem-capable model for `director/edit-plan.json`; `HighlightDirectorPlanValidator` gates it | Still an untrusted creative proposal until validated and reviewed; validation proves contract safety, not artistic quality |
|
||||||
|
| Prepare assets | `HighlightAssetPreparationService`, `HighlightLocalAssetWorker`, `LocalAssetSynthesizer` | File materialization; existence does not prove provenance, intelligibility, fit, or quality |
|
||||||
|
| Render | `HighlightDirectorFlowService` makes one `EditDecision` per highlight; `HighlightFfmpegRenderer` renders and stream-concats | A single source range per highlight, not an internally multi-shot edit |
|
||||||
|
| Report | renderer writes `render-manifest.json` and `qa-report.json`; the highlight path probes duration, black ranges, long silence, and sample peaks | Technical evidence only; loudness, true peak, A/V sync, freeze detection, raster text safety, and creative fit remain unmeasured |
|
||||||
|
|
||||||
|
The two ranking paths are deliberately distinct. `CinematicHighlightAnalyzer` remains the older multi-clip edit-project ranker and writes root-level files. The single-source `HighlightSourceScheduler` now invokes `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Treat its category as model-supported only when independent local visual labels clear the threshold; metadata/fallback evidence stays low-confidence `generic_vlog` and requires media review.
|
||||||
|
|
||||||
|
## Reason about time before touching cuts
|
||||||
|
|
||||||
|
Define these terms once:
|
||||||
|
|
||||||
|
- **PTS** (presentation timestamp): when a decoded frame/sample is presented.
|
||||||
|
- **DTS** (decode timestamp): when it is decoded; reordered codecs may have DTS different from PTS.
|
||||||
|
- **time base**: seconds per timestamp tick. Wall-clock seconds are `timestamp * time_base`.
|
||||||
|
- **CFR/VFR**: constant/variable frame rate. `r_frame_rate` is not proof of CFR and is not a frame clock.
|
||||||
|
- **GOP** (group of pictures): frames between independently decodable keyframes; non-keyframes depend on neighbors.
|
||||||
|
|
||||||
|
Current plans round seconds to milliseconds. `FfmpegClipInspector` does not retain `time_base`, `start_time`, `avg_frame_rate`, frame count, rotation, sample aspect ratio, or VFR evidence. Do not assume `frame = seconds * r_frame_rate`, do not compare decimal timestamps as if they were original PTS, and do not promise frame-accurate cuts from `analysis/ffprobe.json` alone.
|
||||||
|
|
||||||
|
Inspect the missing timing facts without writing media:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
INPUT=path/to/source.mp4
|
||||||
|
ffprobe -v error -show_entries \
|
||||||
|
format=start_time,duration,format_name:stream=index,codec_type,codec_name,time_base,start_time,duration,r_frame_rate,avg_frame_rate,nb_frames,width,height,pix_fmt,sample_aspect_ratio,sample_rate,channel_layout \
|
||||||
|
-of json "$INPUT"
|
||||||
|
ffprobe -v error -select_streams v:0 -show_frames \
|
||||||
|
-show_entries frame=best_effort_timestamp_time,key_frame,pict_type \
|
||||||
|
-of csv=p=0 "$INPUT" | head -n 40
|
||||||
|
```
|
||||||
|
|
||||||
|
Interpret boundaries as half-open ranges `[start,end)`. For every decision, enforce the repository's duration equation:
|
||||||
|
|
||||||
|
```text
|
||||||
|
rendered_duration = (source_end - source_start) / playback_speed
|
||||||
|
```
|
||||||
|
|
||||||
|
`EditPlanValidator` permits only 0.25x through 4.0x and allows 0.05 seconds of equation error. `HighlightDirectorFlowService` independently clamps speed into that range but does not invoke the validator.
|
||||||
|
|
||||||
|
### Copy versus re-encode
|
||||||
|
|
||||||
|
| Operation | Boundary behavior | Repository use |
|
||||||
|
|---|---|---|
|
||||||
|
| `-c copy` trim | Start commonly snaps to dependency/keyframe constraints; no filter can run | Folder clipping may preserve input quality; do not call this an exact creative cut without packet evidence |
|
||||||
|
| Decode + encode trim | Decoder can discard to requested time and encoder creates a new boundary; still quantized to stream/filter time bases | Both edit renderers trim each segment, filter it, and encode H.264/AAC |
|
||||||
|
| Concat demuxer + `-c copy` | Requires stream-compatible segments: codec parameters, time bases, stream layout, and ordering must agree | Both renderers join normalized segments this way; mixed audio-presence/channel layouts remain a risk |
|
||||||
|
| `xfade`/`acrossfade` | Requires overlapping, timestamp-normalized streams and shortens/repositions the combined timeline | Not implemented |
|
||||||
|
|
||||||
|
The string `crossfade` is misleading today. Both renderers translate it into a fade-out on one segment plus a fade-in on the next, then concatenate them without overlap. `HighlightDirectorFlowService` itself emits only `cut`, `fade-in`, and `fade-out`.
|
||||||
|
|
||||||
|
## Understand codecs, containers, and formats here
|
||||||
|
|
||||||
|
Keep three layers separate: a **container** (MP4/MOV/MKV) stores streams; a **codec** (H.264/HEVC/AAC) compresses a stream; a decoded **pixel/sample format** describes raw data.
|
||||||
|
|
||||||
|
The current segment contract is MP4 containing libx264 H.264 at configured resolution and frame rate, `yuv420p`, CRF 18, preset `veryfast`, plus optional AAC at the configured bitrate and 48 kHz default. The configured `video-bitrate` is not used by either editing renderer. `ProxyGenerator` separately emits H.264 CRF 28, width 640 by default, and AAC 96k. `ThumbnailExtractor` emits JPEG at `-q:v 2`.
|
||||||
|
|
||||||
|
Do not infer production compatibility from the `.mp4` suffix. Before concat or delivery, compare stream count, codec/profile/level, pixel format, dimensions, time base, frame rate behavior, audio sample rate, channel layout, and color metadata. The current inspector cannot do that comparison.
|
||||||
|
|
||||||
|
The renderers scale to fit and pad to 1920x1080 by default. This preserves the whole source frame but can letterbox/pillarbox. Any non-`none` visual treatment also applies the same fixed contrast/saturation/brightness, unsharp, vignette, and a 4% center crop in `HighlightFfmpegRenderer`; the treatment text does not select a distinct grade. `HighlightVisualEffectsStage` writes descriptive effect names only. It does not execute LUTs, grain, animated overlays, or prompt-specific effects.
|
||||||
|
|
||||||
|
## Read the visual signals without overclaiming
|
||||||
|
|
||||||
|
| Signal | Exact current calculation | What it can and cannot mean |
|
||||||
|
|---|---|---|
|
||||||
|
| Scene score | FFmpeg `scene` metadata at selected frames; threshold 0.35 | Large frame difference. It can fire on flashes/camera motion and miss semantic changes. `ShotSegment.sceneScore` is the boundary score entering that segment, not its average quality. |
|
||||||
|
| Cut-density “motion” | `clamp(0.25 + ((segments-1)/duration*60)/20)` | Editing/camera-change density, not optical or subject motion. A static multicamera cut sequence can score high; a continuous action shot can score low. |
|
||||||
|
| Blur score | Average `clamp(variance(Laplacian(gray))/500)` over readable thumbnails | Really a sparse sharpness/edge-energy score: higher is sharper. Noise, text, and texture inflate it; intentional shallow focus and low-detail frames depress it. |
|
||||||
|
| Exposure score | Average `clamp(1 - 2*abs(mean(gray)/255 - 0.5))` | Mean proximity to mid-gray. It does not detect clipped highlights, crushed shadows, contrast, skin exposure, HDR transfer, or local exposure. |
|
||||||
|
| Composition | Average of aspect score (0.8 for 1.70-1.90, else 0.55) and thumbnail-presence score (0.7 or 0.45) | An aspect/file-presence prior, not composition analysis. |
|
||||||
|
| Face presence | OpenCV frontal-face Haar cascade; any detection yields `faces_detected` | Sparse, frontal, binary presence only. No identity, consent, expression, tracking, screen area, or false-positive calibration. |
|
||||||
|
| Objects | YOLO max confidence per class over sampled thumbnails, top 10 | Detector confidence on sparse frames, not semantic relevance or continuous presence. Current default `yolov8n.pt` may auto-download. |
|
||||||
|
| Filename heuristic | Tokens such as `porsche`, `car`, `food`, `family` in clip ID/path | Metadata guess only. Confidence constants are authored values, not calibrated probabilities. |
|
||||||
|
|
||||||
|
When OpenCV or thumbnails are unavailable, the worker returns 0.5 quality defaults. When YOLO yields nothing, it returns `unknown` at 0.2. These are sentinel-like values but are not explicitly typed as “unknown”; never aggregate them as measurements. `SourceVisualAnalyzer` marks a local-CV failure with `local_cv_failed_fallback_...`, so gate on `analysisMethod`, not just numeric fields.
|
||||||
|
|
||||||
|
### Worked visual example
|
||||||
|
|
||||||
|
For 13 detected segments in a 120-second source, the current motion heuristic gives:
|
||||||
|
|
||||||
|
```text
|
||||||
|
cuts_per_minute = (13 - 1) / 120 * 60 = 6
|
||||||
|
motion_score = 0.25 + 6 / 20 = 0.55
|
||||||
|
```
|
||||||
|
|
||||||
|
This proves only that the scene detector retained about six boundaries per minute at the configured threshold/minimum duration. It does not prove visible motion or highlight value. Discriminate the causes by reviewing frames around each boundary and a proxy interval on both sides.
|
||||||
|
|
||||||
|
## Evaluate highlight candidates as evidence
|
||||||
|
|
||||||
|
Current `CinematicHighlightAnalyzer`:
|
||||||
|
|
||||||
|
1. Classifies from project name/style/clip path substrings.
|
||||||
|
2. Splits each clip into eight-second windows.
|
||||||
|
3. Caps at 12 windows and spreads them through long clips.
|
||||||
|
4. Scores `min(1, 0.35 + 0.12*(max(.1,sharpness)+max(.1,brightness)+max(.1,motion)) + role + category)`.
|
||||||
|
5. Assigns role from position: first is hook, last is hero, early third is rising action.
|
||||||
|
|
||||||
|
Because `FfmpegClipInspector` initializes all three quality fields to zero and no current enrichment replaces them, quality contributes exactly `0.036`. A first car candidate therefore scores `0.35 + 0.036 + 0.25 + 0.20 = 0.836` without inspecting its pixels or sound. Treat the ranking as a coverage scaffold.
|
||||||
|
|
||||||
|
For a production candidate evaluator, require each candidate to carry independently measured and normalized features plus their availability masks:
|
||||||
|
|
||||||
|
| Dimension | Discriminating evidence | Reject or penalize when |
|
||||||
|
|---|---|---|
|
||||||
|
| Technical usability | sharpness distribution, highlight/shadow clipping, shake, occlusion, audio noise/clipping | defect defeats the intended role unless narratively essential |
|
||||||
|
| Semantic moment | local embeddings/detectors/transcript tied to sampled timestamps | evidence does not support the category/action claim |
|
||||||
|
| Novelty | distance from already selected shots in visual/audio embedding space | angle/action is redundant |
|
||||||
|
| Story function | observable setup, action, reaction, reveal, or payoff | role is assigned only from source position |
|
||||||
|
| Editability | clean handles before/after, usable continuity, compatible motion/audio | cut begins/ends mid-action or cannot bridge coherently |
|
||||||
|
| Audio value | dialogue/event/ambience confidence and intelligibility | silence/noise is mistaken for low/high drama |
|
||||||
|
|
||||||
|
Predict expected metric movement before changing a model or weight. Evaluate on an approved, licensed golden set using timestamp-level relevance, diversity, coverage, false-positive cost, and blinded human pairwise preference. Never tune and certify on the same sources.
|
||||||
|
|
||||||
|
## Engineer sound instead of merely attaching files
|
||||||
|
|
||||||
|
Define the units:
|
||||||
|
|
||||||
|
- **dBFS**: sample amplitude relative to digital full scale; 0 dBFS is the sample ceiling.
|
||||||
|
- **LUFS-I**: program integrated loudness with perceptual weighting and gating.
|
||||||
|
- **dBTP**: reconstructed true peak, which can exceed sample peak between samples.
|
||||||
|
- **LRA**: loudness range, a statistical measure of loudness variation, not simply max minus min.
|
||||||
|
|
||||||
|
`SourceAudioAnalyzer` reports `volumedetect` mean/max dBFS and `silencedetect` ranges at -35 dB for 0.5 seconds by default. It does not report any LUFS, dBTP, LRA, clipping count, speech intelligibility, noise floor, music beats, or semantic events. For missing audio it reports `audioPresent=false` but numeric mean/max values of `0`; never interpret those zeros as loud audio.
|
||||||
|
|
||||||
|
Measure a final file read-only:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FINAL=path/to/final.mp4
|
||||||
|
ffmpeg -hide_banner -nostats -i "$FINAL" \
|
||||||
|
-af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json -f null -
|
||||||
|
ffmpeg -hide_banner -nostats -i "$FINAL" \
|
||||||
|
-af ebur128=peak=true -f null -
|
||||||
|
ffmpeg -hide_banner -nostats -i "$FINAL" \
|
||||||
|
-af silencedetect=n=-35dB:d=0.5 -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
The repository defaults target -16 LUFS-I, -1.5 dBTP, and LRA 11, but these are configuration targets, not certified acceptance thresholds. The renderer uses one-pass `loudnorm`; exact conformance requires measuring the encoded output, and deterministic file normalization normally requires a measured first pass followed by a second pass using the measured fields.
|
||||||
|
|
||||||
|
### Current mix graph and its consequences
|
||||||
|
|
||||||
|
- Source audio is always referenced as `[0:a]`; a video-only timeline can make mixing fail.
|
||||||
|
- Music now uses the plan cue's `gainDb`, start, end, trim, and delay. It still needs final-media loudness measurement and a proven scene-fit generation model.
|
||||||
|
- Each script line is a distinct voiceover asset, trimmed and delayed to its planned `VoiceoverLine` range, then combined into the voice bus at unity gain. Generated waveform duration and intelligibility are not yet measured against that slot.
|
||||||
|
- Music is sidechain-compressed only when voiceover exists, using configured threshold 0.045, ratio 8, attack 20 ms, release 250 ms by default.
|
||||||
|
- SFX are trimmed, gain-adjusted in dB, and delayed to their cue start.
|
||||||
|
- `amix=duration=first` makes the source timeline define output length, then one-pass loudness normalization feeds AAC 192k/48 kHz by default.
|
||||||
|
- Audio-mix and preview failures fail the highlight render. The mastering check proves that the successful command contained `loudnorm`; it does not measure encoded LUFS-I or true peak.
|
||||||
|
|
||||||
|
Therefore asset existence and a passed QA JSON are not proof of audible voiceover/music/SFX. Verify stream presence, timing, loudness, true peak, silence, and content audibly and instrumentally. A max sample peak of 0 dBFS signals risk but does not alone prove intersample clipping; use true-peak analysis.
|
||||||
|
|
||||||
|
For voice timing, derive speech duration from the generated waveform, not `characters / constant`. Place each line with `adelay`, trim/pad deliberately, leave breaths and edit handles, and fail when a line cannot fit its approved slot without unacceptable time-stretch. Duck from the aligned voice bus, not an unaligned full-file narration.
|
||||||
|
|
||||||
|
## Apply color, LUTs, and overlays safely
|
||||||
|
|
||||||
|
Before grading, preserve and inspect transfer, primaries, matrix, range, bit depth, pixel format, and rotation. Current inspection and rendering do not preserve or validate this color contract explicitly. A numeric `eq` preset is not color management and can damage already-graded, log, HDR, or limited-range footage.
|
||||||
|
|
||||||
|
Treat a LUT as executable media data: approve its license and checksum; document expected input/output color spaces; test neutral ramps, skin/brand colors, legal range, banding, and highlight roll-off. Although a LUT folder is configured, neither renderer uses it.
|
||||||
|
|
||||||
|
Current overlays use FFmpeg `drawtext`, 54 px white type, 2 px dark border, and positions around 6% horizontal/8-10% vertical margins. The `animation` field is validated in the multi-clip path but ignored by both renderers. Fonts are not explicitly selected, so output depends on host font resolution. The highlight QA check rejects out-of-timeline and unsupported placement values, but “safe” placement names are heuristics, not proof against glyph width, multiline wrapping, platform UI, subtitles, crops, or localization.
|
||||||
|
|
||||||
|
Require a pinned licensed font, measured text bounds, explicit wrapping, title/action safe policy for target platforms, and raster review at every output aspect ratio. Do not approve an overlay because `text_overlays_safe=true`; it proves plan bounds, not raster safety.
|
||||||
|
|
||||||
|
## Make local inference reproducible
|
||||||
|
|
||||||
|
For every model runtime, record this minimum manifest next to the approved deployment artifact, not as tribal knowledge:
|
||||||
|
|
||||||
|
| Field | Required content |
|
||||||
|
|---|---|
|
||||||
|
| Identity | task, model family, exact revision/weight filename, SHA-256 |
|
||||||
|
| Provenance | authoritative source, retrieval date, license text/version, commercial-use decision, approver |
|
||||||
|
| Runtime | OS/architecture, CPU/GPU/accelerator, driver, Python, native libraries, framework/wheel hashes |
|
||||||
|
| Inference | preprocessing, image/audio sample policy, thresholds, seeds, deterministic settings, precision/device |
|
||||||
|
| Contract | input/output schema, label map/voice identity, expected sample rate/channel/pixel format |
|
||||||
|
| Evidence | golden-set version/hash, accuracy/quality results, latency, memory, known failure slices |
|
||||||
|
|
||||||
|
Use platform-specific artifacts for macOS development, Linux/VPS, and cloud images, but prove semantic parity on the same golden inputs. Bit-identical neural outputs across different accelerators may be unrealistic; define allowed numeric/output drift before testing and compare downstream decisions as well as tensors.
|
||||||
|
|
||||||
|
Current gaps are blocking for production reproducibility: `tools/local_asset_requirements.txt` has no versions/hashes; local CV package versions are pinned but not hash-locked; model names are mutable aliases; no model/asset checksum or license manifest is enforced; the runtime can select OS `say`/`espeak`; AudioCraft `get_pretrained` and YOLO name loading can use network/cache state.
|
||||||
|
|
||||||
|
Inventory pre-provisioned artifacts without downloading anything:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find input/highlights/assets -type f -print 2>/dev/null | sort
|
||||||
|
find . -type f \( -name '*.pt' -o -name '*.pth' -o -name '*.onnx' -o -name '*.bin' \) -print
|
||||||
|
shasum -a 256 path/to/model-or-asset
|
||||||
|
```
|
||||||
|
|
||||||
|
On Linux, use `sha256sum path/to/model-or-asset`. A hash proves identity, not license or quality.
|
||||||
|
|
||||||
|
### Placeholder trap: worked trace
|
||||||
|
|
||||||
|
As of 2026-07-21, unavailable AudioCraft/Piper models return failure, the worker does not emit tones, silence, or host speech, `LocalAssetSynthesizer` deletes failed/inaudible output, and requested missing assets block rendering. The highlight path creates script-specific per-line voice files and delays them to planned reading-time windows before combining the voice bus for ducking. Detecting nonzero PCM is still not speech intelligibility or script-faithfulness validation, so keep approved rendering blocked until semantic audio QA and licensed immutable model provenance exist.
|
||||||
|
|
||||||
|
## Interpret final output with a measured checklist
|
||||||
|
|
||||||
|
Before calling a render production-ready, require all of the following:
|
||||||
|
|
||||||
|
- Source selections trace to timestamped evidence and are diverse, technically usable, and story-relevant.
|
||||||
|
- Every plan range is within the real stream timeline and survives decode-boundary inspection.
|
||||||
|
- Final streams match an approved codec/container/pixel/audio/color contract.
|
||||||
|
- Final duration is probed, not copied from plan arithmetic.
|
||||||
|
- Every required asset has approved provenance, checksum, license, model/runtime manifest, and intended content.
|
||||||
|
- Voiceover is intelligible, fact-grounded, aligned to its slots, and absent only by explicit creative decision.
|
||||||
|
- Music and SFX are aligned, mixed, ducked, and measured; no placeholder or silent fallback exists.
|
||||||
|
- LUFS-I, dBTP, LRA, silence, black frames, freeze/duplication, and A/V sync are measured against approved thresholds.
|
||||||
|
- Overlays are rendered with pinned fonts and checked for bounds/safe areas on each target.
|
||||||
|
- Color treatment is appropriate to the input color space and does not clip or shift protected colors.
|
||||||
|
- A blinded human review compares against the approved baseline; heuristic scores cannot waive this review.
|
||||||
|
- Promotion follows `video-editing-change-control` and evidence is retained under the process defined by `video-editing-validation-and-qa`.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
The implementation claims above come from `FfmpegClipInspector`, `ThumbnailExtractor`, `ProxyGenerator`, `WaveformGenerator`, `ShotSceneSegmenter`, `SourceAudioAnalyzer`, `SourceVisualAnalyzer`, `HeuristicVisualAnalysisProvider`, `LocalCvVisualAnalysisProvider`, `CinematicHighlightAnalyzer`, `HighlightSourceScheduler`, `HighlightDirectorPromptGenerator`, `HighlightDirectorFlowService`, `HighlightVisualEffectsStage`, `HighlightFfmpegRenderer`, `FfmpegEditRenderer`, `EditPlanValidator`, `LocalAssetRuntimeVerifier`, `LocalAssetSynthesizer`, `tools/local_cv_worker.py`, `tools/local_asset_worker.py`, both worker shell scripts, both requirements files, and `application.yml`.
|
||||||
|
|
||||||
|
Re-verify volatile facts with read-only commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'ffmpeg|ffprobe|sceneDetection|silenceThreshold|loudness|ducking|autoStart|strictRuntime|fallback' src/main/java/org/example/videoclips src/main/resources/application.yml
|
||||||
|
rg -n 'filter|libx264|crf|loudnorm|sidechaincompress|amix|drawtext|concat|copy_post_timeline' src/main/java/org/example/videoclips/editing/*Renderer.java
|
||||||
|
rg -n 'Laplacian|brightness|cuts_per_minute|Haar|YOLO|yolov8n|fallback|silence|tone|get_pretrained' tools src/main/java/org/example/videoclips/editing
|
||||||
|
rg -n 'category.json|highlight-candidates.json|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing
|
||||||
|
sed -n '1,140p' src/main/resources/application.yml
|
||||||
|
sed -n '1,120p' tools/local_cv_requirements.txt
|
||||||
|
sed -n '1,120p' tools/local_asset_requirements.txt
|
||||||
|
ffmpeg -version
|
||||||
|
ffprobe -version
|
||||||
|
ffmpeg -hide_banner -filters | rg 'scene|loudnorm|ebur128|xfade|sidechaincompress|lut3d|drawtext'
|
||||||
|
```
|
||||||
|
|
||||||
|
If any command, property, path, formula, or wiring changes, update this skill in the same approved change. Date-stamp the new verification and preserve the distinction between implemented behavior, measurement theory, candidate improvements, and certified evidence.
|
||||||
|
|
@ -0,0 +1,382 @@
|
||||||
|
---
|
||||||
|
name: video-editing-architecture-contract
|
||||||
|
description: Load when changing module boundaries, workflow orchestration, domain models, ports/adapters, project-folder protocols, state transitions, render approval, local-model integration, or persistence in the video-editing service; use it to preserve current contracts while moving the code toward a production-grade modular monolith.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Architecture Contract
|
||||||
|
|
||||||
|
Use this contract before changing a workflow boundary, state transition, persisted JSON shape, project directory, scheduler, renderer, model worker, or adapter. Treat it as a map of the implementation as verified on **2026-07-21**, not as proof that the target architecture already exists.
|
||||||
|
|
||||||
|
## Do not use this skill for
|
||||||
|
|
||||||
|
| Need | Load instead |
|
||||||
|
|---|---|
|
||||||
|
| Decide whether a proposed change is allowed, gated, or ready to merge | `video-editing-change-control` |
|
||||||
|
| Reproduce a build or repair a workstation/runtime | `video-editing-build-and-env` |
|
||||||
|
| Look up a property, default, or environment variable | `video-editing-config-and-flags` |
|
||||||
|
| Diagnose a concrete failure or stuck project | `video-editing-debugging-playbook` |
|
||||||
|
| Decide whether media output meets its evidence bar | `video-editing-validation-and-qa` |
|
||||||
|
| Operate schedulers, inputs, projects, or outputs | `video-editing-run-and-operate` |
|
||||||
|
| Improve highlight selection or cinematic quality | `video-editing-cinematic-highlights-campaign` |
|
||||||
|
|
||||||
|
Do not use architecture language to waive a quality gate. Route every behavior-changing proposal through `video-editing-change-control`.
|
||||||
|
|
||||||
|
## Define the terms
|
||||||
|
|
||||||
|
- **Capability**: a cohesive business responsibility, such as upload-and-clipping or cinematic highlight production.
|
||||||
|
- **Domain**: business concepts and invariants that can be tested without Spring, HTTP, JPA, FFmpeg, or the filesystem.
|
||||||
|
- **Application service**: orchestration for one use case; it calls domain behavior and outbound ports.
|
||||||
|
- **Port**: an interface owned by the core that expresses a required inbound or outbound capability.
|
||||||
|
- **Adapter**: framework or infrastructure code implementing a port, such as JPA, S3, FFmpeg, or a controller.
|
||||||
|
- **Filesystem state machine**: a workflow in which directory location and artifact presence act as queue and state signals.
|
||||||
|
- **Contract artifact**: a persisted file whose name, location, or JSON shape is consumed by another stage.
|
||||||
|
- **Modular monolith**: one deployable application with enforced capability boundaries and inward-pointing dependencies. One Maven module or one JVM process alone does not make the code modular.
|
||||||
|
|
||||||
|
## Start with the honest baseline
|
||||||
|
|
||||||
|
As of 2026-07-21, this is one Spring Boot application and one Maven module. It contains four workflows with different contracts. The upload-and-clipping workflow has recognizable ports and adapters. The folder workflow is a small vertical slice. The `editing` package contains multi-clip and single-source highlight domains, orchestration, filesystem persistence, process execution, and model-worker integration together.
|
||||||
|
|
||||||
|
The project does **not** currently have Spring Modulith, ArchUnit, automated package-boundary rules, a module diagram in code, or enforced clean-architecture dependency direction. `VideoAssetService` imports HTTP request/response DTOs, and `EditProjectController` invokes `EditRenderer` directly. Do not describe the current tree as a completed hexagonal architecture.
|
||||||
|
|
||||||
|
## Map the four workflows
|
||||||
|
|
||||||
|
| Workflow | Trigger and orchestrator | Durable state | Main outputs | Representative tests |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| API upload and clipping | `/v1/video-assets`, `/v1/clip-jobs`, and `/v1/clips`; `VideoAssetService`; `ClipJobQueuePort`; `ClipProcessor` | Repository records, object keys, and optionally `queue_messages` | Uploaded clip objects plus `Clip` records and signed download URLs | `VideoAssetControllerTest`, `DatabaseBackedClipJobQueueAdapterTest` |
|
||||||
|
| Folder segmentation | Poll `input-directory`; `FolderVideoScanScheduler` -> `FolderVideoValidator` -> `FolderFfmpegClipper` | File location: source -> working -> processed/rejected; `.failed` is last-resort quarantine | `output-directory/<source-base>[-N]/clip_%05d.<container>` | `FolderVideoScanSchedulerTest`, `FolderFfmpegIntegrationTest` |
|
||||||
|
| Multi-clip cinematic edit | REST API or poll `editing.local-director.source-directory`; `LocalDirectorScheduler`, `EditPlanInboxScanner`, `FfmpegEditRenderer` | `project.json`, analysis/plan/manifest JSON, inbox file names, and source-folder location | One project `final.mp4`, `render-manifest.json`, `qa-report.json` | `CinematicEditingIntegrationTest`, `EditPlanInboxScannerTest`, `FfmpegEditRendererTest` |
|
||||||
|
| Single-source highlights | Poll `editing.highlight-scheduler.source-directory`; `HighlightSourceScheduler`, `HighlightDirectorPlanScanner`, `HighlightDirectorFlowService` | Highlight project tree, `project.json`, director-plan presence, approval marker, final-file presence | Per-highlight final/preview/QA/manifests plus project `final.mp4` | `HighlightSourceSchedulerTest`, `HighlightDirectorFlowServiceTest`, `HighlightFfmpegRendererTest` |
|
||||||
|
|
||||||
|
Do not connect two workflows merely because both call FFmpeg or share records. First identify the owner of the use case, state, retry semantics, and output contract.
|
||||||
|
|
||||||
|
## Preserve the implemented dependency map
|
||||||
|
|
||||||
|
### Upload and clipping
|
||||||
|
|
||||||
|
Follow this current call direction while improving it incrementally:
|
||||||
|
|
||||||
|
```text
|
||||||
|
api controllers
|
||||||
|
-> application.VideoAssetService
|
||||||
|
-> application.VideoClippingRepository
|
||||||
|
-> queue.ClipJobQueuePort
|
||||||
|
-> storage.ObjectStoragePort
|
||||||
|
|
||||||
|
queue adapters -> processing.ClipProcessor
|
||||||
|
processing.ClipProcessor
|
||||||
|
-> application.VideoClippingRepository
|
||||||
|
-> processing.VideoClipperPort
|
||||||
|
-> storage.ObjectStoragePort
|
||||||
|
|
||||||
|
outbound implementations:
|
||||||
|
infrastructure.InMemoryVideoAssetRepository
|
||||||
|
persistence.JpaVideoClippingRepository
|
||||||
|
queue.InMemoryClipJobQueueAdapter / DatabaseBackedClipJobQueueAdapter
|
||||||
|
processing.StubVideoClipperAdapter / FfmpegVideoClipperAdapter
|
||||||
|
storage.InMemoryObjectStorageAdapter / S3ObjectStorageAdapter
|
||||||
|
```
|
||||||
|
|
||||||
|
Treat `VideoClippingRepository`, `ClipJobQueuePort`, `VideoClipperPort`, and `ObjectStoragePort` as existing contracts. Do not make core business behavior depend on JPA entities, Spring Data repositories, S3 types, or `ProcessBuilder`.
|
||||||
|
|
||||||
|
Current exception: `VideoAssetService` consumes types from `api.dto`. Move transport-to-command mapping to the inbound adapter when changing this surface; preserve API compatibility and add characterization tests first.
|
||||||
|
|
||||||
|
### Folder segmentation
|
||||||
|
|
||||||
|
Treat `org.example.videoclips.folder` as a self-contained file-ingestion capability. It currently calls FFprobe and FFmpeg directly through package-private functional test seams. Do not route folder jobs through the API repository or object storage unless a stated product requirement unifies their ownership and recovery model.
|
||||||
|
|
||||||
|
### Editing workflows
|
||||||
|
|
||||||
|
Recognize these existing interfaces, but do not mistake their package location for enforced isolation:
|
||||||
|
|
||||||
|
| Port-like interface | Current owner and purpose |
|
||||||
|
|---|---|
|
||||||
|
| `EditProjectStore` | Multi-clip project tree and top-level JSON artifacts |
|
||||||
|
| `HighlightProjectStore` | Single-source highlight project tree and nested artifacts |
|
||||||
|
| `EditRenderer` | Render a saved multi-clip project by project ID |
|
||||||
|
| `VoiceoverGenerator` | Produce project voiceover from timed lines |
|
||||||
|
| `EditAssetProvider` / `EditAssetLibrary` | Resolve or select local media assets |
|
||||||
|
| `VisualAnalysisProvider` | Analyze a clip from metadata, thumbnails, and shots |
|
||||||
|
|
||||||
|
Keep application-facing interfaces free of `Process`, HTTP client, and framework-specific return types. Introduce a port only for a concrete substitution, isolation, or testability need.
|
||||||
|
|
||||||
|
## Preserve each state machine
|
||||||
|
|
||||||
|
### API upload and clipping states
|
||||||
|
|
||||||
|
```text
|
||||||
|
VideoAsset: PENDING_UPLOAD -> UPLOADED -> PROCESSING -> READY
|
||||||
|
| failure/retry -> UPLOADED
|
||||||
|
any retained terminal asset -> DELETED
|
||||||
|
|
||||||
|
ClipJob: QUEUED -> RUNNING -> SUCCEEDED
|
||||||
|
| -> FAILED
|
||||||
|
QUEUED or RUNNING -> CANCEL_REQUESTED -> CANCELLED
|
||||||
|
```
|
||||||
|
|
||||||
|
Enforce these invariants:
|
||||||
|
|
||||||
|
- Create a job only for an `UPLOADED` or `READY` asset.
|
||||||
|
- Keep tenant ownership checks at every read or mutation boundary.
|
||||||
|
- Preserve idempotency for create-asset and create-job requests.
|
||||||
|
- Store generated media before exposing a persisted clip as downloadable.
|
||||||
|
- Return an asset to `UPLOADED` after a failed or cancelled processing attempt.
|
||||||
|
- Keep DB-queue retry ownership in `DatabaseBackedClipJobQueueAdapter`; do not persist terminal failure from `ClipProcessor` before retries are exhausted.
|
||||||
|
|
||||||
|
### Folder file states
|
||||||
|
|
||||||
|
```text
|
||||||
|
source candidate --atomic move--> working
|
||||||
|
working --valid + FFmpeg success--> processed
|
||||||
|
working --invalid or processing failure--> rejected
|
||||||
|
working --rejection move failure--> <name>.failed beside active file
|
||||||
|
```
|
||||||
|
|
||||||
|
Preserve one-file-per-scan behavior, natural filename ordering, refusal to overwrite a destination, and exclusion of hidden/partial/failed files. Atomic moves assume directories share a filesystem; treat cross-device deployment as unsupported until tested and designed.
|
||||||
|
|
||||||
|
### Multi-clip edit states
|
||||||
|
|
||||||
|
The implemented local-director path normally follows:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CREATED -> ANALYZING -> ANALYZED -> WAITING_FOR_DIRECTOR
|
||||||
|
-> PLANNED -> RENDERING -> RENDERED
|
||||||
|
any handled workflow/render failure -> FAILED
|
||||||
|
```
|
||||||
|
|
||||||
|
`PLANNING` exists in `EditProjectStatus` but is not part of the normal scheduler sequence. `EditProjectService.updateProject` does not validate legal transitions. Add transition enforcement before relying on status as a concurrency or authorization boundary.
|
||||||
|
|
||||||
|
The source folder also moves `source -> working -> processed/rejected`. An accepted inbox plan moves from `inbox/edit-plan.json` to `inbox/edit-plan.json.accepted`; a rejected plan becomes `.rejected`. Approval is signaled by `inbox/approved.flag` when configured.
|
||||||
|
|
||||||
|
### Single-source highlight states
|
||||||
|
|
||||||
|
The observed new-project path is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CREATED -> WAITING_FOR_DIRECTOR -> RENDERING -> RENDERED
|
||||||
|
invalid empty director plan -> FAILED
|
||||||
|
```
|
||||||
|
|
||||||
|
`ANALYZING` and `PLANNED` exist in `HighlightProjectStatus`, but the primary scheduler/flow does not currently persist them. The render scanner considers a project renderable when `director/edit-plan.json` exists and project `final.mp4` does not. Its in-process `AtomicBoolean` prevents overlapping scans only inside one JVM; it is not a distributed claim.
|
||||||
|
|
||||||
|
Do not infer progress solely from the enum. Check artifacts and logs. Before horizontal scaling, add durable claims, leases, idempotent stages, and recovery tests.
|
||||||
|
|
||||||
|
## Preserve the two editing contracts separately
|
||||||
|
|
||||||
|
### Multi-clip contract
|
||||||
|
|
||||||
|
The root is `video-clipping.editing.project-directory` (default `./output/edit-projects`). A project currently uses:
|
||||||
|
|
||||||
|
```text
|
||||||
|
<project-id>/
|
||||||
|
project.json
|
||||||
|
analysis.json
|
||||||
|
cinematic-highlight-analysis.json
|
||||||
|
category.json
|
||||||
|
highlight-candidates.json
|
||||||
|
thumbnails/ contact-sheets/ proxies/ audio/
|
||||||
|
inbox/
|
||||||
|
edit-plan.json
|
||||||
|
final.mp4
|
||||||
|
render-manifest.json
|
||||||
|
qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`EditProject` describes an input directory containing multiple clips. `EditPlan.decisions` can reference multiple clip IDs and builds one combined timeline.
|
||||||
|
|
||||||
|
### Single-source highlight contract
|
||||||
|
|
||||||
|
The root is `video-clipping.editing.highlight-project-directory` (default `./output/highlight-projects`). `FileSystemHighlightProjectStore` creates:
|
||||||
|
|
||||||
|
```text
|
||||||
|
<project-id>/
|
||||||
|
project.json
|
||||||
|
manifest.json
|
||||||
|
source/
|
||||||
|
analysis/
|
||||||
|
ffprobe.json
|
||||||
|
scene-segments.json
|
||||||
|
audio-analysis.json
|
||||||
|
visual-analysis.json
|
||||||
|
source-analysis.json
|
||||||
|
frames/ contact-sheets/ proxies/ audio/
|
||||||
|
director/
|
||||||
|
director-brief.md
|
||||||
|
director-prompt.md
|
||||||
|
edit-plan.json
|
||||||
|
approved.flag # only when approval is required
|
||||||
|
assets/{voiceover,music,sfx,overlays}/
|
||||||
|
highlights/<highlight-id>/
|
||||||
|
storyboard.md
|
||||||
|
edit-plan.json
|
||||||
|
assets/
|
||||||
|
final.mp4
|
||||||
|
preview.mp4
|
||||||
|
render-manifest.json
|
||||||
|
qa-report.json
|
||||||
|
final.mp4
|
||||||
|
render-manifest.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`HighlightFolderContract.standard()` lists intended names such as `shots.json`, `scenes.json`, `transcript.json`, `audio-events.json`, `category.json`, and `highlight-candidates.json`. The source analyzer writes its five technical analysis JSON files; the scheduler's `HighlightCandidateGenerator` separately writes `category.json` and `highlight-candidates.json`. `shots.json`, `scenes.json`, `transcript.json`, and `audio-events.json` remain manifest intentions, so do not claim every listed artifact exists.
|
||||||
|
|
||||||
|
`HighlightDirectorPlan` selects several windows from exactly one source and carries story purpose, visual treatment, music/SFX directions, voiceover, and overlays. `HighlightDirectorFlowService` converts each item to a single-decision `EditPlan`, renders per-highlight outputs, then concatenates them.
|
||||||
|
|
||||||
|
### Why the contracts remain separate
|
||||||
|
|
||||||
|
Keep them separate because they have different aggregates, input cardinality, director schemas, folder protocols, approval locations, output cardinality, scheduler ownership, and recovery signals. Sharing `EditPlan`, `RenderManifest`, or analysis helpers is implementation reuse, not schema equivalence.
|
||||||
|
|
||||||
|
Do not merge the stores, statuses, or roots until all of these exist:
|
||||||
|
|
||||||
|
- A versioned replacement contract and explicit migration/compatibility decision.
|
||||||
|
- Golden JSON fixtures for both schemas and backward-read tests.
|
||||||
|
- Recovery tests from every durable stage boundary.
|
||||||
|
- A collision strategy for project IDs and artifact names.
|
||||||
|
- An approval model that cannot be bypassed by REST or scheduler entry points.
|
||||||
|
- A rollout and rollback plan approved through `video-editing-change-control`.
|
||||||
|
|
||||||
|
## Enforce load-bearing invariants
|
||||||
|
|
||||||
|
Apply this checklist to every architecture change:
|
||||||
|
|
||||||
|
- [ ] Keep project IDs restricted to `[A-Za-z0-9][A-Za-z0-9._-]{0,127}` and reject `..`.
|
||||||
|
- [ ] Normalize and containment-check every project-relative path before reading or writing it.
|
||||||
|
- [ ] Never accept an absolute path or backslash-containing relative artifact name in the highlight store.
|
||||||
|
- [ ] Refuse overwrite when claiming source files; never silently replace an existing processed/rejected input.
|
||||||
|
- [ ] Treat source media and completed contract artifacts as immutable inputs to later stages.
|
||||||
|
- [ ] Validate project ID, source identity, time ranges, supported effects, and required licensed assets before rendering.
|
||||||
|
- [ ] Persist an artifact successfully before advertising the status that promises it exists.
|
||||||
|
- [ ] Make retries idempotent: never duplicate a rendered highlight, queue message, clip record, or asset charge.
|
||||||
|
- [ ] Record model identity/version, input hash, asset provenance/license, FFmpeg command, and output hash for reproducibility.
|
||||||
|
- [ ] Derive acceptance from measured media probes and approved creative review, never file existence or visual impression alone.
|
||||||
|
- [ ] Keep transport DTOs, JPA entities, and worker-process details outside the domain.
|
||||||
|
- [ ] Keep framework annotations out of new domain objects unless a decision record justifies them.
|
||||||
|
|
||||||
|
## Enforce the local-runtime and render prohibitions
|
||||||
|
|
||||||
|
The following are project directives, even where current defaults or code contradict them:
|
||||||
|
|
||||||
|
1. Do not download dependencies or models automatically at startup, test time, or job time.
|
||||||
|
2. Do not call external AI services. Model/media inference must use in-process calls or approved non-network inter-process communication; loopback HTTP is still network and is not a certified path.
|
||||||
|
3. Do not permit model/media inference or acquisition to use any network, including loopback. Pre-provision and verify dependencies, model weights, and licenses during an approved artifact-build process. Production API, database, storage, and telemetry integrations remain separately approved and secured network boundaries.
|
||||||
|
4. Do not ingest or generate assets without recorded licensing and provenance.
|
||||||
|
5. Do not substitute silence, sine tones, or placeholder media for missing voiceover, music, or SFX.
|
||||||
|
6. Do not render without explicit approval bound to the exact plan and source digest.
|
||||||
|
7. Do not change production-facing defaults without change-control approval, compatibility analysis, and rollback evidence.
|
||||||
|
|
||||||
|
Current disqualifying gaps include:
|
||||||
|
|
||||||
|
- `tools/run_local_asset_worker.sh` and `tools/run_local_cv_worker.sh` install Python dependencies in `auto` mode; the CV script can load/download YOLO weights, and AudioCraft `get_pretrained` can fetch models.
|
||||||
|
- The 2026-07-21 working tree removes worker silence/tone/host-speech fallbacks and makes requested assets mandatory at both renderer boundaries. Standalone bootstrap scripts and local-CV heuristic fallback remain noncompliant gaps.
|
||||||
|
- `application.yml` enables highlight ingestion but now defaults highlight rendering off and director approval on. Approval remains a bare, non-digest-bound file.
|
||||||
|
- `POST /v1/edit-projects/{projectId}:render` invokes the renderer without an approval check.
|
||||||
|
- `HighlightFfmpegRenderer` can copy video forward after audio mixing fails, and several QA checks are hard-coded `true` rather than measured.
|
||||||
|
|
||||||
|
Treat these as open architecture defects. Do not operate them as production behavior, and do not hide them with documentation. Repair them only through the change-control and validation skills.
|
||||||
|
|
||||||
|
## Know the current weak points
|
||||||
|
|
||||||
|
| Weak point | Evidence and consequence | Required direction |
|
||||||
|
|---|---|---|
|
||||||
|
| No enforced modules | One Maven module; no ArchUnit/Modulith dependency test | Add capability-oriented boundaries and fail the build on violations |
|
||||||
|
| Anemic state records | Public records allow unchecked states; update methods accept arbitrary status | Put invariants and legal transitions in domain/application code |
|
||||||
|
| Filesystem is database and queue | Direct JSON writes, artifact-presence polling, local `AtomicBoolean` guards | Add atomic publish, durable claim/lease, restart recovery, retention, and concurrency tests |
|
||||||
|
| Highlight analysis is incomplete | New-source analyzer does not call `CinematicHighlightAnalyzer`; prompt tolerates absent category/candidates | Make multimodal candidate production an explicit, measured stage |
|
||||||
|
| Director plan contract is gated but creative truth is not proven | `HighlightDirectorPlanValidator` validates identity, category/candidate linkage, safe IDs, ranges, configured bounds, and required directions; it does not verify visible claims, asset licenses, or artistic fit | Keep media review and explicit render approval; add provenance and content-grounding evidence before claiming production readiness |
|
||||||
|
| Local worker boundary is porous | Shell bootstrap downloads; loopback endpoint is configurable; models are named rather than content-addressed | Package locked runtime/model artifacts; verify hashes; replace loopback HTTP with in-process or approved non-network IPC and deny inference networking |
|
||||||
|
| QA can overstate success | Highlight checks assert duration, overlays, assets, and mastering without measuring output | Probe outputs and fail closed on production-required checks |
|
||||||
|
| Renderer configuration leaks | Project concatenation uses literal `ffmpeg`/`ffprobe` while other stages use typed properties | Route executables and process policy through an outbound media-tool port |
|
||||||
|
| OS assumptions leak | macOS `say`, local paths, atomic rename, and worker virtualenvs are runtime details | Prove equivalent macOS dev and Linux/VPS/cloud behavior; remove OS-specific production fallbacks |
|
||||||
|
| API/core dependency points outward | Application service imports `api.dto`; controller directly calls renderer | Map DTOs at inbound adapters and expose application use cases |
|
||||||
|
| Local project state blocks horizontal safety | Edit/highlight projects require shared local paths and have no distributed ownership | Define durable object/project storage before claiming cloud scalability |
|
||||||
|
|
||||||
|
## Move toward the target modular monolith
|
||||||
|
|
||||||
|
Use capabilities, not technical-layer-only packages, as the top-level target. Keep one deployable unless measured requirements justify distribution.
|
||||||
|
|
||||||
|
```text
|
||||||
|
org.example.videoclips
|
||||||
|
clipping/ # upload sessions, assets, clip jobs, clip publication
|
||||||
|
folderingestion/ # watched-folder segmentation workflow
|
||||||
|
multiedit/ # multi-source project, plan, render lifecycle
|
||||||
|
highlights/ # single-source analysis, direction, assets, render lifecycle
|
||||||
|
media/ # approved FFmpeg/FFprobe and local-model runtime adapters
|
||||||
|
platform/ # configuration, persistence wiring, security, observability
|
||||||
|
```
|
||||||
|
|
||||||
|
Within each capability, point dependencies inward:
|
||||||
|
|
||||||
|
```text
|
||||||
|
inbound adapters -> application use cases -> domain
|
||||||
|
outbound adapters -> application-owned outbound ports
|
||||||
|
configuration -> all adapters for composition only
|
||||||
|
domain -> no Spring, HTTP, persistence, filesystem, process, or sibling adapter package
|
||||||
|
```
|
||||||
|
|
||||||
|
Expose cross-capability access through a small documented application API or durable event. Do not import another capability's adapter or persistence entity. Do not create a generic base service, universal repository, universal mapper, CQRS infrastructure, event bus, or microservice without a measured requirement.
|
||||||
|
|
||||||
|
### Execute boundary migration safely
|
||||||
|
|
||||||
|
1. Characterize the current API, JSON, file names, state changes, and restart behavior with tests.
|
||||||
|
2. Write an ADR and route it through `video-editing-change-control`.
|
||||||
|
3. Define the destination capability and its public application API.
|
||||||
|
4. Move domain types without changing behavior; keep compatibility mapping at the old boundary.
|
||||||
|
5. Move outbound interfaces inward, then adapt JPA, filesystem, FFmpeg, and model workers outside them.
|
||||||
|
6. Add ArchUnit or Spring Modulith verification only when its rule corresponds to the documented dependency direction.
|
||||||
|
7. Run narrow tests, full tests, clean-checkout verification, and artifact contract tests.
|
||||||
|
8. Remove compatibility code only after persisted-project and API migration evidence is approved.
|
||||||
|
|
||||||
|
For macOS development, Linux/VPS production, and cloud deployment, use the same platform-neutral application artifact and contract versions. Build signed platform-specific runtime/model/image bundles, then promote each bundle without rebuild for the same OS/CPU target. Permit environment-specific adapters/configuration, not environment-specific domain behavior.
|
||||||
|
|
||||||
|
## Record every major decision
|
||||||
|
|
||||||
|
For every module split, new dependency, persistence change, state change, model/runtime choice, or contract version, record:
|
||||||
|
|
||||||
|
1. Requirement being addressed.
|
||||||
|
2. Selected approach.
|
||||||
|
3. Alternatives considered.
|
||||||
|
4. Benefits and trade-offs.
|
||||||
|
5. Operational consequences.
|
||||||
|
6. Security implications.
|
||||||
|
7. Automated and manual verification.
|
||||||
|
8. Revisit conditions.
|
||||||
|
|
||||||
|
Label the target as `proposed`, `accepted`, `implemented`, or `verified`. Never call it production-ready until clean-checkout build, security, observability, recovery, load, media-quality, and operational gates all pass.
|
||||||
|
|
||||||
|
## Review checklist
|
||||||
|
|
||||||
|
- [ ] Name the affected workflow and aggregate.
|
||||||
|
- [ ] List every changed API, JSON, file, state, metric, and approval contract.
|
||||||
|
- [ ] Show dependency direction before and after.
|
||||||
|
- [ ] Keep domain behavior testable without Spring.
|
||||||
|
- [ ] Add a port only where a real adapter boundary exists.
|
||||||
|
- [ ] Prove restart and retry behavior at the changed durable boundary.
|
||||||
|
- [ ] Prove path containment and tenant/project isolation.
|
||||||
|
- [ ] Prove no startup/job-time downloads or external network calls occur.
|
||||||
|
- [ ] Prove all assets and models are pre-provisioned, hashed, and licensed.
|
||||||
|
- [ ] Prove approval binds source, plan, configuration, model, and asset digests.
|
||||||
|
- [ ] Prove QA from measurements; reject placeholder audio and degraded render fallback.
|
||||||
|
- [ ] Run the relevant workflow tests plus the full change-control gates.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Re-verify the package inventory: `find src/main/java/org/example/videoclips -type d | sort`
|
||||||
|
|
||||||
|
Re-verify application wiring and schedulers: `rg -n '@SpringBootApplication|@EnableAsync|@EnableScheduling|@Scheduled|@ConditionalOn' src/main/java`
|
||||||
|
|
||||||
|
Re-verify existing ports: `rg -n '^public interface ' src/main/java/org/example/videoclips/{application,processing,queue,storage,editing}`
|
||||||
|
|
||||||
|
Re-verify outward application imports: `rg -n '^import org\.example\.videoclips\.(api|persistence|storage|queue|processing)' src/main/java/org/example/videoclips/{domain,application}`
|
||||||
|
|
||||||
|
Re-verify statuses: `for f in src/main/java/org/example/videoclips/domain/ClipJobStatus.java src/main/java/org/example/videoclips/editing/{EditProjectStatus,HighlightProjectStatus}.java; do sed -n '1,120p' "$f"; done`
|
||||||
|
|
||||||
|
Re-verify filesystem writes and render signals: `rg -n 'writeJson\(|Files\.(move|copy|write)|final\.mp4|approved\.flag' src/main/java/org/example/videoclips/{folder,editing}`
|
||||||
|
|
||||||
|
Re-verify highlight artifacts against the declared folder contract: `rg -n 'analysis/.*\.json|director/.*\.json|HighlightFolderContract' src/main/java/org/example/videoclips/editing`
|
||||||
|
|
||||||
|
Re-verify prohibited downloads and fallbacks: `rg -n 'pip install|get_pretrained|YOLO\(|write_silence|fallback[_ -]tone|anullsrc|sine=' tools src/main/java src/main/resources`
|
||||||
|
|
||||||
|
Re-verify production-facing defaults: `sed -n '1,180p' src/main/resources/application.yml && sed -n '1,120p' src/main/resources/application.properties`
|
||||||
|
|
||||||
|
Re-verify architecture enforcement dependencies/tests: `rg -n 'archunit|spring-modulith|ApplicationModules|ArchRule' pom.xml src/test || true`
|
||||||
|
|
||||||
|
Re-run focused contract tests after architecture changes: `mvn -o -Dtest=VideoAssetControllerTest,DatabaseBackedClipJobQueueAdapterTest,FolderVideoScanSchedulerTest,EditPlanInboxScannerTest,FileSystemEditProjectStoreTest,FileSystemHighlightProjectStoreTest,HighlightDirectorFlowServiceTest test`
|
||||||
|
|
@ -0,0 +1,345 @@
|
||||||
|
---
|
||||||
|
name: video-editing-build-and-env
|
||||||
|
description: Load this skill when reproducing or repairing the build environment on macOS, Linux/VPS, or a cloud build runner; when Maven, Java 21, FFmpeg/ffprobe, Python runtimes, local model files, executable permissions, architecture compatibility, offline dependency resolution, clean-checkout reproducibility, or test working-directory assumptions are involved; or when designing the dependency-locked and network-isolated build supply chain. Do not load it for an already-built service's runtime symptom.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Build and Environment
|
||||||
|
|
||||||
|
Use this runbook from the repository root. Treat every path and version below as a verified description of `HEAD` on **2026-07-21**, unless it is explicitly labeled `TARGET`.
|
||||||
|
|
||||||
|
## Use this skill for
|
||||||
|
|
||||||
|
- Recreating a build environment on macOS, Linux/VPS, or a cloud build runner.
|
||||||
|
- Deciding whether a failure comes from Java, Maven, FFmpeg, Python, a missing local model, permissions, CPU architecture, or an unclean workspace.
|
||||||
|
- Running focused tests, the current Maven gate, or a clean archived-checkout experiment.
|
||||||
|
- Designing the future reproducible, dependency-locked, offline build and runtime supply chain.
|
||||||
|
|
||||||
|
## Do not use this skill for
|
||||||
|
|
||||||
|
| Need | Load instead |
|
||||||
|
|---|---|
|
||||||
|
| Starting schedulers, processing media, locating outputs, or deployment operation | `video-editing-run-and-operate` |
|
||||||
|
| Changing application properties or worker flags | `video-editing-config-and-flags` |
|
||||||
|
| Deciding what evidence proves media or creative quality | `video-editing-validation-and-qa` |
|
||||||
|
| Diagnosing a runtime symptom after the environment is known good | `video-editing-debugging-playbook` |
|
||||||
|
| Learning why the clean-checkout and local-asset traps exist | `video-editing-failure-archaeology` |
|
||||||
|
| Promoting a toolchain, dependency, image, model, or default change | `video-editing-change-control` |
|
||||||
|
|
||||||
|
Do not use a successful compile or test run as proof of cinematic quality, production readiness, security, or operability.
|
||||||
|
|
||||||
|
Terms: **SBOM** means Software Bill of Materials; **OCI** means Open Container Initiative image format.
|
||||||
|
|
||||||
|
## Non-negotiable environment boundary
|
||||||
|
|
||||||
|
Enforce these rules in development, CI, image construction, and production:
|
||||||
|
|
||||||
|
1. Do not allow application startup or validation to install a package, fetch a model, contact an external AI service, or access the network.
|
||||||
|
2. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` as setup commands. Their default `BOOTSTRAP_MODE=auto` can create virtual environments and invoke `pip`; the CV launcher can also cause Ultralytics to fetch YOLO weights.
|
||||||
|
3. Provision Maven artifacts, Python wheels, FFmpeg, local model weights, fonts, LUTs, music, and sound effects only in an approved build/image process. Record version, SHA-256 checksum, origin, license, target OS, target CPU architecture, and approval for every non-source artifact.
|
||||||
|
4. Run the delivered service and its validation offline. Missing dependencies or models must fail closed; do not replace them with downloads, silence, tones, host speech utilities, heuristics, or unlicensed assets.
|
||||||
|
5. Do not change production-facing defaults to make a build pass. Route such a change through `video-editing-change-control`.
|
||||||
|
|
||||||
|
The current source still violates parts of this boundary: the standalone launchers bootstrap automatically and local CV can fall back to heuristics. Asset generation itself now rejects missing local models and placeholder output, and strict asset readiness fails startup. Treat the remaining launcher/CV behavior as open defects, not approved procedures.
|
||||||
|
|
||||||
|
## Current baseline and gaps
|
||||||
|
|
||||||
|
| Component | Current repository fact | Production interpretation |
|
||||||
|
|---|---|---|
|
||||||
|
| Java | `pom.xml` sets `java.version` to `21`; compilation uses Java release 21. | Use a supported Java 21 LTS JDK for the current code. Do not infer vendor or patch level from the POM. |
|
||||||
|
| Spring Boot | Parent is `org.springframework.boot:spring-boot-starter-parent:3.3.2`. | This is the implemented version, not a claim that it is the latest supported release. Upgrade only through change control with compatibility evidence. |
|
||||||
|
| Build tool | Maven project with `pom.xml`; no `mvnw`, Maven wrapper metadata, or Gradle files exist. | A host Maven installation is currently required and its version is not repository-controlled. |
|
||||||
|
| Project version | `org.example:video-editing:1.0-SNAPSHOT`. | The application artifact itself is a snapshot. The POM contains no snapshot/prerelease dependency version, but no automated enforcer proves that property. |
|
||||||
|
| Dependency management | Spring Boot manages most versions; AWS SDK uses BOM `2.28.29`; JaCoCo is `0.8.12`. | There is no dependency lock, repository mirror policy, SBOM plugin, vulnerability gate, or update automation in the POM. |
|
||||||
|
| Test runner | Surefire activates Spring profile `test`; JaCoCo checks 100% instruction, line, and branch coverage only for `org.example.videoclips.folder` during `verify`. | `test` does not execute the JaCoCo check. No project-wide coverage threshold is implemented. |
|
||||||
|
| FFmpeg | Application defaults call `ffmpeg` and `ffprobe` from `PATH`. | Version, build flags, codecs, and checksum are unpinned. Some real-media tests skip when these tools are missing. |
|
||||||
|
| CV Python | Four top-level packages are version-pinned in `tools/local_cv_requirements.txt`; transitive dependencies and hashes are not locked. | `.venv-local-cv` and YOLO weights are local, platform-specific, untracked state. |
|
||||||
|
| Asset Python | `torch`, `audiocraft`, `soundfile`, and `numpy` are unpinned in `tools/local_asset_requirements.txt`. | `.venv-local-asset`, Piper, and music/SFX/voice models are not reproducibly provisioned. |
|
||||||
|
| Packaging/deploy | No Dockerfile, Compose file, Maven wrapper, CI workflow, or deployment manifest is tracked. | There is no certified Linux/VPS/cloud build path or OCI image yet. |
|
||||||
|
|
||||||
|
## Inspect before building
|
||||||
|
|
||||||
|
Run only read-only checks first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pwd
|
||||||
|
test -f pom.xml
|
||||||
|
git status --short
|
||||||
|
java -version
|
||||||
|
javac -version
|
||||||
|
mvn -version
|
||||||
|
python3 --version
|
||||||
|
command -v ffmpeg
|
||||||
|
command -v ffprobe
|
||||||
|
ffmpeg -version | sed -n '1,4p'
|
||||||
|
ffprobe -version | sed -n '1,3p'
|
||||||
|
uname -s
|
||||||
|
uname -m
|
||||||
|
git ls-files -s tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Require these observations:
|
||||||
|
|
||||||
|
| Check | Accept now | If different |
|
||||||
|
|---|---|---|
|
||||||
|
| Repository root | `pom.xml`, `src/`, and `tools/` are present. | Stop and change to the repository root. Relative worker and data paths depend on the process working directory. |
|
||||||
|
| Java | Runtime and compiler both report major version 21. | Select a Java 21 LTS JDK. Do not silently compile with a newer release. |
|
||||||
|
| Maven | A host Maven is available. | Stop. There is no wrapper to recover a missing or incompatible Maven. Record the selected Maven version in evidence. |
|
||||||
|
| Media tools | Both commands resolve and report versions. | Unit tests may still pass, while real FFmpeg integration tests skip. Do not accept that as media validation. |
|
||||||
|
| OS/CPU | Record, do not normalize away. | Build or select Python wheels, FFmpeg, and model runtime packages for this exact OS/architecture. |
|
||||||
|
| Script modes | CV launcher is `100755`; asset launcher is `100644` at current `HEAD`. | The asset bootstrap cannot be directly executed by `ProcessBuilder` on POSIX. Do not fix locally with `chmod` and call the environment reproducible; change the tracked mode through change control. |
|
||||||
|
|
||||||
|
Inspect source declarations without invoking Maven plugins:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '1,180p' pom.xml
|
||||||
|
sed -n '1,120p' tools/local_cv_requirements.txt
|
||||||
|
sed -n '1,120p' tools/local_asset_requirements.txt
|
||||||
|
sed -n '1,120p' tools/run_local_cv_worker.sh
|
||||||
|
sed -n '1,120p' tools/run_local_asset_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not run `mvn dependency:*` or `mvn help:*` as an offline preflight unless the corresponding plugin is already present in the approved Maven cache. On the observed workstation, `mvn -o help:evaluate` fails because the help plugin is absent from the local cache.
|
||||||
|
|
||||||
|
## Platform setup contract
|
||||||
|
|
||||||
|
Provision tools outside application startup. This section names the required result; it intentionally does not use `brew`, `apt`, `curl`, `pip`, or a model hub.
|
||||||
|
|
||||||
|
### macOS development
|
||||||
|
|
||||||
|
Provide all of the following before running Maven:
|
||||||
|
|
||||||
|
- A supported Java 21 LTS JDK and a known Maven distribution.
|
||||||
|
- `ffmpeg` and `ffprobe` from the same approved FFmpeg build.
|
||||||
|
- Python runtimes and wheel sets matching `uname -m` (`arm64` or `x86_64`). Never reuse a virtual environment created for the other architecture.
|
||||||
|
- Pre-provisioned, licensed local model files. Configure explicit absolute paths; do not rely on a model name that a library may resolve online.
|
||||||
|
- A network-denied validation environment.
|
||||||
|
|
||||||
|
Do not rely on macOS `say`. The former host-dependent fallback was removed; it is not the approved local voiceover engine and is unavailable on Linux.
|
||||||
|
|
||||||
|
### Linux/VPS production
|
||||||
|
|
||||||
|
Build and validate on the same Linux CPU architecture and compatible runtime family used in production. Provide Java 21, Maven only in the build stage, FFmpeg/ffprobe, Python runtimes, wheels, Piper, and model files from the approved artifact store. Keep the runtime artifact set read-only where practical.
|
||||||
|
|
||||||
|
Do not rely on `espeak`. The former fallback was removed and was never a production voiceover contract. Do not copy a macOS virtual environment, native Python wheel, FFmpeg binary, or Maven build output that embeds host-specific paths into Linux.
|
||||||
|
|
||||||
|
### Cloud build and image pipeline
|
||||||
|
|
||||||
|
Split acquisition from verification:
|
||||||
|
|
||||||
|
| Stage | Network | Required output |
|
||||||
|
|---|---|---|
|
||||||
|
| Approved acquisition | Restricted to approved artifact repositories | Versioned Maven cache, hashed Python wheelhouse, hashed FFmpeg distribution, hashed model/asset bundle, licenses, SBOM inputs. |
|
||||||
|
| Build | Denied | Compiled and tested application from immutable inputs. |
|
||||||
|
| Runtime image assembly | Denied | Java runtime, application, media tools, Python runtime, models, assets, and manifest for one OS/CPU target. |
|
||||||
|
| Runtime/smoke validation | Denied | Fail-closed startup and functional evidence; no cache writes or downloads. |
|
||||||
|
|
||||||
|
Build distinct artifacts for `linux/amd64` and `linux/arm64` until cross-platform equivalence is demonstrated. GPU, CUDA, Metal/MPS, and CPU model runtimes are not interchangeable. Record device/runtime selection and numerical or quality acceptance results; never assume model availability means equivalent output.
|
||||||
|
|
||||||
|
## Safe build commands
|
||||||
|
|
||||||
|
All Maven commands write under `target/`. They must run with network access denied. Maven `-o` requests offline resolution but is not itself a network sandbox.
|
||||||
|
|
||||||
|
### Compile without executing tests
|
||||||
|
|
||||||
|
Use this only to isolate compilation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -DskipTests package
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected current artifact name after success: `target/video-editing-1.0-SNAPSHOT.jar`. This is not a quality gate.
|
||||||
|
|
||||||
|
### Run a focused test
|
||||||
|
|
||||||
|
Use a focused class while developing a narrow change:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -Dtest=EditPlanValidatorTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace the class name only after locating it under `src/test/java`. Focused tests are evidence for that scope only. Tests that directly construct production process adapters may invoke host tools even though Surefire activates the `test` profile.
|
||||||
|
|
||||||
|
### Run the current full Maven gate
|
||||||
|
|
||||||
|
The current repository-wide command is:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o verify
|
||||||
|
```
|
||||||
|
|
||||||
|
`verify` compiles main and test sources, runs Surefire tests, applies the folder-package JaCoCo check, and creates a coverage report. It does **not** provide dependency locking, formatting, static analysis, architecture enforcement, vulnerability scanning, secret scanning, API compatibility, container scanning, SBOM generation, signing, or deployment validation.
|
||||||
|
|
||||||
|
Do not report this gate as green from the current source. A clean archived checkout fails as documented below. A pass in a long-lived workspace can be caused by untracked `.venv-local-asset` state and may invoke a model library capable of downloading weights.
|
||||||
|
|
||||||
|
## Reproduce the clean-checkout gate
|
||||||
|
|
||||||
|
Use an archive so untracked virtual environments, models, generated media, and local ignore rules cannot contaminate the result. Point Maven at an explicitly approved, pre-populated cache. The command performs no Git mutation and no source-tree write:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
repo_root="$(git rev-parse --show-toplevel)"
|
||||||
|
archive_root="$(mktemp -d "${TMPDIR:-/tmp}/video-editing-clean.XXXXXX")"
|
||||||
|
git -C "$repo_root" archive --format=tar HEAD | tar -xf - -C "$archive_root"
|
||||||
|
(
|
||||||
|
cd "$archive_root"
|
||||||
|
test ! -e .venv-local-cv
|
||||||
|
test ! -e .venv-local-asset
|
||||||
|
test -d "${MAVEN_REPO:?set MAVEN_REPO to the approved offline Maven cache}"
|
||||||
|
mvn -o -Dmaven.repo.local="$MAVEN_REPO" verify
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Interpret the result:
|
||||||
|
|
||||||
|
| Observation | Meaning | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| Maven cannot resolve an artifact or plugin in offline mode | The approved cache is incomplete. | Fix the approved acquisition manifest/cache. Do not remove `-o` or enable network. |
|
||||||
|
| Both FFmpeg integration tests are skipped | `ffmpeg` or `ffprobe` is unavailable. | Fail media-capable validation; provision the approved pair. |
|
||||||
|
| `LocalAssetGenerationStageTest.reusesExistingSharedAssetsAndSynthesizesMissingVoiceover` fails | This is the valid verified clean-archive result when Maven runs from inside the archive. | Do not seed an untracked virtual environment to hide it. Make the test hermetic and make the runtime fail closed through change control. |
|
||||||
|
| The clean archive passes but the workspace fails | Workspace-generated state or source changes affect the result. | Compare tracked changes and environment; do not delete user work. |
|
||||||
|
| The workspace passes but the clean archive fails | The workspace relies on untracked state. | Treat clean reproducibility as failed. |
|
||||||
|
|
||||||
|
### Verified clean result on 2026-07-21
|
||||||
|
|
||||||
|
With Java `21.0.10`, Maven `3.6.3`, FFmpeg/ffprobe `7.1.1`, and a populated host Maven cache, the valid run changed directory into the archived checkout, compiled 157 main sources, and ran 216 tests in 56 test classes. It finished with **one failed assertion**: `LocalAssetGenerationStageTest.reusesExistingSharedAssetsAndSynthesizesMissingVoiceover` expected `voiceover.wav` to exist.
|
||||||
|
|
||||||
|
An earlier run reported two failures because it invoked Maven with `-f` from the long-lived workspace. That violated the test's process-working-directory assumption and is not valid clean-checkout evidence. The workspace's 216 tests passed while `.venv-local-asset` existed locally but was untracked. Together these observations establish a reproducibility defect, not the exact behavior of every clean machine. The macOS, Maven, Java, and FFmpeg patch versions are observations from one machine, not repository requirements.
|
||||||
|
|
||||||
|
## Working-directory and executable traps
|
||||||
|
|
||||||
|
The defaults contain repository-relative paths such as `./tools/run_local_cv_worker.sh`, `./tools/local_asset_worker.py`, and `./.venv-local-asset/bin/python`. Java converts some of these to absolute paths relative to the process working directory.
|
||||||
|
|
||||||
|
Follow these rules:
|
||||||
|
|
||||||
|
1. Run current development Maven commands from the repository root.
|
||||||
|
2. Do not start the packaged JAR from an arbitrary directory while retaining relative worker paths.
|
||||||
|
3. In a production artifact, configure absolute paths inside the immutable runtime layout.
|
||||||
|
4. Verify every directly executed POSIX script with `test -x`; a shebang does not compensate for a missing execute bit.
|
||||||
|
5. Invoke `.py` files through the approved Python interpreter. Do not make source scripts executable as an ad hoc workaround.
|
||||||
|
|
||||||
|
Use these diagnostics without starting workers:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -r tools/local_asset_worker.py
|
||||||
|
test -x tools/run_local_cv_worker.sh
|
||||||
|
test -x tools/run_local_asset_worker.sh
|
||||||
|
bash -n tools/run_local_cv_worker.sh
|
||||||
|
bash -n tools/run_local_asset_worker.sh
|
||||||
|
git ls-files .venv-local-cv .venv-local-asset
|
||||||
|
```
|
||||||
|
|
||||||
|
At current `HEAD`, the third command is expected to fail and the final command prints nothing. That is evidence about tracked state, not permission to alter it locally.
|
||||||
|
|
||||||
|
## Known test signals and warnings
|
||||||
|
|
||||||
|
| Signal | Current interpretation |
|
||||||
|
|---|---|
|
||||||
|
| `Standard Commons Logging discovery... remove commons-logging.jar` | Observed classpath warning. Track it as dependency hygiene; do not suppress it as a build fix. |
|
||||||
|
| Byte Buddy dynamic-agent warning | Mockito currently loads an agent dynamically; a future JDK may disallow this default. Java 21 tests currently continue. |
|
||||||
|
| Flyway warns that H2 `2.2.224` is newer than its tested H2 support | Current tests use H2 for Spring contexts. This warning does not prove PostgreSQL behavior. |
|
||||||
|
| `spring.jpa.open-in-view is enabled by default` during a test context | Investigate effective test configuration; main `application.properties` declares it false. Do not dismiss the mismatch. |
|
||||||
|
| Local-CV fallback warnings in unit tests | Some tests intentionally exercise fallback behavior. They do not certify a production local CV model. |
|
||||||
|
| FFmpeg test skipped | The two `@EnabledIf` integration classes skip when tools are unavailable. A green count with skips is weaker evidence. |
|
||||||
|
|
||||||
|
## TARGET: reproducible, dependency-locked build campaign
|
||||||
|
|
||||||
|
Nothing in this section is implemented merely because it is listed. Complete and promote each phase through `video-editing-change-control`.
|
||||||
|
|
||||||
|
### Phase 1: pin the build toolchain
|
||||||
|
|
||||||
|
- [ ] Add a Maven wrapper with an exact Maven distribution URL and verified distribution checksum.
|
||||||
|
- [ ] Pin the Java 21 vendor/distribution and patch policy in CI/image metadata; enforce Java major version and Maven version at build start.
|
||||||
|
- [ ] Centralize deliberate plugin and dependency version overrides; preserve Spring Boot dependency management unless an ADR proves why an override is required.
|
||||||
|
- [ ] Reject dependency versions containing `SNAPSHOT`, milestone, RC, or other prerelease markers. Treat the project's own `1.0-SNAPSHOT` separately and replace it with traceable release versioning before release.
|
||||||
|
- [ ] Set reproducible archive metadata and prove two isolated builds of the same commit and inputs produce identical artifact hashes.
|
||||||
|
|
||||||
|
Gate: a clean checkout executes one documented wrapper command with no host Maven dependency, and the build records Java, Maven, OS, CPU, commit, and artifact hash.
|
||||||
|
|
||||||
|
### Phase 2: freeze dependency acquisition
|
||||||
|
|
||||||
|
- [ ] Resolve Maven only through an approved immutable repository mirror; capture the complete dependency/plugin graph and checksums.
|
||||||
|
- [ ] Produce a Python lock per supported OS/CPU/runtime, including transitive versions and hashes.
|
||||||
|
- [ ] Build an approved wheelhouse; install with offline and hash-required semantics. Never run unconstrained `pip install` in service startup.
|
||||||
|
- [ ] Generate an SBOM and license inventory for Java, Python, FFmpeg, native libraries, models, fonts, LUTs, music, and SFX.
|
||||||
|
- [ ] Add vulnerability and license policy gates. Zero unresolved critical/high findings may ship without documented risk acceptance.
|
||||||
|
|
||||||
|
Gate: remove all repository and model-hub access, delete mutable caches, rebuild only from the approved inputs, and obtain the same dependency inventory.
|
||||||
|
|
||||||
|
### Phase 3: provision local models and media tools
|
||||||
|
|
||||||
|
- [ ] Give every model an immutable identifier, exact file list, SHA-256 checksums, source, license/usage rights, architecture/runtime compatibility, and acceptance evidence.
|
||||||
|
- [ ] Configure models by explicit local paths. Prohibit library shorthand such as `yolov8n.pt` or `musicgen-small` when it can trigger online resolution.
|
||||||
|
- [ ] Pin FFmpeg/ffprobe to one approved build per platform, including configure flags and codec/filter inventory.
|
||||||
|
- [ ] Remove or fail closed on automatic bootstrap, heuristic substitution, host speech fallback, silence, and tone generation.
|
||||||
|
- [ ] Verify local runtime readiness before accepting work; absence or checksum mismatch must produce a non-ready state and actionable error.
|
||||||
|
|
||||||
|
Gate: startup and a representative highlight run succeed with egress denied and read-only dependency/model directories; removing one required model causes a deterministic readiness failure and no placeholder output.
|
||||||
|
|
||||||
|
### Phase 4: make tests hermetic and cross-platform
|
||||||
|
|
||||||
|
- [ ] Replace `LocalAssetGenerationStageTest` host-process coupling with controlled test doubles or checked test fixtures; separately test the real packaged local runtimes.
|
||||||
|
- [ ] Require explicit FFmpeg integration execution instead of silently accepting skips in the full gate.
|
||||||
|
- [ ] Test PostgreSQL behavior with a real approved PostgreSQL test environment; current Spring tests use H2.
|
||||||
|
- [ ] Run the same build portfolio on macOS developer architectures and supported Linux production architectures.
|
||||||
|
- [ ] Add architecture tests, formatting, static analysis, API checks, security scans, secrets scanning, and deterministic end-to-end smoke tests only where each closes a stated acceptance criterion.
|
||||||
|
|
||||||
|
Gate: the clean archive passes from an empty workspace on every supported target without user-home state, network, host speech tools, or untracked files.
|
||||||
|
|
||||||
|
### Phase 5: certify the release artifact
|
||||||
|
|
||||||
|
- [ ] Build an OCI-compatible, non-root runtime image or equivalent immutable artifact for each supported architecture.
|
||||||
|
- [ ] Keep build tools out of the runtime; include only the approved Java runtime, application, FFmpeg/ffprobe, Python runtime, models, and licensed assets needed for operation.
|
||||||
|
- [ ] Generate provenance linking commit, dependency/model manifests, SBOM, tests, scans, and image digest.
|
||||||
|
- [ ] Sign the release artifact and verify the signature before deployment.
|
||||||
|
- [ ] Validate offline startup, graceful shutdown, readiness, resource limits, and representative media execution in the production-like environment.
|
||||||
|
|
||||||
|
Gate: promotion consumes the already-built digest; it does not rebuild per environment. Rollback and roll-forward use known digests and compatible data migrations.
|
||||||
|
|
||||||
|
## Evidence checklist
|
||||||
|
|
||||||
|
Attach this information to any environment or build claim:
|
||||||
|
|
||||||
|
- [ ] Commit identifier and `git status --short` output.
|
||||||
|
- [ ] `java -version`, `javac -version`, and `mvn -version`.
|
||||||
|
- [ ] OS and CPU architecture.
|
||||||
|
- [ ] FFmpeg and ffprobe versions plus build configuration.
|
||||||
|
- [ ] Maven cache manifest/digest and confirmation that offline mode/network denial were active.
|
||||||
|
- [ ] Python lock/wheelhouse digest and exact interpreter version, if local workers were in scope.
|
||||||
|
- [ ] Model and licensed-asset manifest/digest, if cinematic processing was in scope.
|
||||||
|
- [ ] Exact Maven command, test totals, skips, failures, and JaCoCo gate result.
|
||||||
|
- [ ] Application JAR/image digest and SBOM/provenance references, when implemented.
|
||||||
|
- [ ] A clear `CURRENT FAILURE`, `CANDIDATE`, or `PROMOTED` label. Never call an unimplemented target production-ready.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Re-verify the Java, Spring Boot, AWS BOM, project, and JaCoCo declarations:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '<java.version>|spring-boot-starter-parent|aws.sdk.version|<version>1.0-SNAPSHOT</version>|jacoco-maven-plugin|<minimum>1.00</minimum>' pom.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify build-system and delivery gaps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find . -maxdepth 3 -type f \( -name mvnw -o -name mvnw.cmd -o -name maven-wrapper.properties -o -name 'Dockerfile*' -o -name 'compose*.yml' -o -path './.github/workflows/*' -o -name '*.lock' \) -print
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify Python pins, bootstrap/download paths, placeholders, and test coupling:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '^[A-Za-z0-9_.-]+(\[.*\])?(==.*)?$|pip install|YOLO\(|get_pretrained|write_silence|write_fallback_tone|fallbackTone|runSpeechFallback|LocalAssetSynthesizer' tools src/main/java src/test/java/org/example/videoclips/editing/LocalAssetGenerationStageTest.java
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify relative paths, startup defaults, and test safety overrides:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'auto-start:|enabled:|render-enabled:|fallback-to-heuristic:|bootstrap-script:|python-binary:|script:' src/main/resources/application.yml src/test/resources/application.properties src/main/java/org/example/videoclips/config/VideoClippingProperties.java
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify tracked permissions and untracked-runtime assumptions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git ls-files -s tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh tools/local_asset_worker.py
|
||||||
|
git ls-files .venv-local-cv .venv-local-asset
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-run the archived offline experiment from **Reproduce the clean-checkout gate** after build, test, Python, model, or worker changes. Update the dated result only from that isolated evidence; do not copy counts or versions from a long-lived workspace.
|
||||||
|
|
@ -0,0 +1,280 @@
|
||||||
|
---
|
||||||
|
name: video-editing-change-control
|
||||||
|
description: Load before proposing, implementing, reviewing, merging, releasing, or rolling back any change to this video-editing repository, especially changes to runtime behavior, configuration defaults, APIs, persistence, FFmpeg rendering, highlight selection, local CV/TTS/music/SFX models, assets, security, deployment, or production promotion. Also load when deciding what evidence and approvals a change requires.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Change Control
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Control every change from intent through evidence and promotion. Treat **promotion** as moving a change into a more trusted environment or declaring it production-ready. Treat a **gate** as evidence or approval that must exist before promotion.
|
||||||
|
|
||||||
|
Use the repository state dated **2026-07-21** as the current baseline. The Fortune 500 requirements below are the target acceptance standard, not a description of the repository today.
|
||||||
|
|
||||||
|
## When not to use this skill
|
||||||
|
|
||||||
|
- Do not use this as a configuration catalog. Load `video-editing-config-and-flags`.
|
||||||
|
- Do not use this to diagnose a failure. Load `video-editing-debugging-playbook`, then return here before changing behavior.
|
||||||
|
- Do not use this as the test procedure. Load `video-editing-validation-and-qa`, then attach its evidence here.
|
||||||
|
- Do not use this to reconstruct an old failure. Load `video-editing-failure-archaeology`.
|
||||||
|
- Do not use this to learn module boundaries. Load `video-editing-architecture-contract`.
|
||||||
|
- Do not use this to operate schedulers or renders. Load `video-editing-run-and-operate`; never use an operating procedure to bypass the approval gates here.
|
||||||
|
- Do not use this to judge cinematic quality by eye. Load the highlight-quality campaign and proof/analysis skills; bring their measurements back to this gate.
|
||||||
|
|
||||||
|
## Establish the baseline first
|
||||||
|
|
||||||
|
Run these read-only checks from the repository root before editing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short
|
||||||
|
git branch --show-current
|
||||||
|
git log -10 --oneline --decorate
|
||||||
|
git diff --stat
|
||||||
|
git diff
|
||||||
|
```
|
||||||
|
|
||||||
|
Record the starting commit, existing uncommitted files, scope, and expected observable outcome. Do not overwrite, reset, clean, or reformat unrelated work.
|
||||||
|
|
||||||
|
Confirm important current facts rather than trusting a plan checkbox:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -f pom.xml
|
||||||
|
test -f README.md
|
||||||
|
test -x mvnw
|
||||||
|
find .github -maxdepth 3 -type f -print 2>/dev/null
|
||||||
|
find src/test/java -name '*Test.java' | sort
|
||||||
|
rg -n 'enabled:|auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:|strict-runtime:' src/main/resources
|
||||||
|
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone' tools src/main
|
||||||
|
```
|
||||||
|
|
||||||
|
As of 2026-07-21, `pom.xml` exists; `README.md`, Maven Wrapper, `.github/` CI, container definitions, and deployment definitions do not. The current working tree passes `mvn -q -o verify` with 245 tests in 62 test classes. Phase 1 also reproduced a clean archived-checkout `mvn verify` failure in `LocalAssetGenerationStageTest` because that revision could depend on an untracked local asset virtual environment or an audible TTS fallback. The working tree removed that fallback dependency, but do not call the build reproducible until a fresh clean, offline run passes on macOS and Linux.
|
||||||
|
|
||||||
|
## Apply the non-negotiables
|
||||||
|
|
||||||
|
Stop a change immediately when it violates any of the seven user prohibitions below. **These prohibitions have no waiver or risk-acceptance path.** Existing violations are gaps to remove, never precedent. `video-editing-change-control` is the authoritative home for this policy; sibling skills state only workflow-specific consequences.
|
||||||
|
|
||||||
|
| Rule | Required behavior | Why and repository evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| No automatic dependency or model downloads | Provision signed, checksummed dependencies and model weights before startup. Model/media inference and acquisition must make no network connection, including loopback remote procedure calls (RPCs). Fail startup or the affected job when an approved local artifact is absent. | `tools/run_local_asset_worker.sh` still creates a virtualenv and runs `pip install`; do not invoke it. The working-tree runtime verifier no longer bootstraps and the asset worker accepts only existing local model paths with offline hub flags. |
|
||||||
|
| No external AI service in the production media path | Keep highlight identification, direction, CV, voiceover, music, and SFX inference within the approved runtime using pre-provisioned local models. | `docs/cinematic-highlight-operator-checklist.md` currently instructs an operator to run Codex, Claude, or another filesystem-capable AI. That is a manual development workflow, not the target production architecture. |
|
||||||
|
| No unlicensed asset | Require machine-readable origin, license identifier/terms, checksum, and allowed use for every music, SFX, voice, font, LUT, and model artifact before render approval. | `AssetLicensePolicy` now excludes media without a nonblank adjacent `.license.txt`, preserves that sidecar during copies, and render preflight requires licensed requested audio. A sidecar string alone still does not prove origin, checksum, allowed use, or commercial rights. |
|
||||||
|
| No placeholder silence or tones | Fail closed when requested voiceover, music, or SFX cannot be generated or resolved. Never promote synthetic silence or a diagnostic sine tone as a finished asset. | The 2026-07-21 working tree removes Python/Java silence, tone, and host-speech success paths and tests missing-asset render rejection. Preserve that gate. |
|
||||||
|
| No unapproved rendering | Require a validated plan, complete licensed assets, measured QA, and an explicit approval artifact or auditable approval record before FFmpeg starts. | Highlight defaults now keep rendering disabled and require `approved.flag` if enabled. The flag is not bound to source/plan/configuration digests, and `POST /v1/edit-projects/{projectId}:render` still has no approval gate, so neither path is production authorization. |
|
||||||
|
| No unapproved network access | Use in-process calls or explicitly designed non-network inter-process communication for model/media inference. The current loopback CV HTTP worker is noncompliant and cannot be a certified path. Separately, API, PostgreSQL, object-storage, and telemetry connections are production integrations: each requires explicit approval, authentication/authorization where applicable, transport security, least privilege, bounded timeouts, and tested failure behavior. Permit no external AI or unapproved egress. | Local CV currently uses configured loopback HTTP at `127.0.0.1:8091`; S3 and HTTP APIs also exist. Loopback is still network, and network presence is not approval. |
|
||||||
|
| No silent production-default change | List old value, new value, environments affected, migration, rollback, and operator impact. Require explicit review for every default or `matchIfMissing` change. | `application.yml` currently enables folder scheduling, editing, local workers, the local director, and highlight ingestion by default. Highlight rendering now defaults off, but a normal start can still consume files and start the network-capable CV bootstrap path. |
|
||||||
|
| No claim without measured evidence | Label plans, generated artifacts, and heuristic checks accurately. A passing test or existing MP4 is not proof of cinematic or production quality. | Commit `5d889b0` explicitly recorded “working version but not cinematic.” Several highlight QA checks in `HighlightFfmpegRenderer.buildQaReport` are hard-coded `true`, including duration, asset, overlay, and mastering claims. |
|
||||||
|
|
||||||
|
## Classify the change
|
||||||
|
|
||||||
|
Choose the highest applicable class. Splitting a large change does not lower its class when the parts jointly alter behavior.
|
||||||
|
|
||||||
|
| Class | Examples here | Minimum route |
|
||||||
|
|---|---|---|
|
||||||
|
| C0: knowledge-only | Skills, docs, comments, diagnostic scripts that cannot execute in a build or runtime | Ground-truth review, command/path verification, docs consistency review |
|
||||||
|
| C1: isolated implementation | Internal refactor with identical API, config, storage, scheduling, render, and media output behavior; focused deterministic test | Focused tests, full offline unit suite, compatibility review |
|
||||||
|
| C2: behavior or contract | API/DTO/error change, config/flag, dependency/plugin, scheduler state, filesystem contract, queue/retry, observability, performance, or renderer command | C1 plus integration/contract evidence, migration and rollback, security and operations review |
|
||||||
|
| C3: high-impact | Highlight ranking, director plan semantics, local model/runtime, asset generation/licensing, automatic rendering, auth, persistence migration, deployment, data deletion, production default | All gates; adversarial validation; macOS and Linux evidence; explicit security, data, operations, and product/creative approval |
|
||||||
|
| Emergency | Active confidentiality, integrity, availability, or legal incident | Use the smallest reversible mitigation, preserve evidence, obtain incident authority, test the failure path, and complete C2/C3 follow-up before normal promotion |
|
||||||
|
|
||||||
|
Treat any unknown impact as C3 until discriminating evidence lowers it.
|
||||||
|
|
||||||
|
## Write the change record
|
||||||
|
|
||||||
|
For every C2/C3 decision, add or update an ADR when the repository has an ADR location. Until that location exists, include this record in the pull request or review artifact; do not invent a docs-of-record location silently.
|
||||||
|
|
||||||
|
```text
|
||||||
|
Requirement:
|
||||||
|
Selected approach:
|
||||||
|
Alternatives considered:
|
||||||
|
Benefits and trade-offs:
|
||||||
|
Operational consequences:
|
||||||
|
Security implications:
|
||||||
|
Verification evidence:
|
||||||
|
Revisit conditions:
|
||||||
|
Owner and review date:
|
||||||
|
Change class and affected environments:
|
||||||
|
Rollback or roll-forward procedure:
|
||||||
|
```
|
||||||
|
|
||||||
|
State what will not change. Map each claim to a test, measurement, schema comparison, scan, or reviewed artifact.
|
||||||
|
|
||||||
|
## Execute the gates
|
||||||
|
|
||||||
|
### Gate 1: scope and dependency direction
|
||||||
|
|
||||||
|
- Keep the modular monolith unless measured business requirements justify distribution.
|
||||||
|
- Put business rules in domain/application code; keep controllers, persistence, HTTP, filesystem, FFmpeg, and model processes at adapter boundaries.
|
||||||
|
- Do not add a dependency, abstraction, repository base class, mapper layer, CQRS infrastructure, cache, circuit breaker, virtual thread, or service merely to satisfy a pattern checklist.
|
||||||
|
- For each new dependency, record maintenance status, stable version, license, vulnerability result, why Spring Boot dependency management is insufficient if overridden, and removal plan.
|
||||||
|
- Block snapshots, milestones, release candidates, deprecated dependencies, and undocumented overrides.
|
||||||
|
- Require automated architecture rules before claiming module boundaries, no cycles, or inward dependencies are enforced. None were found in the 2026-07-21 test inventory.
|
||||||
|
|
||||||
|
### Gate 2: forbidden-path scan
|
||||||
|
|
||||||
|
Run before review:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'pip install|curl |wget |get_pretrained|from_pretrained|snapshot_download' . --glob '!target/**' --glob '!.git/**'
|
||||||
|
rg -n 'write_silence|write_fallback_tone|fallback-to-heuristic|fallback.*copy' tools src/main docs
|
||||||
|
rg -n 'render-enabled:|auto-render|require.*approval|auto-start:|matchIfMissing' src/main src/test docs
|
||||||
|
rg -n -i 'api[_-]?key|secret|password|token|private[_-]?key' . --glob '!target/**' --glob '!.git/**'
|
||||||
|
rg -n '<version>.*(SNAPSHOT|M[0-9]+|RC[0-9]*)</version>' pom.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
Classify every hit. Existing violations do not authorize new ones. Any new runtime download, placeholder, approval bypass, secret, or prerelease dependency blocks the change.
|
||||||
|
|
||||||
|
### Gate 3: deterministic verification
|
||||||
|
|
||||||
|
Use the locally provisioned toolchain without network access:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o test
|
||||||
|
mvn -o verify
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
`mvn verify` currently runs Surefire and JaCoCo; JaCoCo enforces 100% instruction, line, and branch coverage only for `org.example.videoclips.folder`. It does not provide the target 90% domain/application or 80% overall gate. Add focused tests at the lowest useful layer and use real FFmpeg/model/PostgreSQL infrastructure where behavior depends on it.
|
||||||
|
|
||||||
|
For clean-checkout evidence without modifying the current tree:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tmp_dir="$(mktemp -d)"; git archive HEAD | tar -x -C "$tmp_dir"; (cd "$tmp_dir" && mvn -o verify)
|
||||||
|
```
|
||||||
|
|
||||||
|
Run that only after Maven artifacts and approved model/runtime assets have been provisioned outside application startup. Record OS, architecture, Java, Maven, FFmpeg/ffprobe, Python, model checksums, command, duration, and result. A cache-dependent pass is not a reproducible-build proof.
|
||||||
|
|
||||||
|
### Gate 4: contract, data, security, and resilience
|
||||||
|
|
||||||
|
For affected changes, require all applicable evidence:
|
||||||
|
|
||||||
|
- Validate every public endpoint's success, validation, authentication, authorization, idempotency, bounded-collection, and failure cases against a versioned OpenAPI contract.
|
||||||
|
- Use RFC 9457 Problem Details without internal paths, exceptions, stack traces, storage keys, secrets, or sensitive payloads.
|
||||||
|
- Exercise Flyway migrations from an empty real PostgreSQL Testcontainer; verify constraints, indexes, transaction/concurrency behavior, backward-compatible rollout, backup/recovery, and roll-forward. H2 compatibility mode is not PostgreSQL evidence.
|
||||||
|
- Threat-model trust boundaries and apply server-side authorization, least privilege, restrictive CORS/headers, secret management, audit events, and sensitive-log redaction.
|
||||||
|
- Verify explicit timeouts, bounded safe retries, idempotency, partial failure, and resource limits for each network/process integration.
|
||||||
|
- Produce dependency, static, secret, container, and Software Bill of Materials (SBOM) evidence. Release with zero unresolved critical/high findings.
|
||||||
|
|
||||||
|
The current `pom.xml` has no Spring Security, OpenAPI, Testcontainers, ArchUnit/Spring Modulith, mutation testing, dependency lock, or security scanning integration. Do not mark this gate passed from intent or documentation.
|
||||||
|
|
||||||
|
### Gate 5: media and local-model evidence
|
||||||
|
|
||||||
|
For highlight, render, or asset changes, require more than structural output:
|
||||||
|
|
||||||
|
- Pin every local model and asset by immutable version/checksum and approved license; pre-provision it for macOS development and the target Linux/VPS/cloud runtime.
|
||||||
|
- Start and run model/media inference with all network access denied, including loopback RPC. Use in-process or approved non-network IPC. Missing or corrupt models must fail closed with a diagnosable, non-sensitive error.
|
||||||
|
- Validate highlight choices against versioned representative footage and annotations. Report precision/recall or an explicitly defined ranking metric, category slices, negative cases, and regressions.
|
||||||
|
- Probe actual output duration, codecs, streams, frame rate, resolution, black frames, freeze frames, clipping, integrated loudness, true peak, long silence, asset presence, and A/V synchronization.
|
||||||
|
- Verify voiceover intelligibility and timing, licensed music/SFX provenance, ducking, and that no silence/tone placeholder entered the final mix.
|
||||||
|
- Compare against a named baseline using predicted numeric outcomes and a blinded human rubric. Preserve manifests, commands, model versions, inputs or legal fixture references, measurements, and reviewer scores.
|
||||||
|
- Inspect each `qa-report.json` check's mechanism. Current duration/black/silence/sample-peak checks are measured; asset, mastering-filter, and overlay-bound checks remain structural, and the report is not a creative-quality certificate.
|
||||||
|
|
||||||
|
### Gate 6: operability and promotion
|
||||||
|
|
||||||
|
Promote the same immutable, platform-neutral application artifact across environments for a given release. Package it into signed, platform-specific runtime/model/image bundles for each supported OS/CPU target. Promote a bundle unchanged between environments with the same target; do not rebuild it per environment.
|
||||||
|
|
||||||
|
Require:
|
||||||
|
|
||||||
|
- typed validated external configuration; no embedded secret; safe disabled defaults; an environment diff;
|
||||||
|
- structured logs, bounded metrics, traces, secured health/readiness/liveness, business service-level indicators (SLIs), actionable alerts, and linked runbooks;
|
||||||
|
- non-root reproducible Open Container Initiative (OCI) image, vulnerability scan, read-only filesystem where practical, graceful shutdown, resource limits, and validated deployment definitions;
|
||||||
|
- measured workload with p50/p95/p99, throughput, errors, CPU, memory, GC, pools, threads, a 60-minute endurance run, downstream slowdown, and 2x expected peak behavior;
|
||||||
|
- commit-to-artifact provenance, SBOM, controlled approval, smoke/acceptance tests, and tested rollback or roll-forward;
|
||||||
|
- named owner, Recovery Time Objective (RTO), Recovery Point Objective (RPO), backup/restore, dependency-outage, migration-failure, and rollback procedures where applicable.
|
||||||
|
|
||||||
|
The user-provided default API targets of 200 requests/second/instance, p95 below 200 ms, p99 below 500 ms, error rate below 0.1%, no endurance memory growth, and graceful behavior at 2x peak are candidate baselines. Adjust them to the measured asynchronous video workload through an approved decision; do not claim they are currently met.
|
||||||
|
|
||||||
|
## Review the change
|
||||||
|
|
||||||
|
Require an independent reviewer for C2 and at least domain/architecture plus security/operations reviewers for C3. Add a creative-quality reviewer for highlight/render/audio changes and a data owner for persistence changes.
|
||||||
|
|
||||||
|
Review in this order:
|
||||||
|
|
||||||
|
1. Reproduce the original symptom or requirement.
|
||||||
|
2. Verify that one mechanism explains positive and negative observations.
|
||||||
|
3. Attempt an adversarial refutation: missing model, denied egress, corrupt asset, invalid plan, duplicate job, process timeout, disk pressure, restart, and approval absence as applicable.
|
||||||
|
4. Inspect the complete diff, configuration delta, schema/API delta, dependency tree, generated artifacts, and rollback.
|
||||||
|
5. Re-run the narrow test, then all required gates from a clean checkout.
|
||||||
|
6. Record residual risk and explicit approval; never infer approval from silence, a passing build, or an existing flag value.
|
||||||
|
|
||||||
|
Use the target human rubric as the final release scorecard: score architectural clarity, domain modeling, maintainability, security, tests, API, data, resilience, observability, performance, cloud operation, CI/CD, developer experience, docs, and operational readiness from 0 to 4 with concrete evidence. Block reference-architecture status if any category is below 3, the average is below 3.5, or security, data integrity, testing, or operational readiness is below 3.
|
||||||
|
|
||||||
|
## Promote or reject
|
||||||
|
|
||||||
|
Promote only when every applicable gate is green and the artifact is approved. The repository does **not** meet the production/reference-architecture definition as of 2026-07-21: the clean build is not reproducible, required security/API/architecture/CI/container/deployment gates are absent, and the production-quality local-model highlight flow has known fail-open behavior.
|
||||||
|
|
||||||
|
Reject or retain behind a disabled experimental flag when evidence is incomplete. Give every experimental flag an owner, safe default, expiry, removal criterion, and telemetry. Never change a production-facing default merely to make an experiment run.
|
||||||
|
|
||||||
|
For rollback:
|
||||||
|
|
||||||
|
- Prefer disabling a new path with an already-reviewed safe flag or rolling forward with a corrective artifact.
|
||||||
|
- Never edit an already released Flyway migration; write a new recovery migration and test it on production-shaped data.
|
||||||
|
- Preserve render manifests, audit records, failing inputs where legally permitted, and model/artifact checksums.
|
||||||
|
- Re-run smoke, integrity, and observability checks after rollback. A process restart alone is not rollback evidence.
|
||||||
|
|
||||||
|
## Historical rationale
|
||||||
|
|
||||||
|
Use these incidents to challenge recurring shortcuts; load `video-editing-failure-archaeology` for the full chronology.
|
||||||
|
|
||||||
|
| Incident | Lesson enforced here |
|
||||||
|
|---|---|
|
||||||
|
| Upload completion was initially a metadata flip; commit `7e8a214` routed it through object storage. | A state label must reflect the external side effect, not intention. Test the adapter boundary. |
|
||||||
|
| Generated clips were metadata-only until commit `66e998e` persisted real object locations. Signed URLs later derived fake paths until commit `7307082` used the persisted `objectKey`. | Verify artifact existence, identity, and retrieval end to end. Do not synthesize locations. |
|
||||||
|
| Worker files accumulated until commit `8f4804c` added cleanup in a `finally` block. | Test cleanup on success, failure, retry, and interruption; include disk saturation in operations review. |
|
||||||
|
| Internal storage locations leaked through API/events until commit `0d45f89` removed and tested them. | Treat paths, storage keys, and process output as boundary-sensitive data. |
|
||||||
|
| Retryable database-queue exceptions became terminal failures until commit `1737d8b` separated retry from DLQ exhaustion. | Model transient and terminal states separately; prove duplicate and retry behavior. |
|
||||||
|
| Commit `5d889b0` documented a renderer that worked but was not cinematic. Later work added category planning, validation, effects, QA, and approval. | Technical completion is not product quality. Predict and measure creative outcomes independently. |
|
||||||
|
| Commit `9536928` added approval to the local-director path; later highlight work initially defaulted to rendering without approval. The working tree now defaults highlight rendering off and approval on. | Apply policy to every workflow; similar names and flags do not imply equivalent gates. Keep tests on both property defaults. |
|
||||||
|
| Commit `97ba827` added automatic bootstrap and local generation with runtime installation/model-resolution and successful silence/tone fallbacks. The working tree removed asset-runtime bootstrap and placeholder success, but standalone bootstrap and the CV path remain network-capable. | “Local” is not the same as offline, provisioned, licensed, or fail-closed. Test absence and denied-egress paths. |
|
||||||
|
|
||||||
|
## Definition of done
|
||||||
|
|
||||||
|
Check every applicable item; do not weaken the list in a feature branch.
|
||||||
|
|
||||||
|
- [ ] Requirement, class, scope, affected environments, and eight-part decision record are explicit.
|
||||||
|
- [ ] No forbidden runtime download, external AI, unlicensed asset, placeholder, unapproved render, egress dependency, or silent default change remains.
|
||||||
|
- [ ] Core business rules remain independent of adapters and automated architecture tests enforce the intended boundaries.
|
||||||
|
- [ ] Clean offline one-command build passes on supported macOS and Linux from committed sources plus separately provisioned approved artifacts.
|
||||||
|
- [ ] Focused, full, architecture, API, persistence, integration, security, resilience, and end-to-end tests pass as applicable.
|
||||||
|
- [ ] API/schema/config compatibility, migration, idempotency, failure semantics, and rollback are verified.
|
||||||
|
- [ ] Dependency, static, secret, container, vulnerability, license, SBOM, signing/provenance, and reproducibility evidence passes as applicable.
|
||||||
|
- [ ] Logs, metrics, traces, health, SLI/dashboard/alert, capacity, shutdown, recovery, and runbooks are verified in production-like staging.
|
||||||
|
- [ ] Media/model output passes objective probes, offline/fail-closed tests, baseline comparison, representative/negative fixtures, and blinded human review.
|
||||||
|
- [ ] The immutable application artifact and target-specific signed bundle are approved and traceable from commit through controlled promotion without per-environment rebuild.
|
||||||
|
- [ ] Each target rubric category has linked evidence; no score is below 3 and the average is at least 3.5.
|
||||||
|
- [ ] Docs, OpenAPI, diagrams, configuration reference, threat model, testing/deployment/operations guides, and ADRs match implementation.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
This skill was grounded in repository source, configuration, tests, docs, and Git history on **2026-07-21**. Re-verify volatile facts before relying on them:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log -12 --oneline --decorate
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find src/test/java -name '*Test.java' | wc -l; awk -F'[,: ]+' '/Tests run:/{sum+=$3;fail+=$5;err+=$7;skip+=$9} END{print "tests="sum,"failures="fail,"errors="err,"skipped="skip}' target/surefire-reports/*.txt 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for p in README.md mvnw .mvn .github Dockerfile compose.yml docker-compose.yml; do test -e "$p" && echo "present $p" || echo "absent $p"; done
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'enabled:|auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:|strict-runtime:' src/main/resources/application*.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|local_asset_runtime_degraded' tools src/main
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'duration_matches_timeline|required_assets_resolved|text_overlays_safe|audio_mastering_applied' src/main/java/org/example/videoclips/editing/*Renderer.java
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'spring-security|springdoc|testcontainers|archunit|modulith|dependency-check|cyclonedx|pitest' pom.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o verify
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,476 @@
|
||||||
|
---
|
||||||
|
name: video-editing-cinematic-highlights-campaign
|
||||||
|
description: "Load this skill when planning, implementing, debugging, or promoting the single-source cinematic highlight campaign: local-only highlight identification and editing with production visuals, music, SFX, and voiceover; especially for candidate generation, an offline director, model packaging, director-plan safety, measured media QA, resumability, source-to-final tests, or macOS/Linux parity."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Cinematic Highlights Campaign
|
||||||
|
|
||||||
|
Status verified against the repository on **2026-07-21**. Treat every path or component marked
|
||||||
|
`[PROPOSED]` as work that does not exist yet. Introduce it only through
|
||||||
|
`video-editing-change-control`.
|
||||||
|
|
||||||
|
## Use this campaign correctly
|
||||||
|
|
||||||
|
Use this skill to drive the hardest live problem: turn one source video into correctly selected,
|
||||||
|
high-cinematic-quality highlights with production visuals, music, sound effects (SFX), and
|
||||||
|
voiceover, using only pre-provisioned local models resident in the service runtime.
|
||||||
|
|
||||||
|
Do **not** use it for:
|
||||||
|
|
||||||
|
| Need | Use instead |
|
||||||
|
|---|---|
|
||||||
|
| A routine, already-understood patch | `video-editing-change-control` |
|
||||||
|
| A symptom with no established root cause | `video-editing-debugging-playbook` |
|
||||||
|
| Media theory or FFmpeg fundamentals | `cinematic-media-engineering-reference` |
|
||||||
|
| General test evidence or existing test inventory | `video-editing-validation-and-qa` |
|
||||||
|
| Existing measurement commands and artifact inspection | `video-editing-diagnostics-and-tooling` |
|
||||||
|
| Post-triage causal proof or first-principles derivation | `video-editing-proof-and-analysis-toolkit` |
|
||||||
|
| Environment setup or normal operation | `video-editing-build-and-env` or `video-editing-run-and-operate` |
|
||||||
|
|
||||||
|
Never use this campaign to route around approval, licensing, security, or change control.
|
||||||
|
|
||||||
|
## Terms and hard constraints
|
||||||
|
|
||||||
|
| Term | Meaning here |
|
||||||
|
|---|---|
|
||||||
|
| Certified fixture | Rights-cleared source media plus immutable annotations, checksum, provenance, and an assigned development or locked-holdout split. |
|
||||||
|
| Temporal IoU (tIoU) | Intersection duration divided by union duration for a predicted and annotated highlight interval. |
|
||||||
|
| Recall@3 | Fraction of annotated highlights matched by one of the top three predictions at the stated tIoU. |
|
||||||
|
| Macro-F1 | Unweighted mean of per-category F1 scores, so a large category cannot hide a weak one. |
|
||||||
|
| nDCG@3 | Normalized discounted cumulative gain for the top three ranked candidates; higher relevance near rank one receives more credit. |
|
||||||
|
| SLO | Service-level objective: an approved target for a measured service-level indicator over a stated workload/window. |
|
||||||
|
| SBOM | Software Bill of Materials: an inventory of shipped software components and their identities. |
|
||||||
|
| Fail closed | Stop before rendering or promotion when required evidence, models, assets, approval, or QA is missing. |
|
||||||
|
| Local model | A model whose weights, loader, tokenizer, and runtime dependencies are pre-provisioned in the deployed artifact or mounted runtime bundle; loading it performs no network operation. |
|
||||||
|
| Creative rubric | A blinded 0-4 human score for story, selection, pacing, visual craft, sound design, voiceover, and factual grounding. |
|
||||||
|
|
||||||
|
The no-waiver rules are authoritative in `video-editing-change-control`; these are their campaign consequences:
|
||||||
|
|
||||||
|
- Make no automatic dependency or model downloads. Do not call external AI services.
|
||||||
|
- Make no network connection during the certified offline run, including loopback and model
|
||||||
|
resolution. The current CV worker uses loopback HTTP; replace that transport under change control
|
||||||
|
before certification.
|
||||||
|
- Use only assets with recorded licenses and permitted production use.
|
||||||
|
- Produce no placeholder silence, tones, generic assets, or unrequested fallback render.
|
||||||
|
- Render only after explicit approval. Do not change packaged production-facing defaults.
|
||||||
|
- Do not treat a successful FFmpeg exit or a JSON field set to `true` as quality evidence.
|
||||||
|
- Keep every experiment isolated behind an explicit, disabled-by-default campaign control. Promotion
|
||||||
|
still follows `video-editing-change-control`.
|
||||||
|
|
||||||
|
## Current truth: do not design from the plan documents
|
||||||
|
|
||||||
|
| Repository fact | Evidence and consequence |
|
||||||
|
|---|---|
|
||||||
|
| `HighlightSourceScheduler` calls `HighlightCandidateGenerator` after source analysis and before `HighlightDirectorPromptGenerator`. | The single-source flow now persists `analysis/category.json` and `analysis/highlight-candidates.json`; missing files mean candidate generation failed or the project predates this change. |
|
||||||
|
| `HighlightCandidateGenerator` is a deterministic baseline; `CinematicHighlightAnalyzer` remains the older multi-clip ranker. | The single-source generator ranks shot/coverage windows from source-level visual scores, FFmpeg silence sections, scene score, duration, and position. It explicitly downweights heuristic/fallback visual evidence; do not call it semantic highlight understanding. |
|
||||||
|
| No local director executes the generated highlight prompt. | `director/director-brief.md` tells an operator to use Codex, Claude, or another agent; rendering begins only when `director/edit-plan.json` appears. |
|
||||||
|
| The highlight plan has a dedicated `HighlightDirectorPlanValidator`; it does not use the multi-clip `EditPlanValidator`. | It requires persisted category/candidates, safe unique IDs, candidate-contained time ranges, configured duration bounds, and nonblank visual/music/SFX/voiceover direction before asset work. |
|
||||||
|
| Plan-derived `highlightId` and mutable asset-request paths are validated. | `HighlightLocalAssetWorker` requires exact project/highlight identity, supported types, safe keys, bounded duration, contract-exact request/target paths, project containment, and no symlink component before materialization. Adjacent nonblank asset-license sidecars are required and preserved; authenticity, checksum, origin, allowed-use validation, and request-file integrity signatures remain open. |
|
||||||
|
| Standalone bootstrap scripts remain network-capable. | Both worker shell scripts run `pip install` in `auto` mode; the CV script defaults to `yolov8n.pt`. `LocalAssetRuntimeVerifier` no longer invokes the asset bootstrap, but operators must still not run either launcher in a certified environment. |
|
||||||
|
| Strict asset readiness now fails startup. | The 2026-07-21 working tree requires pre-provisioned Piper, MusicGen, and AudioGen paths and throws when strict readiness is incomplete. No such three-model bundle is present in this repository. |
|
||||||
|
| Placeholder asset success was removed. | The Python and Java asset workers now return failure and delete invalid output instead of emitting OS speech, silence, or tones. Missing requested music, SFX, or voiceover blocks rendering. |
|
||||||
|
| Audio and preview failures now fail the highlight render. | The renderer no longer copies the pre-mix timeline or final output after those failures. Prepared SFX keys and rendered SFX keys are shared. |
|
||||||
|
| Highlight QA now includes selected media probes but is not a creative-quality gate. | Final duration, black ranges, long silence, and sample peaks are probed. Integrated loudness, true peak, A/V sync, freeze detection, raster safe area, asset semantics, and human creative acceptance remain open. |
|
||||||
|
| The source-to-final test is incomplete. | `HighlightDirectorFlowServiceTest` uses a mocked renderer and dummy files. `CinematicEditingIntegrationTest` covers the separate multi-clip flow. |
|
||||||
|
| Processing is not stage-resumable. | The scanner skips only projects with root `final.mp4`; there is no per-stage journal, input digest, lease, or verified checkpoint. |
|
||||||
|
|
||||||
|
Reconfirm those claims before opening a campaign change:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n "HighlightCandidateGenerator|category.json|highlight-candidates.json" \
|
||||||
|
src/main/java/org/example/videoclips/editing/{HighlightSourceScheduler,HighlightCandidateGenerator,HighlightDirectorPromptGenerator}.java
|
||||||
|
rg -n "planValidator|HighlightDirectorPlanValidator|highlightId\(\)|targetPath\(\)" \
|
||||||
|
src/main/java/org/example/videoclips/editing/{HighlightDirectorFlowService,HighlightDirectorPlanValidator,HighlightLocalAssetWorker}.java
|
||||||
|
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded|duration_matches_timeline" \
|
||||||
|
tools src/main/java/org/example/videoclips/editing
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected on the 2026-07-21 working tree: matches in all three commands; explicit scheduler wiring to
|
||||||
|
`HighlightCandidateGenerator`; no placeholder-success implementation in the asset synthesizers; network-capable
|
||||||
|
launcher matches remain non-zero. Request target strings remain in the code but are now checked against exact
|
||||||
|
project-local paths before use. If that changes, update this campaign before proceeding.
|
||||||
|
|
||||||
|
## Campaign gate map
|
||||||
|
|
||||||
|
Do not start a later phase until the preceding gate is green.
|
||||||
|
|
||||||
|
| Phase | Deliverable | Numeric gate |
|
||||||
|
|---:|---|---|
|
||||||
|
| 0 | Frozen offline baseline | 245 tests in 62 test classes; the full offline Maven suite has 0 failures, 0 errors, and 0 skips |
|
||||||
|
| 1 | Certified fixture and annotation set | 4 current categories represented; 2 independent annotations per source; 100% checksums/licenses/splits present |
|
||||||
|
| 2 | Single-source category and candidate wiring | Macro-F1 and Recall@3 thresholds calibrated and recorded; every processed fixture emits exactly 1 category file and 1 candidate file |
|
||||||
|
| 3 | Resident local director and safe plan | 0 network operations; 100% schema-valid plans; all adversarial plans rejected before filesystem/render work |
|
||||||
|
| 4 | Fail-closed model and asset runtime | 100% manifest/checksum/license/canary pass; any one missing capability prevents readiness and render |
|
||||||
|
| 5 | Production render semantics | Every requested effect/audio/VO action either has execution evidence or fails the render; 0 silent degradations |
|
||||||
|
| 6 | Measured technical and creative QA | 0 ERROR checks; creative average >=3.5/4 and no dimension <3; paired-preference lower 95% bound >50% |
|
||||||
|
| 7 | Resumability and real source-to-final test | 1 final per requested highlight, 0 duplicate side effects, and exact recovery after every injected stage interruption |
|
||||||
|
| 8 | Offline platform, security, performance, operations | macOS and Linux/VPS outputs pass identical gates; 0 attempted network calls; 0 path escapes; workload SLOs met |
|
||||||
|
| 9 | Controlled promotion | All change-control evidence present; human reference-architecture rubric has no score <3 and average >=3.5 |
|
||||||
|
|
||||||
|
## Phase 0: freeze an offline baseline
|
||||||
|
|
||||||
|
1. Work from the repository root. Record, but do not mutate, source and tool state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short
|
||||||
|
git rev-parse HEAD
|
||||||
|
java -version
|
||||||
|
mvn -version
|
||||||
|
ffmpeg -version | head -n 1
|
||||||
|
ffprobe -version | head -n 1
|
||||||
|
printf 'test_methods='; rg -n '^\s*@Test\b' src/test/java | wc -l
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Run the current tests that do not invoke the network-capable model/asset bootstrap paths:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -q -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Gate 0 expectations for the current working tree: Maven exits `0`. Treat generated test media as
|
||||||
|
disposable, not certified evidence. A 2026-07-21 `mvn -q -o verify` run executed all 245 tests in 62 test classes with
|
||||||
|
zero failures, errors, or skips and passed the bound JaCoCo gate. Maven offline mode does not prevent
|
||||||
|
subprocess networking, so call this offline dependency-resolution evidence, not no-egress certification.
|
||||||
|
|
||||||
|
- If offline Maven reports a missing artifact, branch to `video-editing-build-and-env`; pre-provision a
|
||||||
|
locked artifact repository or build cache. Do not enable network.
|
||||||
|
- If a test fails because local voice/TTS behavior differs, branch to Phase 4. The current test suite
|
||||||
|
can consume an untracked local environment and is not yet clean-checkout reproducible.
|
||||||
|
- If the worktree is dirty, preserve unrelated changes. Record their paths and isolate campaign
|
||||||
|
evidence; never reset them.
|
||||||
|
|
||||||
|
## Phase 1: create certified fixtures and annotations
|
||||||
|
|
||||||
|
The repository has no certified creative golden inventory. Do not promote existing `input/` or
|
||||||
|
`output/` media to goldens by assumption.
|
||||||
|
|
||||||
|
Under change control, add `[PROPOSED]` fixture metadata and small rights-cleared media, or an
|
||||||
|
authenticated internal fixture-bundle mount whose immutable digest is recorded. The metadata must
|
||||||
|
contain: fixture ID, SHA-256, byte size, duration, codecs, category, rights/license ID, permitted use,
|
||||||
|
provenance, split, and annotation version. Keep weight files and large/proprietary footage out of Git.
|
||||||
|
|
||||||
|
For each source, collect two independent annotations containing category, positive highlight
|
||||||
|
intervals, excluded intervals, story role, visible facts allowed in voiceover, unsafe/blurred frames,
|
||||||
|
speech intervals, and music/SFX intent. An adjudicator resolves disagreement without seeing model
|
||||||
|
output. Lock the holdout split before tuning.
|
||||||
|
|
||||||
|
Starting floor (a campaign minimum, not a SOTA claim): at least 12 sources, at least 3 each for
|
||||||
|
`car_vlog`, `food_vlog`, `family_vlog`, and `generic_vlog`; at least one no-audio and one low-quality
|
||||||
|
negative source; 2 annotators per source. Record a larger statistically powered sample before making
|
||||||
|
external quality claims.
|
||||||
|
|
||||||
|
Gate 1 `[PROPOSED]` command; implement a repository script only through change control:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x tools/validate-highlight-fixtures && tools/validate-highlight-fixtures --offline --strict
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: exit `0`, `sources>=12`, `categories=4`, `annotators_per_source>=2`,
|
||||||
|
`missing_checksums=0`, `missing_licenses=0`, `split_overlap=0`. If the script is absent, stop: do not
|
||||||
|
replace it with visual inspection. If rights are unclear, quarantine the fixture and branch to legal/
|
||||||
|
asset governance.
|
||||||
|
|
||||||
|
## Phase 2: wire category and candidate generation
|
||||||
|
|
||||||
|
Implement a single-source application use case between `HighlightSourceAnalyzer.analyze` and prompt
|
||||||
|
generation. It must consume persisted source, scene, audio, and visual analysis; classify using actual
|
||||||
|
local evidence; generate shot-boundary-aligned candidates; and atomically write exactly:
|
||||||
|
|
||||||
|
- `analysis/category.json`
|
||||||
|
- `analysis/highlight-candidates.json`
|
||||||
|
|
||||||
|
Do not merely copy the metadata-keyword classifier from the multi-clip path. First define a
|
||||||
|
deterministic heuristic baseline, then compare local learned scorers against it. Candidate IDs must be
|
||||||
|
stable for the same source/config/model digests. Durations must honor the configured 8-35 second
|
||||||
|
range; at most the configured top 3 proceed to planning. Uncertain classification is a review state,
|
||||||
|
not permission to render generic content.
|
||||||
|
|
||||||
|
Before running, predict category confusion, candidate Recall@3 at `tIoU>=0.5`, and ranking nDCG@3 on
|
||||||
|
development fixtures. Record predicted numbers, then run once. Set final thresholds from baseline plus
|
||||||
|
confidence intervals in an ADR; until that ADR exists, use the campaign floors: macro-F1 `>=0.85`,
|
||||||
|
Recall@3 `>=0.80`, nDCG@3 `>=0.80`, and invalid/out-of-range candidates `=0`. These are candidate
|
||||||
|
promotion thresholds, not current results.
|
||||||
|
|
||||||
|
Gate 2 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -Dtest=HighlightSourceCandidatePipelineTest,HighlightCandidateEvaluationTest test
|
||||||
|
test -x tools/evaluate-highlight-selection && \
|
||||||
|
tools/evaluate-highlight-selection --fixtures certified --split holdout --offline
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: both exit `0`; one category and one candidate artifact per source; the four numeric floors
|
||||||
|
above pass. If category passes but Recall@3 fails, branch to segmentation/fusion experiments. If Recall
|
||||||
|
passes but nDCG fails, keep candidates and work only on ranking. If all offline metrics pass but human
|
||||||
|
selected moments remain weak, inspect annotation agreement before changing the model.
|
||||||
|
|
||||||
|
## Phase 3: implement a resident local director and validate plans
|
||||||
|
|
||||||
|
Replace the operator-mediated external-agent step with an explicit application port and a local
|
||||||
|
adapter. The adapter must use a pre-provisioned model path, constrained JSON decoding, fixed schema
|
||||||
|
version, bounded context/output, deterministic seed where supported, timeout, cancellation, and no
|
||||||
|
socket or model-hub resolution. Preserve human approval; the model may propose but never approve.
|
||||||
|
|
||||||
|
Validate `HighlightDirectorPlan` before creating storyboard, asset requests, or directories:
|
||||||
|
|
||||||
|
| Required validation | Reject when |
|
||||||
|
|---|---|
|
||||||
|
| Identity | Project/source mismatch; unknown candidate; duplicate/unsafe highlight ID |
|
||||||
|
| Numeric safety | NaN/infinity; negative/reversed/out-of-source time; target outside configured range |
|
||||||
|
| Structural bounds | Zero highlights; more than configured maximum; empty/oversized strings or arrays |
|
||||||
|
| Semantics | Unknown category, story role, effect, transition, render profile, voice, or asset type |
|
||||||
|
| Grounding | Voiceover asserts a fact absent from certified visible/transcript evidence |
|
||||||
|
| Path safety | Absolute path, `..`, separator, symlink escape, or caller-supplied target path |
|
||||||
|
| Asset closure | A requested music/SFX/voice/font/LUT capability has no licensed local resolution |
|
||||||
|
|
||||||
|
Derive target paths server-side from validated IDs. Resolve against the project root, normalize,
|
||||||
|
verify containment, reject symlinks at trust boundaries, and create through the store. Make the JSON
|
||||||
|
schema version explicit and validate it during the Maven build.
|
||||||
|
|
||||||
|
Gate 3 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -Dtest=LocalHighlightDirectorContractTest,HighlightDirectorPlanValidatorTest,HighlightPathSafetyTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: exit `0`; valid plans accepted `=100%`; malformed, unknown-field, timestamp, NaN/infinity,
|
||||||
|
duplicate-ID, `../`, absolute-path, backslash, symlink, oversized-text, unknown-effect, and missing-asset
|
||||||
|
cases rejected `=100%`; filesystem writes after a rejected plan `=0`; network operations `=0`.
|
||||||
|
|
||||||
|
If constrained decoding still emits invalid JSON, do not repair it silently: record the failure and
|
||||||
|
stop. If factual grounding cannot be established, omit the line or route to human review.
|
||||||
|
|
||||||
|
## Phase 4: package and preflight all local capabilities
|
||||||
|
|
||||||
|
Add `[PROPOSED]` typed configuration for a runtime model root and a signed/immutable model manifest;
|
||||||
|
do not overload the current free-form model-name properties. For every CV, ASR, director, TTS, music,
|
||||||
|
and SFX model record: purpose, relative path, exact bytes, SHA-256, loader/runtime version, tokenizer/
|
||||||
|
config paths, license identifier and license-text digest, approved use, attribution, memory estimate,
|
||||||
|
and a deterministic canary input/output predicate.
|
||||||
|
|
||||||
|
Pre-provision Python wheels and weights in the build/deployment supply chain. Use locked versions and
|
||||||
|
hashes. Production startup must never run `pip`, `YOLO("yolov8n.pt")`, or
|
||||||
|
`get_pretrained("facebook/...")`. A readiness preflight must verify regular-file containment,
|
||||||
|
checksums, licenses, loader compatibility, capacity, and one inference canary for every enabled
|
||||||
|
capability. Any failure makes readiness false and blocks project claims and rendering.
|
||||||
|
|
||||||
|
Gate 4 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x tools/verify-local-model-bundle && \
|
||||||
|
tools/verify-local-model-bundle --manifest "${VIDEO_EDITING_MODEL_MANIFEST:?set manifest}" --offline --strict
|
||||||
|
mvn -o -Dtest=LocalModelManifestTest,OfflineCapabilityPreflightTest,FailClosedAssetGenerationTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `models_checked>=6`, `checksum_mismatches=0`, `license_failures=0`, `canary_failures=0`,
|
||||||
|
`network_attempts=0`; Maven exits `0`. Re-run after corrupting one byte in a disposable test copy:
|
||||||
|
expected preflight exit non-zero, readiness false, renders started `=0`. If a license is absent or
|
||||||
|
ambiguous, the model/asset is unavailable, not experimental.
|
||||||
|
|
||||||
|
Remove production paths to `say`, `espeak`, tones, and silence. Keep negative tests proving these
|
||||||
|
cannot satisfy a required asset. Do not edit packaged defaults during the campaign; activate new
|
||||||
|
components only with explicit environment configuration in approved environments.
|
||||||
|
|
||||||
|
## Phase 5: make render instructions real
|
||||||
|
|
||||||
|
Implement an allowlisted renderer capability registry. A plan may name only behavior the renderer
|
||||||
|
can prove it executed. Today, any non-empty visual treatment maps to the same crop/grade and fades are
|
||||||
|
per-segment. Audio and preview failures now propagate; keep regression tests on that fail-closed behavior.
|
||||||
|
Do not relabel the remaining generic visual behaviors as cinematic.
|
||||||
|
|
||||||
|
| Plane | Required semantics and evidence |
|
||||||
|
|---|---|
|
||||||
|
| Selection/edit | Multiple decisions when the story requires them; source/timeline arithmetic verified; true inter-shot transitions rather than independent fades |
|
||||||
|
| Visual | Explicit color transform, stabilization, reframing, speed ramp, overlay/font, and transition parameters; command/filter evidence plus output measurements |
|
||||||
|
| Music | Licensed/generated asset digest; deliberate in/out points, trim/loop policy, gain envelope, and final mix contribution |
|
||||||
|
| SFX | One existing local asset per cue; sample-accurate or measured timing; no missing-path attempt |
|
||||||
|
| Voiceover | One grounded script version, approved local voice/model, line timing/alignment, intelligibility measurement, and ducking evidence |
|
||||||
|
| Master | Two-pass or otherwise measured loudness normalization; output codec, geometry, cadence, duration, and channel layout probed from the file |
|
||||||
|
|
||||||
|
Make every requested production asset blocking. On FFmpeg, preview, or mix failure, mark the attempt
|
||||||
|
failed and publish no `final.mp4`. Write to a temporary name and atomically promote only after QA.
|
||||||
|
Record tool versions, complete argv, exit codes, input/output hashes, models/assets/licenses, timing,
|
||||||
|
and effective config in `render-manifest.json`.
|
||||||
|
|
||||||
|
Gate 5 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -Dtest=HighlightRendererIntegrationTest,HighlightAudioMixIntegrationTest,HighlightEffectContractTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: exit `0`; requested actions `= executed actions`; unresolved actions `=0`; silent
|
||||||
|
degradations `=0`; failure injection for each FFmpeg stage publishes final files `=0`. If a filter is
|
||||||
|
unsupported on macOS or Linux, do not substitute one; branch to Phase 8 and either standardize FFmpeg
|
||||||
|
or remove the capability through change control.
|
||||||
|
|
||||||
|
## Phase 6: replace asserted QA with observed QA
|
||||||
|
|
||||||
|
Each QA check must include expected value, observed value, unit, tool/algorithm version, command or
|
||||||
|
method, artifact path/digest, severity, and pass/fail. Probe the actual final file. At minimum measure:
|
||||||
|
stream presence, duration delta, 1920x1080 geometry, 30 fps cadence, H.264/AAC codecs, 48 kHz audio,
|
||||||
|
black/freeze spans, silence spans, clipping/true peak, integrated loudness, overlay bounds, asset
|
||||||
|
contribution, voice intelligibility/alignment, and requested-effect execution.
|
||||||
|
|
||||||
|
Use the current configured mastering targets as initial technical gates: integrated loudness
|
||||||
|
`-16 LUFS +/-1`, true peak `<=-1.5 dBTP`, duration error `<=0.25 s`, missing requested streams/assets
|
||||||
|
`=0`, and ERROR checks `=0`. Calibrate black/silence/freeze tolerances by content and annotations;
|
||||||
|
do not apply a universal threshold that rejects intentional black or quiet moments.
|
||||||
|
|
||||||
|
Operator spot-check commands (diagnostic only, not a replacement for automated parsing):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FINAL="${FINAL:?set final mp4}"
|
||||||
|
ffprobe -v error -show_entries format=duration:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$FINAL"
|
||||||
|
ffmpeg -hide_banner -i "$FINAL" -vf "blackdetect=d=0.25:pix_th=0.10" -af "silencedetect=n=-45dB:d=1.0" -f null -
|
||||||
|
ffmpeg -hide_banner -nostats -i "$FINAL" -filter_complex "ebur128=peak=true" -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
Gate 6 technical command is `[PROPOSED]`; implement the parser through change control rather than
|
||||||
|
copying console text into `qa-report.json`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x tools/verify-highlight-render && \
|
||||||
|
tools/verify-highlight-render --final "$FINAL" --manifest "${MANIFEST:?set render manifest}" --strict
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: exit `0`, `error_checks=0`, `missing_requested_assets=0`, `duration_error_seconds<=0.25`,
|
||||||
|
`integrated_lufs` in `[-17,-15]`, and `true_peak_dbtp<=-1.5`. If the tool is absent, Gate 6 is red.
|
||||||
|
|
||||||
|
Run blinded creative review against the frozen deterministic baseline. Use at least 3 reviewers per
|
||||||
|
output; score the seven rubric dimensions 0-4; randomize A/B order; retain disagreements. Gate:
|
||||||
|
average `>=3.5`, every dimension median `>=3`, factual errors `=0`, and the 95% confidence interval
|
||||||
|
lower bound for preference over baseline `>50%`. If technical QA passes but creative QA fails, do not
|
||||||
|
tune FFmpeg blindly: branch according to reviewer tags (selection -> Phase 2, story/VO -> Phase 3,
|
||||||
|
craft/audio -> Phase 5).
|
||||||
|
|
||||||
|
## Phase 7: add resumability and a real source-to-final test
|
||||||
|
|
||||||
|
Add a durable stage journal with state, attempt, lease owner/expiry, started/completed timestamps,
|
||||||
|
code/config/model/input digests, output paths/digests, and failure classification. Required stages:
|
||||||
|
claim, analyze, classify, rank, direct, validate, approve, prepare-assets, render, QA, publish. Commit
|
||||||
|
each checkpoint atomically. Resume only when all input digests match; otherwise invalidate that stage
|
||||||
|
and every dependent stage. Never infer completion from file existence alone.
|
||||||
|
|
||||||
|
Build a real source-to-final test using a tiny rights-cleared fixture and pre-provisioned test models/
|
||||||
|
assets. Do not mock the analyzer, director, asset generator, renderer, FFmpeg, or FFprobe. Test restart
|
||||||
|
after every stage, duplicate scans, stale leases, corrupt checkpoints, changed model digest, invalid
|
||||||
|
plan, missing asset, FFmpeg failure, and crash between temporary output and atomic publish.
|
||||||
|
|
||||||
|
Gate 7 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -Dtest=HighlightSourceToFinalIT,HighlightWorkflowResumeIT,HighlightWorkflowFailureIT test
|
||||||
|
mvn -o verify
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: exit `0`; final outputs equal requested approved highlights; duplicate renders/uploads `=0`;
|
||||||
|
published partial files `=0`; each injected interruption resumes at the first invalid/incomplete stage;
|
||||||
|
changing any input digest recomputes all dependent stages. If the test uses a mock renderer or dummy
|
||||||
|
text as MP4, it does not satisfy this gate.
|
||||||
|
|
||||||
|
## Phase 8: prove offline parity, security, performance, and operations
|
||||||
|
|
||||||
|
Run the same immutable fixture bundle, model bundle, service artifact, config digest, and FFmpeg build
|
||||||
|
on macOS development and Linux/VPS. Cloud deployment may mount the same pre-provisioned bundle but
|
||||||
|
must not fetch it at application startup. Container work is `[PROPOSED]`; the repository has no current
|
||||||
|
container definition, so do not claim `--network none` validation until one is approved.
|
||||||
|
|
||||||
|
Define the representative workload before measuring: source minutes, resolution/codecs, highlight
|
||||||
|
count, concurrent jobs, hardware/accelerator, warm/cold state, and storage. Record per-stage p50/p95/
|
||||||
|
p99, real-time factor (wall seconds/source seconds), CPU, accelerator, RSS, disk peak, queue depth,
|
||||||
|
failures, and output bytes. Establish SLOs from measured capacity and business demand; never import the
|
||||||
|
200-RPS synchronous-API example into a media pipeline.
|
||||||
|
|
||||||
|
Gate 8 `[PROPOSED]`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x tools/run-offline-highlight-certification && \
|
||||||
|
tools/run-offline-highlight-certification --fixtures certified --models "${VIDEO_EDITING_MODEL_MANIFEST:?}" --strict
|
||||||
|
mvn -o -Dtest=HighlightOfflineSecurityIT,HighlightCapacityIT,HighlightGracefulShutdownIT test
|
||||||
|
mvn -o verify
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected on both platforms: technical and creative Gate 6 pass; `network_attempts=0`; path escapes
|
||||||
|
and symlink escapes `=0`; checksum/license failures `=0`; orphan child processes `=0`; corrupt or
|
||||||
|
partial finals `=0`; measured p95 and resource peaks remain within the approved workload SLO. Compare
|
||||||
|
semantic decisions and QA outcomes, not bit-identical encoded video unless the codec/toolchain is
|
||||||
|
fully standardized. If platform results diverge, capture tool/model/config digests and stop promotion.
|
||||||
|
|
||||||
|
Security review must threat-model malicious media, decompression/resource exhaustion, hostile model
|
||||||
|
output, path traversal/symlinks, command injection, poisoned weights, unlicensed assets, prompt
|
||||||
|
injection in transcripts/metadata, denial of service, and sensitive media/log leakage. FFmpeg commands
|
||||||
|
remain argv lists, never shell strings. Run workers with least privilege, bounded CPU/memory/time/disk,
|
||||||
|
read-only model mounts, project-scoped writable storage, and no network capability.
|
||||||
|
|
||||||
|
Operations must expose stage duration/failure, queue age/depth, active jobs, model readiness,
|
||||||
|
candidate counts, approval waits, render real-time factor, QA failures, disk pressure, and worker
|
||||||
|
restarts without source filenames, prompt text, or project IDs as metric labels. Define alerts,
|
||||||
|
retention, cancellation, graceful shutdown, recovery, and model rollback before production.
|
||||||
|
|
||||||
|
## Ranked solution menu
|
||||||
|
|
||||||
|
Choose with measured evidence, not model popularity.
|
||||||
|
|
||||||
|
| Rank | Candidate | Use when | Required derivation and disproof |
|
||||||
|
|---:|---|---|---|
|
||||||
|
| 1 | Local multimodal director over deterministic shot/audio candidates | Resident model fits capacity and materially improves grounded story decisions | Predict memory, latency, Recall@3-to-plan loss, grounding error, and creative preference. Disprove with locked holdout, ablation without visual/audio inputs, adversarial transcripts, and local-only trace. |
|
||||||
|
| 2 | Local feature models + deterministic ranker + smaller constrained text director | A full multimodal director is too slow or weakly grounded | Derive feature normalization, fusion weights, calibration, context budget, and failure states. Ablate each feature; require one mechanism to explain positive and negative fixtures. |
|
||||||
|
| 3 | Local learned temporal scorer + template/constrained planner | Director variance blocks schema safety but selection can be learned | Prove scorer improves Recall/nDCG and templates improve repeatability without lowering blinded creative scores. Reject if category styles collapse to generic edits. |
|
||||||
|
| 4 | Deterministic heuristics and fixed renderer | Establish the baseline and retain an emergency diagnostic comparator only | Specify every weight/rule and sensitivity. Never promote as “high cinematic quality” without independently passing all creative gates. |
|
||||||
|
|
||||||
|
For any chosen option, the decision record must state: requirement; selected approach; alternatives;
|
||||||
|
benefits/trade-offs; operational consequences; security implications; verification; revisit condition.
|
||||||
|
No new database, queue, service, framework, or model is justified merely by this campaign.
|
||||||
|
|
||||||
|
## Known wrong paths: fence these off
|
||||||
|
|
||||||
|
- Do not use filename/project-style keywords as proof of visual content category.
|
||||||
|
- Do not ask an external agent to write the plan and call the workflow local-only.
|
||||||
|
- Do not let model libraries resolve names such as `yolov8n.pt` or `facebook/musicgen-small` at runtime.
|
||||||
|
- Do not interpret a non-strict readiness log as certification. Strict mode now fails startup, but its
|
||||||
|
sidecar and file-presence checks do not establish provenance authenticity, checksums, allowed use, or quality.
|
||||||
|
- Do not accept silence, sine tones, OS `say`/`espeak`, missing optional assets, or source-audio-only
|
||||||
|
copies when the approved plan requests production audio.
|
||||||
|
- Do not use `EditPlanValidator` unchanged for highlight projects; it reads the multi-clip store and
|
||||||
|
analysis contract. Reuse validation concepts, not the wrong persistence contract.
|
||||||
|
- Do not resolve untrusted plan IDs or request target paths directly with `Path.resolve`/`Path.of`.
|
||||||
|
- Do not add more prompt adjectives to fix missing analysis, weak selection, or unsupported effects.
|
||||||
|
- Do not call a fixed 4% crop a dynamic crop, per-segment fades a crossfade, or hard-coded `true` QA.
|
||||||
|
- Do not tune on the locked holdout, promote unlabeled output, or judge cinematic quality by eye.
|
||||||
|
- Do not retry permanent validation/model/license failures; do not resume from file existence alone.
|
||||||
|
- Do not alter packaged production defaults to expose campaign behavior.
|
||||||
|
|
||||||
|
## Phase 9: promotion protocol
|
||||||
|
|
||||||
|
Route every implementation and final promotion through `video-editing-change-control`. Promotion
|
||||||
|
requires one evidence bundle containing baseline commit/tool digests; fixture/annotation version and
|
||||||
|
rights; model/dependency SBOM, hashes, licenses, and signatures; predictions made before experiments;
|
||||||
|
selection/director/renderer/QA results including negatives and ablations; offline network-denial proof;
|
||||||
|
macOS/Linux parity; performance/capacity; threat model; recovery/graceful-shutdown evidence; human
|
||||||
|
review data; runbook/alerts; rollback; and ADRs.
|
||||||
|
|
||||||
|
The project is not campaign-complete until every gate passes on the locked holdout, the Fortune 500
|
||||||
|
human-review rubric has no category below 3 and average at least 3.5, security/testing/operational
|
||||||
|
readiness are each at least 3, and no critical behavior relies on an undocumented assumption. A model
|
||||||
|
upgrade restarts Phases 4, 6, 7, and 8. A schema/renderer change restarts Phases 3, 5, 6, 7, and 8.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Primary code: `HighlightSourceScheduler`, `HighlightSourceAnalyzer`,
|
||||||
|
`HighlightDirectorPromptGenerator`, `HighlightDirectorPlanScanner`, `HighlightDirectorFlowService`,
|
||||||
|
`HighlightDirectorPlan`, `HighlightLocalAssetWorker`, `LocalAssetSynthesizer`,
|
||||||
|
`LocalAssetRuntimeVerifier`, `HighlightFfmpegRenderer`, `FileSystemHighlightProjectStore`, and the two
|
||||||
|
worker scripts. Planning documents are context, not implementation truth.
|
||||||
|
|
||||||
|
Re-verify volatile facts in one line each:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '^\s*@Test\b' src/test/java | wc -l
|
||||||
|
rg -n "category.json|highlight-candidates.json" src/main/java/org/example/videoclips/editing
|
||||||
|
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded" tools src/main/java/org/example/videoclips/editing
|
||||||
|
rg -n "duration_matches_timeline|required_assets_resolved|audio_mastering_applied" src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java
|
||||||
|
mvn -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
Update the dated truth table, baseline counts, thresholds/ADR links, fixture inventory, model manifest
|
||||||
|
contract, and platform/SLO evidence whenever those commands or campaign decisions change.
|
||||||
|
|
@ -0,0 +1,418 @@
|
||||||
|
---
|
||||||
|
name: video-editing-config-and-flags
|
||||||
|
description: Load when inspecting, setting, adding, renaming, reviewing, or debugging video-clipping.*, Spring profiles, scheduler/render guards, adapter selectors, filesystem paths, retention, local CV or local asset worker settings, their environment-variable mappings, or configuration-default drift. Use before any configuration or production-default change.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Configuration and Flags
|
||||||
|
|
||||||
|
Use this runbook to determine the effective configuration before starting the service or changing behavior. Treat every value below as repository state verified on **2026-07-21**, not as a production endorsement.
|
||||||
|
|
||||||
|
## Scope and routing
|
||||||
|
|
||||||
|
Use this skill for property ownership, precedence, defaults, environment mappings, validation, and flag-change review.
|
||||||
|
|
||||||
|
Do **not** use it to:
|
||||||
|
|
||||||
|
- start or operate a workflow; use `video-editing-run-and-operate`;
|
||||||
|
- rebuild local environments or provision models; use `video-editing-build-and-env`;
|
||||||
|
- diagnose a failed render or worker; use `video-editing-debugging-playbook` and `video-editing-diagnostics-and-tooling`;
|
||||||
|
- decide whether a behavior change may ship; use `video-editing-change-control`;
|
||||||
|
- judge output quality; use `video-editing-validation-and-qa`.
|
||||||
|
|
||||||
|
Definitions used here:
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| Java default | Field initializer in `VideoClippingProperties`; applies when no higher-precedence source supplies the key. |
|
||||||
|
| Packaged effective default | Value obtained from packaged `application.properties` plus `application.yml` with no profile and no environment override. This is what a plain service start sees. |
|
||||||
|
| Guard | Spring condition or runtime test that enables a component or destructive action. |
|
||||||
|
| Adapter selector | String property used by `@ConditionalOnProperty` to choose one implementation. |
|
||||||
|
| Pre-provisioned | Dependency, executable, model, and licensed asset already present in the runtime image/filesystem; startup must not fetch it. |
|
||||||
|
| Fail-closed | Missing or invalid required capability prevents processing or promotion; it must not silently substitute lower-quality output. |
|
||||||
|
|
||||||
|
## Non-negotiable configuration policy
|
||||||
|
|
||||||
|
Enforce these user-supplied constraints for every environment:
|
||||||
|
|
||||||
|
1. Prohibit automatic dependency or model downloads, external AI services, network access, unlicensed assets, placeholder silence or tones, and unapproved rendering.
|
||||||
|
2. Do not change a production-facing default without `video-editing-change-control`, an ADR-style decision record, tests, rollback instructions, and environment impact analysis.
|
||||||
|
3. Provision local models and dependencies before runtime. Pin their identity and checksum outside behavioral configuration, then verify them before accepting work.
|
||||||
|
4. Production startup now fails under `strict-runtime=true` when Piper, MusicGen, AudioGen, their local paths, or the Python stack are absent. Keep that behavior and add license/checksum manifest verification before calling the runtime certified.
|
||||||
|
5. Keep all automatic render controls disabled and require an approval artifact at the render boundary.
|
||||||
|
6. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` with their current default `BOOTSTRAP_MODE=auto` in a network-prohibited runtime. Both scripts can create virtualenvs and invoke `pip`; the CV script can also make Ultralytics resolve/download `yolov8n.pt`.
|
||||||
|
|
||||||
|
## Resolve the effective value
|
||||||
|
|
||||||
|
Use Spring Boot precedence: command-line arguments and environment-derived properties override packaged config; profile files override non-profile packaged files. This repository also embeds explicit `${CUSTOM_ENV:default}` placeholders in `application.yml`.
|
||||||
|
|
||||||
|
Follow this procedure:
|
||||||
|
|
||||||
|
1. Record active profiles and all `VIDEO_EDITING_*`, `FOLDER_SCHEDULER_*`, `LOCAL_ASSET_*`, and `LOCAL_CV_*` variables without exposing secrets.
|
||||||
|
2. Locate the key in `application-<profile>.*`, then `application.yml`, then `application.properties`, then the Java initializer.
|
||||||
|
3. Check the component guard below. A bound value can exist while its component is absent.
|
||||||
|
4. Check whether Java and packaged defaults differ. Packaged configuration wins in a normal application start.
|
||||||
|
5. For a generic Spring environment override, convert dots and hyphens to underscores and uppercase, for example `video-clipping.cleanup.enabled` to `VIDEO_CLIPPING_CLEANUP_ENABLED`. Prefer an explicit placeholder already present in `application.yml`; it documents the supported operational name.
|
||||||
|
|
||||||
|
Read configuration without starting the service:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '1,220p' src/main/resources/application.properties
|
||||||
|
sed -n '1,220p' src/main/resources/application.yml
|
||||||
|
sed -n '1,220p' src/main/resources/application-jpa.properties
|
||||||
|
sed -n '1,160p' src/main/resources/application-cinematic-editing-local.yml
|
||||||
|
sed -n '1,160p' src/main/resources/application-folder-scheduler-local.yml
|
||||||
|
sed -n '1,220p' src/test/resources/application.properties
|
||||||
|
```
|
||||||
|
|
||||||
|
## Critical drift: plain startup is unsafe
|
||||||
|
|
||||||
|
The packaged effective defaults activate multiple independent consumers and local bootstrap paths:
|
||||||
|
|
||||||
|
| Axis | Java default | Packaged effective default | Risk/status |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| folder scheduler enabled | `false` | `true` | Consumes files under `./input/source`. Unsafe for an exploratory start. |
|
||||||
|
| editing enabled | `true` | `true` | Creates editing components and starts runtime checks. |
|
||||||
|
| local director enabled | `true` | `true` | Polls editing input. Auto-render remains off in base config. |
|
||||||
|
| highlight scheduler enabled | `true` | `true` | Polls and moves highlight sources. |
|
||||||
|
| highlight render enabled | `false` | `false` | Render scan is fail-closed by default. |
|
||||||
|
| highlight approval required | `true` | `true` | Requires the configured flag when rendering is explicitly enabled; the bare flag is not production authorization. |
|
||||||
|
| visual provider | `heuristic` | `local-cv` | Packaged YAML selects the local HTTP adapter. |
|
||||||
|
| visual heuristic fallback | `true` | `true` | Quality may degrade without failing. Not production-ready. |
|
||||||
|
| local CV auto-start | `false` | `true` | Launches bootstrap script, which may install/download. Prohibited at runtime. |
|
||||||
|
| local asset auto-start | `false` | `true` | Runs resident-runtime verification at application startup; it no longer invokes the bootstrap script. |
|
||||||
|
| local asset strict runtime | `false` | `true` | Throws at startup when required local binaries, Python imports, model paths, or adjacent model-license sidecars are missing. |
|
||||||
|
| local director auto-render | `false` | `false` | Safe in base config. |
|
||||||
|
| local director approval required | `true` | `true` | Approval applies only if auto-render is enabled. |
|
||||||
|
|
||||||
|
The `cinematic-editing-local` profile sets local-director auto-render to `true` while requiring `approved.flag`. Treat the profile as a local workflow convenience, not a production profile. The `folder-scheduler-local` profile enables folder segmentation. Never activate either without checking the input directories.
|
||||||
|
|
||||||
|
## Complete application property catalog
|
||||||
|
|
||||||
|
Notation: `J` is the Java default; `P` is the packaged effective default when it differs. `@Min` is the only bean-validation constraint currently declared. Values without `@Min` generally lack startup validation.
|
||||||
|
|
||||||
|
### API, adapters, S3, queue, and processing
|
||||||
|
|
||||||
|
| Property | J / P default | Meaning, values, guard |
|
||||||
|
|---|---|---|
|
||||||
|
| `video-clipping.max-file-size-bytes` | `53687091200` | Max declared source size; `@Min(1)`. |
|
||||||
|
| `video-clipping.multipart-part-size-bytes` | `104857600` | Multipart chunk size; `@Min(5242880)`. |
|
||||||
|
| `video-clipping.upload-url-ttl-minutes` | `60` | Upload URL lifetime; `@Min(1)`. |
|
||||||
|
| `video-clipping.download-url-ttl-minutes` | `15` | Download URL lifetime; `@Min(1)`. |
|
||||||
|
| `video-clipping.allowed-content-types` | `video/mp4`, `video/quicktime`, `video/webm` | Boundary allow-list; no non-empty validation. |
|
||||||
|
| `video-clipping.repository` | no Java field / `memory` | Conditional selector only: `memory` (also match-if-missing) or `jpa`. It is present in packaged properties but absent from `VideoClippingProperties`. |
|
||||||
|
| `video-clipping.storage` | `memory` | `memory` (also match-if-missing) or `s3`; other values leave no storage adapter. |
|
||||||
|
| `video-clipping.queue` | `memory` | `memory` (also match-if-missing) or `db`; `db` enables polling and queue metrics. |
|
||||||
|
| `video-clipping.processing` | `stub` | `stub` (also match-if-missing) or `ffmpeg`; `stub` is not production output. |
|
||||||
|
| `video-clipping.s3.region` | `us-east-1` | S3 client region. |
|
||||||
|
| `video-clipping.s3.bucket` | `video-clipping-dev` | Object bucket; development name is not a production default. |
|
||||||
|
| `video-clipping.s3.endpoint` | unset | Optional endpoint override. |
|
||||||
|
| `video-clipping.s3.path-style` | `true` | Path-style addressing. |
|
||||||
|
| `video-clipping.database-queue.poll-interval-ms` | `1000` | Database queue scheduled delay; `@Min(100)`. |
|
||||||
|
| `video-clipping.database-queue.batch-size` | `5` | Max messages claimed per poll; `@Min(1)`. |
|
||||||
|
| `video-clipping.database-queue.visibility-timeout-ms` | `900000` | Expired-claim threshold; `@Min(1000)`. |
|
||||||
|
| `video-clipping.database-queue.max-attempts` | `3` | Dead-letter threshold; `@Min(1)`. |
|
||||||
|
| `video-clipping.database-queue.retry-backoff-ms` | `5000` | Retry delay; `@Min(0)`. |
|
||||||
|
| `video-clipping.quotas.max-active-jobs-per-tenant` | `5` | Active-job admission cap; `0` is permitted and rejects all new active work; `@Min(0)`. |
|
||||||
|
|
||||||
|
`application-jpa.properties` selects `repository=jpa`, `queue=db`, and `processing=stub`, then configures an H2 database in PostgreSQL compatibility mode with Flyway and Hibernate validation. It is not a PostgreSQL production profile.
|
||||||
|
|
||||||
|
Packaged Spring/platform settings are also configuration axes:
|
||||||
|
|
||||||
|
| Property | Packaged value | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| `spring.application.name` | `video-clipping-service` | Service identity. |
|
||||||
|
| `spring.mvc.problemdetails.enabled` | `true` | Enables Spring Problem Details handling. |
|
||||||
|
| `management.endpoints.web.exposure.include` | `health,info,metrics,prometheus` | Exposure is not the same as authorization; production must secure management endpoints. |
|
||||||
|
| `spring.jpa.open-in-view` | `false` | Deliberate persistence boundary. |
|
||||||
|
| `logging.level.org.example.videoclips.folder` | `INFO` | Base YAML; explicit env `FOLDER_SCHEDULER_LOG_LEVEL`. |
|
||||||
|
| `spring.datasource.url/driver-class-name/username/password` | H2 memory URL / H2 driver / `sa` / empty | `application-jpa.properties` only; development/test convenience. |
|
||||||
|
| `spring.jpa.hibernate.ddl-auto` / `spring.jpa.show-sql` | `validate` / `false` | JPA profile schema validation and SQL logging. |
|
||||||
|
| `spring.flyway.enabled` | `true` | JPA profile migration control. |
|
||||||
|
|
||||||
|
Classify selectors and modes conservatively:
|
||||||
|
|
||||||
|
| Classification | Current values |
|
||||||
|
|---|---|
|
||||||
|
| Development/test only | `repository=memory`, `storage=memory`, `queue=memory`, `processing=stub`, H2 JPA profile, heuristic analysis, all local convenience profiles. |
|
||||||
|
| Experimental/unsafe for promoted output | heuristic fallback, network-capable local-CV auto-bootstrap, named/defaulted YOLO analysis, non-strict asset readiness, and any explicitly enabled render path whose approval is not authenticated and digest-bound. Historical Python/Java placeholder-audio fallbacks are removed in the 2026-07-21 working tree. |
|
||||||
|
| Production candidates requiring evidence | `repository=jpa` with real PostgreSQL, `storage=s3`, `queue=db`, `processing=ffmpeg`, pre-provisioned local CV/assets/director. Candidate means not certified by this repository. |
|
||||||
|
|
||||||
|
### API FFmpeg workspace, cleanup, and retention
|
||||||
|
|
||||||
|
| Property | J / P default | Meaning and validation |
|
||||||
|
|---|---|---|
|
||||||
|
| `video-clipping.ffmpeg.ffmpeg-binary` | `ffmpeg` | API clipper executable. |
|
||||||
|
| `video-clipping.ffmpeg.input-directory` | `./tmp/ffmpeg-input` | Materialized source workspace. |
|
||||||
|
| `video-clipping.ffmpeg.output-directory` | `./tmp/ffmpeg-output` | Generated clip workspace. |
|
||||||
|
| `video-clipping.ffmpeg.exact-preset` | `veryfast` | FFmpeg encoding preset used by the API clipper; absent from packaged config, so the Java default applies. |
|
||||||
|
| `video-clipping.ffmpeg.cleanup-local-files` | `true` | Per-job worker cleanup control. |
|
||||||
|
| `video-clipping.cleanup.enabled` | `true` | Runtime check inside both scheduled cleanup jobs; jobs still wake when false. |
|
||||||
|
| `video-clipping.cleanup.local-artifact-poll-interval-ms` | `300000` | Local cleanup delay; `@Min(30000)`. |
|
||||||
|
| `video-clipping.cleanup.local-artifact-retention-hours` | `24` | Age for FFmpeg and fixed temporary roots; `@Min(1)`. |
|
||||||
|
| `video-clipping.cleanup.retention-poll-interval-ms` | `3600000` | storage/repository cleanup delay; `@Min(30000)`. |
|
||||||
|
| `video-clipping.cleanup.source-retention-hours` | `168` | Expiry assigned to new source assets; `@Min(1)`. |
|
||||||
|
| `video-clipping.cleanup.clip-retention-hours` | `168` | Expiry assigned to clips; `@Min(1)`. |
|
||||||
|
|
||||||
|
Local artifact cleanup also targets fixed `tmp/in-memory-storage` and `tmp/stub-output`; those roots are not configurable. Relative paths resolve from the process working directory. Validate mount ownership, capacity, backups, retention, and path separation on macOS, Linux/VPS, and cloud runtimes.
|
||||||
|
|
||||||
|
### Folder segmentation scheduler
|
||||||
|
|
||||||
|
| Property | J / P default | Meaning and validation |
|
||||||
|
|---|---|---|
|
||||||
|
| `video-clipping.folder-scheduler.enabled` | J `false`; P `true` | Creates initializer, validator, clipper, and poller. |
|
||||||
|
| `.input-directory` | J `/input/source`; P `./input/source` | Inbox; scheduler moves candidates. Env: `FOLDER_SCHEDULER_INPUT_DIRECTORY`. |
|
||||||
|
| `.output-directory` | J `/output/clips`; P `./output/clips` | Segmented output. Env: `FOLDER_SCHEDULER_OUTPUT_DIRECTORY`. |
|
||||||
|
| `.processed-directory` | J `/input/processed`; P `./input/processed` | Successful source destination. Env: `FOLDER_SCHEDULER_PROCESSED_DIRECTORY`. |
|
||||||
|
| `.rejected-directory` | J `/input/rejected`; P `./input/rejected` | Failed source destination. Env: `FOLDER_SCHEDULER_REJECTED_DIRECTORY`. |
|
||||||
|
| `.working-directory` | J `/input/working`; P `./input/working` | Claimed source directory. Env: `FOLDER_SCHEDULER_WORKING_DIRECTORY`. |
|
||||||
|
| `.poll-interval-ms` | `5000` | Initial/fixed delay; `@Min(1000)`. Env: `FOLDER_SCHEDULER_POLL_INTERVAL_MS`. |
|
||||||
|
| `.segment-duration-seconds` | `8` | Segment target; `@Min(1)`; no explicit env placeholder. |
|
||||||
|
| `.ffmpeg-binary` / `.ffprobe-binary` | `ffmpeg` / `ffprobe` | Executables; no explicit env placeholders. |
|
||||||
|
| `.output-container` / `.exact-preset` | `mp4` / `veryfast` | FFmpeg output choices; no enum/allow-list validation. |
|
||||||
|
| `.preserve-input-quality` | `true` | Selects preservation behavior. Env: `FOLDER_SCHEDULER_PRESERVE_INPUT_QUALITY`. |
|
||||||
|
|
||||||
|
`FOLDER_SCHEDULER_ENABLED` controls the packaged guard; `FOLDER_SCHEDULER_LOG_LEVEL` controls only `org.example.videoclips.folder` logging.
|
||||||
|
|
||||||
|
### Editing, render, analysis, and assets
|
||||||
|
|
||||||
|
All keys below start with `video-clipping.editing`.
|
||||||
|
|
||||||
|
| Suffix | J / P default | Meaning / explicit environment variable |
|
||||||
|
|---|---|---|
|
||||||
|
| `.enabled` | `true` | Master component guard; `VIDEO_EDITING_ENABLED`. |
|
||||||
|
| `.project-directory` | `./output/edit-projects` | Multi-clip projects; `VIDEO_EDITING_PROJECT_DIRECTORY`. |
|
||||||
|
| `.highlight-project-directory` | `./output/highlight-projects` | Single-source projects; `VIDEO_EDITING_HIGHLIGHT_PROJECT_DIRECTORY`. |
|
||||||
|
| `.ffmpeg-binary` / `.ffprobe-binary` | `ffmpeg` / `ffprobe` | Editing executables; `VIDEO_EDITING_FFMPEG_BINARY`, `VIDEO_EDITING_FFPROBE_BINARY`. |
|
||||||
|
| `.thumbnail-count-per-clip` / `.contact-sheet-columns` | `5` / `5` | Inspection density; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.proxy-enabled` / `.proxy-width` | `true` / `640` | Proxy generation and width; width `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.scene-detection-threshold` | `0.35` | FFmpeg scene threshold; no range validation; `VIDEO_EDITING_SCENE_DETECTION_THRESHOLD`. |
|
||||||
|
| `.minimum-scene-duration-seconds` | `1.0` | Segment merge threshold; no validation; `VIDEO_EDITING_MINIMUM_SCENE_DURATION_SECONDS`. |
|
||||||
|
| `.silence-threshold-db` / `.silence-minimum-duration-seconds` | `-35.0` / `0.5` | Source audio analysis; no validation; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.target-duration-seconds` | `600` | Multi-clip target; `@Min(1)`; `VIDEO_EDITING_TARGET_DURATION_SECONDS`. |
|
||||||
|
| `.output-width` / `.output-height` / `.output-frame-rate` | `1920` / `1080` / `30` | Render geometry/rate; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.audio-sample-rate` | `48000` | Render audio rate; `@Min(1)`; `VIDEO_EDITING_AUDIO_SAMPLE_RATE`. |
|
||||||
|
| `.video-bitrate` / `.audio-bitrate` | `12000k` / `192k` | FFmpeg bitrate strings; no format validation; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.voiceover-provider` | `local` | Bound and tested but not consumed by main Java code; it is not a working remote-provider selector. Env: `VIDEO_EDITING_VOICEOVER_PROVIDER`. External AI is prohibited. |
|
||||||
|
| `.loudness-target-i` / `.loudness-true-peak` / `.loudness-range` | `-16.0` / `-1.5` / `11.0` | `loudnorm` parameters; no range validation; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.music-ducking-threshold` / `.music-ducking-ratio` | `0.045` / `8.0` | `sidechaincompress` parameters; no range validation; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.music-ducking-attack-ms` / `.music-ducking-release-ms` | `20` / `250` | Ducking timing; each `@Min(1)`; matching `VIDEO_EDITING_*` variable. |
|
||||||
|
| `.assets.music-folder` | `./input/highlights/assets/music` | Intended location for approved, licensed local music; directory membership is not license proof. `VIDEO_EDITING_ASSETS_MUSIC_FOLDER`. |
|
||||||
|
| `.assets.sfx-folder` | `./input/highlights/assets/sfx` | Intended location for approved, licensed local SFX; directory membership is not license proof. `VIDEO_EDITING_ASSETS_SFX_FOLDER`. |
|
||||||
|
| `.assets.fonts-folder` | `./input/highlights/assets/fonts` | Intended location for approved, licensed local fonts; directory membership is not license proof. `VIDEO_EDITING_ASSETS_FONTS_FOLDER`. |
|
||||||
|
| `.assets.luts-folder` | `./input/highlights/assets/luts` | Intended location for approved, licensed local LUTs; directory membership is not license proof. `VIDEO_EDITING_ASSETS_LUTS_FOLDER`. |
|
||||||
|
| `.assets.voiceover-folder` | `./output/highlight-projects/_voiceover-cache` | Generated voiceover cache; `VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER`. |
|
||||||
|
|
||||||
|
Treat render, loudness, ducking, scene, and duration changes as output behavior changes. Require objective measurements and representative media evidence; configuration binding tests alone are insufficient.
|
||||||
|
|
||||||
|
Exact environment names for grouped editing rows:
|
||||||
|
|
||||||
|
| Property suffixes | Environment variables, in the same order |
|
||||||
|
|---|---|
|
||||||
|
| `.thumbnail-count-per-clip`, `.contact-sheet-columns` | `VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP`, `VIDEO_EDITING_CONTACT_SHEET_COLUMNS` |
|
||||||
|
| `.proxy-enabled`, `.proxy-width` | `VIDEO_EDITING_PROXY_ENABLED`, `VIDEO_EDITING_PROXY_WIDTH` |
|
||||||
|
| `.silence-threshold-db`, `.silence-minimum-duration-seconds` | `VIDEO_EDITING_SILENCE_THRESHOLD_DB`, `VIDEO_EDITING_SILENCE_MINIMUM_DURATION_SECONDS` |
|
||||||
|
| `.output-width`, `.output-height`, `.output-frame-rate` | `VIDEO_EDITING_OUTPUT_WIDTH`, `VIDEO_EDITING_OUTPUT_HEIGHT`, `VIDEO_EDITING_OUTPUT_FRAME_RATE` |
|
||||||
|
| `.video-bitrate`, `.audio-bitrate` | `VIDEO_EDITING_VIDEO_BITRATE`, `VIDEO_EDITING_AUDIO_BITRATE` |
|
||||||
|
| `.loudness-target-i`, `.loudness-true-peak`, `.loudness-range` | `VIDEO_EDITING_LOUDNESS_TARGET_I`, `VIDEO_EDITING_LOUDNESS_TRUE_PEAK`, `VIDEO_EDITING_LOUDNESS_RANGE` |
|
||||||
|
| `.music-ducking-threshold`, `.music-ducking-ratio` | `VIDEO_EDITING_MUSIC_DUCKING_THRESHOLD`, `VIDEO_EDITING_MUSIC_DUCKING_RATIO` |
|
||||||
|
| `.music-ducking-attack-ms`, `.music-ducking-release-ms` | `VIDEO_EDITING_MUSIC_DUCKING_ATTACK_MS`, `VIDEO_EDITING_MUSIC_DUCKING_RELEASE_MS` |
|
||||||
|
|
||||||
|
### Visual analysis and local CV worker
|
||||||
|
|
||||||
|
All keys start with `video-clipping.editing.visual-analysis`.
|
||||||
|
|
||||||
|
| Suffix | J / P default | Meaning / explicit environment variable |
|
||||||
|
|---|---|---|
|
||||||
|
| `.provider` | J `heuristic`; P `local-cv` | Recognized routing is `local-cv`; any other value uses heuristic. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER`. |
|
||||||
|
| `.endpoint` | `http://127.0.0.1:8091/v1/analyze-visuals` | Current local HTTP endpoint. Loopback is network, so this is a noncompliant implementation gap, not an approved certified path. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT`. |
|
||||||
|
| `.timeout-ms` | `30000` | Total HTTP call timeout; `@Min(1)`. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS`. |
|
||||||
|
| `.fallback-to-heuristic` | `true` | On local-CV failure, return weaker heuristic analysis. Must be false in the fail-closed production target. Env: `VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC`. |
|
||||||
|
| `.local-cv-worker.auto-start` | J `false`; P `true` | Starts `script` only when provider is `local-cv`. Runtime auto-start/bootstrap is prohibited. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START`. |
|
||||||
|
| `.local-cv-worker.script` | `./tools/run_local_cv_worker.sh` | Launched executable. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT`. |
|
||||||
|
| `.local-cv-worker.startup-wait-ms` | `0` | `0` skips health waiting; otherwise timeout; `@Min(0)`. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS`. |
|
||||||
|
| `.local-cv-worker.health-path` | `/health` | Health URI path. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH`. |
|
||||||
|
| `.local-cv-worker.health-check-interval-ms` | `1000` | Readiness poll and per-call cap; `@Min(1)`. Env: `VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS`. |
|
||||||
|
|
||||||
|
Worker-only environment variables are not Spring properties:
|
||||||
|
|
||||||
|
| Variable | Default/source | Effect and policy |
|
||||||
|
|---|---|---|
|
||||||
|
| `LOCAL_CV_HOST`, `LOCAL_CV_PORT` | `127.0.0.1`, `8091`; manager derives both from endpoint | Uvicorn bind address/port. Keep local to the service trust boundary. |
|
||||||
|
| `LOCAL_CV_BOOTSTRAP_MODE` | `auto` | `always` or first-use `auto` invokes pip. Runtime must use a pre-provisioned path/mode that cannot install. |
|
||||||
|
| `LOCAL_CV_PRELOAD_ONLY` | `false` | Exit after bootstrap/model preload. Preload can still download. |
|
||||||
|
| `LOCAL_CV_YOLO_MODEL` | `yolov8n.pt` | Ultralytics model path/name. A name can trigger model retrieval; production must supply a verified local path. |
|
||||||
|
| `LOCAL_CV_DISABLE_YOLO` | `false` | Skips YOLO and returns weaker/default labels. Experimental/debug only, never production-quality success. |
|
||||||
|
| `LOCAL_CV_LOG_LEVEL` | `INFO` | Python worker logging level. |
|
||||||
|
|
||||||
|
The worker health response reports library availability but returns HTTP 200 even if OpenCV or YOLO is absent. Do not equate `/health` success with production analysis readiness.
|
||||||
|
|
||||||
|
### Local asset runtime
|
||||||
|
|
||||||
|
All keys start with `video-clipping.editing.local-asset-worker`.
|
||||||
|
|
||||||
|
| Suffix | J / P default | Meaning / explicit environment variable |
|
||||||
|
|---|---|---|
|
||||||
|
| `.auto-start` | J `false`; P `true` | Runs bootstrap verification, not a persistent asset server. Must be false at production runtime under the no-download rule. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START`. |
|
||||||
|
| `.strict-runtime` | J `false`; P `true` | Makes startup fail when the pre-provisioned local asset runtime is incomplete. It requires adjacent nonblank model-license sidecars but does not verify authenticity, origin, checksums, or allowed use. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_STRICT_RUNTIME`. |
|
||||||
|
| `.bootstrap-script` | `./tools/run_local_asset_worker.sh` | Startup verifier command; `VIDEO_EDITING_LOCAL_ASSET_BOOTSTRAP_SCRIPT`. |
|
||||||
|
| `.script` | `./tools/local_asset_worker.py` | Per-asset CLI worker; `VIDEO_EDITING_LOCAL_ASSET_WORKER_SCRIPT`. |
|
||||||
|
| `.startup-wait-ms` | `0` | Bound and tested but not consumed by current asset verifier/synthesizer; `@Min(0)`. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_STARTUP_WAIT_MS`. |
|
||||||
|
| `.health-path` | `/health` | Bound and tested but no local asset HTTP server consumes it. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_PATH`. |
|
||||||
|
| `.health-check-interval-ms` | `1000` | Bound and tested but unused; `@Min(1)`. Env: `VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_CHECK_INTERVAL_MS`. |
|
||||||
|
| `.python-binary` | `./.venv-local-asset/bin/python` | Python used for import readiness and worker invocation; `VIDEO_EDITING_LOCAL_ASSET_PYTHON_BINARY`. |
|
||||||
|
| `.piper-binary` | `piper` | Voice model executable; `VIDEO_EDITING_LOCAL_ASSET_PIPER_BINARY`. |
|
||||||
|
| `.piper-model-path` | empty | Required in practice for Piper; `VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH`. |
|
||||||
|
| `.music-model` | `musicgen-small` | AudioCraft model name; `VIDEO_EDITING_LOCAL_ASSET_MUSIC_MODEL`. A short name is normalized to `facebook/<name>` and may download. |
|
||||||
|
| `.sfx-model` | `audiogen-medium` | AudioCraft model name; `VIDEO_EDITING_LOCAL_ASSET_SFX_MODEL`. A short name may download. |
|
||||||
|
|
||||||
|
Worker-only environment variables:
|
||||||
|
|
||||||
|
| Variable | Current behavior |
|
||||||
|
|---|---|
|
||||||
|
| `LOCAL_ASSET_BOOTSTRAP_MODE` | Script default `auto`; `auto`/`always` invokes pip. The Java verifier forcibly sets `auto`, overriding an inherited safer value. This blocks compliant runtime auto-start today. |
|
||||||
|
| `LOCAL_ASSET_PRELOAD_ONLY` | Java verifier sets `true`; script exits after dependency bootstrap. |
|
||||||
|
| `LOCAL_ASSET_PIPER_BINARY` | Java verifier passes configured binary; Python CLI also reads it by default. |
|
||||||
|
| `LOCAL_ASSET_PIPER_MODEL_PATH` | Java verifier exports it when nonblank; generation passes the configured model explicitly. |
|
||||||
|
| `LOCAL_ASSET_HOST`, `LOCAL_ASSET_PORT` | Script reads defaults `127.0.0.1:8092` but never uses them; there is no HTTP asset server. |
|
||||||
|
|
||||||
|
The Python and Java fallback success paths were removed in the 2026-07-21 working tree. Continue rejecting any generated asset whose production provenance, local model identity, checksum, license, and semantic fit are not verified; an audible WAV alone is insufficient.
|
||||||
|
|
||||||
|
### Local director and highlight scheduler
|
||||||
|
|
||||||
|
| Property suffix | Default | Meaning / explicit environment variable |
|
||||||
|
|---|---|---|
|
||||||
|
| `.local-director.enabled` | `true` | With editing enabled, polls multi-clip inbox; `VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED`. |
|
||||||
|
| `.local-director.source/working/processed/rejected-directory` | `./input/editing/<state>` | Move-based lifecycle; matching `VIDEO_EDITING_LOCAL_DIRECTOR_*_DIRECTORY`. |
|
||||||
|
| `.local-director.poll-interval-ms` | `5000` | Initial/fixed delay; `@Min(1000)`; `VIDEO_EDITING_LOCAL_DIRECTOR_POLL_INTERVAL_MS`. |
|
||||||
|
| `.local-director.director-prompt-file-name` | `ai-director-prompt.md` | Prompt artifact; `VIDEO_EDITING_LOCAL_DIRECTOR_PROMPT_FILE_NAME`. |
|
||||||
|
| `.local-director.expected-plan-file-name` | `edit-plan.json` | Plan inbox filename; `VIDEO_EDITING_LOCAL_DIRECTOR_EXPECTED_PLAN_FILE_NAME`. |
|
||||||
|
| `.local-director.auto-render-when-plan-appears` | `false` | Enables render after plan scan; `VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER`. Keep false unless explicitly approved. |
|
||||||
|
| `.local-director.require-approval-before-render` | `true` | Requires approval file when auto-rendering; `VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL`. |
|
||||||
|
| `.local-director.approval-file-name` | `approved.flag` | Approval artifact name; `VIDEO_EDITING_LOCAL_DIRECTOR_APPROVAL_FILE_NAME`. |
|
||||||
|
| `.highlight-scheduler.enabled` | `true` | With editing enabled, enables both source and plan/render scanners; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED`. |
|
||||||
|
| `.highlight-scheduler.source/working/processed/rejected-directory` | `./input/highlights/<state>` | Move-based source lifecycle; matching `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_*_DIRECTORY`. |
|
||||||
|
| `.highlight-scheduler.poll-interval-ms` | `5000` | Shared by source and render scanners; `@Min(1000)`; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS`. |
|
||||||
|
| `.highlight-scheduler.render-enabled` | `false` | Runtime render scan gate; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED`. Keep false until the approval workflow is proven. |
|
||||||
|
| `.highlight-scheduler.require-director-approval` | `true` | Requires director approval file before flow; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL`. The bare file is not digest-bound authorization. |
|
||||||
|
| `.highlight-scheduler.approval-file-name` | `approved.flag` | Approval artifact under director directory; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME`. |
|
||||||
|
| `.highlight-scheduler.max-highlights-per-source` | `3` | Plan item cap; `@Min(1)`; `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE`. |
|
||||||
|
| `.highlight-scheduler.highlight-min-duration-seconds` | `8.0` | Lower duration clamp; `@Min(1)`; matching environment variable. |
|
||||||
|
| `.highlight-scheduler.highlight-max-duration-seconds` | `35.0` | Upper duration clamp; `@Min(1)`; matching environment variable. No validation ensures max >= min. |
|
||||||
|
|
||||||
|
Exact environment names for grouped scheduler rows:
|
||||||
|
|
||||||
|
| Property suffixes | Environment variables, in the same order |
|
||||||
|
|---|---|
|
||||||
|
| `.local-director.source/working/processed/rejected-directory` | `VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_WORKING_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_PROCESSED_DIRECTORY`, `VIDEO_EDITING_LOCAL_DIRECTOR_REJECTED_DIRECTORY` |
|
||||||
|
| `.highlight-scheduler.source/working/processed/rejected-directory` | `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_SOURCE_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_WORKING_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY` |
|
||||||
|
| `.highlight-scheduler.highlight-min/max-duration-seconds` | `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS`, `VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS` |
|
||||||
|
|
||||||
|
## Guards and coupled settings
|
||||||
|
|
||||||
|
| Behavior | Required condition | Additional gate |
|
||||||
|
|---|---|---|
|
||||||
|
| Folder segmentation | `folder-scheduler.enabled=true` | None beyond input eligibility; it moves files. |
|
||||||
|
| Editing API/core | `editing.enabled=true` or missing | Many editing beans use `matchIfMissing=true`. |
|
||||||
|
| Multi-clip director polling | editing and local-director enabled | Directory candidate rules. |
|
||||||
|
| Multi-clip auto-render | above plus `auto-render-when-plan-appears=true` | Approval file only when `require-approval-before-render=true`. |
|
||||||
|
| Highlight ingestion | editing and highlight scheduler enabled | Moves one eligible video at a time. |
|
||||||
|
| Highlight plan rendering | same | `render-enabled=true`; approval only when `require-director-approval=true`. |
|
||||||
|
| Local CV process | editing enabled | provider exactly `local-cv` and worker auto-start true. |
|
||||||
|
| Local CV fallback | provider `local-cv` fails | Returns heuristic only when fallback is true; otherwise throws. |
|
||||||
|
| Local asset readiness check | editing enabled | Local asset auto-start true; strict mode fails startup when readiness is incomplete. The bootstrap script is not invoked by this verifier. |
|
||||||
|
| Cleanup | scheduled unconditionally | Work skipped when cleanup enabled is false. |
|
||||||
|
| DB queue poller | `queue=db` | Requires JPA repository/infrastructure coherence. |
|
||||||
|
|
||||||
|
Before changing one setting, inspect all settings in its row. In particular, enabling an approval boolean without disabling render during rollout leaves a race window unless deployment/configuration is atomic.
|
||||||
|
|
||||||
|
## Profiles and tests
|
||||||
|
|
||||||
|
| Source | Overrides | Classification |
|
||||||
|
|---|---|---|
|
||||||
|
| `application.properties` | API defaults, adapters, S3, DB queue, FFmpeg workspace, quotas, cleanup, actuator exposure | Packaged base; currently development-oriented. |
|
||||||
|
| `application.yml` | Folder scheduler and complete editing tree | Packaged base; current all-on behavior is unsafe. |
|
||||||
|
| `application-jpa.properties` | JPA/db queue/stub plus H2, Flyway, Hibernate validation | Local/integration convenience, not production PostgreSQL. |
|
||||||
|
| `application-folder-scheduler-local.yml` | Enables local folder scheduler paths | Local only. |
|
||||||
|
| `application-cinematic-editing-local.yml` | Disables folder scheduler; enables editing/director auto-render with approval | Local only; rendering side effects. |
|
||||||
|
| `src/test/resources/application.properties` | Disables folder scheduler, director, highlight scheduler, CV auto-start, asset auto-start; sets asset strict false | Test safety override. |
|
||||||
|
| `src/test/resources/application-test.properties` | Disables folder scheduler only | Narrow test-profile override; do not assume it disables other consumers. |
|
||||||
|
|
||||||
|
`VideoClippingPropertiesTest` verifies most editing Java defaults and representative binding overrides. It does not exhaustively assert every top-level, folder, cleanup, queue, or highlight setting. Profile tests parse YAML values but do not prove safe runtime composition.
|
||||||
|
|
||||||
|
## Production target, not current implementation
|
||||||
|
|
||||||
|
For macOS development, Linux/VPS production, and cloud infrastructure, converge through change control on this behavior:
|
||||||
|
|
||||||
|
- Ship one immutable, platform-neutral application artifact, then create signed platform-specific runtime/model/image bundles. Promote the same bundle unchanged between environments for the same OS/CPU target; inject paths and secrets externally.
|
||||||
|
- Disable every scheduler, worker auto-start, and render path by default. Enable only the deployed capability.
|
||||||
|
- Run local model workers as pre-provisioned supervised processes or explicit adapters with no installation/download code in the request/startup path.
|
||||||
|
- Replace model-worker HTTP with in-process calls or explicitly designed non-network IPC. Deny model/media inference networking, including loopback. Treat API, database, storage, and telemetry as separately approved production integrations.
|
||||||
|
- Require verified local model paths, checksums, licenses, writable/output mounts, capacity limits, and health that validates model readiness.
|
||||||
|
- Make missing FFmpeg, FFprobe, Python stack, Piper model, AudioCraft model, CV model, or licensed asset fail startup/readiness for any enabled dependent capability.
|
||||||
|
- Disable heuristic, silence, tone, macOS `say`, and `espeak` substitutions for production results.
|
||||||
|
- Require approval for every render path and record the approver/artifact; a bare filesystem flag is a current mechanism, not sufficient Fortune 500 authorization evidence.
|
||||||
|
- Use distinct durable/shared project storage or enforce single-instance scheduling. Current filesystem queues/projects and move-based inboxes are not automatically horizontally scalable.
|
||||||
|
|
||||||
|
Do not silently implement this target by editing defaults. Record each gap as a controlled change with migration and rollback.
|
||||||
|
|
||||||
|
## Add or change a property
|
||||||
|
|
||||||
|
Complete every checkbox:
|
||||||
|
|
||||||
|
- [ ] State the requirement, owner, consumers, supported values/units, safe default, production classification, expiry/revisit condition, and whether it changes output or side effects.
|
||||||
|
- [ ] Search for an existing property before adding one; avoid aliases and speculative flags.
|
||||||
|
- [ ] Add the typed field under `VideoClippingProperties`; use an immutable value type where practical and add Jakarta validation for every meaningful range, nonblank path, enum, URI, or cross-field invariant.
|
||||||
|
- [ ] Decide whether omission must fail. Never give production-critical model, license, secret, endpoint, or approval settings an insecure fallback.
|
||||||
|
- [ ] Add the packaged property exactly once. If operations need an explicit environment name, use a documented placeholder and preserve Spring relaxed-binding consistency.
|
||||||
|
- [ ] Classify the default change through `video-editing-change-control`. Treat enabling a scheduler/worker/render, changing an adapter, endpoint, path, retention, quality threshold, model, codec, or fallback as production-facing.
|
||||||
|
- [ ] Map environment impact for macOS, Linux/VPS, and cloud: paths/mounts, permissions, filesystem persistence, process supervision, network policy, resources, secrets, rollout ordering, backward compatibility, and rollback value.
|
||||||
|
- [ ] Check coupled guards and all profiles. Make production fail closed; keep runtime downloads, external AI, network fetching, unlicensed assets, placeholders, and unapproved renders impossible.
|
||||||
|
- [ ] Add default, valid-override, invalid-value, and application-context guard tests. Add a cross-field test for min/max or mutually dependent settings.
|
||||||
|
- [ ] Add behavior tests proving the consumer uses the property. For output changes, add objective QA evidence; binding alone is not proof.
|
||||||
|
- [ ] Update this catalog and the configuration reference/documents of record. Explain every deliberate Spring Boot default deviation.
|
||||||
|
- [ ] Run the focused tests, full clean build prescribed by `video-editing-validation-and-qa`, configuration drift scans, and `git diff --check`.
|
||||||
|
- [ ] Provide rollout order, observable signals, rollback command/value, and removal date for temporary flags. Do not leave permanent dual paths without ownership.
|
||||||
|
|
||||||
|
## Review a proposed environment
|
||||||
|
|
||||||
|
Reject the configuration if any answer is “no”:
|
||||||
|
|
||||||
|
- [ ] Are only the intended scheduler and adapter beans enabled?
|
||||||
|
- [ ] Are model/dependency downloads impossible after image/build promotion?
|
||||||
|
- [ ] Do all model identifiers resolve to verified local artifacts rather than remote names?
|
||||||
|
- [ ] Are external AI and arbitrary remote worker endpoints impossible?
|
||||||
|
- [ ] Are all input, working, processed, rejected, project, cache, and temporary paths distinct, mounted, permissioned, capacity-limited, and retained intentionally?
|
||||||
|
- [ ] Does any missing model or asset fail closed without heuristic, silence, tone, `say`, or `espeak` substitution?
|
||||||
|
- [ ] Is every render disabled until authenticated approval is present?
|
||||||
|
- [ ] Are queue/repository/storage selectors coherent and production adapters, not `memory`/`stub`/H2 conveniences?
|
||||||
|
- [ ] Are cleanup and retention consistent with recovery, audit, and legal requirements?
|
||||||
|
- [ ] Are local worker health checks capability-aware, and are timeout/resource limits proven?
|
||||||
|
- [ ] Is the same application artifact and same signed runtime bundle used without rebuild across environments for each platform target, with auditable external configuration?
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Primary sources: `VideoClippingProperties`, packaged/profile/test configuration, all `@ConditionalOnProperty`/`@ConditionalOnExpression` guards, scheduler annotations, local worker managers, shell launchers, Python workers, and configuration/profile tests. Re-verify volatile facts after any configuration, worker, or dependency change.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List declared Java defaults and validation annotations.
|
||||||
|
rg -n '^ public static class|^[[:space:]]+@(Min|Max|NotBlank|NotNull|Pattern)|^[[:space:]]+private ' src/main/java/org/example/videoclips/config/VideoClippingProperties.java
|
||||||
|
|
||||||
|
# Regenerate the packaged/profile/test key inventory.
|
||||||
|
rg -n '^[[:space:]]*[A-Za-z0-9_.-]+[=:]' src/main/resources src/test/resources
|
||||||
|
|
||||||
|
# Find configuration consumers and guards.
|
||||||
|
rg -n '@ConditionalOn(Property|Expression)|@Scheduled|video-clipping\.' src/main/java src/test/java
|
||||||
|
|
||||||
|
# Regenerate explicit environment mappings and worker-only variables.
|
||||||
|
rg -n '\$\{[A-Z][A-Z0-9_]*:|LOCAL_(ASSET|CV)_[A-Z0-9_]+' src/main/resources tools src/main/java
|
||||||
|
|
||||||
|
# Recheck fallback and download hazards without executing them.
|
||||||
|
rg -n 'pip install|get_pretrained|YOLO\(|fallback|write_silence|write_fallback_tone|strictRuntime|strict-runtime' tools src/main/java src/main/resources
|
||||||
|
|
||||||
|
# Run focused binding/profile tests only after confirming test resources disable side effects.
|
||||||
|
mvn -o -Dtest=VideoClippingPropertiesTest,CinematicEditingLocalProfileTest,FolderSchedulerLocalProfileTest,FolderSchedulerSpringContextTest test
|
||||||
|
|
||||||
|
# Check documentation formatting and the repository diff without mutation.
|
||||||
|
wc -l .claude/skills/video-editing-config-and-flags/SKILL.md
|
||||||
|
git diff --check -- .claude/skills/video-editing-config-and-flags
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,330 @@
|
||||||
|
---
|
||||||
|
name: video-editing-debugging-playbook
|
||||||
|
description: "Load when a concrete video-editing symptom needs symptom-to-branch triage: startup/readiness failure, missing or stalled highlight output, inconsistent project state, missing category/candidate artifact, rejected director plan, FFmpeg/audio/codec failure, scheduler quarantine, API queue/DLQ/storage/JPA incident, profile drift, or clean-checkout failure. Use it to choose discriminating read-only experiments; move to the proof toolkit only after triage identifies competing causal mechanisms."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Debug the Video Editing Service
|
||||||
|
|
||||||
|
Use this runbook from the repository root. Treat code, tests, runtime artifacts, and structured logs as evidence; do not judge media quality by eye alone.
|
||||||
|
|
||||||
|
Facts marked **current** were verified against `main` on 2026-07-21. **Target** means the requested Fortune 500 production architecture, not implemented behavior.
|
||||||
|
|
||||||
|
## Safety boundary
|
||||||
|
|
||||||
|
The no-waiver policy lives in `video-editing-change-control`. During triage, make no dependency/model download, external-AI call, model/media network call (including loopback), unlicensed-asset use, placeholder acceptance, unapproved render, or production-default change. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh`: both can install packages automatically, and the CV script can fetch a YOLO model.
|
||||||
|
|
||||||
|
Keep the first pass read-only. A local FFprobe or decode-to-null command measures an existing file; it does not render or publish a replacement.
|
||||||
|
|
||||||
|
Route every proposed code, dependency, configuration, asset, migration, redrive, or render change through `video-editing-change-control`. Use `video-editing-failure-archaeology` before revisiting a historical fix. Never mutate queue rows during triage.
|
||||||
|
|
||||||
|
## Know the terms
|
||||||
|
|
||||||
|
| Term | Meaning here |
|
||||||
|
|---|---|
|
||||||
|
| Highlight project | Single-source filesystem workflow rooted by default at `output/highlight-projects/<project-id>/`. |
|
||||||
|
| Edit project | Multi-clip filesystem workflow rooted by default at `output/edit-projects/<project-id>/`. It has a different plan scanner and validator. |
|
||||||
|
| Candidate | Ranked source time range proposed for a highlight. Its expected single-source file is `analysis/highlight-candidates.json`. |
|
||||||
|
| Director plan | Human/local-director-authored `director/edit-plan.json` consumed by the single-source renderer. |
|
||||||
|
| Strict runtime | `video-clipping.editing.local-asset-worker.strict-runtime`; when asset auto-start is enabled, incomplete resident model/runtime readiness throws during startup. |
|
||||||
|
| DLQ | Dead-letter queue; terminal database-queue state after the configured maximum attempts. |
|
||||||
|
| Marker | A failed folder input renamed in place to `<name>.failed` when movement to the rejected directory also fails. |
|
||||||
|
| Discriminating experiment | A check whose possible outcomes select different causes or next actions. |
|
||||||
|
|
||||||
|
## Start with a read-only evidence bundle
|
||||||
|
|
||||||
|
Set shell variables only to shorten commands; do not infer project IDs from display names.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
project_id='<exact-project-directory-name>'
|
||||||
|
project="output/highlight-projects/$project_id"
|
||||||
|
logs='<path-to-captured-service-log>'
|
||||||
|
git status --short
|
||||||
|
git rev-parse --short HEAD
|
||||||
|
java -version
|
||||||
|
mvn -version
|
||||||
|
ffmpeg -version | sed -n '1,3p'
|
||||||
|
ffprobe -version | sed -n '1,3p'
|
||||||
|
find "$project" -maxdepth 4 -type f -print | sort
|
||||||
|
sed -n '1,220p' "$project/project.json"
|
||||||
|
rg 'event=(highlight|local_cv|local_asset|visual_analysis)' "$logs" | tail -n 250
|
||||||
|
```
|
||||||
|
|
||||||
|
If a command reports a missing path, preserve that as evidence. Do not create the path to make the check pass.
|
||||||
|
|
||||||
|
## Fast symptom map
|
||||||
|
|
||||||
|
| Symptom | First discriminator | If X, branch to | If not X, branch to |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Startup hangs/fails | Search `local_cv_worker_*` and `local_asset_*` events | Bootstrap/readiness | Spring/profile/config |
|
||||||
|
| Source disappears; no final | Inventory source, working, processed, rejected, project | Scheduler claim/analysis | Wrong directory/profile |
|
||||||
|
| `CREATED` persists | Check `analysis/source-analysis.json` and failure logs | Partial analysis | Prompt/status persistence |
|
||||||
|
| `WAITING_FOR_DIRECTOR` persists | Check plan, approval, asset request files | Plan/approval/assets | Scanner disabled or wrong root |
|
||||||
|
| `RENDERING` persists | Check renderer logs and per-highlight `final.mp4` | FFmpeg/publish failure | Stale status/artifact mismatch |
|
||||||
|
| `FAILED` | Read `failureMessage` and correlate the same project | Empty highlight list | Unrecorded exception; do not invent cause |
|
||||||
|
| No category/candidates | Compare single-source artifacts with generator wiring | Known pipeline gap | Corrupt/mislocated JSON |
|
||||||
|
| Folder input is rejected/marked | Inspect validator and movement events | Invalid media/collision | FFmpeg failure |
|
||||||
|
| API jobs stall/fail | Confirm queue adapter and queue row states | DB queue/DLQ | Memory queue/process-local loss |
|
||||||
|
| Works locally, clean build fails | Run isolated offline archive build | Untracked runtime dependency | Toolchain/cache mismatch |
|
||||||
|
|
||||||
|
## Triage startup and local models
|
||||||
|
|
||||||
|
### Local CV worker
|
||||||
|
|
||||||
|
**Current:** `application.yml` selects `local-cv`, enables auto-start, sets `startup-wait-ms: 0`, and permits heuristic fallback. A zero wait logs `local_cv_worker_ready` without an HTTP health check. The launch script defaults `LOCAL_CV_BOOTSTRAP_MODE=auto`, installs packages when its virtualenv is incomplete, and loads `yolov8n.pt`, which may trigger a model fetch.
|
||||||
|
|
||||||
|
Run only local, non-starting checks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x .venv-local-cv/bin/python && echo python-present || echo python-missing
|
||||||
|
.venv-local-cv/bin/python -c 'import fastapi,uvicorn,cv2,numpy,ultralytics; print("cv-imports-ready")'
|
||||||
|
test -f yolov8n.pt && stat -f '%N %z bytes' yolov8n.pt 2>/dev/null || stat -c '%n %s bytes' yolov8n.pt 2>/dev/null
|
||||||
|
rg 'local_cv_worker_(start|health|process)|local_cv_visual_analysis_(completed|fallback|failed)' "$logs"
|
||||||
|
```
|
||||||
|
|
||||||
|
- If imports or the model are missing, stop. The runtime image/environment is not preprovisioned; use `video-editing-build-and-env`. Do not invoke bootstrap.
|
||||||
|
- If `local_cv_worker_ready` appears with `local_cv_worker_health_wait_skipped`, readiness is unproved.
|
||||||
|
- If `local_cv_visual_analysis_fallback` appears, inspect `analysis/visual-analysis.json`. An `analysisMethod` beginning `local_cv_failed_fallback_` proves degraded heuristic output.
|
||||||
|
- If the worker exited unexpectedly, inspect preceding `local_cv_worker_output`; do not merely restart it.
|
||||||
|
- **Target:** immutable, licensed, checksummed model artifacts inside the approved runtime; startup must prove readiness without downloads.
|
||||||
|
|
||||||
|
### Local asset runtime
|
||||||
|
|
||||||
|
**Current working tree:** application startup no longer invokes asset bootstrap. Strict readiness requires existing Piper, MusicGen, and AudioGen paths and throws when incomplete. Java/Python generation returns failure and removes invalid output instead of using host speech, silence, or tones. The standalone launcher still runs `pip install` in `auto` mode; never use it for triage or certified operation.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x .venv-local-asset/bin/python && echo python-present || echo python-missing
|
||||||
|
.venv-local-asset/bin/python -c 'import torch,audiocraft,soundfile,numpy; print("audio-imports-ready")'
|
||||||
|
test -n "${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:-}" && test -f "$VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH" && echo piper-model-present || echo piper-model-missing
|
||||||
|
command -v piper || true
|
||||||
|
rg 'local_asset_(bootstrap|runtime|python_stack|worker|generation|voiceover)' "$logs"
|
||||||
|
```
|
||||||
|
|
||||||
|
- If `local_asset_runtime_check_failed` appears, do not render or promote. In strict mode the application must fail startup; in non-strict mode it logs and continues.
|
||||||
|
- The current asset verifier does not invoke its bootstrap script. Treat any bootstrap log as a historical revision or a separately invoked prohibited shell launcher.
|
||||||
|
- Successful Python imports do not prove model weights are present. Require the approved runtime inventory/checksum evidence; do not discover availability by invoking `get_pretrained`.
|
||||||
|
- If logs say `local_asset_worker_completed`, verify the waveform; exit zero is not proof of a model-generated asset.
|
||||||
|
- **Target:** missing or invalid required local models fail startup; every produced asset carries model/provenance evidence; placeholders are impossible.
|
||||||
|
|
||||||
|
## Detect prohibited silence and tones
|
||||||
|
|
||||||
|
Apply to every generated voiceover, music, and SFX file. First correlate generation events; legacy `strategy=silence`, `strategy=fallback-tone`, `model=silence`, or `model=ffmpeg-sine` in an older artifact is an immediate failure. The current worker must fail instead of emitting those strategies.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
audio='<generated-audio-file>'
|
||||||
|
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels,duration -of default=noprint_wrappers=1 "$audio"
|
||||||
|
ffmpeg -hide_banner -nostats -i "$audio" -af 'astats=metadata=1:reset=0' -f null - 2>&1 | rg 'Peak level dB|RMS level dB|Zero crossings rate'
|
||||||
|
ffmpeg -hide_banner -nostats -i "$audio" -af 'silencedetect=noise=-50dB:d=0.5' -f null - 2>&1 | rg 'silence_(start|end|duration)'
|
||||||
|
```
|
||||||
|
|
||||||
|
- If peak/RMS is `-inf`, reject as silence.
|
||||||
|
- If generation failed and a narrow periodic signal appears, treat it as suspected fallback tone; confirm against the log strategy or compare its dominant behavior with the documented 110 Hz music / 880 Hz SFX implementation. Do not approve by listening.
|
||||||
|
- If speech is audible but came from macOS `say` or `espeak`, it is a fallback, not proof that the configured Piper model ran.
|
||||||
|
- Route deterministic asset certification to `video-editing-validation-and-qa`.
|
||||||
|
|
||||||
|
**Costly trap:** commit `97ba827` added strict runtime after placeholder-producing paths already existed. The 2026-07-21 working tree now fails strict startup and removes placeholder success, but file existence, a license sidecar, or exit code zero still does not prove production audio quality, provenance authenticity, or allowed use.
|
||||||
|
|
||||||
|
## Diagnose the single-source highlight flow
|
||||||
|
|
||||||
|
### Establish the artifact frontier
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for f in project.json analysis/source-analysis.json analysis/category.json analysis/highlight-candidates.json director/director-prompt.md director/edit-plan.json final.mp4 render-manifest.json; do
|
||||||
|
test -f "$project/$f" && printf 'present %s\n' "$f" || printf 'missing %s\n' "$f"
|
||||||
|
done
|
||||||
|
find "$project/highlights" -maxdepth 3 -type f -print 2>/dev/null | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
Interpret the last present artifact as the frontier, then inspect the first missing producer. Do not use downstream absence as a root cause.
|
||||||
|
|
||||||
|
### State-specific branches
|
||||||
|
|
||||||
|
| State | Current meaning | Discriminating branch |
|
||||||
|
|---|---|---|
|
||||||
|
| `CREATED` | Project record was written. Source analysis may not have completed. | If `analysis/source-analysis.json` is missing and source moved to rejected, inspect `highlight_analysis_step_failed`. If analysis exists but state remains `CREATED`, prompt generation/status update did not complete. |
|
||||||
|
| `ANALYZING` | Enum exists, but the single-source scheduler does not currently write it. | Treat it as external/stale data unless code history proves otherwise. |
|
||||||
|
| `WAITING_FOR_DIRECTOR` | Prompt generation updated status; no validated plan has entered the flow. | If a plan exists, inspect approval and validator rejection in `failureMessage`/logs. |
|
||||||
|
| `PLANNED` | The director plan passed `HighlightDirectorPlanValidator`; required assets may still be pending. | Inspect asset request JSON, immutable local model inventory, and worker failure logs. Do not send it back to director unless the plan itself changes. |
|
||||||
|
| `RENDERING` | Flow set status immediately before renderer invocation. | Missing final plus FFmpeg exception means interrupted render; current flow does not catch all exceptions to mark `FAILED`. |
|
||||||
|
| `RENDERED` | Project-level final and manifest were written before status update. | Still probe both media and manifest; status alone is insufficient QA. |
|
||||||
|
| `FAILED` | Single-source flow explicitly writes this for an empty highlight list. | Read `failureMessage`; other exceptions may escape without this state. |
|
||||||
|
|
||||||
|
### No category or candidates
|
||||||
|
|
||||||
|
**Current behavior:** `HighlightSourceAnalyzer` writes FFprobe, scene, audio, visual, and source analysis. `HighlightSourceScheduler` then invokes `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Missing files on a newly created project are therefore a failed/incomplete candidate stage; older projects may predate this wiring.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'HighlightCandidateGenerator|category\.json|highlight-candidates\.json' src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java src/main/java/org/example/videoclips/editing/HighlightCandidateGenerator.java src/main/java/org/example/videoclips/editing/HighlightDirectorPromptGenerator.java
|
||||||
|
rg -n 'writeJson.*(category|highlight-candidates)' src/main/java/org/example/videoclips/editing
|
||||||
|
```
|
||||||
|
|
||||||
|
- If files are absent on a new project, inspect `highlight_candidates_started`, `highlight_candidate_scored`, and `highlight_candidates_completed`; do not continue to director or render.
|
||||||
|
- If files exist, parse and validate their project ID, category confidence, nonempty candidate list, bounds, and scores against `analysis/source-analysis.json`.
|
||||||
|
- Do not copy multi-clip root files (`category.json`, `highlight-candidates.json`) into the single-source `analysis/` tree as a workaround.
|
||||||
|
|
||||||
|
### Director plan, approval, and pending assets
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '1,260p' "$project/director/edit-plan.json"
|
||||||
|
rg 'highlight_flow_(waiting_for_approval|waiting_for_assets|started|completed)|highlight_render_scan' "$logs" | rg "$project_id"
|
||||||
|
find "$project" -path '*/requests/*.json' -type f -print -exec sed -n '1,180p' {} \;
|
||||||
|
```
|
||||||
|
|
||||||
|
- If plan is absent, remain `WAITING_FOR_DIRECTOR`; do not fabricate one during incident response.
|
||||||
|
- If `require-director-approval=true` and the configured approval file is absent, this is an intentional gate.
|
||||||
|
- If request JSON remains pending, inspect the matching target path and worker logs. Never satisfy it with placeholder media.
|
||||||
|
- If the plan has an empty `highlights` array, the flow marks `FAILED`.
|
||||||
|
- If JSON is malformed, deserialization throws. Semantic validation is performed by `HighlightDirectorPlanValidator`, which requires category/candidate artifacts, safe IDs, candidate-contained ranges, duration bounds, and the production creative directions. A rejection marks the project `FAILED`; read `failureMessage` before changing the plan.
|
||||||
|
- If `contentCategory` is missing or unknown, rendering silently maps it to `GENERIC_VLOG`; treat this as invalid production input, not successful classification.
|
||||||
|
|
||||||
|
**Costly trap:** history labeled commit `5d889b0` “working version but not cinematic.” A completed render proved mechanical execution, not creative correctness. Require candidate evidence, certified assets, media probes, and the QA rubric.
|
||||||
|
|
||||||
|
### Multi-clip manual plans are different
|
||||||
|
|
||||||
|
Use this branch only for `output/edit-projects/<project-id>/inbox/edit-plan.json`, not highlight projects.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
edit_project='output/edit-projects/<project-id>'
|
||||||
|
find "$edit_project" -maxdepth 2 -type f -print | sort
|
||||||
|
rg 'event=edit_plan_(inbox_detected|rejected|imported|saved)|event=edit_render_waiting' "$logs" | tail -n 100
|
||||||
|
```
|
||||||
|
|
||||||
|
- Invalid JSON or validation failure moves the inbox plan to `edit-plan.json.rejected`; read the associated rejection message.
|
||||||
|
- A valid import writes root `edit-plan.json`, renames the inbox file `.accepted`, and writes `PLANNED` before any approved automatic render.
|
||||||
|
- The validator checks project/style identity, source and timeline bounds, playback speed, transitions, audio cues, voiceover, overlays, target duration, and minimum cinematic richness. It also requires referenced SFX files to exist.
|
||||||
|
- If auto-render is disabled, `PLANNED` is expected. If approval is required, the configured file belongs in the project `inbox/` directory.
|
||||||
|
- Do not move `.rejected` back or edit it in place during triage. Correct the producer through change control and submit a new reviewed plan.
|
||||||
|
|
||||||
|
## Diagnose FFmpeg, filters, audio, and codecs
|
||||||
|
|
||||||
|
Probe the source and output before changing commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
media='<source-or-output-media>'
|
||||||
|
ffprobe -v error -show_entries format=duration,size:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$media"
|
||||||
|
ffmpeg -v error -i "$media" -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
- If FFprobe fails, isolate corrupt/unreadable media or an unsupported container before inspecting filter graphs.
|
||||||
|
- If FFprobe succeeds but decode-to-null fails, isolate codec/decoder or damaged packets.
|
||||||
|
- If decode succeeds and rendering fails, capture the exact logged FFmpeg command/output. Test that exact command only in an approved disposable workspace; do not simplify it until the failing filter is identified.
|
||||||
|
- If failure names `drawtext`, verify the selected font file and escaping. If it names `loudnorm`, `amix`, or `sidechaincompress`, inventory actual audio inputs and durations. If concat copy fails, compare codecs, time bases, resolution, and audio layouts across each per-highlight final.
|
||||||
|
- If per-highlight outputs exist but project `final.mp4` does not, inspect concat. The project concatenates with `-c copy`, so incompatible streams are a separate cause from segment rendering.
|
||||||
|
- A `qa-report.json` pass is not a creative certificate. Existing QA contains structural checks and warning-level probes; use `video-editing-validation-and-qa` for acceptance evidence.
|
||||||
|
|
||||||
|
## Diagnose folder scheduler quarantine
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find input/source input/working input/processed input/rejected -maxdepth 1 -type f -print 2>/dev/null | sort
|
||||||
|
find output/clips -maxdepth 2 -type f -print 2>/dev/null | sort
|
||||||
|
rg 'event=(candidate|validation|ffprobe|ffmpeg|processing|failed_source|failure_)' "$logs" | tail -n 200
|
||||||
|
```
|
||||||
|
|
||||||
|
- Source only: candidate may be ignored because it is hidden, unsupported, `.tmp`, `.part`, `.download`, or `.failed`, or scheduler/profile is disabled.
|
||||||
|
- Working then rejected: validation or clipping failed; use event ordering to distinguish them.
|
||||||
|
- Rejected collision: the scheduler refuses overwrite. If quarantine then also fails, it renames the working file to `.failed`; an existing marker causes `failure_marker_skipped` and leaves the source unchanged.
|
||||||
|
- Processed collision after clips were created: source is moved to rejected. Inventory the output directory before retrying to avoid duplicate artifacts.
|
||||||
|
- `ffmpeg_completed exit_code=0` with zero clips is still failure.
|
||||||
|
|
||||||
|
Do not delete markers or move quarantined files back during diagnosis. Route recovery through `video-editing-run-and-operate` and change control.
|
||||||
|
|
||||||
|
## Diagnose API queue, DLQ, storage, and JPA
|
||||||
|
|
||||||
|
First identify adapters from resolved configuration/logs. **Current defaults:** memory repository, memory storage, memory queue, and stub processing. `application-jpa.properties` opts into JPA and DB queue but uses H2 in PostgreSQL compatibility mode; it is not a production PostgreSQL profile.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '^video-clipping\.(repository|storage|queue|processing)=|^spring\.datasource|^spring\.jpa|^spring\.flyway' src/main/resources/application*.properties
|
||||||
|
rg 'DatabaseBackedClipJobQueueAdapter|InMemoryClipJobQueueAdapter|JpaVideoClippingRepository|S3ObjectStorageAdapter' "$logs"
|
||||||
|
```
|
||||||
|
|
||||||
|
For an approved read-only PostgreSQL session, run:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
select status, count(*) from queue_messages group by status order by status;
|
||||||
|
select id, job_id, status, attempt_count, available_at, claimed_at, last_error
|
||||||
|
from queue_messages where status in ('PENDING','PROCESSING','DLQ') order by available_at limit 50;
|
||||||
|
select id, asset_id, status, attempt_count, progress_percent, error_message
|
||||||
|
from clip_jobs where id = '<job-id>';
|
||||||
|
select event_type, message, created_at from job_events
|
||||||
|
where aggregate_id = '<job-id>' order by created_at;
|
||||||
|
```
|
||||||
|
|
||||||
|
- Rising `PENDING`: worker capacity/polling or a downstream bottleneck.
|
||||||
|
- Old `PROCESSING`: compare age with `visibility-timeout-ms` (current default 900000 ms); reclaim is expected only after expiry.
|
||||||
|
- Retry below max attempts: job should return to `QUEUED`, not terminal `FAILED`.
|
||||||
|
- `DLQ`: current code marks the job `FAILED` and records `DLQ` and `FAILED` events. Fix cause before any controlled redrive.
|
||||||
|
- Memory queue: state is process-local and disappears at restart; DB runbooks do not apply.
|
||||||
|
- Storage metadata without a retrievable object is failure. Verify the persisted `source_object_key`/clip `object_key` against the configured adapter using an approved provider procedure; do not infer an object key from an ID.
|
||||||
|
- Actuator custom health indicators currently report adapter identity/capability without exercising repository, queue, or storage. `UP` is not dependency reachability evidence.
|
||||||
|
|
||||||
|
**Historical traps:** commits `66e998e` and `7307082` fixed metadata-only clip artifacts and fabricated download paths; `1737d8b` fixed retryable DB failures being marked terminal on the first exception. Read `video-editing-failure-archaeology` before changing these contracts.
|
||||||
|
|
||||||
|
## Detect profile and documentation drift
|
||||||
|
|
||||||
|
Resolve behavior from code plus the active profile, never from a plan checkbox alone.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'spring\.profiles\.active|application-[^.]+\.(yml|properties)' pom.xml src/main/resources src/test/resources
|
||||||
|
rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml
|
||||||
|
rg -n '@ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
|
||||||
|
git log -1 --format='%h %cI %s' -- src/main/resources/application.yml docs
|
||||||
|
```
|
||||||
|
|
||||||
|
- Tests force the `test` profile via Surefire; test resources disable schedulers and worker auto-start. A passing test does not prove default application startup safety.
|
||||||
|
- Base `application.yml` currently enables the folder scheduler, editing, local CV auto-start, local asset auto-start, local director, and highlight ingestion; highlight rendering defaults off and director approval defaults on.
|
||||||
|
- `application-cinematic-editing-local.yml` changes multi-clip local-director behavior, not the entire single-source highlight contract.
|
||||||
|
- When docs and executable code differ, record the discrepancy and route doc correction to `video-editing-docs-and-writing`; do not silently choose the more convenient claim.
|
||||||
|
|
||||||
|
## Reproduce clean-checkout build failures without network
|
||||||
|
|
||||||
|
**Current as of 2026-07-21:** the working tree passes 245 tests in 62 test classes with `mvn -q -o verify`. An earlier archived clean tree failed `LocalAssetGenerationStageTest` because it depended on untracked local synthesis/host fallback; the test is now deterministic and fail-closed, but a new clean-archive run is still required. The repository has no Maven wrapper or CI configuration.
|
||||||
|
|
||||||
|
Use an already populated Maven cache and forbid network:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
git archive HEAD | tar -x -C "$tmp"
|
||||||
|
(cd "$tmp" && mvn -o -q verify)
|
||||||
|
git ls-files mvnw .mvn .github .gitlab-ci.yml README.md
|
||||||
|
git check-ignore -v .venv-local-asset .venv-local-cv target
|
||||||
|
```
|
||||||
|
|
||||||
|
- Workspace passes, archive fails: find untracked runtime/tool/asset assumptions; do not copy them into the archive as a fix.
|
||||||
|
- Both fail identically: diagnose the first failing test, not the final Maven summary.
|
||||||
|
- Offline dependency resolution fails before tests: report the cache limitation separately; it does not prove source failure.
|
||||||
|
- Archive passes: compare JDK, Maven, FFmpeg, locale, and environment variables with the failing environment.
|
||||||
|
- Route reproducibility changes to `video-editing-build-and-env` and change control.
|
||||||
|
|
||||||
|
## Do not use this skill when
|
||||||
|
|
||||||
|
- Designing a fix or changing defaults: use `video-editing-change-control` and `video-editing-architecture-contract`.
|
||||||
|
- Cataloging configuration: use `video-editing-config-and-flags`.
|
||||||
|
- Running an approved production operation, redrive, or render: use `video-editing-run-and-operate`.
|
||||||
|
- Certifying cinematic/audio quality: use `video-editing-validation-and-qa`.
|
||||||
|
- Advancing candidate identification and production local-model rendering: use `video-editing-cinematic-highlights-campaign`.
|
||||||
|
- Researching whether an old approach was rejected: use `video-editing-failure-archaeology`.
|
||||||
|
|
||||||
|
## Exit checklist
|
||||||
|
|
||||||
|
- [ ] Record commit, active profile, resolved adapters, tool versions, project/job ID, and timestamps.
|
||||||
|
- [ ] Identify the first missing or invalid artifact, not merely the last visible symptom.
|
||||||
|
- [ ] Correlate state, artifact, and log evidence; note contradictions explicitly.
|
||||||
|
- [ ] Run at least one experiment that separates the leading hypotheses.
|
||||||
|
- [ ] Reject downloads, external calls, unlicensed assets, placeholders, unapproved renders, and default changes.
|
||||||
|
- [ ] State current behavior separately from the production target.
|
||||||
|
- [ ] Route any mutation through change control with a test and rollback plan.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Re-verify state writers: `rg -n 'HighlightProjectStatus\.|markStatus|markFailed' src/main/java/org/example/videoclips/editing`
|
||||||
|
|
||||||
|
Re-verify scheduler/default guards: `rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml src/test/resources/application*.properties`
|
||||||
|
|
||||||
|
Re-verify prohibited bootstrap behavior: `rg -n 'pip install|YOLO\(|write_silence|write_fallback_tone|fallbackTone|writeSilence' tools src/main/java/org/example/videoclips/editing`
|
||||||
|
|
||||||
|
Re-verify single-source artifact producers: `rg -n 'writeJson.*(source-analysis|category|highlight-candidates)|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing`
|
||||||
|
|
||||||
|
Re-verify queue states and retry rules: `rg -n 'PENDING|PROCESSING|COMPLETED|DLQ|maxAttempts|visibilityTimeout|markRetry|markTerminal' src/main/java/org/example/videoclips/queue src/main/java/org/example/videoclips/processing`
|
||||||
|
|
||||||
|
Re-verify build/profile behavior: `rg -n 'spring.profiles.active|maven-surefire|jacoco|java.version|spring-boot-starter-parent' pom.xml && git ls-files mvnw .mvn .github .gitlab-ci.yml README.md`
|
||||||
|
|
||||||
|
Re-verify historical incidents: `git show --stat --oneline 1737d8b 7307082 66e998e bd620fc 5d889b0 97ba827`
|
||||||
|
|
@ -0,0 +1,283 @@
|
||||||
|
---
|
||||||
|
name: video-editing-diagnostics-and-tooling
|
||||||
|
description: "Load this skill when you need a read-only measurement tool: inventory edit/highlight project state, verify a render manifest or QA report against media, run FFprobe/FFmpeg probes, lexically audit runtime hazards/defaults, inspect existing logs/metrics/health captures, or rerun an existing benchmark harness offline. Do not load it for interactive symptom-to-branch triage or for causal proof after triage."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Diagnostics and Tooling
|
||||||
|
|
||||||
|
Use measurement before changing code. Treat every `passed` field, health response, log line, and benchmark document as a claim with a known evidence boundary.
|
||||||
|
|
||||||
|
Verified against the repository on **2026-07-21**.
|
||||||
|
|
||||||
|
## Do not use this skill when
|
||||||
|
|
||||||
|
| Need | Load this sibling instead |
|
||||||
|
| --- | --- |
|
||||||
|
| Classify, approve, implement, or promote a behavior change | `video-editing-change-control` |
|
||||||
|
| Start schedulers, workers, or the service; deploy or recover it | `video-editing-run-and-operate` |
|
||||||
|
| Look up a configuration property or add a flag | `video-editing-config-and-flags` |
|
||||||
|
| Choose acceptance thresholds or certify a render/release | `video-editing-validation-and-qa` |
|
||||||
|
| Triage a known symptom interactively | `video-editing-debugging-playbook` |
|
||||||
|
| Explain a settled failure or rejected fix | `video-editing-failure-archaeology` |
|
||||||
|
| Prove a new algorithm or quality mechanism | `video-editing-proof-and-analysis-toolkit` |
|
||||||
|
|
||||||
|
Do not use a diagnostic result to bypass change control. Diagnostics may falsify a claim; they do not authorize a render, default change, dependency installation, model acquisition, or production promotion.
|
||||||
|
|
||||||
|
## Diagnostic safety boundary
|
||||||
|
|
||||||
|
The canonical prohibitions are in `video-editing-change-control`; the configuration catalog is `video-editing-config-and-flags`, and acceptance evidence is defined by `video-editing-validation-and-qa`. Never perform these actions from a diagnostic session:
|
||||||
|
|
||||||
|
- Download a dependency or model automatically.
|
||||||
|
- Call an external AI service or use network transport for model/media work, including loopback.
|
||||||
|
- use an asset without recorded license/provenance evidence.
|
||||||
|
- Accept silence, a generated tone, or an OS speech fallback as production voiceover/music/SFX.
|
||||||
|
- Trigger rendering without recorded approval.
|
||||||
|
- change a production-facing default.
|
||||||
|
- Start the application merely to inspect project files. Current defaults enable the folder scheduler, editing, local director, highlight ingestion, and two local-worker auto-start paths; highlight rendering itself defaults off.
|
||||||
|
|
||||||
|
The local worker shell scripts are **not safe diagnostic commands**. `tools/run_local_asset_worker.sh` and `tools/run_local_cv_worker.sh` run `pip install` in their default `auto` bootstrap mode. The CV script also constructs an Ultralytics `YOLO` model from the default `yolov8n.pt`, which may initiate model acquisition when the file is absent. The asset runtime verifier no longer calls its bootstrap script and strict readiness now fails startup when resident model paths are missing.
|
||||||
|
|
||||||
|
## Start here
|
||||||
|
|
||||||
|
Run these from the repository root. They are offline and read-only.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py --repo . --fail-on never
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py --repo . --fail-on never
|
||||||
|
```
|
||||||
|
|
||||||
|
Then verify a specific render:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py \
|
||||||
|
--repo . \
|
||||||
|
--project-dir output/edit-projects/PROJECT_ID
|
||||||
|
```
|
||||||
|
|
||||||
|
For a per-highlight render, point `--project-dir` to its directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py \
|
||||||
|
--repo . \
|
||||||
|
--project-dir output/highlight-projects/PROJECT_ID/highlights/HIGHLIGHT_ID \
|
||||||
|
--deep
|
||||||
|
```
|
||||||
|
|
||||||
|
`--deep` reads the entire media file three times through FFmpeg. Run it on a production copy or isolated worker, not on a latency-sensitive service volume.
|
||||||
|
|
||||||
|
## Shipped tools
|
||||||
|
|
||||||
|
| Tool | Question answered | Dependencies | Exit codes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `scripts/inventory_projects.py` | Which edit/highlight projects are partial, inconsistent, stale, or backed only by asserted QA? | Python 3 standard library | `0` no finding at selected threshold; `2` findings; `3` invocation/read error |
|
||||||
|
| `scripts/verify_render.py` | Does a manifest agree with the real output duration and streams? Were QA checks measured or asserted? | Python 3, `ffprobe`; `ffmpeg` with `--deep` | `0` no error finding; `2` verification error; `3` tool/input error |
|
||||||
|
| `scripts/audit_runtime_safety.py` | Where do forbidden downloads, placeholders, auto-start, auto-render, approval bypass, or asset-license gaps remain? | Python 3 standard library | `0` no selected-severity finding; `2` finding; `3` invocation/read error |
|
||||||
|
|
||||||
|
All output ordering is deterministic. JSON mode is suitable for later CI integration:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py \
|
||||||
|
--repo . --stale-hours 24 --format json --fail-on warning
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py \
|
||||||
|
--repo . --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
`--stale-hours` compares file modification times with the invocation time. Omit it when reproducibility across dates matters. An asset request count means request JSON files exist; it does not prove that every request remains unresolved.
|
||||||
|
|
||||||
|
## Interpret project state
|
||||||
|
|
||||||
|
The edit project root defaults to `output/edit-projects`; the highlight project root defaults to `output/highlight-projects`. Override either root explicitly when diagnosing a mounted production copy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py \
|
||||||
|
--repo . \
|
||||||
|
--edit-root /mounted-copy/edit-projects \
|
||||||
|
--highlight-root /mounted-copy/highlight-projects \
|
||||||
|
--format table --fail-on error
|
||||||
|
```
|
||||||
|
|
||||||
|
The tool only treats a child as a project when `project.json` exists. It ignores caches such as `_voiceover-cache`.
|
||||||
|
|
||||||
|
| Observation | Meaning | Next discriminating check |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `RENDERED_OUTPUT_MISSING` | Persisted state claims completion, but `final.mp4` is absent. | Inspect storage/copy logs; do not repair state by hand. |
|
||||||
|
| `OUTPUT_STATUS_MISMATCH` | A final file exists while project state is nonterminal/failed. | Verify the file and manifest, then inspect the state-transition failure. |
|
||||||
|
| `MANIFEST_OUTPUT_MISSING` | Provenance exists for an absent local output. | Resolve whether cleanup, relocation, or failed publication occurred. |
|
||||||
|
| `DIRECTOR_PLAN_MISSING` | A planned/rendering/rendered highlight has no director plan at the folder-contract path. | Check move/copy history and `highlight_render_scan_*` events. |
|
||||||
|
| `HIGHLIGHT_QA_ASSERTED` | A highlight QA report exists but did not run FFmpeg media probes. | Run `verify_render.py --deep`; do not certify from the report. |
|
||||||
|
| `NONTERMINAL_STALE` | No project file changed inside the chosen time window. | Identify the expected scanner/worker and correlate its last event. |
|
||||||
|
| `RENDERED_QA_MISSING` | A render has no persisted QA report. | Verify it independently; absence is not a pass or failure by itself. |
|
||||||
|
|
||||||
|
Statuses are `CREATED`, `ANALYZING`, `ANALYZED` (edit only), `WAITING_FOR_DIRECTOR`, `PLANNING` (edit only), `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`. Only `RENDERED` and `FAILED` are terminal for stale detection.
|
||||||
|
|
||||||
|
## Separate measured QA from asserted QA
|
||||||
|
|
||||||
|
This distinction is load-bearing.
|
||||||
|
|
||||||
|
| Renderer | Current report behavior | Evidence actually present |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `FfmpegEditRenderer` | Runs `blackdetect`, `silencedetect`, and `astats`; also checks files, timeline arithmetic, assets, overlays, command recording, and configured mastering filters. | Media probes plus structural assertions. A warning does not make overall `passed` false; only failed `ERROR` checks do. |
|
||||||
|
| `HighlightFfmpegRenderer` | Marks duration, assets, overlays, and mastering checks true from plan/file assumptions; runs no black/silence/peak probe in `buildQaReport`. | Assertions only. Its `passed: true` is not measured cinematic, visual, or audio quality. |
|
||||||
|
|
||||||
|
The older edit duration check compares the edit plan to the duration value passed into report construction. It does **not** probe the rendered container duration. Always compare `render-manifest.json.durationSeconds` with FFprobe independently.
|
||||||
|
|
||||||
|
`verify_render.py` checks:
|
||||||
|
|
||||||
|
- manifest `outputPath` resolution;
|
||||||
|
- FFprobe readability;
|
||||||
|
- manifest duration versus container duration, default tolerance `0.10s`;
|
||||||
|
- presence of video and audio streams;
|
||||||
|
- codec, dimensions, pixel format, frame-rate expression, sample rate, channels, and file size;
|
||||||
|
- declared QA failure and measured-versus-asserted classification;
|
||||||
|
- optionally, black ranges, silence ranges, and audio peaks using the repository's current thresholds.
|
||||||
|
|
||||||
|
It does **not** prove that highlights are correctly selected, pacing is cinematic, speech is intelligible, music/SFX are appropriate or licensed, overlays are readable, colors are intentional, or source-to-output continuity is correct. Load `video-editing-validation-and-qa` for certification.
|
||||||
|
|
||||||
|
## Direct media measurements
|
||||||
|
|
||||||
|
Use an explicit input path. These commands read media and write only standard output/error.
|
||||||
|
|
||||||
|
### Container and streams
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffprobe -v error -show_streams -show_format -of json -- path/to/final.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
Interpretation checklist:
|
||||||
|
|
||||||
|
- Confirm exactly the stream topology required by the acceptance contract; this project normally emits H.264 video and optional/expected AAC audio depending on the flow.
|
||||||
|
- Compare `format.duration` with the manifest and planned timeline. Container duration, stream duration, and planned duration can differ.
|
||||||
|
- Confirm dimensions, pixel format, average frame rate, audio sample rate, channels, and codec rather than inferring them from a filename/profile string.
|
||||||
|
- Treat successful decode metadata as structural evidence, not creative-quality evidence.
|
||||||
|
|
||||||
|
### Black, silence, clipping, freeze, and loudness
|
||||||
|
|
||||||
|
Repository-equivalent probes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffmpeg -hide_banner -v info -i path/to/final.mp4 -vf 'blackdetect=d=0.5:pic_th=0.98' -an -f null -
|
||||||
|
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'silencedetect=noise=-45dB:d=2' -vn -f null -
|
||||||
|
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'astats=metadata=1:reset=1' -vn -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
Additional investigative probes, not current application gates:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffmpeg -hide_banner -v info -i path/to/final.mp4 -vf 'freezedetect=n=-60dB:d=2' -an -f null -
|
||||||
|
ffmpeg -hide_banner -v info -i path/to/final.mp4 -af 'ebur128=peak=true' -vn -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
Interpret the event timestamps, not just whether a word appears. Intentional fades can trigger black detection; pauses can trigger silence detection; a static composition can trigger freeze detection. Compare each interval with the edit plan and source. `astats` peaks at or above `-0.1 dBFS` fail the older renderer's audio-peak predicate. The configured mastering targets are integrated loudness `-16 LUFS`, true peak `-1.5 dBTP`, and loudness range `11 LU`; the renderer invokes `loudnorm`, but only measurement can show the result.
|
||||||
|
|
||||||
|
## Logs: event IDs and correlation
|
||||||
|
|
||||||
|
Application logs are SLF4J messages containing stable-looking `event=<id>` key/value tokens. They are not configured as JSON in the repository. Do not call them machine-readable structured logs until a formatter/parser contract is implemented and tested.
|
||||||
|
|
||||||
|
Search source-defined IDs without guessing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -o 'event=[a-zA-Z0-9_{}-]+' src/main/java | sed 's/.*event=//' | sort -u
|
||||||
|
```
|
||||||
|
|
||||||
|
| Investigation | Start events | Completion/failure events | Correlation fields present in relevant paths |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Edit analysis | `edit_analysis_started` | `edit_analysis_completed` | `project_id`, elapsed/count fields |
|
||||||
|
| Edit render | `edit_render_started` | `edit_render_completed`, `edit_render_failed` | `project_id`, elapsed, output, duration, size/error |
|
||||||
|
| Highlight ingest/analysis | `highlight_scan_started`, `highlight_analysis_started` | `highlight_scan_completed`, `highlight_processing_failed`, `highlight_analysis_completed` | `scan_id`, `project_id`, source/candidate fields vary by event |
|
||||||
|
| Highlight render flow | `highlight_flow_started`, `highlight_render_started` | `highlight_flow_completed`, `highlight_render_completed` | `flow_id`, `scan_id`, `project_id`, `highlight_id` across relevant events |
|
||||||
|
| Local CV | `local_cv_request_started`, `local_cv_worker_starting` | `local_cv_visual_analysis_completed`, `local_cv_visual_analysis_failed`, worker lifecycle events | clip/endpoint/status/elapsed fields vary |
|
||||||
|
| Local assets | `local_asset_runtime_check_started`, `local_asset_generation_started` | `local_asset_generation_completed`, `local_asset_generation_failed` | project/type/model/target/strategy fields vary |
|
||||||
|
|
||||||
|
Discriminating searches:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg 'event=highlight_(flow|render|asset)|event=local_asset_' path/to/application.log
|
||||||
|
rg 'project_id=PROJECT_ID|flow_id=PROJECT_ID:' path/to/application.log
|
||||||
|
rg 'fallback|degraded|pending|failed|silence|ffmpeg-sine' path/to/application.log
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not assume trace correlation exists. The build has no Micrometer Tracing/OpenTelemetry/Brave dependency and source has no tracer instrumentation as of the verification date. Documentation says `traceId` “if available”; that is not implementation evidence.
|
||||||
|
|
||||||
|
Never put media paths containing sensitive names, prompt bodies, transcript/voiceover text, tokens, tenant/user IDs, or exception payloads into newly added broad logs. Existing logs include paths and failure messages; assess redaction before production use.
|
||||||
|
|
||||||
|
## Metrics, dashboards, and alerts
|
||||||
|
|
||||||
|
Actuator exposes `health`, `info`, `metrics`, and `prometheus`. Prometheus metric names translate dots to underscores.
|
||||||
|
|
||||||
|
| Metric family in code | Type | Interpretation |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `video.clipping.queue.pending`, `.processing`, `.dlq`, `.oldest.pending.age.seconds` | DB-queue-only gauges | Backlog state; absent when queue mode is not `db`. |
|
||||||
|
| `video.clipping.edit.projects.created` | Counter | Process-local project creations since restart. |
|
||||||
|
| `video.clipping.edit.analysis.duration` | Timer | Analysis attempts that reached `analysisCompleted`; failures before it may be absent. |
|
||||||
|
| `video.clipping.edit.analysis.clip.count`, `.error.count` | Distribution summaries | Per-completed-analysis counts, not current gauges. |
|
||||||
|
| `video.clipping.edit.render.duration` | Timer | Both completed and failed renders record duration. |
|
||||||
|
| `video.clipping.edit.render.failures` | Counter | Render exceptions observed by instrumented renderers. |
|
||||||
|
| `video.clipping.edit.render.output.duration.seconds`, `.size.bytes` | Distribution summaries | Completed render output claims passed to observability, not FFprobe measurements. |
|
||||||
|
| `video.clipping.edit.director.prompt.tokens`, `.asset.generation.cost.usd` | Distribution summaries | Estimates, not tokenizer/provider invoices. |
|
||||||
|
|
||||||
|
The checked-in Grafana dashboard currently covers DB queue depth/age/DLQ/processing, API request rate and p95, CPU, and heap. It does not visualize edit/highlight render metrics or creative/business quality. The checked-in alert document covers queue age, DLQ, API 5xx, target down, and process CPU. It explicitly records missing job-failure, FFmpeg-exit, storage-failure, signed-URL, and disk metrics.
|
||||||
|
|
||||||
|
Do not invent a render success ratio: there is no render-success counter. A candidate alert such as `increase(video_clipping_edit_render_failures_total[15m]) > 0` can detect recorded exceptions, but it misses silent audio/preview fallbacks and needs change-control approval, deployment labels, threshold evidence, and a linked runbook before adoption.
|
||||||
|
|
||||||
|
### Cardinality rules
|
||||||
|
|
||||||
|
- Never tag metrics with `project_id`, `highlight_id`, `flow_id`, `scan_id`, file path, prompt, model path, tenant/user ID, request ID, trace ID, or raw exception message.
|
||||||
|
- Put per-project correlation in logs/traces; use metrics only for bounded dimensions such as workflow, outcome, stage, or a reviewed error class.
|
||||||
|
- Enumerate every allowed tag value in the design review. Reject free-form strings.
|
||||||
|
- Verify emitted series count and actual Prometheus names in a production-like scrape before adding panels or alerts.
|
||||||
|
- Do not expose health/details or Prometheus publicly. Exposure in `application.properties` is endpoint availability, not proof of authorization or network policy.
|
||||||
|
|
||||||
|
## Health is not readiness proof
|
||||||
|
|
||||||
|
`RepositoryHealthIndicator`, `QueueHealthIndicator`, and `ObjectStorageHealthIndicator` always return `UP` with adapter/capability details. They do not execute a repository query, publish/claim a message, access object storage, invoke FFmpeg/FFprobe, verify disk capacity, or check local models. Consequently:
|
||||||
|
|
||||||
|
- `UP` proves only that Spring constructed those adapter beans and called the indicator.
|
||||||
|
- The DB-queue gauges query repository state, but their presence is not an end-to-end queue test.
|
||||||
|
- Local asset/CV worker `/health` handling is separate from Actuator health.
|
||||||
|
- No repository evidence establishes distinct liveness/readiness groups or an authenticated actuator policy.
|
||||||
|
|
||||||
|
Treat a health response as one signal. Correlate it with queue age, real project progress, dependency metrics, logs, and an approved synthetic transaction.
|
||||||
|
|
||||||
|
## Benchmark harnesses
|
||||||
|
|
||||||
|
These Maven tests are reproducible local planning harnesses. They write reports under `target/benchmarks`; they are not read-only and they are not production load tests.
|
||||||
|
|
||||||
|
| Command | What it measures | What it does not prove |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `mvn -q -o -Dtest=FfmpegPresetBenchmarkHarness test` | Real FFmpeg preset wall time/output bytes on generated 16s 720p input | Production throughput or highlight quality |
|
||||||
|
| `mvn -q -o -Dtest=WorkerBenchmarkHarness test` | Synchronous stub worker CPU/wall/local/uploaded bytes for modeled 600s input | Real FFmpeg or object-store behavior |
|
||||||
|
| `mvn -q -o -Dtest=ObjectStorageBandwidthBenchmarkHarness test` | Local copy and throttled transfer throughput | Live S3/network performance |
|
||||||
|
| `mvn -q -o -Dtest=CostModelBenchmarkHarness test` | Formula output using checked-in assumptions | Current vendor pricing or invoice forecast |
|
||||||
|
|
||||||
|
Run a harness only after confirming it cannot trigger forbidden model/dependency acquisition in the prepared environment. Record hardware, OS, Java, Maven, FFmpeg, source commit, warmup, sample count, input, and raw report. Never compare current output to checked-in numbers without normalizing those variables. No checked-in load generator or executed production-like end-to-end load result exists as of 2026-07-21.
|
||||||
|
|
||||||
|
## Evidence checklist for a diagnostic report
|
||||||
|
|
||||||
|
- [ ] Record repository commit and dirty-worktree state without mutating Git.
|
||||||
|
- [ ] Record OS/architecture, Java, Maven, FFmpeg, and FFprobe versions.
|
||||||
|
- [ ] State whether files are originals or read-only copies and identify the project/highlight IDs.
|
||||||
|
- [ ] Attach inventory JSON and render-verifier JSON.
|
||||||
|
- [ ] Distinguish an asserted application field from an independently measured value.
|
||||||
|
- [ ] State thresholds before running the measurement.
|
||||||
|
- [ ] Preserve negative observations and failed commands.
|
||||||
|
- [ ] Redact secrets, personal data, transcripts, voiceover text, signed URLs, and sensitive paths.
|
||||||
|
- [ ] Route every proposed behavior/config/instrumentation change through `video-editing-change-control`.
|
||||||
|
- [ ] Route certification through `video-editing-validation-and-qa`; never certify by eye or from `qa-report.json` alone.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Re-verify volatile facts from the repository root; none of these commands starts the application or accesses the network:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:' src/main/resources/application*.yml
|
||||||
|
rg -n 'pip install|YOLO\(' tools src/main/java
|
||||||
|
rg -n 'fallbackTone|writeSilence|anullsrc|ffmpeg-sine|strategy=silence' src/main/java tools
|
||||||
|
rg -n 'RenderQaCheck|blackdetect|silencedetect|astats' src/main/java/org/example/videoclips/editing/{FfmpegEditRenderer,HighlightFfmpegRenderer}.java
|
||||||
|
rg -n 'registry\.(counter|timer|summary)|Gauge.builder' src/main/java
|
||||||
|
rg -n 'micrometer-tracing|opentelemetry|zipkin|brave|ObservationRegistry|Tracer' pom.xml src/main || true
|
||||||
|
rg -n 'management.endpoints.web.exposure.include' src/main/resources
|
||||||
|
jq -r '.panels[]? | [.title,.type] | @tsv' dashboards/video-clipping-overview-grafana.json
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/inventory_projects.py --help
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/verify_render.py --help
|
||||||
|
python3 .claude/skills/video-editing-diagnostics-and-tooling/scripts/audit_runtime_safety.py --help
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,93 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Lexically audit known runtime safety hazards without executing project code."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
RULES = (
|
||||||
|
("BLOCK", "AUTO_DEPENDENCY_INSTALL", re.compile(r"\bpip\s+install\b"), ("tools/*.sh",)),
|
||||||
|
("BLOCK", "REMOTE_MODEL_RESOLUTION", re.compile(r"\bget_pretrained\s*\("), ("tools/*.py",)),
|
||||||
|
("BLOCK", "NAMED_YOLO_ACQUISITION", re.compile(r"(?:LOCAL_CV_YOLO_MODEL.*:-|os\.getenv\([^,]+,)\s*[\"']yolo[^\"']*\.pt|YOLO\([^\n]*LOCAL_CV_YOLO_MODEL"), ("tools/*.sh", "tools/*.py")),
|
||||||
|
("WARN", "YOLO_LOAD_REQUIRES_REVIEW", re.compile(r"\bYOLO\s*\("), ("tools/*.py",)),
|
||||||
|
("BLOCK", "PLACEHOLDER_TONE", re.compile(r"fallbackTone|fallback-tone|ffmpeg-sine"), ("src/main/java/**/*.java",)),
|
||||||
|
("BLOCK", "PLACEHOLDER_TONE", re.compile(r"write_fallback_tone\s*\("), ("tools/*.py",)),
|
||||||
|
("BLOCK", "PLACEHOLDER_SILENCE", re.compile(r"writeSilence|strategy=silence|anullsrc="), ("src/main/java/**/*.java",)),
|
||||||
|
("BLOCK", "PLACEHOLDER_SILENCE", re.compile(r"write_silence\s*\("), ("tools/*.py",)),
|
||||||
|
("BLOCK", "AUTO_START_DEFAULT_TRUE", re.compile(r"auto-start:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
|
||||||
|
("BLOCK", "RENDER_DEFAULT_TRUE", re.compile(r"render-enabled:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
|
||||||
|
("BLOCK", "APPROVAL_DEFAULT_FALSE", re.compile(r"require-director-approval:\s*\$\{[^}:]+:false\}"), ("src/main/resources/application*.yml",)),
|
||||||
|
("BLOCK", "HEURISTIC_FALLBACK_DEFAULT_TRUE", re.compile(r"fallback-to-heuristic:\s*\$\{[^}:]+:true\}"), ("src/main/resources/application*.yml",)),
|
||||||
|
("WARN", "CONFIGURED_HTTP_NON_LOOPBACK", re.compile(r"https?://(?!127\.0\.0\.1|localhost)", re.IGNORECASE), ("src/main/resources/application*",)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Read-only lexical audit for runtime hazards; findings require human classification and a clean result is not compliance proof."
|
||||||
|
)
|
||||||
|
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root (default: current directory).")
|
||||||
|
parser.add_argument("--format", choices=("text", "json"), default="text")
|
||||||
|
parser.add_argument(
|
||||||
|
"--fail-on", choices=("block", "warn", "never"), default="block",
|
||||||
|
help="Exit 2 on BLOCK findings, on BLOCK or WARN findings, or never because of findings (default: block).",
|
||||||
|
)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def paths_for(repo: Path, globs: tuple[str, ...]) -> list[Path]:
|
||||||
|
found: set[Path] = set()
|
||||||
|
for pattern in globs:
|
||||||
|
found.update(path for path in repo.glob(pattern) if path.is_file())
|
||||||
|
return sorted(found, key=lambda path: str(path.relative_to(repo)))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
opts = parse_args()
|
||||||
|
repo = opts.repo.resolve()
|
||||||
|
if not (repo / "pom.xml").is_file():
|
||||||
|
print(f"error: not a recognized repository root (pom.xml missing): {repo}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
findings: list[dict] = []
|
||||||
|
try:
|
||||||
|
for severity, code, pattern, globs in RULES:
|
||||||
|
for path in paths_for(repo, globs):
|
||||||
|
for number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
|
||||||
|
if pattern.search(line):
|
||||||
|
findings.append({"severity": severity, "code": code,
|
||||||
|
"path": str(path.relative_to(repo)), "line": number,
|
||||||
|
"evidence": line.strip()[:240]})
|
||||||
|
asset_root = repo / "input" / "highlights" / "assets"
|
||||||
|
if asset_root.is_dir():
|
||||||
|
license_markers = list(asset_root.glob("LICENSE*")) + list(asset_root.glob("**/*.license*"))
|
||||||
|
media = [p for p in asset_root.rglob("*") if p.is_file() and p.suffix.lower() in {".wav", ".mp3", ".flac", ".aif", ".aiff", ".ttf", ".otf", ".cube"}]
|
||||||
|
if media and not license_markers:
|
||||||
|
findings.append({"severity": "WARN", "code": "ASSET_LICENSE_MARKER_MISSING",
|
||||||
|
"path": str(asset_root.relative_to(repo)), "line": 0,
|
||||||
|
"evidence": f"{len(media)} media/font/LUT files and no LICENSE* or *.license* marker; directory membership and marker files are not license proof"})
|
||||||
|
except OSError as exc:
|
||||||
|
print(f"error:goal f {exc}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
findings.sort(key=lambda item: (item["path"], item["line"], item["code"]))
|
||||||
|
if opts.format == "json":
|
||||||
|
print(json.dumps({"repo": str(repo), "findings": findings}, indent=2, sort_keys=True))
|
||||||
|
else:
|
||||||
|
for item in findings:
|
||||||
|
location = f"{item['path']}:{item['line']}" if item["line"] else item["path"]
|
||||||
|
print(f"{item['severity']} {item['code']} {location} {item['evidence']}")
|
||||||
|
print(f"SUMMARY block={sum(f['severity'] == 'BLOCK' for f in findings)} warn={sum(f['severity'] == 'WARN' for f in findings)}")
|
||||||
|
severities = {item["severity"] for item in findings}
|
||||||
|
if opts.fail_on == "block" and "BLOCK" in severities:
|
||||||
|
return 2
|
||||||
|
if opts.fail_on == "warn" and severities & {"BLOCK", "WARN"}:
|
||||||
|
return 2
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -0,0 +1,207 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Inventory edit/highlight project state without modifying it."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
TERMINAL = {"RENDERED", "FAILED"}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Inventory partial, inconsistent, and optionally stale edit/highlight projects."
|
||||||
|
)
|
||||||
|
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root (default: current directory).")
|
||||||
|
parser.add_argument("--edit-root", type=Path, help="Override edit-project root.")
|
||||||
|
parser.add_argument("--highlight-root", type=Path, help="Override highlight-project root.")
|
||||||
|
parser.add_argument(
|
||||||
|
"--allow-missing-roots", action="store_true",
|
||||||
|
help="Do not report missing edit/highlight roots as errors (use only when the absent workflow is intentional).",
|
||||||
|
)
|
||||||
|
parser.add_argument("--stale-hours", type=float, help="Flag nonterminal projects not modified within this many hours.")
|
||||||
|
parser.add_argument("--format", choices=("table", "json"), default="table")
|
||||||
|
parser.add_argument(
|
||||||
|
"--fail-on", choices=("error", "warning", "never"), default="error",
|
||||||
|
help="Return 2 at or above this finding severity (default: error).",
|
||||||
|
)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: Path) -> tuple[dict | None, str | None]:
|
||||||
|
try:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
return None, "top-level JSON value is not an object"
|
||||||
|
return value, None
|
||||||
|
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
|
||||||
|
return None, str(exc)
|
||||||
|
|
||||||
|
|
||||||
|
def qa_mode(path: Path) -> str:
|
||||||
|
if not path.is_file():
|
||||||
|
return "missing"
|
||||||
|
value, error = load_json(path)
|
||||||
|
if error or value is None:
|
||||||
|
return "invalid"
|
||||||
|
names = {str(check.get("name")) for check in value.get("checks", []) if isinstance(check, dict)}
|
||||||
|
measured = {"black_frames", "long_silence", "audio_clipping"}
|
||||||
|
return "measured" if measured <= names else "partial"
|
||||||
|
|
||||||
|
|
||||||
|
def finding(severity: str, code: str, detail: str) -> dict:
|
||||||
|
return {"severity": severity, "code": code, "detail": detail}
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_project(kind: str, directory: Path, now: float, stale_hours: float | None) -> dict:
|
||||||
|
project_file = directory / "project.json"
|
||||||
|
project, error = load_json(project_file)
|
||||||
|
findings: list[dict] = []
|
||||||
|
if error:
|
||||||
|
findings.append(finding("error", "PROJECT_JSON_INVALID", error))
|
||||||
|
project = {}
|
||||||
|
status = str(project.get("status", "UNKNOWN"))
|
||||||
|
final = directory / "final.mp4"
|
||||||
|
render_manifest = directory / "render-manifest.json"
|
||||||
|
qa_report = directory / "qa-report.json"
|
||||||
|
if status == "RENDERED":
|
||||||
|
if not final.is_file():
|
||||||
|
findings.append(finding("error", "RENDERED_OUTPUT_MISSING", "status is RENDERED but final.mp4 is absent"))
|
||||||
|
if not render_manifest.is_file():
|
||||||
|
findings.append(finding("error", "RENDERED_MANIFEST_MISSING", "status is RENDERED but render-manifest.json is absent"))
|
||||||
|
if kind != "highlight" and not qa_report.is_file():
|
||||||
|
findings.append(finding("warning", "RENDERED_QA_MISSING", "status is RENDERED but qa-report.json is absent"))
|
||||||
|
elif final.is_file():
|
||||||
|
findings.append(finding("error", "OUTPUT_STATUS_MISMATCH", f"final.mp4 exists while status is {status}"))
|
||||||
|
if render_manifest.is_file() and not final.is_file():
|
||||||
|
findings.append(finding("error", "MANIFEST_OUTPUT_MISSING", "render-manifest.json exists but final.mp4 is absent"))
|
||||||
|
if status == "FAILED" and not (project.get("failureReason") or project.get("failureMessage")):
|
||||||
|
findings.append(finding("warning", "FAILURE_REASON_MISSING", "FAILED project has no recorded failure reason"))
|
||||||
|
if stale_hours is not None and status not in TERMINAL:
|
||||||
|
try:
|
||||||
|
newest = max((path.stat().st_mtime for path in directory.rglob("*") if path.is_file()), default=directory.stat().st_mtime)
|
||||||
|
age_hours = (now - newest) / 3600.0
|
||||||
|
if age_hours >= stale_hours:
|
||||||
|
findings.append(finding("warning", "NONTERMINAL_STALE", f"newest artifact is {age_hours:.1f} hours old"))
|
||||||
|
except OSError as exc:
|
||||||
|
findings.append(finding("error", "STAT_FAILED", str(exc)))
|
||||||
|
|
||||||
|
pending_assets = 0
|
||||||
|
highlight_count = 0
|
||||||
|
qa_modes: list[str] = []
|
||||||
|
if kind == "highlight":
|
||||||
|
director_plan = directory / "director" / "edit-plan.json"
|
||||||
|
if status in {"PLANNED", "RENDERING", "RENDERED"} and not director_plan.is_file():
|
||||||
|
findings.append(finding("error", "DIRECTOR_PLAN_MISSING", f"status is {status} but director/edit-plan.json is absent"))
|
||||||
|
highlights = directory / "highlights"
|
||||||
|
if highlights.is_dir():
|
||||||
|
for child in sorted((p for p in highlights.iterdir() if p.is_dir()), key=lambda p: p.name):
|
||||||
|
highlight_count += 1
|
||||||
|
mode = qa_mode(child / "qa-report.json")
|
||||||
|
if mode != "missing":
|
||||||
|
qa_modes.append(mode)
|
||||||
|
requests = child / "assets" / "requests"
|
||||||
|
if requests.is_dir():
|
||||||
|
pending_assets += sum(1 for p in requests.glob("*.json") if p.is_file())
|
||||||
|
if status == "RENDERED" and not qa_report.is_file():
|
||||||
|
if qa_modes:
|
||||||
|
findings.append(finding(
|
||||||
|
"warning", "HIGHLIGHT_AGGREGATE_QA_MISSING",
|
||||||
|
"project-root aggregate qa-report.json is absent; per-highlight QA exists but does not certify the concatenated final.mp4",
|
||||||
|
))
|
||||||
|
else:
|
||||||
|
findings.append(finding(
|
||||||
|
"warning", "RENDERED_QA_MISSING",
|
||||||
|
"no project-root aggregate or per-highlight qa-report.json was found",
|
||||||
|
))
|
||||||
|
if "partial" in qa_modes:
|
||||||
|
findings.append(finding("warning", "HIGHLIGHT_QA_PARTIAL", "per-highlight QA lacks one or more expected FFmpeg media probes"))
|
||||||
|
if "invalid" in qa_modes:
|
||||||
|
findings.append(finding("error", "HIGHLIGHT_QA_INVALID", "one or more per-highlight QA reports are invalid"))
|
||||||
|
else:
|
||||||
|
mode = qa_mode(qa_report)
|
||||||
|
if mode != "missing":
|
||||||
|
qa_modes.append(mode)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"kind": kind,
|
||||||
|
"projectId": str(project.get("id") or directory.name),
|
||||||
|
"status": status,
|
||||||
|
"path": str(directory),
|
||||||
|
"finalExists": final.is_file(),
|
||||||
|
"manifestExists": render_manifest.is_file(),
|
||||||
|
"qaModes": sorted(set(qa_modes)),
|
||||||
|
"highlightCount": highlight_count,
|
||||||
|
"assetRequestCount": pending_assets,
|
||||||
|
"findings": findings,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def scan_root(kind: str, root: Path, now: float, stale_hours: float | None) -> list[dict]:
|
||||||
|
if not root.exists():
|
||||||
|
return []
|
||||||
|
if not root.is_dir():
|
||||||
|
raise ValueError(f"{kind} root is not a directory: {root}")
|
||||||
|
projects = []
|
||||||
|
for directory in sorted((p for p in root.iterdir() if p.is_dir()), key=lambda p: p.name):
|
||||||
|
if (directory / "project.json").is_file():
|
||||||
|
projects.append(inspect_project(kind, directory, now, stale_hours))
|
||||||
|
return projects
|
||||||
|
|
||||||
|
|
||||||
|
def print_table(projects: list[dict]) -> None:
|
||||||
|
print("KIND\tPROJECT\tSTATUS\tFINAL\tMANIFEST\tQA\tASSET_REQUESTS\tFINDINGS")
|
||||||
|
for item in projects:
|
||||||
|
codes = ",".join(f["severity"][0].upper() + ":" + f["code"] for f in item["findings"]) or "-"
|
||||||
|
print("\t".join((item["kind"], item["projectId"], item["status"], str(item["finalExists"]).lower(),
|
||||||
|
str(item["manifestExists"]).lower(), ",".join(item["qaModes"]) or "-",
|
||||||
|
str(item["assetRequestCount"]), codes)))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parse_args()
|
||||||
|
if args.stale_hours is not None and args.stale_hours < 0:
|
||||||
|
print("error: --stale-hours must be nonnegative", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
repo = args.repo.resolve()
|
||||||
|
if not (repo / "pom.xml").is_file():
|
||||||
|
print(f"error: not a recognized repository root (pom.xml missing): {repo}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
roots = {
|
||||||
|
"edit": (args.edit_root or repo / "output" / "edit-projects").resolve(),
|
||||||
|
"highlight": (args.highlight_root or repo / "output" / "highlight-projects").resolve(),
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
now = time.time()
|
||||||
|
missing_roots = [finding("error", "PROJECT_ROOT_MISSING", f"{kind} root is absent: {root}")
|
||||||
|
for kind, root in roots.items() if not root.exists()]
|
||||||
|
if args.allow_missing_roots:
|
||||||
|
missing_roots = []
|
||||||
|
projects = scan_root("edit", roots["edit"], now, args.stale_hours)
|
||||||
|
projects += scan_root("highlight", roots["highlight"], now, args.stale_hours)
|
||||||
|
except (OSError, ValueError) as exc:
|
||||||
|
print(f"error: {exc}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
if args.format == "json":
|
||||||
|
print(json.dumps({"roots": {k: str(v) for k, v in roots.items()}, "rootFindings": missing_roots,
|
||||||
|
"projects": projects}, indent=2, sort_keys=True))
|
||||||
|
else:
|
||||||
|
for item in missing_roots:
|
||||||
|
print(f"ROOT\t-\t-\t-\t-\t-\t-\t{item['severity'][0].upper()}:{item['code']} {item['detail']}")
|
||||||
|
print_table(projects)
|
||||||
|
severities = {f["severity"] for f in missing_roots}
|
||||||
|
severities.update(f["severity"] for p in projects for f in p["findings"])
|
||||||
|
if args.fail_on == "error" and "error" in severities:
|
||||||
|
return 2
|
||||||
|
if args.fail_on == "warning" and severities & {"warning", "error"}:
|
||||||
|
return 2
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -0,0 +1,223 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Verify render manifests and media using local ffprobe/ffmpeg only."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="Verify one edit or highlight render without modifying it.")
|
||||||
|
parser.add_argument("--repo", type=Path, default=Path("."), help="Repository root used to resolve manifest paths.")
|
||||||
|
parser.add_argument("--project-dir", type=Path, required=True, help="Directory containing render-manifest.json.")
|
||||||
|
parser.add_argument("--manifest", type=Path, help="Manifest override (default: PROJECT_DIR/render-manifest.json).")
|
||||||
|
parser.add_argument("--qa-report", type=Path, help="QA report override (default: PROJECT_DIR/qa-report.json).")
|
||||||
|
parser.add_argument("--media", type=Path, help="Media override; otherwise use manifest outputPath.")
|
||||||
|
parser.add_argument("--duration-tolerance", type=float, default=0.10, help="Allowed manifest/probe difference in seconds.")
|
||||||
|
parser.add_argument("--deep", action="store_true", help="Also run read-only black, silence, and audio-peak FFmpeg probes.")
|
||||||
|
parser.add_argument("--format", choices=("text", "json"), default="text")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def read_object(path: Path) -> dict:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError(f"top-level JSON is not an object: {path}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def run(command: list[str]) -> subprocess.CompletedProcess[str]:
|
||||||
|
return subprocess.run(command, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_media(value: str, repo: Path, project: Path) -> tuple[Path | None, list[str], list[str]]:
|
||||||
|
raw = Path(value)
|
||||||
|
# Current manifests record either an absolute path, a repository-relative path such as
|
||||||
|
# output/edit-projects/<id>/final.mp4, or a project-relative path such as final.mp4.
|
||||||
|
candidates = [raw] if raw.is_absolute() else [repo / raw, project / raw, project / raw.name]
|
||||||
|
normalized: list[Path] = []
|
||||||
|
for candidate in candidates:
|
||||||
|
resolved = candidate.resolve()
|
||||||
|
if resolved not in normalized:
|
||||||
|
normalized.append(resolved)
|
||||||
|
existing = [candidate for candidate in normalized if candidate.is_file()]
|
||||||
|
chosen = existing[0] if len(existing) == 1 else None
|
||||||
|
return chosen, [str(path) for path in normalized], [str(path) for path in existing]
|
||||||
|
|
||||||
|
|
||||||
|
def qa_summary(path: Path) -> dict:
|
||||||
|
if not path.is_file():
|
||||||
|
return {"present": False, "mode": "missing", "declaredPassed": None, "failedChecks": []}
|
||||||
|
report = read_object(path)
|
||||||
|
checks = [c for c in report.get("checks", []) if isinstance(c, dict)]
|
||||||
|
names = {str(c.get("name")) for c in checks}
|
||||||
|
expected = {"black_frames", "long_silence", "audio_clipping"}
|
||||||
|
mode = "measured" if expected <= names else "partial"
|
||||||
|
return {
|
||||||
|
"present": True,
|
||||||
|
"mode": mode,
|
||||||
|
"declaredPassed": report.get("passed"),
|
||||||
|
"failedChecks": [str(c.get("name")) for c in checks if c.get("passed") is False],
|
||||||
|
"invalidChecks": [str(c.get("name")) for c in checks if not isinstance(c.get("passed"), bool)],
|
||||||
|
"missingProbes": sorted(expected - names),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def deep_probes(ffmpeg: str, media: Path) -> tuple[dict, list[dict]]:
|
||||||
|
findings: list[dict] = []
|
||||||
|
probes: dict = {}
|
||||||
|
commands = {
|
||||||
|
"black": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-vf", "blackdetect=d=0.5:pic_th=0.98", "-an", "-f", "null", "-"],
|
||||||
|
"silence": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-af", "silencedetect=noise=-45dB:d=2", "-vn", "-f", "null", "-"],
|
||||||
|
"peaks": [ffmpeg, "-hide_banner", "-v", "info", "-i", str(media), "-af", "astats=metadata=1:reset=1", "-vn", "-f", "null", "-"],
|
||||||
|
}
|
||||||
|
for name, command in commands.items():
|
||||||
|
result = run(command)
|
||||||
|
output = result.stdout + result.stderr
|
||||||
|
probes[name] = {"exitCode": result.returncode}
|
||||||
|
if result.returncode != 0:
|
||||||
|
findings.append({"severity": "error", "code": f"FFMPEG_{name.upper()}_FAILED", "detail": f"exit={result.returncode}"})
|
||||||
|
continue
|
||||||
|
if name == "black":
|
||||||
|
count = len(re.findall(r"black_start:", output))
|
||||||
|
probes[name]["rangeCount"] = count
|
||||||
|
if count:
|
||||||
|
findings.append({"severity": "error", "code": "BLACK_RANGE_DETECTED", "detail": f"ranges={count}; threshold d=0.5,pic_th=0.98"})
|
||||||
|
elif name == "silence":
|
||||||
|
count = len(re.findall(r"silence_start:", output))
|
||||||
|
probes[name]["rangeCount"] = count
|
||||||
|
if count:
|
||||||
|
findings.append({"severity": "warning", "code": "LONG_SILENCE_DETECTED", "detail": f"ranges={count}; threshold -45dB for 2s"})
|
||||||
|
else:
|
||||||
|
peaks = []
|
||||||
|
for value in re.findall(r"Peak level dB:\s*([^\s]+)", output):
|
||||||
|
try:
|
||||||
|
peaks.append(float(value))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
maximum = max(peaks) if peaks else None
|
||||||
|
probes[name]["maximumPeakDbfs"] = maximum
|
||||||
|
if maximum is None:
|
||||||
|
findings.append({"severity": "warning", "code": "AUDIO_PEAK_UNMEASURED", "detail": "astats emitted no numeric peak"})
|
||||||
|
elif maximum >= -0.1:
|
||||||
|
findings.append({"severity": "warning", "code": "AUDIO_PEAK_UNSAFE", "detail": f"maximum={maximum} dBFS"})
|
||||||
|
return probes, findings
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
opts = args()
|
||||||
|
if opts.duration_tolerance < 0:
|
||||||
|
print("error: --duration-tolerance must be nonnegative", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
repo = opts.repo.resolve()
|
||||||
|
project = opts.project_dir.resolve()
|
||||||
|
manifest_path = (opts.manifest or project / "render-manifest.json").resolve()
|
||||||
|
qa_path = (opts.qa_report or project / "qa-report.json").resolve()
|
||||||
|
findings: list[dict] = []
|
||||||
|
try:
|
||||||
|
manifest = read_object(manifest_path)
|
||||||
|
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
|
||||||
|
print(f"error: {exc}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
try:
|
||||||
|
qa = qa_summary(qa_path)
|
||||||
|
except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc:
|
||||||
|
qa = {"present": True, "mode": "invalid", "declaredPassed": None, "failedChecks": [],
|
||||||
|
"invalidChecks": [], "missingProbes": []}
|
||||||
|
findings.append({"severity": "error", "code": "QA_REPORT_INVALID", "detail": str(exc)})
|
||||||
|
if qa["mode"] == "missing":
|
||||||
|
findings.append({"severity": "error", "code": "QA_REPORT_MISSING", "detail": str(qa_path)})
|
||||||
|
elif qa["mode"] == "partial":
|
||||||
|
findings.append({"severity": "warning", "code": "QA_PARTIAL", "detail": f"missing probes={','.join(qa['missingProbes'])}"})
|
||||||
|
if qa["present"] and not isinstance(qa["declaredPassed"], bool):
|
||||||
|
findings.append({"severity": "error", "code": "QA_PASSED_INVALID", "detail": "passed must be a JSON boolean"})
|
||||||
|
if qa.get("invalidChecks"):
|
||||||
|
findings.append({"severity": "error", "code": "QA_CHECK_RESULT_INVALID", "detail": ",".join(qa["invalidChecks"])})
|
||||||
|
if qa["declaredPassed"] is True and qa["failedChecks"]:
|
||||||
|
findings.append({"severity": "error", "code": "QA_RESULT_INCONSISTENT", "detail": "passed=true with failed checks: " + ",".join(qa["failedChecks"])})
|
||||||
|
if qa["declaredPassed"] is False:
|
||||||
|
findings.append({"severity": "error", "code": "QA_DECLARED_FAILED", "detail": ",".join(qa["failedChecks"]) or "report passed=false"})
|
||||||
|
media_value = str(opts.media) if opts.media else str(manifest.get("outputPath") or "")
|
||||||
|
if not media_value:
|
||||||
|
print("error: no --media and manifest outputPath is empty", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
if opts.media:
|
||||||
|
explicit = opts.media.resolve()
|
||||||
|
media, candidates, existing = (explicit if explicit.is_file() else None), [str(explicit)], ([str(explicit)] if explicit.is_file() else [])
|
||||||
|
else:
|
||||||
|
media, candidates, existing = resolve_media(media_value, repo, project)
|
||||||
|
if len(existing) > 1:
|
||||||
|
findings.append({"severity": "error", "code": "MEDIA_RESOLUTION_AMBIGUOUS", "detail": "multiple candidates exist: " + ",".join(existing)})
|
||||||
|
if media is None:
|
||||||
|
if not existing:
|
||||||
|
findings.append({"severity": "error", "code": "MANIFEST_OUTPUT_MISSING", "detail": f"candidates={','.join(candidates)}"})
|
||||||
|
report = {"manifest": str(manifest_path), "media": None, "resolutionCandidates": candidates,
|
||||||
|
"qa": qa, "probe": None, "deepProbes": None, "findings": findings}
|
||||||
|
if opts.format == "json":
|
||||||
|
print(json.dumps(report, indent=2, sort_keys=True))
|
||||||
|
else:
|
||||||
|
for item in findings:
|
||||||
|
print(f"{item['severity'].upper()} {item['code']} {item['detail']}")
|
||||||
|
return 2
|
||||||
|
ffprobe = shutil.which("ffprobe")
|
||||||
|
if not ffprobe:
|
||||||
|
print("error: ffprobe is not available on PATH", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
result = run([ffprobe, "-v", "error", "-show_streams", "-show_format", "-of", "json", str(media)])
|
||||||
|
if result.returncode != 0:
|
||||||
|
print(f"error: ffprobe exited {result.returncode}: {result.stderr.strip()}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
try:
|
||||||
|
probe = json.loads(result.stdout)
|
||||||
|
duration = float(probe.get("format", {}).get("duration"))
|
||||||
|
except (TypeError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
print(f"error: invalid ffprobe result: {exc}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
expected = manifest.get("durationSeconds")
|
||||||
|
if isinstance(expected, bool) or not isinstance(expected, (int, float)) or float(expected) < 0:
|
||||||
|
findings.append({"severity": "error", "code": "MANIFEST_DURATION_INVALID", "detail": f"durationSeconds={expected!r}"})
|
||||||
|
elif abs(float(expected) - duration) > opts.duration_tolerance:
|
||||||
|
findings.append({"severity": "error", "code": "DURATION_MISMATCH", "detail": f"manifest={expected} probe={duration:.6f} tolerance={opts.duration_tolerance}"})
|
||||||
|
streams = probe.get("streams", [])
|
||||||
|
video_streams = [s for s in streams if s.get("codec_type") == "video"]
|
||||||
|
audio_streams = [s for s in streams if s.get("codec_type") == "audio"]
|
||||||
|
if not video_streams:
|
||||||
|
findings.append({"severity": "error", "code": "VIDEO_STREAM_MISSING", "detail": "ffprobe found no video stream"})
|
||||||
|
if not audio_streams:
|
||||||
|
findings.append({"severity": "warning", "code": "AUDIO_STREAM_MISSING", "detail": "ffprobe found no audio stream"})
|
||||||
|
deep = None
|
||||||
|
if opts.deep:
|
||||||
|
ffmpeg = shutil.which("ffmpeg")
|
||||||
|
if not ffmpeg:
|
||||||
|
print("error: ffmpeg is not available on PATH", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
deep, deep_findings = deep_probes(ffmpeg, media)
|
||||||
|
findings.extend(deep_findings)
|
||||||
|
summary = {
|
||||||
|
"durationSeconds": duration,
|
||||||
|
"sizeBytes": int(probe.get("format", {}).get("size", media.stat().st_size)),
|
||||||
|
"video": [{k: s.get(k) for k in ("codec_name", "width", "height", "pix_fmt", "avg_frame_rate")} for s in video_streams],
|
||||||
|
"audio": [{k: s.get(k) for k in ("codec_name", "sample_rate", "channels", "channel_layout")} for s in audio_streams],
|
||||||
|
}
|
||||||
|
report = {"manifest": str(manifest_path), "media": str(media), "resolutionCandidates": candidates,
|
||||||
|
"qa": qa, "probe": summary, "deepProbes": deep, "findings": findings}
|
||||||
|
if opts.format == "json":
|
||||||
|
print(json.dumps(report, indent=2, sort_keys=True))
|
||||||
|
else:
|
||||||
|
print(f"media={media}")
|
||||||
|
print(f"duration_seconds={duration:.6f} video_streams={len(video_streams)} audio_streams={len(audio_streams)} qa_mode={qa['mode']}")
|
||||||
|
for item in findings:
|
||||||
|
print(f"{item['severity'].upper()} {item['code']} {item['detail']}")
|
||||||
|
if not findings:
|
||||||
|
print("OK manifest and measured media properties agree")
|
||||||
|
return 2 if any(item["severity"] == "error" for item in findings) else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -0,0 +1,400 @@
|
||||||
|
---
|
||||||
|
name: video-editing-docs-and-writing
|
||||||
|
description: Load when creating, correcting, reviewing, or retiring this repository's README, plans, runbooks, diagrams, ADRs, API/configuration references, release evidence, operational documents, prompts, or milestone checklists; also load when a command, path, status, production-readiness claim, or implementation description in docs may have drifted from source.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Documentation And Writing
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Write documentation that a zero-context mid-level engineer or Sonnet-class model can execute without guessing. Separate implemented behavior from target architecture, and make every important claim traceable to source, a test, a measurement, or an approved decision.
|
||||||
|
|
||||||
|
Treat repository facts in this skill as verified on **2026-07-21**. The Fortune 500 reference-architecture requirements are the target acceptance standard; they do not describe the repository's current state.
|
||||||
|
|
||||||
|
Compact glossary: **SBOM** is Software Bill of Materials; **SLI** is service-level indicator; **SLO** is service-level objective; **RTO** is Recovery Time Objective; **RPO** is Recovery Point Objective; **OCI** is Open Container Initiative image format.
|
||||||
|
|
||||||
|
## When not to use this skill
|
||||||
|
|
||||||
|
| Need | Load instead |
|
||||||
|
|---|---|
|
||||||
|
| Decide whether a behavior/default may change or a production claim may be promoted | `video-editing-change-control` |
|
||||||
|
| Find the implemented architecture and load-bearing invariants | `video-editing-architecture-contract` |
|
||||||
|
| Catalog or change properties, profiles, flags, and guards | `video-editing-config-and-flags` |
|
||||||
|
| Recreate the toolchain or prove a clean build | `video-editing-build-and-env` |
|
||||||
|
| Start schedulers, render media, or locate runtime artifacts | `video-editing-run-and-operate` |
|
||||||
|
| Define tests, creative-quality evidence, or promotion thresholds | `video-editing-validation-and-qa` |
|
||||||
|
| Diagnose a symptom or interpret diagnostic output | `video-editing-debugging-playbook` and `video-editing-diagnostics-and-tooling` |
|
||||||
|
| Explain why an old decision or failure happened | `video-editing-failure-archaeology` |
|
||||||
|
|
||||||
|
Do not use documentation to authorize a change, bypass approval, or turn intent into evidence.
|
||||||
|
|
||||||
|
## Apply the documentation safety boundary
|
||||||
|
|
||||||
|
Enforce these rules in every document, example, diagram, and generated artifact:
|
||||||
|
|
||||||
|
1. Do not instruct runtime startup to download dependencies or model weights. Do not prescribe external AI services, unapproved network access, unlicensed assets, placeholder silence/tones, or unapproved rendering.
|
||||||
|
2. Do not present a production-facing default change as a documentation correction. Route the implementation and its documentation through `video-editing-change-control`.
|
||||||
|
3. Mark the current automatic bootstrap/download, heuristic fallback, placeholder audio, and approval-bypass behavior as defects or development-only behavior, never as approved production practice.
|
||||||
|
4. Do not paste secrets, credentials, private hosts, user home paths, local virtual environments, proprietary footage, or unredacted production payloads into documentation.
|
||||||
|
5. Do not claim production-ready, secure, scalable, reproducible, cinematic, licensed, validated, signed off, or complete without the required evidence and approval.
|
||||||
|
6. Treat documentation-only work as C0 only when it cannot change runtime/build behavior. A command, configuration, API, schema, deployment, or operational contract correction that reveals or requires behavior change inherits the applicable C2/C3 route.
|
||||||
|
|
||||||
|
## Start from the current repository truth
|
||||||
|
|
||||||
|
Run these read-only checks from the repository root before writing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short
|
||||||
|
git log -10 --oneline --decorate
|
||||||
|
git ls-files 'docs/**' 'src/main/resources/**' 'tools/**' 'dashboards/**' pom.xml | sort
|
||||||
|
find .claude/skills -maxdepth 2 -name SKILL.md -print | sort
|
||||||
|
find src/main/java src/test/java -type f | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
As of 2026-07-21:
|
||||||
|
|
||||||
|
| Current fact | Documentation consequence |
|
||||||
|
|---|---|
|
||||||
|
| There is no tracked `README.md`, contributor guide, OpenAPI document, ADR set, architecture diagram, CI workflow, container definition, or deployment definition. | List each as missing target work. Never write “see” or “validated by” one of these artifacts. |
|
||||||
|
| `docs/` contains plans, prompts, runbooks, operating checklists, benchmark baselines, and readiness templates without an explicit authority hierarchy. | Classify each document before relying on it. A plan is not an implementation contract. |
|
||||||
|
| `pom.xml` uses Java 21 and Spring Boot 3.3.2; no Maven Wrapper is tracked. | State implemented versions and current prerequisites. Do not call Spring Boot 3.3.2 “latest” without a separately approved, date-stamped primary-source check. |
|
||||||
|
| Base `application.yml` enables ingestion schedulers and local-worker auto-start, while highlight rendering defaults off and approval defaults on; current CV/bootstrap scripts may install packages or resolve models. | Do not recommend a plain startup as a harmless documentation-verification command. Use the operating/build siblings and explicit safe overrides. |
|
||||||
|
| `application-jpa.properties` uses H2 in PostgreSQL compatibility mode. | Do not describe current repository integration tests as real PostgreSQL/Testcontainers evidence. |
|
||||||
|
| `production-cinematic-highlight-editing-plan.md` checks all milestones, while `highlight-rendering-gap-closure-plan.md` remains unchecked and `cinematic-highlight-operator-checklist.md` says rendering is not wired. Current source now contains `HighlightDirectorFlowService` and `HighlightFfmpegRenderer`. | Label the documents contradictory/stale until source, tests, and an executed flow establish one current description. Do not settle the conflict by trusting the newest prose or a checkbox. |
|
||||||
|
| `video-clipping-service-implementation-plan.md` checks “Load-test signoff,” while `load-test-signoff.md` says execution evidence is pending and no load generator/results are checked in. | State “documentation template exists; load signoff unvalidated.” |
|
||||||
|
| `input-folder-scheduler-plan.md` names a nonexistent `FolderSchedulerProperties.java`; properties are nested under `VideoClippingProperties`. | Correct or label the proposed path; do not send an engineer to it. |
|
||||||
|
| Existing benchmark documents describe local baselines and generated files under `target/`. | Preserve environment/date/commit qualifiers. Never convert a workstation baseline into an SLO or production capacity claim. |
|
||||||
|
|
||||||
|
## Use the authority hierarchy
|
||||||
|
|
||||||
|
Resolve conflicts in this order. Higher authority establishes current behavior; it does not automatically prove quality or production readiness.
|
||||||
|
|
||||||
|
| Rank | Authority | What it can establish |
|
||||||
|
|---:|---|---|
|
||||||
|
| 1 | Executable source, migrations, packaged resource files, build manifest | What the checked-out artifact is designed to do and its configured defaults. |
|
||||||
|
| 2 | Deterministic tests and retained command output tied to a commit/environment | What was exercised, within the test's actual assertions and prerequisites. |
|
||||||
|
| 3 | Versioned generated contract or measured report tied to its generator and inputs | The generated schema/measurement, if regeneration matches. `target/` alone is disposable, not a record. |
|
||||||
|
| 4 | Approved ADR, API contract, operations record, or release evidence | The accepted decision/contract for its stated scope and date. These record approval, not code existence. |
|
||||||
|
| 5 | Current reference/runbook checked against ranks 1-4 | How to understand or operate the verified behavior. |
|
||||||
|
| 6 | Plan, prompt, issue-shaped note, checklist, proposed diagram | Intent. A checked box still needs evidence. |
|
||||||
|
| 7 | Historical or retired document | Context only; never an operating instruction. |
|
||||||
|
|
||||||
|
When code and an approved contract differ, report the mismatch. Do not silently rewrite the contract to match code or treat the code as approved.
|
||||||
|
|
||||||
|
## Give each fact one home
|
||||||
|
|
||||||
|
Keep the detailed fact in one canonical home. Cross-reference it elsewhere with a short reason; do not copy tables that will drift.
|
||||||
|
|
||||||
|
| Fact | Current/target canonical home | Cross-reference rule |
|
||||||
|
|---|---|---|
|
||||||
|
| Entry point, prerequisites, safe local quick start, repository map | **TARGET:** `README.md` (currently absent) | Other docs link to the relevant README heading once it exists. Until then, state that the entry-point doc is missing. |
|
||||||
|
| Dependency versions and build plugin behavior | `pom.xml`; explanation in build/environment docs | Never duplicate a version without “verified YYYY-MM-DD” and a recheck command. |
|
||||||
|
| Runtime property defaults and profile overrides | Source/resources; `video-editing-config-and-flags` is the maintained explanatory catalog | Runbooks name only the few overrides needed for that procedure and link to the catalog. |
|
||||||
|
| REST resources, schemas, status/errors, compatibility | **TARGET:** versioned OpenAPI document (currently absent) | Controller prose is an implementation note, not the public contract. |
|
||||||
|
| Database schema and migration order | `src/main/resources/db/migration/` | Data/runbook docs link to migrations and explain operation; do not reproduce full DDL. |
|
||||||
|
| Module responsibilities and dependency direction | **TARGET:** architecture overview plus context/module diagrams (currently absent) | ADRs explain decisions; code maps and README link to the overview. |
|
||||||
|
| Important design decision and alternatives | **TARGET:** one immutable-numbered ADR per decision (currently absent) | Plans and PRs link to the ADR; do not restate the decision history. |
|
||||||
|
| Alert response and recovery procedure | `docs/runbooks.md` or one linked specialized runbook | Alert definitions link to the procedure; dashboards do not duplicate it. |
|
||||||
|
| Test and quality acceptance | `video-editing-validation-and-qa` until a tracked testing strategy exists | Plans list intended coverage only; releases link to retained evidence. |
|
||||||
|
| Historical failure | `video-editing-failure-archaeology` | Current docs link to the incident/lesson rather than retelling it. |
|
||||||
|
| Change gates and promotion | `video-editing-change-control` | Every behavior-changing plan routes to it. |
|
||||||
|
| Release-specific versions, evidence, approvals, known limits | **TARGET:** immutable release record (currently absent) | README/runbooks link to the current supported release record. |
|
||||||
|
|
||||||
|
Propose any new canonical path through documentation review. Do not create a second home merely because the target home is absent.
|
||||||
|
|
||||||
|
## Label status and evidence explicitly
|
||||||
|
|
||||||
|
Put this block near the top of every maintained reference, runbook, plan, diagram, or report:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Status: CURRENT | TARGET | CANDIDATE | HISTORICAL
|
||||||
|
Scope: <workflow, environment, and audience>
|
||||||
|
Last verified: YYYY-MM-DD
|
||||||
|
Applies to: commit <full-or-short-sha> | release <identifier> | proposed
|
||||||
|
Owner: <team or role; UNKNOWN if not assigned>
|
||||||
|
Evidence: <source paths, tests, measurement record, ADR, or NONE>
|
||||||
|
Next review: YYYY-MM-DD | event that forces review
|
||||||
|
```
|
||||||
|
|
||||||
|
Use labels strictly:
|
||||||
|
|
||||||
|
| Label | Meaning | Required wording |
|
||||||
|
|---|---|---|
|
||||||
|
| `CURRENT` | Matches source and has evidence for the bounded claim. | Say exactly what was verified, where, when, and on which environment. |
|
||||||
|
| `TARGET` | Accepted requirement not yet demonstrated. | Use “must,” “target,” or “not implemented”; never “supports” or “provides.” |
|
||||||
|
| `CANDIDATE` | Proposed/hypothesized and awaiting decision or evidence. | Name owner, experiment/gate, and expiry or revisit condition. |
|
||||||
|
| `HISTORICAL` | Superseded, rejected, or incident context. | Name replacement/status and prohibit operational use. |
|
||||||
|
|
||||||
|
Qualify evidence separately: `SOURCE-INSPECTED`, `TESTED`, `MEASURED`, `REVIEWED`, or `UNVERIFIED`. A source-inspected branch is not tested; a passing structure test is not cinematic quality; a document checklist is not execution evidence.
|
||||||
|
|
||||||
|
Never use a bare `[x]` for a milestone. Use a table:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
| Milestone | Status | Evidence | Last verified | Remaining gate |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Local-model voiceover | TARGET | `NONE` | 2026-07-21 | Offline licensed model; intelligibility and timing measurements; approval |
|
||||||
|
```
|
||||||
|
|
||||||
|
## Follow the maintenance workflow
|
||||||
|
|
||||||
|
1. **Classify the document.** Choose reference, runbook, contract, decision, plan, evidence report, prompt, or historical record. Do not combine target design and current operation without visibly separate sections.
|
||||||
|
2. **Select one fact owner.** Use the table above. If the home is missing, add it to the backlog; do not scatter temporary copies.
|
||||||
|
3. **Build a claim ledger.** For each normative/current sentence, record claim, status, evidence, verification command, and volatility.
|
||||||
|
4. **Inspect implementation.** Read the relevant source, resources, tests, migrations, and history. A class name proves existence only, not wiring or success.
|
||||||
|
5. **Write in imperative order.** State purpose and safety boundary, prerequisites, exact steps, expected observations, failure branches, rollback/stop condition, and evidence to retain.
|
||||||
|
6. **Synchronize all affected surfaces.** Use the matrices below. If an expected contract is absent, state the gap instead of inventing it.
|
||||||
|
7. **Verify every literal.** Check paths, class names, commands, flags, defaults, endpoint mappings, filenames, event names, and output fields against the current tree.
|
||||||
|
8. **Run non-mutating audits.** Do not start schedulers, bootstrap workers, download dependencies/models, render, use network, or mutate Git merely to validate prose.
|
||||||
|
9. **Route claims.** Send behavior/default changes and production-readiness claims through `video-editing-change-control`; attach validation evidence rather than declaring success in prose.
|
||||||
|
10. **Record maintenance.** Date volatile facts and add one-line re-verification commands at the end.
|
||||||
|
|
||||||
|
## Synchronize contracts
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
For any property statement, compare all applicable locations:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'class VideoClippingProperties|@ConfigurationProperties|enabled|auto-start|render-enabled|require.*approval|fallback|strict-runtime' src/main/java/org/example/videoclips/config src/main/resources src/test
|
||||||
|
rg -n 'VIDEO_EDITING_|FOLDER_SCHEDULER_|video-clipping\.' docs tools src/main/resources src/main/java
|
||||||
|
```
|
||||||
|
|
||||||
|
Check the Java default, packaged default, profile override, environment mapping, validation, component guard, tests, operational risk, and rollback. Defer the full catalog to `video-editing-config-and-flags`.
|
||||||
|
|
||||||
|
### API
|
||||||
|
|
||||||
|
For any endpoint statement, compare controllers, DTOs, exception mapping, application service, and API tests:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '@(RequestMapping|GetMapping|PostMapping|PutMapping|PatchMapping|DeleteMapping)|ProblemDetail' src/main/java/org/example/videoclips/api
|
||||||
|
find src/main/java/org/example/videoclips/api/dto src/test/java/org/example/videoclips/api -type f | sort
|
||||||
|
find . -maxdepth 3 -type f \( -iname '*openapi*' -o -iname '*swagger*' \) -print
|
||||||
|
```
|
||||||
|
|
||||||
|
The last command currently returns nothing. Do not claim OpenAPI publication, build validation, compatibility checking, authentication, pagination, or idempotency unless separately evidenced.
|
||||||
|
|
||||||
|
### Runtime file layout
|
||||||
|
|
||||||
|
For any input/output path, compare packaged configuration, folder-contract/constants, initializer, store, scheduler, tests, and the runbook:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'Path |resolve\(|createDirectories|source-directory|working-directory|processed-directory|rejected-directory|project-directory|highlight-project-directory' src/main src/test
|
||||||
|
rg -n 'input/|output/|target/|project.json|edit-plan.json|render-manifest.json|qa-report.json' docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Distinguish repository-relative defaults, configurable paths, generated/disposable `target/` output, and production volume contracts. Never use a private absolute path.
|
||||||
|
|
||||||
|
### Operations, metrics, and deployment
|
||||||
|
|
||||||
|
Compare event/metric source with alert, dashboard, and runbook names:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'event=|Counter|Timer|Gauge|Observation|MeterRegistry|HealthIndicator' src/main/java
|
||||||
|
rg -n 'video_clipping_|http_server_|event=|/actuator/' docs dashboards
|
||||||
|
find . -maxdepth 3 -type f \( -iname 'Dockerfile*' -o -iname '*compose*' -o -iname '*helm*' -o -iname '*k8s*' \) -print
|
||||||
|
```
|
||||||
|
|
||||||
|
The deployment-file search currently returns nothing. Label Kubernetes, HPA/KEDA, cloud, backup, restore, and rollout material as target guidance or environment procedure unless executed evidence exists.
|
||||||
|
|
||||||
|
## Run the documentation audits
|
||||||
|
|
||||||
|
### Broken or stale path audit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git ls-files | sort > /tmp/video-editing-tracked-files.txt
|
||||||
|
rg -n '`(src|docs|tools|dashboards)/[^`]+`|`pom.xml`|`README.md`' docs
|
||||||
|
rg -n 'FolderSchedulerProperties|README|openapi|swagger|ADR|Dockerfile|compose|helm|k8s' docs
|
||||||
|
find docs src/main/resources tools dashboards -type f -print | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
Inspect every literal path against the tracked-file list. Treat placeholders such as `<project-id>` as patterns, not files. Flag references to generated `target/` artifacts unless the document provides a reproducible generator and explicitly calls them disposable.
|
||||||
|
|
||||||
|
### Milestone and overclaim audit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '^- \[[ xX]\]|complete|implemented|production[- ]ready|validated|verified|signed off|supports|automatic today' docs
|
||||||
|
rg -n 'pending|not implemented|not yet|still manual|limitation|known gap|unchecked|heuristic|fallback|placeholder|proposed|intended' docs
|
||||||
|
git log --format='%h %ad %s' --date=short -- docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Pair every positive claim with contrary qualifiers and implementation evidence. Mandatory current findings include the highlight-plan/rendering contradiction and the load-signoff contradiction listed above. Preserve old milestones as `HISTORICAL` when useful; do not erase a failed or superseded approach.
|
||||||
|
|
||||||
|
### Source drift audit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git diff --name-only HEAD -- pom.xml src/main src/test tools dashboards docs
|
||||||
|
rg -n '<java.version>|spring-boot-starter-parent|<version>|<artifactId>' pom.xml
|
||||||
|
rg -n '^video-clipping:|^[[:space:]]+[a-z0-9-]+:' src/main/resources/application*.yml
|
||||||
|
find src/main/resources/db/migration -maxdepth 1 -type f -print | sort
|
||||||
|
find src/test/java -name '*Test.java' -print | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
When code changes without docs, identify the canonical document affected. When docs change without code, verify that they remain intent-only or accurately describe existing behavior. Do not use line/test counts as stable claims unless date-stamped and regenerated.
|
||||||
|
|
||||||
|
### Command and example audit
|
||||||
|
|
||||||
|
For each command, verify executable, working directory, required environment, side effects, network behavior, expected exit/result, and cleanup. Prefer a focused read-only or test command. Never recommend `mvn spring-boot:run` as a documentation lint because current defaults can consume input, start workers, install dependencies, resolve models, and render.
|
||||||
|
|
||||||
|
Use these syntax/content checks after editing Markdown:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git diff --check -- .claude/skills docs README.md 2>/dev/null
|
||||||
|
rg -n '^```' .claude/skills docs
|
||||||
|
rg -n '[/]Users/|[/]home/|[A-Za-z]:\\\\|api[_-]?key|secret|password|token' .claude/skills docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Review fence pairs manually or with an approved Markdown linter if one is later added. No Markdown lint/build is configured in `pom.xml` today.
|
||||||
|
|
||||||
|
## Use these document templates
|
||||||
|
|
||||||
|
### ADR
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# ADR-NNN: <Decision>
|
||||||
|
Status: CANDIDATE | ACCEPTED | SUPERSEDED
|
||||||
|
Date: YYYY-MM-DD
|
||||||
|
Owners: <roles>
|
||||||
|
|
||||||
|
## Context
|
||||||
|
Requirement, constraints, current evidence, and trust boundaries.
|
||||||
|
## Decision
|
||||||
|
Selected approach and dependency direction.
|
||||||
|
## Alternatives Considered
|
||||||
|
Option, rejection reason, and evidence.
|
||||||
|
## Consequences
|
||||||
|
Benefits, trade-offs, operational consequences, and security implications.
|
||||||
|
## Verification
|
||||||
|
Tests, measurements, scans, and approval required.
|
||||||
|
## Revisit Conditions
|
||||||
|
Specific signal, date, dependency change, or failed threshold.
|
||||||
|
```
|
||||||
|
|
||||||
|
For every major decision include the requirement, selected approach, alternatives, benefits/trade-offs, operational consequences, security implications, verification, and revisit conditions. Do not mark an ADR accepted without the authority required by change control.
|
||||||
|
|
||||||
|
### Incident record
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# Incident: <Observed symptom>
|
||||||
|
Status: OPEN | MITIGATED | RESOLVED
|
||||||
|
Window/Environment/Build: <UTC times, environment, commit/release>
|
||||||
|
Impact: <measured user/data/media effect>
|
||||||
|
Detection: <alert, report, or operator observation>
|
||||||
|
Evidence preserved: <logs, traces, metrics, manifests, checksums; redacted>
|
||||||
|
Timeline: <UTC event/action/result>
|
||||||
|
Root cause: <mechanism, or UNKNOWN>
|
||||||
|
Contributing conditions: <bounded facts>
|
||||||
|
Rejected hypotheses: <experiment and result>
|
||||||
|
Mitigation and recovery: <approved action and verification>
|
||||||
|
Corrective actions: <owner, due date, gate>
|
||||||
|
Recurrence test: <exact automated or drill evidence>
|
||||||
|
```
|
||||||
|
|
||||||
|
Move the durable symptom -> cause -> evidence -> status lesson into `video-editing-failure-archaeology`; keep sensitive operational data outside the repository.
|
||||||
|
|
||||||
|
### Operational runbook
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# <Alert/Symptom> Runbook
|
||||||
|
Status/Owner/Last verified/Applies to: <metadata>
|
||||||
|
Trigger and user impact: <measurable conditions>
|
||||||
|
Safety boundary: <actions requiring approval; data to preserve>
|
||||||
|
Prerequisites: <role, environment, tools>
|
||||||
|
Triage: <read-only command -> expected result -> branch>
|
||||||
|
Mitigation: <smallest approved reversible action>
|
||||||
|
Verification: <metrics, traces, state, and user journey>
|
||||||
|
Escalation: <threshold, role, evidence bundle>
|
||||||
|
Rollback/Recovery: <procedure and stop condition>
|
||||||
|
Known gaps: <explicit target work>
|
||||||
|
```
|
||||||
|
|
||||||
|
Never place an unapproved database mutation, file deletion, scheduler start, render, dependency/model download, or production-default change in a “quick fix.”
|
||||||
|
|
||||||
|
### Diagram
|
||||||
|
|
||||||
|
Use text-source diagrams that can be reviewed in diffs. Label every box as implemented, target, or external; show trust boundaries, protocols, state stores, local model processes, ownership, and dependency arrows. A context diagram shows users/external systems. A module/container diagram shows deployable/process/module boundaries. A runtime-flow diagram shows ordering, state transitions, failure paths, approval, retry/idempotency, and observability.
|
||||||
|
|
||||||
|
Validate every node against source or an accepted ADR. Do not draw target PostgreSQL, OAuth/OIDC, OpenTelemetry, containers, Kubernetes, model registry, or cloud services as current.
|
||||||
|
|
||||||
|
### Release record
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# Release <immutable identifier>
|
||||||
|
Commit/artifact digest/build provenance/SBOM: <references>
|
||||||
|
Environment promotions and approvals: <records>
|
||||||
|
Schema/API/config/default deltas: <old -> new>
|
||||||
|
Model/asset versions, checksums, origins, licenses: <inventory>
|
||||||
|
Security and dependency scan result: <evidence>
|
||||||
|
Test/quality/load/resilience evidence: <evidence and thresholds>
|
||||||
|
Known limitations/risk acceptances/expiry: <records>
|
||||||
|
Migration/rollback/smoke verification: <commands and results>
|
||||||
|
Owners/RTO/RPO/support window: <facts>
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not create release notes from commit subjects alone. Verify user-visible behavior and retained evidence.
|
||||||
|
|
||||||
|
## Maintain the Fortune 500 documentation backlog
|
||||||
|
|
||||||
|
Record these as **TARGET/MISSING**, not complete. Create them only through reviewed work; keep one fact home as defined above.
|
||||||
|
|
||||||
|
| Priority | Required artifact | Current evidence/gap |
|
||||||
|
|---:|---|---|
|
||||||
|
| P0 | README with prerequisites, safe local setup, architecture map, test/config/deploy links, representative API requests, and troubleshooting | No `README.md`. |
|
||||||
|
| P0 | Versioned OpenAPI plus build validation and breaking-change gate | Controllers/tests exist; no OpenAPI artifact/plugin. |
|
||||||
|
| P0 | Architecture overview: context, modules/dependency direction, and critical runtime flows | No architecture diagram or enforced module document. |
|
||||||
|
| P0 | Security/threat-model summary: trust boundaries, assets, threats, mitigations, auth, secrets, audit/redaction | No threat model; Spring Security is not in the current POM. |
|
||||||
|
| P0 | Testing strategy and evidence inventory: unit, architecture, slices, PostgreSQL/Testcontainers, contracts, E2E, mutation, coverage | Tests exist, but no comprehensive strategy or required enterprise gates. |
|
||||||
|
| P0 | Local development guide for macOS and reproducible Linux/VPS/cloud build/runtime | Existing runbooks are workflow-specific; clean-checkout reproducibility is not established. |
|
||||||
|
| P0 | Deployment/operations guide: immutable OCI artifact, config/secrets, health, graceful shutdown, resources, rolling deploy, rollback/roll-forward | No image or deployment definitions; existing operational docs are templates/guidance. |
|
||||||
|
| P0 | Local-model/media supply-chain and licensing record | Current workers can bootstrap/download/fallback; model and asset provenance is not production-certified. |
|
||||||
|
| P1 | Configuration reference generated/checked against typed properties, resources, profiles, guards, and secret policy | Use `video-editing-config-and-flags` now; no product-facing reference artifact. |
|
||||||
|
| P1 | ADR index and ADRs for modular-monolith boundaries, persistence/queue/storage, local model isolation, approval gate, file contracts, observability, and deployment | No ADRs. |
|
||||||
|
| P1 | Observability/SLI/alert/runbook map and dashboard validation | Docs/dashboard exist; traces, deployed evidence, owners, and full business SLIs remain incomplete. |
|
||||||
|
| P1 | Data operations: schema strategy, immutable migrations, UTC/concurrency/index decisions, backup/restore, retention, RTO/RPO | Flyway V1-V6 and procedure prose exist; no real PostgreSQL restore evidence or assigned RTO/RPO. |
|
||||||
|
| P1 | CI/CD, quality-gate, dependency-update, branch-review, SBOM/signing/provenance, and promotion policy | No CI workflow or release provenance record. |
|
||||||
|
| P1 | Performance/capacity methodology and executed workload reports | Local baselines exist; no checked-in production-like load generator/signoff. |
|
||||||
|
| P2 | Ownership/contribution policy, glossary, support/escalation map, changelog/release records, and documentation review cadence | No contributor/ownership/release docs found. |
|
||||||
|
|
||||||
|
Treat existing plans as inputs to this backlog, not substitutes for these records.
|
||||||
|
|
||||||
|
## Apply the house style
|
||||||
|
|
||||||
|
- Address the operator directly with imperative verbs: “Run,” “Verify,” “Stop,” and “Escalate.”
|
||||||
|
- Define each project/domain term once. Link to its owner thereafter.
|
||||||
|
- Put prerequisites before commands and expected observations immediately after them.
|
||||||
|
- Give discriminating branches: “If X, do Y; if not, stop and collect Z.”
|
||||||
|
- Use repository-relative paths and exact case. Put literals in backticks.
|
||||||
|
- Use UTC for incidents/releases; use ISO `YYYY-MM-DD` for volatile fact dates.
|
||||||
|
- Separate `CURRENT`, `TARGET`, `CANDIDATE`, and `HISTORICAL` content visibly.
|
||||||
|
- Use tables for catalogs/comparisons and checklists for actions, not for proof.
|
||||||
|
- State negative facts plainly: “No OpenAPI artifact exists,” not “OpenAPI is forthcoming.”
|
||||||
|
- Keep examples realistic but sanitized. Label illustrative IDs, hosts, values, and outputs.
|
||||||
|
- Explain rationale and evidence; do not narrate obvious syntax or advertise the architecture.
|
||||||
|
- Preserve limitations, failed experiments, and uncertainty. Write `UNKNOWN` rather than guessing.
|
||||||
|
|
||||||
|
## Definition of done for a documentation change
|
||||||
|
|
||||||
|
- [ ] The document has one type, owner, status, scope, date, applicable commit/release, evidence, and review trigger.
|
||||||
|
- [ ] Every current claim is bounded and evidenced; every target/candidate statement is labeled.
|
||||||
|
- [ ] The fact has one canonical home and sibling documents cross-reference rather than duplicate it.
|
||||||
|
- [ ] Paths, commands, config values, endpoints, event/metric names, file layouts, versions, and tests were rechecked.
|
||||||
|
- [ ] Commands state prerequisites, side effects, expected observations, failure branch, and avoid prohibited actions.
|
||||||
|
- [ ] API/config/schema/file-layout/operations surfaces were synchronized or an explicit drift issue was recorded.
|
||||||
|
- [ ] Security, licensing, privacy, approval, and network boundaries are explicit where applicable.
|
||||||
|
- [ ] Behavior/default/production claims went through `video-editing-change-control` with evidence from the correct sibling skill.
|
||||||
|
- [ ] No private path, secret, proprietary input, generated-only source, or unverified production claim was embedded.
|
||||||
|
- [ ] `git diff --check` and the audits above pass or each remaining finding is recorded with owner/status.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Repository baseline and volatile findings were source-inspected on **2026-07-21**. Re-verify before use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log -1 --format='%H %cs %s'
|
||||||
|
test -f README.md; find .github -maxdepth 3 -type f -print 2>/dev/null; find . -maxdepth 3 -type f \( -iname '*openapi*' -o -iname '*adr*' -o -iname 'Dockerfile*' -o -iname '*compose*' \) -not -path './target/*' -print
|
||||||
|
rg -n '^- \[[ xX]\]|production[- ]ready|signed off|execution evidence pending|still manual|not wired|limitation|known gap' docs
|
||||||
|
rg -n 'FolderSchedulerProperties|HighlightDirectorFlowService|HighlightFfmpegRenderer|render-enabled|require-director-approval' docs src/main src/test
|
||||||
|
rg -n '@(RequestMapping|GetMapping|PostMapping|PutMapping|PatchMapping|DeleteMapping)|ProblemDetail' src/main/java/org/example/videoclips/api
|
||||||
|
rg -n '<java.version>|spring-boot-starter-parent|jacoco-maven-plugin' pom.xml
|
||||||
|
find src/main/resources/db/migration -maxdepth 1 -type f -print | sort
|
||||||
|
git diff --check -- .claude/skills docs README.md 2>/dev/null
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,318 @@
|
||||||
|
---
|
||||||
|
name: video-editing-external-positioning
|
||||||
|
description: Load this skill when drafting or reviewing public claims, release notes, papers, architecture case studies, model cards, benchmark comparisons, production-readiness statements, Fortune 500 reference-architecture assessments, or claims that this video-editing service is cinematic, local, offline, secure, scalable, reproducible, or state of the art. Use it to assemble the evidence package and prevent repository capabilities from being overstated.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Position This Project Externally
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Turn repository facts into claims that an independent reviewer can reproduce. Treat code,
|
||||||
|
plans, tests, executed measurements, and accepted claims as different evidence levels.
|
||||||
|
|
||||||
|
Use the repository state and rules dated **2026-07-21**:
|
||||||
|
|
||||||
|
- The target is a production-grade reference architecture suitable for a Fortune 500
|
||||||
|
engineering organization.
|
||||||
|
- The hardest live problem is identifying highlights and editing them to high cinematic
|
||||||
|
quality with production-ready visuals, sound, music, and voiceover using models packaged
|
||||||
|
inside the service runtime.
|
||||||
|
- Automatic dependency or model downloads, external AI services, unlicensed assets,
|
||||||
|
placeholder silence or tones, unapproved rendering, network access, and changes to
|
||||||
|
production-facing defaults are prohibited.
|
||||||
|
- Required environments are macOS development, Linux/VPS production, and cloud
|
||||||
|
infrastructure.
|
||||||
|
|
||||||
|
Do not weaken these rules to make a release or paper easier to announce.
|
||||||
|
|
||||||
|
## When Not To Use This Skill
|
||||||
|
|
||||||
|
| Need | Load this sibling instead |
|
||||||
|
|---|---|
|
||||||
|
| Classify, gate, or approve a code/config change | `video-editing-change-control` |
|
||||||
|
| Decide whether tests and measurements meet acceptance thresholds | `video-editing-validation-and-qa` |
|
||||||
|
| Turn a hypothesis into an experiment and survive adversarial review | `video-editing-research-methodology` |
|
||||||
|
| Plan work on cinematic highlight quality | `video-editing-cinematic-highlights-campaign` |
|
||||||
|
| Document build prerequisites or reproduce the runtime | `video-editing-build-and-env` |
|
||||||
|
| Maintain repository documentation rather than external claims | `video-editing-docs-and-writing` |
|
||||||
|
|
||||||
|
Do not use this skill to approve a claim about your own change. Require an independent
|
||||||
|
reviewer and the gates below.
|
||||||
|
|
||||||
|
**SBOM** means Software Bill of Materials.
|
||||||
|
|
||||||
|
## Define The Claim Vocabulary
|
||||||
|
|
||||||
|
Use these terms consistently:
|
||||||
|
|
||||||
|
| Term | Meaning here |
|
||||||
|
|---|---|
|
||||||
|
| Technique | A known engineering method used by the repository, such as FFmpeg filtering, Flyway migrations, ports/adapters, or Micrometer metrics. Its presence is not novelty. |
|
||||||
|
| Capability | Behavior demonstrated by code plus an executable test or retained run artifact for a named commit and environment. |
|
||||||
|
| Candidate | A plausible design, result, or novelty that has not met its evidence gate. |
|
||||||
|
| Claim | A bounded statement with named scope, baseline, metric, threshold, commit, environment, and evidence owner. |
|
||||||
|
| Evidence package | Immutable inputs, commands, outputs, measurements, provenance, negative results, and independent review needed to audit a claim. |
|
||||||
|
| Production-ready | Verified against the entire production definition of done, not merely able to start or render an MP4. |
|
||||||
|
| Local | Computation executes on the declared host or contained runtime. This does not imply offline, bundled models, no telemetry, or no download. |
|
||||||
|
| Offline | A clean, isolated run completes with egress denied and all dependencies, weights, and assets supplied by an approved artifact. |
|
||||||
|
| Reproducible | An independent operator can recreate a declared result from an immutable revision and artifact set within stated tolerances. Do not imply byte-for-byte identity unless tested. |
|
||||||
|
| Cinematic quality | A preregistered technical and blinded-human evaluation passes. File validity or the presence of effects is insufficient. |
|
||||||
|
| State of the art (SOTA) | A statistically defensible improvement over current, relevant external baselines established from approved primary evidence. |
|
||||||
|
|
||||||
|
## Start From The Current Claim Ledger
|
||||||
|
|
||||||
|
Treat this table as the default posture until newer retained evidence replaces it.
|
||||||
|
|
||||||
|
| Area | Repository fact | Allowed wording | Forbidden inference |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Platform | `pom.xml` declares Java 21, Spring Boot 3.3.2, Maven, PostgreSQL, Flyway, S3, Actuator, Prometheus, H2, and test dependencies. | "The repository currently declares ..." | "Latest," "supported," "locked," "hardened," or "vulnerability-free." No approved current external version/vulnerability evidence exists. |
|
||||||
|
| Editing | Code contains source analysis, candidate/director flow, FFmpeg renderers, local asset integration, manifests, and QA report types. | "Implements an in-repository cinematic highlight pipeline candidate." | "Produces production-ready cinematic edits." |
|
||||||
|
| Visual models | A local HTTP CV adapter and worker exist; configuration defaults to `local-cv` and permits heuristic fallback. | "Supports a local CV worker adapter with configurable heuristic fallback." | "All highlight decisions are model-driven," "fully local," or "offline." |
|
||||||
|
| Audio models | `tools/local_asset_worker.py` invokes Piper when configured and AudioCraft model loaders for music/SFX. | "Contains adapters for local TTS and generative audio runtimes." | "Models reside in the service runtime," "no downloads," or "production voice/music/SFX." |
|
||||||
|
| Fallbacks | Historical output may contain silence, tones, or host speech. The 2026-07-21 working tree removes those success paths and rejects missing requested assets. | Identify the exact source revision and asset provenance. | Any cinematic or real-asset claim for an older output or an output without model/asset manifests. |
|
||||||
|
| Downloads | Worker bootstrap scripts run unpinned `pip install`; model APIs can resolve named weights. Local CV documentation says startup may download dependencies/YOLO weights. | "Current bootstrap paths may access package/model networks." | "Hermetic," "offline," "dependency-locked," or "no automatic downloads." |
|
||||||
|
| Approval | Highlight rendering now defaults off and approval on; the separate local-director profile also requires a flag. The REST edit-render endpoint and bare flag design still lack production authorization. | State the exact workflow, effective configuration, and approval limitation. | "All rendering requires authenticated, digest-bound human approval." |
|
||||||
|
| QA | `HighlightFfmpegRenderer.buildQaReport` now probes duration, black ranges, long silence, and sample peaks; asset/mastering and overlay checks remain structural or plan-level. | "Writes a technical QA report with measured defect probes and structural checks." | "QA-certified," "validated loudness," "raster-safe overlays," or "measured cinematic quality." |
|
||||||
|
| Tests | The suite contains unit, Spring, and FFmpeg-backed tests. A 2026-07-21 offline working-tree `mvn -q -o verify` run passed 245 tests in 62 test classes. | State the command, tree state, date, and attached report. | "Clean-checkout reproducible," "network-isolated," or cross-platform. An earlier clean archived checkout failed because an asset-generation test depended on untracked runtime state or an audible TTS fallback. |
|
||||||
|
| Operations | Plans, benchmark harnesses, runbooks, alerts, and a Grafana dashboard exist. `docs/load-test-signoff.md` explicitly says execution evidence is pending. | "Operational planning artifacts exist." | "Load-tested," "cloud-ready," "operationally signed off," or "scalable." |
|
||||||
|
| Enterprise controls | No checked-in CI definition, Maven wrapper, container/deployment manifest, OpenAPI document, architecture-boundary gate, Testcontainers setup, SBOM/signing gate, Spring Security dependency, or threat model was found in Phase 1. | List each as a current gap. | "Secure," "Fortune 500 reference architecture," "cloud-native," or "release governed." |
|
||||||
|
| Licensing | Asset lookup can copy adjacent `.license.txt` files, but the repository has no complete model/asset/license inventory. | "Supports an adjacent license-file convention for selected assets." | "All assets/models are licensed" or "commercially usable." |
|
||||||
|
| Novelty | The repository combines known local analysis, planning, asset, and FFmpeg techniques around filesystem project contracts. | "Repository-specific integration candidate." | "Novel architecture," "research contribution," or "SOTA." No novelty search or external comparison is approved. |
|
||||||
|
|
||||||
|
Plans and checked boxes prove intent or implementation bookkeeping only. They do not replace
|
||||||
|
executed evidence. Generated files under `target/`, `input/`, `output/`, or local virtual
|
||||||
|
environments are not certified evidence unless an evidence manifest names and hashes them.
|
||||||
|
|
||||||
|
## Apply The Claim Ladder
|
||||||
|
|
||||||
|
Promote wording one level at a time. Never skip a level.
|
||||||
|
|
||||||
|
| Level | Required support | Permitted verbs |
|
||||||
|
|---|---|---|
|
||||||
|
| 0: Intended | Plan or prompt only | "plans," "targets," "proposes" |
|
||||||
|
| 1: Implemented | Reviewed code at an immutable commit | "contains," "implements," "configures" |
|
||||||
|
| 2: Tested | Deterministic automated test passes from a controlled checkout | "tests demonstrate," with scope |
|
||||||
|
| 3: Measured | Retained representative artifacts and preregistered metrics pass | "measured," with numbers and uncertainty |
|
||||||
|
| 4: Independently reproduced | Separate reviewer/environment recreates the result | "reproduced on ..." |
|
||||||
|
| 5: Production qualified | Every applicable enterprise gate passes and operations approve | "production-qualified for [declared scope]" |
|
||||||
|
|
||||||
|
Reject adjectives without a level and scope. Replace "fast" with throughput/latency under a
|
||||||
|
named workload; replace "high quality" with rubric scores and confidence intervals; replace
|
||||||
|
"secure" with the verified control set and unresolved risk count.
|
||||||
|
|
||||||
|
## Gate High-Risk Claims
|
||||||
|
|
||||||
|
Build the evidence listed below before using the claim in a paper, release, presentation, or
|
||||||
|
README. A result can satisfy one row without satisfying another.
|
||||||
|
|
||||||
|
| Claim | Minimum evidence package |
|
||||||
|
|---|---|
|
||||||
|
| **Production-ready output** | Immutable source rights; no placeholder assets; model/asset provenance; approved director plan; measured media QA; blinded human review; failure/recovery results; retained final, manifest, logs, config, and checksums. |
|
||||||
|
| **Cinematic quality** | Preregistered category-balanced dataset; baseline outputs; randomized blinded review by declared reviewers; shot relevance, narrative coherence, pacing, visual finish, speech intelligibility, music/SFX fit, mix quality, and overall preference metrics; inter-rater agreement; confidence intervals; failure examples; technical QA. |
|
||||||
|
| **Local models in runtime** | Image/package inventory; exact model IDs, weight hashes, licenses, paths, memory/accelerator needs; startup/run traces showing local processes; egress-denied success; no host-only fallback; no runtime installation or model resolution. |
|
||||||
|
| **Offline** | Clean environment with network namespace/egress deny; preloaded approved bundle; packet/connection evidence; cold-start plus full source-to-final run; proof that caches were not inherited; failure if an undeclared artifact is removed. |
|
||||||
|
| **Licensed assets** | Source, owner, SPDX or verbatim license, commercial/derivative rights, territory/expiry if applicable, file hash, attribution obligation, and approval for every source, font, LUT, music, SFX, voice, model code, and weight. Generated output also requires model/weight/output-use analysis. |
|
||||||
|
| **Secure** | Threat model; authentication/authorization tests; OWASP ASVS 5.0 scope and evidence; secret/static/dependency/container scans; zero unresolved critical/high findings or recorded risk acceptance; least-privilege runtime; data/log redaction; audit evidence; secured actuator surface. |
|
||||||
|
| **Scalable** | Documented representative workload and hardware; p50/p95/p99, throughput, error rate, CPU, memory, GC, disk, threads, pools, queue age, and model saturation; 60-minute endurance; 2x peak; slowed-dependency test; horizontal-scale evidence; bottleneck and capacity limits. Do not inherit the user's example API numbers without adapting them to media workloads. |
|
||||||
|
| **Reproducible** | Clean checkout; one documented command; locked Java/Maven/Python/native/model inputs; checksums; no network for model/media work; macOS and Linux/VPS runs; one platform-neutral application artifact plus signed target-specific runtime/model/image bundles promoted without rebuild; deterministic seeds where applicable; declared numeric/media tolerance; independent reproduction. |
|
||||||
|
| **SOTA** | Approved primary-source review with dated search protocol; strongest relevant public baseline reproduced or obtained from auditable artifacts; same rights-cleared dataset and budget; preregistered metrics; uncertainty/significance; ablations; negative results; external review. |
|
||||||
|
| **Fortune 500 reference architecture** | Entire definition of done passes. Score all 15 human-review categories 0-4 with concrete evidence: architecture, domain, maintainability, security, testing, API, data, resilience, observability, performance, cloud operation, CI/CD, developer experience, documentation, and operations. No score below 3; average at least 3.5; security, data, testing, and operations each at least 3. |
|
||||||
|
|
||||||
|
For the Fortune 500 claim, also require clean build/run, architecture gates, API contract,
|
||||||
|
database migration from empty PostgreSQL, logs/metrics/traces/health/dashboards, load and
|
||||||
|
resilience results, secure container evidence, deployment and rollback exercises, ADRs, and
|
||||||
|
zero critical behavior dependent on undocumented assumptions.
|
||||||
|
|
||||||
|
## Build A Defensible Cinematic Comparison
|
||||||
|
|
||||||
|
Do not choose only flattering footage. Freeze the evaluation before rendering.
|
||||||
|
|
||||||
|
1. Define intended users, distribution format, input constraints, output duration, content
|
||||||
|
categories, hardware budget, latency budget, and prohibited content.
|
||||||
|
2. Create a rights-cleared, immutable, checksum-indexed dataset covering family, food, car,
|
||||||
|
generic, low light, sparse events, noisy audio, no audio, long takes, and failure cases.
|
||||||
|
3. Freeze these internal baselines where applicable:
|
||||||
|
- Source excerpt or uniform-window selection: measures whether highlight identification adds value.
|
||||||
|
- Heuristic visual provider: measures the local CV model's contribution.
|
||||||
|
- Deterministic plan: measures the director's contribution.
|
||||||
|
- Source-audio-only render: measures generated or licensed audio assets.
|
||||||
|
- Historical `5d889b0` ("working version but not cinematic"): use only after its runtime and
|
||||||
|
exact outputs are reproducibly reconstructed.
|
||||||
|
4. Add an external baseline only after approved primary evidence and artifacts are vendored or
|
||||||
|
cited. Network access is prohibited during this workflow; do not rely on memory for current
|
||||||
|
products, versions, papers, benchmarks, or licenses.
|
||||||
|
5. Match source clips, output duration, compute budget, allowed assets, and human-review process.
|
||||||
|
6. Preregister metrics, thresholds, exclusions, stopping rule, statistical method, and all
|
||||||
|
hypotheses. Predict expected numbers before execution under
|
||||||
|
`video-editing-research-methodology`.
|
||||||
|
7. Randomize and blind human review. Hide system names, filenames, and render order. Retain raw
|
||||||
|
scores and rejected samples.
|
||||||
|
8. Run ablations: remove local CV, director, generated music, SFX, voiceover, and each visual
|
||||||
|
treatment separately. Require one mechanism to explain positive and negative observations.
|
||||||
|
9. Report distributions and confidence intervals, not only averages or a montage of best cases.
|
||||||
|
10. Send the result through `video-editing-validation-and-qa`, then change control, before
|
||||||
|
promotion.
|
||||||
|
|
||||||
|
No cinematic acceptance threshold is currently agreed or evidenced. Keep the claim at
|
||||||
|
"candidate" until the campaign establishes thresholds and passes them.
|
||||||
|
|
||||||
|
## Record Artifact, Model, And License Provenance
|
||||||
|
|
||||||
|
Create one immutable manifest per evidence package. Store it in the approved evidence location
|
||||||
|
defined by change control; do not silently add repository paths from this skill.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
claim_id: <stable-id>
|
||||||
|
claim_text: <bounded proposed wording>
|
||||||
|
claim_level: <0-5>
|
||||||
|
commit: <40-character-sha>
|
||||||
|
created_utc: <timestamp>
|
||||||
|
owner: <team-or-role>
|
||||||
|
independent_reviewer: <team-or-role>
|
||||||
|
environments:
|
||||||
|
- os: <macOS-or-Linux-distribution-and-version>
|
||||||
|
architecture: <arm64-or-amd64>
|
||||||
|
cpu: <model-and-count>
|
||||||
|
memory_bytes: <integer>
|
||||||
|
accelerator: <model-driver-runtime-or-none>
|
||||||
|
commands:
|
||||||
|
- <exact-command>
|
||||||
|
network_policy: <egress-denied-evidence-reference>
|
||||||
|
inputs:
|
||||||
|
- path: <logical-artifact-name>
|
||||||
|
sha256: <hex>
|
||||||
|
source: <approved-origin>
|
||||||
|
license: <SPDX-or-verbatim-reference>
|
||||||
|
models:
|
||||||
|
- id: <model-and-revision>
|
||||||
|
weights_sha256: <hex>
|
||||||
|
code_license: <reference>
|
||||||
|
weights_license: <reference>
|
||||||
|
output_use_review: <approval-reference>
|
||||||
|
outputs:
|
||||||
|
- path: <logical-artifact-name>
|
||||||
|
sha256: <hex>
|
||||||
|
metrics: <raw-and-summary-result-reference>
|
||||||
|
negative_results: <retained-result-reference>
|
||||||
|
limitations: <known-boundaries>
|
||||||
|
decision: <rejected-candidate-or-approved>
|
||||||
|
```
|
||||||
|
|
||||||
|
Reject provenance that says only "local," "open source," "free," or a mutable model alias. A
|
||||||
|
cache hit is not proof that a model is bundled, approved, or available on a clean machine.
|
||||||
|
|
||||||
|
## Prove Cross-Environment Reproducibility
|
||||||
|
|
||||||
|
Use the same immutable application and model/asset bundle in all environments.
|
||||||
|
|
||||||
|
| Environment | Required proof |
|
||||||
|
|---|---|
|
||||||
|
| macOS development | Clean checkout; documented toolchain; no IDE dependency; egress denied; cold cache; source-to-final workflow; test and artifact logs. Record Apple/Intel architecture. |
|
||||||
|
| Linux/VPS production | Non-root runtime; explicit CPU/RAM/GPU/disk; egress denied; read-only filesystem except declared work volumes; signal/shutdown behavior; cold start; repeated render; recovery after restart. |
|
||||||
|
| Cloud | Same release artifact; externalized configuration/secrets; least privilege; readiness/liveness; resource limits; telemetry; rolling deploy and rollback; dependency outage and scale tests; no environment rebuild. |
|
||||||
|
|
||||||
|
Define reproducibility tolerance before execution. For media output, compare timeline decisions,
|
||||||
|
streams/codecs, duration, frames or perceptual hashes, loudness, clipping/silence, and human
|
||||||
|
scores as appropriate. Never promise byte identity across FFmpeg, codec, CPU, or accelerator
|
||||||
|
variants without proving it.
|
||||||
|
|
||||||
|
## Review Every Claim
|
||||||
|
|
||||||
|
Copy this checklist into the claim review:
|
||||||
|
|
||||||
|
- [ ] Quote the exact proposed sentence and intended audience.
|
||||||
|
- [ ] Bound the workflow, environment, content categories, commit, and date.
|
||||||
|
- [ ] Assign a claim-ladder level and an evidence owner.
|
||||||
|
- [ ] Separate repository facts, measurements, interpretations, and candidates.
|
||||||
|
- [ ] Name the fairest baseline and equalize inputs, compute, time, and rights.
|
||||||
|
- [ ] Preregister metrics, thresholds, hypotheses, exclusions, and stopping rules.
|
||||||
|
- [ ] Attach exact commands, raw logs, configs, immutable inputs/outputs, and SHA-256 hashes.
|
||||||
|
- [ ] Inventory every dependency, binary, model, weight, font, LUT, source, music, SFX, and voice license.
|
||||||
|
- [ ] Prove no automatic download, external service, network access, placeholder, or unapproved render occurred.
|
||||||
|
- [ ] Include negative results, known failure cases, uncertainty, and scope limits.
|
||||||
|
- [ ] Reproduce on every environment named by the claim.
|
||||||
|
- [ ] Obtain independent factual, security/license, validation, and change-control approval.
|
||||||
|
- [ ] Make release wording no stronger than the weakest required evidence item.
|
||||||
|
|
||||||
|
Reject the claim if any required box is unknown. "Unknown" is a result, not permission to infer.
|
||||||
|
|
||||||
|
## Write Release Notes Without Oversell
|
||||||
|
|
||||||
|
Use this template:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## <Release or experiment name>
|
||||||
|
|
||||||
|
Revision: `<commit>`
|
||||||
|
Status: <experimental | candidate | qualified for bounded scope>
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- <observable behavior, not aspiration>
|
||||||
|
|
||||||
|
### Verified
|
||||||
|
- <command, environment, dataset, measured result, threshold, evidence ID>
|
||||||
|
|
||||||
|
### Not verified
|
||||||
|
- <production, security, scale, offline, license, quality, or platform gaps>
|
||||||
|
|
||||||
|
### Models and assets
|
||||||
|
- <immutable IDs/hashes, licenses, approval, whether bundled>
|
||||||
|
|
||||||
|
### Compatibility and operations
|
||||||
|
- <configuration/API/data change, rollout, observability, rollback>
|
||||||
|
|
||||||
|
### Negative results and limitations
|
||||||
|
- <failures, excluded cases, uncertainty, known weak points>
|
||||||
|
|
||||||
|
### Claim decision
|
||||||
|
- Proposed wording: <exact sentence>
|
||||||
|
- Evidence level: <0-5>
|
||||||
|
- Reviewers: <roles>
|
||||||
|
- Decision: <rejected | candidate | approved for exact scope>
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not describe a plan, prompt, unchecked runtime path, synthetic fixture, or mocked test as an
|
||||||
|
end-user result. Link a release to retained evidence, not to a moving working directory.
|
||||||
|
|
||||||
|
## Route Claim Promotion
|
||||||
|
|
||||||
|
1. Open the candidate under `video-editing-research-methodology` when the claim depends on a
|
||||||
|
quality, novelty, mechanism, or comparative result.
|
||||||
|
2. Classify the implementation and any production-default impact under
|
||||||
|
`video-editing-change-control`. Never change defaults to make an evaluation pass.
|
||||||
|
3. Generate the complete evidence package without network access or prohibited fallbacks.
|
||||||
|
4. Ask `video-editing-validation-and-qa` for threshold and evidence review.
|
||||||
|
5. Assign an adversarial reviewer to reproduce the result and seek counterexamples.
|
||||||
|
6. Run security/license review for any public, customer, or commercial use.
|
||||||
|
7. Approve only the exact bounded wording. Record rejected wording and the reason.
|
||||||
|
8. Publish limitations and negative results beside the claim.
|
||||||
|
9. Reopen the claim when dependencies, models, datasets, runtime environments, defaults, or
|
||||||
|
relevant external baselines change.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
This skill was grounded on repository revision `97ba827d507292e38387c188a4b88157fd119ddb`
|
||||||
|
and the project owner's criteria on **2026-07-21**. Primary repository evidence included
|
||||||
|
`pom.xml`, `src/main/resources/application.yml`, local worker scripts, editing/rendering code,
|
||||||
|
tests, operational documents, highlight plans/checklists, and Git history. No network research
|
||||||
|
was performed; current external SOTA, version support, vulnerabilities, and licenses remain
|
||||||
|
unverified.
|
||||||
|
|
||||||
|
Re-verify volatile facts before every claim review:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git rev-parse HEAD
|
||||||
|
sed -n '1,260p' pom.xml
|
||||||
|
sed -n '1,180p' src/main/resources/application.yml
|
||||||
|
rg -n "pip install|get_pretrained|write_silence|fallback_tone|fallbackTone|writeSilence" tools src/main/java/org/example/videoclips/editing
|
||||||
|
rg -n "buildQaReport|new RenderQaCheck" src/main/java/org/example/videoclips/editing
|
||||||
|
find . -maxdepth 3 -type f \( -name 'mvnw*' -o -iname '*docker*' -o -iname '*compose*' -o -iname '*openapi*' -o -iname '*sbom*' \) -print
|
||||||
|
find . -maxdepth 3 -type f \( -path './.github/*' -o -name '.gitlab-ci.yml' -o -name 'Jenkinsfile' \) -print
|
||||||
|
rg -n "spring-security|SecurityFilterChain|ArchUnit|Modulith|Testcontainers|CycloneDX|dependency-check|OpenTelemetry" pom.xml src
|
||||||
|
sed -n '120,190p' docs/load-test-signoff.md
|
||||||
|
git log --oneline --all --decorate -80
|
||||||
|
```
|
||||||
|
|
||||||
|
Run build, media, security, load, offline, and cross-platform commands only through their
|
||||||
|
sibling runbooks. Retain their logs and hashes; a command printed here is not evidence that it
|
||||||
|
passed.
|
||||||
|
|
@ -0,0 +1,388 @@
|
||||||
|
---
|
||||||
|
name: video-editing-failure-archaeology
|
||||||
|
description: Load this skill when investigating whether a video-editing failure, tempting fix, stale generated artifact, completed checklist item, deleted test, or unusual Git object has already been encountered in this repository. Use it before reopening storage, queue/DLQ, local-file cleanup, FFmpeg segment timing, cinematic-quality, local-model asset, port/configuration, or single-source highlight incidents, and before treating historical output as proof.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Failure Archaeology
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Reconstruct what the repository actually proves before changing behavior. Treat a **root cause** as the mechanism demonstrated by code, a patch, a test, or a reproducible observation. Treat a commit subject, plan checkbox, and generated media as leads rather than proof.
|
||||||
|
|
||||||
|
Facts and workspace observations in this skill were checked on **2026-07-21**.
|
||||||
|
|
||||||
|
## Use This Skill When
|
||||||
|
|
||||||
|
- A symptom resembles an earlier storage, queue, filesystem, FFmpeg, render, or local-model problem.
|
||||||
|
- A plan says an item is complete but production evidence is unclear.
|
||||||
|
- An ignored or committed runtime artifact appears to prove a workflow works.
|
||||||
|
- Git history contains a deletion, rewrite, unreachable commit, or apparently abandoned approach.
|
||||||
|
- A proposed fix repeats an earlier partial fix.
|
||||||
|
|
||||||
|
## Do Not Use This Skill When
|
||||||
|
|
||||||
|
- Triaging a live symptom from scratch: use `video-editing-debugging-playbook` first, then return here after identifying the subsystem.
|
||||||
|
- Changing code, configuration, dependencies, defaults, or runtime behavior: use `video-editing-change-control`; this skill never authorizes a change.
|
||||||
|
- Deciding whether output meets acceptance thresholds: use `video-editing-validation-and-qa`.
|
||||||
|
- Looking up current defaults or environment wiring: use `video-editing-config-and-flags`.
|
||||||
|
- Operating or recovering a deployed instance: use `video-editing-run-and-operate`.
|
||||||
|
|
||||||
|
## Non-Negotiable Investigation Rules
|
||||||
|
|
||||||
|
1. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh` during archaeology. They can create virtual environments, install packages, and download model weights.
|
||||||
|
2. Do not use network access, external AI services, unlicensed assets, placeholder silence or tones, or unapproved rendering to reproduce a result.
|
||||||
|
3. Do not change a production-facing default to test a theory.
|
||||||
|
4. Do not mutate Git history. Use `git log`, `git show`, `git diff`, and `git fsck`; never reset, rebase, clean, or delete artifacts.
|
||||||
|
5. Do not promote ignored media, a manifest, or a green structural test as cinematic-quality evidence.
|
||||||
|
6. Route every behavior change through `video-editing-change-control`, including a retry change, port change, fallback change, or FFmpeg flag change.
|
||||||
|
|
||||||
|
## Evidence Labels
|
||||||
|
|
||||||
|
Use these labels in incident notes and pull requests:
|
||||||
|
|
||||||
|
| Label | Meaning | Required evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| `FIXED` | The demonstrated mechanism was changed and regression coverage exists. | Patch plus current code/test location. |
|
||||||
|
| `ACTIVE` | The mechanism remains in current code or reproducibly fails. | Current source or dated experiment. |
|
||||||
|
| `PARTIAL` | One failure mode was fixed, but adjacent obligations remain. | Explicit boundary and remaining gap. |
|
||||||
|
| `HISTORICAL` | The evidence describes an earlier tree, not necessarily current behavior. | Commit hash and date. |
|
||||||
|
| `OPEN` | Evidence shows a concern but not its cause or intent. | Known facts and the missing discriminator. |
|
||||||
|
| `NOT EVIDENCE` | The artifact cannot support the claimed conclusion. | Reason it is non-representative or unverified. |
|
||||||
|
|
||||||
|
## Fast History Triage
|
||||||
|
|
||||||
|
Run read-only checks from the repository root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short --ignored
|
||||||
|
git log --oneline --decorate --all --reverse
|
||||||
|
git log --all --name-status -- src/main src/test docs
|
||||||
|
git fsck --no-reflogs --unreachable --no-progress
|
||||||
|
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|open question|fallback|silence|tone' docs src tools
|
||||||
|
```
|
||||||
|
|
||||||
|
Before accepting a historical claim, inspect the exact patch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export COMMIT="${COMMIT:?set COMMIT to a full commit ID}"
|
||||||
|
export FILE_PATH="${FILE_PATH:?set FILE_PATH to a repository-relative path}"
|
||||||
|
export OLDER_COMMIT="${OLDER_COMMIT:?set OLDER_COMMIT to a full commit ID}"
|
||||||
|
export NEWER_COMMIT="${NEWER_COMMIT:?set NEWER_COMMIT to a full commit ID}"
|
||||||
|
git show --format=fuller --stat "$COMMIT"
|
||||||
|
git show --format=fuller --no-ext-diff "$COMMIT" -- "$FILE_PATH"
|
||||||
|
git diff "$OLDER_COMMIT" "$NEWER_COMMIT" -- "$FILE_PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Chronicle At A Glance
|
||||||
|
|
||||||
|
| Date | Symptom or risk | Demonstrated mechanism | Status on 2026-07-21 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 2026-07-08/09 | Upload completion and clips existed as state/metadata without a complete storage path. | Storage operations and persisted object keys were absent from earlier paths. | `FIXED` for the implemented adapters; production S3 proof remains separate. |
|
||||||
|
| 2026-07-09 | Internal object locations leaked through normal API responses and logs. | `objectKey` and staged local path were returned/recorded directly. | `FIXED` by `0d45f89`; regression assertions remain. |
|
||||||
|
| 2026-07-09 | Worker files accumulated after success or failure. | Per-job staging/output lacked guaranteed deletion. | `PARTIAL`: finally cleanup and scheduled local cleanup exist; errors are swallowed and other editing trees are outside that job. |
|
||||||
|
| 2026-07-09 | Retryable DB-queue failure became terminal on the first exception. | `ClipProcessor` persisted `FAILED` before the queue decided retry versus DLQ. | `FIXED` by `1737d8b` for the DB queue contract. |
|
||||||
|
| 2026-07-10 | Folder clips could preserve quality or hit exact boundaries, but not both. | Stream copy cuts on source keyframes; exact cuts force keyframes by transcoding. | `ACTIVE` design trade-off; default selects preservation. |
|
||||||
|
| 2026-07-11 | Renderer worked but was explicitly “not cinematic.” | Exact cause was not recorded; one patch added a uniform grade, followed by broader planning/render/QA work. | `OPEN`: no certified creative-quality evidence. |
|
||||||
|
| 2026-07-12 onward | Single-source projects stopped at `CREATED` or `WAITING_FOR_DIRECTOR`. | Local artifacts lack a director plan/final output; current scanner requires `director/edit-plan.json`. | `ACTIVE` missing local-director capability; artifacts alone do not prove a regression. |
|
||||||
|
| 2026-07-21 | Workspace build passed while clean archived checkout failed local-asset tests. | Tests invoke the default untracked Python runtime and real host fallbacks instead of a hermetic fake. | `ACTIVE`; clean-checkout reproducibility is not established. |
|
||||||
|
|
||||||
|
## Case 1: Metadata-Only Storage Grew Into A Real Data Path
|
||||||
|
|
||||||
|
**Symptom.** API state could say upload or clip work completed while the service did not yet prove a durable source-to-generated-object path.
|
||||||
|
|
||||||
|
**Demonstrated mechanism.** The sequence is visible in the patches:
|
||||||
|
|
||||||
|
| Commit | Added missing responsibility |
|
||||||
|
|---|---|
|
||||||
|
| `7e8a214` | `uploads:complete` called `ObjectStoragePort.completeMultipartUpload` instead of only changing metadata. |
|
||||||
|
| `9df9736` | `VideoAsset.sourceObjectKey` was persisted and passed to clipping. |
|
||||||
|
| `bd620fc` | Source objects were materialized into local staging before FFmpeg. |
|
||||||
|
| `66e998e` | Generated clips carried local paths, were uploaded, and persisted an `objectKey`. |
|
||||||
|
| `7307082` | Signed download URLs used the persisted clip key instead of deriving a fake key from `clipId`. |
|
||||||
|
|
||||||
|
**Rejected partial fixes.** Do not reintroduce any of these:
|
||||||
|
|
||||||
|
- Marking upload complete without completing the provider upload.
|
||||||
|
- Saving clip rows before uploading the generated files.
|
||||||
|
- Deriving storage keys from public IDs when the persisted key is authoritative.
|
||||||
|
- Treating the in-memory adapter's no-op completion as proof that S3 completion works.
|
||||||
|
|
||||||
|
**Current invariant.** `ClipProcessor` must follow materialize source -> generate local clips -> upload each clip -> save each clip record -> mark the job successful. `ObjectStoragePort` owns materialization, upload, delete, and signed-download operations.
|
||||||
|
|
||||||
|
**Discriminate metadata drift from storage failure.** Read the job events and compare the clip record with backend existence. Do not expose the key in a client response merely to debug it. Inspect code with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show --stat 7e8a214 9df9736 bd620fc 66e998e 7307082
|
||||||
|
rg -n 'materializeSourceObject|uploadGeneratedClip|createClipDownloadUrl' src/main/java
|
||||||
|
rg -n 'source_object_key|object_key' src/main/resources/db/migration
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `FIXED` at the code-contract level. No checked-in production S3 end-to-end evidence makes the external system production-certified.
|
||||||
|
|
||||||
|
## Case 2: Object-Key And Local-Path Disclosure
|
||||||
|
|
||||||
|
**Symptom.** Normal API payloads exposed `sourceObjectKey`/clip `objectKey`, and a processing event included the staged local path.
|
||||||
|
|
||||||
|
**Root cause.** Commit `66e998e` initially added `objectKey` to `toClipResponse`; earlier asset mapping also exposed the source key. Internal routing identifiers crossed the API boundary. The materialization event also included a filesystem path.
|
||||||
|
|
||||||
|
**Fix.** Commit `0d45f89` removed those fields from normal responses and removed the path from the event message. Current assertions in `VideoAssetControllerTest` require `sourceObjectKey` and clip `objectKey` not to exist. Clients obtain a signed URL through the download endpoint.
|
||||||
|
|
||||||
|
**Wrong path.** Do not return a key “temporarily,” put it in Problem Details, or log the full local path. Diagnose through authorized backend tooling and redacted correlation identifiers.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show --no-ext-diff 0d45f89 -- src/main/java src/test/java
|
||||||
|
rg -n 'sourceObjectKey.*doesNotExist|objectKey.*doesNotExist' src/test/java/org/example/videoclips/api
|
||||||
|
rg -n 'toAssetResponse|toClipResponse' src/main/java/org/example/videoclips/application/VideoAssetService.java
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `FIXED` for the tested response paths. Any new DTO, log, manifest, or error mapping must repeat the disclosure review through change control.
|
||||||
|
|
||||||
|
## Case 3: Retry And DLQ State Split
|
||||||
|
|
||||||
|
**Symptom.** A transient DB-queue processing exception marked the clip job `FAILED` on the first attempt even though the queue intended to retry it.
|
||||||
|
|
||||||
|
**Root cause.** The processor owned terminal failure persistence while the queue adapter owned attempt policy. The lower layer finalized state before the policy layer knew whether the attempt was terminal.
|
||||||
|
|
||||||
|
**Fix.** Commit `1737d8b` added `processNow(jobId, false)` for DB-queue attempts. The queue adapter now increments attempts, schedules a non-terminal `PENDING` retry with `RETRY_SCHEDULED`, and calls `markTerminalFailure` only when `max-attempts` is reached; terminal events are `DLQ` and `FAILED`.
|
||||||
|
|
||||||
|
**Do not repeat.** Do not mark a job failed in an adapter-independent catch block when a caller owns retry policy. Do not blindly redrive DLQ rows. `docs/dlq-redrive-procedure.md` requires root-cause correction, source existence, a small sample, and transactional row changes; the repository has no built-in redrive command.
|
||||||
|
|
||||||
|
**Discriminating observations.** Before the last attempt, expect message `PENDING`, job `QUEUED`, and a `RETRY_SCHEDULED` event. At exhaustion, expect message `DLQ`, job `FAILED`, then `DLQ` and `FAILED` events. A job already `FAILED` while its message is retryable indicates the old split-brain symptom or another writer.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show --no-ext-diff 1737d8b -- src/main/java/org/example/videoclips/processing/ClipProcessor.java src/main/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java
|
||||||
|
rg -n 'RETRY_SCHEDULED|markTerminalFailure|setStatus\("DLQ"\)' src/main/java src/test/java
|
||||||
|
sed -n '1,320p' docs/dlq-redrive-procedure.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `FIXED` for the database-backed queue. The memory queue and any future external queue require their own semantics and tests.
|
||||||
|
|
||||||
|
## Case 4: Worker Temporary-File Accumulation
|
||||||
|
|
||||||
|
**Symptom.** Staged sources and generated local clips accumulated across successful and failed jobs.
|
||||||
|
|
||||||
|
**Root cause.** The processing path had no guaranteed cleanup boundary.
|
||||||
|
|
||||||
|
**Fixes.** Commit `8f4804c` put job-local deletion in `ClipProcessor.finally`, gated by `video-clipping.ffmpeg.cleanup-local-files`. Commit `3d961f2` added scheduled age-based cleanup for FFmpeg input/output, in-memory storage, and stub output.
|
||||||
|
|
||||||
|
**Limitations.** `LocalArtifactCleanupJob` catches and ignores `IOException`; deletion failure can be silent. Its roots do not include the cinematic edit and highlight project trees. Retention of durable storage/database records belongs to `RetentionCleanupJob`, not this local sweep.
|
||||||
|
|
||||||
|
**Wrong path.** Do not disable cleanup to make a failing test inspectable in production. Preserve a controlled diagnostic copy outside the worker path under an approved procedure.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show --stat 8f4804c 3d961f2
|
||||||
|
rg -n 'cleanupLocalFiles|cleanupLocalArtifacts|local-artifact-retention' src/main/java src/main/resources src/test/java
|
||||||
|
du -sh tmp input output 2>/dev/null
|
||||||
|
find tmp -type f -mtime +1 -print 2>/dev/null | sed -n '1,100p'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `PARTIAL`: job-local cleanup and a safety sweep exist, but cleanup observability and all editing artifact retention remain open.
|
||||||
|
|
||||||
|
## Case 5: Preserve Quality Versus Exact Segment Timing
|
||||||
|
|
||||||
|
**Symptom.** An “8-second clip” expectation conflicts with clips cut at nearby keyframes.
|
||||||
|
|
||||||
|
**Mechanism.** `FolderFfmpegClipper` has two mutually exclusive command shapes:
|
||||||
|
|
||||||
|
| `preserve-input-quality` | FFmpeg mode | Consequence |
|
||||||
|
|---|---|---|
|
||||||
|
| `true` (packaged default) | `-c copy` plus segment muxer | No generation loss; boundaries can drift to source keyframes. |
|
||||||
|
| `false` | `libx264`, CRF 20, forced keyframes, AAC | More exact segment boundaries; re-encode cost and generation loss. |
|
||||||
|
|
||||||
|
The trade-off is documented in `docs/input-folder-scheduler-plan.md`. The plan still asks whether exact timing or keyframe alignment is required, so “exact” is not a settled product requirement for this workflow.
|
||||||
|
|
||||||
|
**Wrong paths.** Do not call stream-copy output exact. Do not switch the default to transcoding to satisfy one fixture. Do not infer visual preservation solely from container/codec names.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'preserve-input-quality|force_key_frames|segment_time|qualityMode' src/main docs/input-folder-scheduler-plan.md
|
||||||
|
git show --no-ext-diff aaacd7f -- src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `ACTIVE` intentional trade-off. Any default or acceptance-threshold decision is a behavior change and must pass change control and measured validation.
|
||||||
|
|
||||||
|
## Case 6: “Working Version But Not Cinematic”
|
||||||
|
|
||||||
|
**Symptom.** Commit `5d889b0` records the only direct historical judgment: the render worked but was not cinematic.
|
||||||
|
|
||||||
|
**What the patch proves.** That commit added a fixed contrast/saturation/sharpen/vignette filter for non-empty visual treatments, plus compatibility/test wiring. It does not record a controlled comparison, reviewer rubric, or measured root cause.
|
||||||
|
|
||||||
|
**What followed.** Later commits added category-aware planning (`5006dfd`), stricter plan validation (`3348ae8`), local asset selection (`376bc41`), overlays (`43fccb7`), audio mastering (`bb23ac8`), dynamic crops (`2d0c331`), QA reports (`9b56e89`), and approval gating (`9536928`). This sequence identifies areas the project invested in; it does not prove each was causally necessary or that current output is cinematic.
|
||||||
|
|
||||||
|
**Rejected explanation.** “It needed a LUT/grade” is not established. A uniform grade was the immediate change, yet the repository then required planning, sound, motion, QA, and approval work.
|
||||||
|
|
||||||
|
**Current status.** `OPEN`. Structural render success, `qa-report.json`, and a committed manifest do not establish creative quality. Use the cinematic campaign and `video-editing-validation-and-qa`; require a licensed asset inventory, local-model provenance, objective media probes, and scored human review. Never fill missing audio with silence or tones.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show --no-ext-diff 5d889b0 -- docs src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
|
||||||
|
git log --oneline 5d889b0..9536928
|
||||||
|
git show --stat 5006dfd 3348ae8 376bc41 43fccb7 bb23ac8 2d0c331 9b56e89 9536928
|
||||||
|
```
|
||||||
|
|
||||||
|
## Case 7: Single-Source Highlight Projects Stalled
|
||||||
|
|
||||||
|
**Observed on 2026-07-21.** Ignored workspace projects under `output/highlight-projects/` include one `CREATED` project and one `WAITING_FOR_DIRECTOR` project from 2026-07-12. Neither contains a final output. The later project contains analysis and a director prompt but no `director/edit-plan.json`.
|
||||||
|
|
||||||
|
**Separate stale-artifact trap.** Git tracks `output/edit-projects/source-clips/project.json` and `render-manifest.json`, which claim `RENDERED` and refer to `final.mp4`; Git does not track that media file. A clean checkout therefore contains success metadata without the claimed output. The `/input/`, `/output/`, and `/tmp/` ignore rules live in clone-local `.git/info/exclude`, not a shared repository ignore file. Do not use either the tracked manifest or this clone's ignored media as certified evidence.
|
||||||
|
|
||||||
|
**Current gate.** `HighlightDirectorPlanScanner` only selects a project with `director/edit-plan.json` and without `final.mp4`. `HighlightDirectorFlowService` returns `missing_director_plan` without that file and may also return `approval_missing` or `assets_pending`.
|
||||||
|
|
||||||
|
**Evidence boundary.** These projects were created after `618ba9a` and before `97ba827`. Their state is consistent with the documented missing-plan gate, so it is not proof that the flow implementation regressed. Current code generates a prompt and watches for a plan file; no runtime local-director model consumes the prompt and writes that plan. Their `manifest.json` also declares analysis filenames that do not exactly match every generated filename. Treat them as diagnostic snapshots, not goldens.
|
||||||
|
|
||||||
|
**Discriminate the gate without rendering.** Inspect only metadata and file presence:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find output/highlight-projects -maxdepth 3 -type f -print 2>/dev/null | sort
|
||||||
|
find output/highlight-projects -name project.json -exec sed -n '1,100p' {} \; 2>/dev/null
|
||||||
|
find output/highlight-projects -path '*/director/edit-plan.json' -o -name final.mp4 2>/dev/null
|
||||||
|
rg -n 'missing_director_plan|approval_missing|assets_pending|findNextRenderableProject' src/main/java/org/example/videoclips/editing
|
||||||
|
git ls-files output input tmp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** Artifact state is `HISTORICAL`; the absence of an in-runtime local director is `ACTIVE`, and end-to-end production readiness is `OPEN`. Reproduce only with approved, licensed local inputs and pre-provisioned local models. Do not let startup download anything or render without approval.
|
||||||
|
|
||||||
|
## Case 8: Local Asset Build Is Not Clean-Checkout Hermetic
|
||||||
|
|
||||||
|
**Observed on 2026-07-21.** The workspace suite passed, and its Surefire report shows `LocalAssetGenerationStageTest` invoked `./.venv-local-asset/bin/python`. That virtual environment exists locally but is untracked. A valid run from inside a clean archived checkout ran 216 tests and failed one assertion, `reusesExistingSharedAssetsAndSynthesizesMissingVoiceover`. An earlier two-failure run used `mvn -f` from the workspace and is invalid clean-checkout evidence because it violated the test working-directory assumption.
|
||||||
|
|
||||||
|
**Root cause.** The archived tests constructed the production `LocalAssetSynthesizer` with default configuration. They expected missing voiceover/SFX to become render-ready but did not provide a fake process. The default Python path pointed into the untracked virtual environment, host `say`/`espeak` availability affected voiceover, and the former worker could return success after writing silence or tones. **Status 2026-07-21:** the working tree removed those success fallbacks, deletes failed/inaudible output, and makes the affected test deterministic and fail-closed.
|
||||||
|
|
||||||
|
**Consequences.** The historical workspace pass was environment-coupled. The 2026-07-21 working tree removes placeholder success and makes strict readiness fail, but the bootstrap script can still install dependencies automatically and no complete resident model bundle is present. Re-run the clean archive before declaring the incident closed.
|
||||||
|
|
||||||
|
**Wrong paths.** Do not commit the virtual environment, enable network during tests, weaken strict mode, accept placeholder audio, or skip the test. Make tests hermetic and make production readiness fail closed through change control.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git ls-files .venv-local-asset tools/local_asset_worker.py tools/run_local_asset_worker.sh
|
||||||
|
rg -n 'new LocalAssetSynthesizer|readyForRender' src/test/java/org/example/videoclips/editing/LocalAssetGenerationStageTest.java
|
||||||
|
rg -n 'write_silence|write_fallback_tone|strict_runtime|strict-runtime|pip install' tools src/main src/main/resources
|
||||||
|
sed -n '1,120p' target/surefire-reports/org.example.videoclips.editing.LocalAssetGenerationStageTest.txt 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `ACTIVE`, release-blocking under the clean-checkout and no-placeholder requirements. Do not re-run bootstrap as a “fix.”
|
||||||
|
|
||||||
|
## Case 9: Port And Runtime-Shape Drift
|
||||||
|
|
||||||
|
**Observed configuration.** API examples assume Spring Boot's default port `8080`. The local CV HTTP worker defaults to `127.0.0.1:8091`, and the process manager derives host/port from its configured endpoint. `tools/run_local_asset_worker.sh` defines `LOCAL_ASSET_HOST` and `LOCAL_ASSET_PORT` defaulting to `8092`, but it launches a one-shot CLI script, not an HTTP server; those variables are unused by `local_asset_worker.py`.
|
||||||
|
|
||||||
|
**Interpretation.** This is verified semantic drift, not a documented outage. Do not diagnose local asset generation by probing port 8092, and do not “fix” it by moving another service to that port. First identify whether the component is HTTP (`local-cv`) or process-invoked (`local-asset`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n '8080|8091|8092|LOCAL_CV_PORT|LOCAL_ASSET_PORT|server.port' docs src tools
|
||||||
|
lsof -nP -iTCP -sTCP:LISTEN | rg ':(8080|8091|8092)\b'
|
||||||
|
rg -n 'ProcessBuilder|local_asset_worker.py|uvicorn' src/main/java/org/example/videoclips/editing tools
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `ACTIVE` documentation/configuration debt. No causal history establishes that a port collision caused a project failure.
|
||||||
|
|
||||||
|
## Case 10: Deleted Tenant Quota Test
|
||||||
|
|
||||||
|
**Evidence.** `TenantQuotaControllerTest` was added with tenant quota work and existed when `1737d8b` reported its pass. Commit `7fd92a7` deleted it while converting folder-scheduler configuration from properties to YAML. The commit message does not explain the deletion. A file named `input/source/TenantQuotaControllerTest.java.failed` is ignored local runtime debris, not the deleted test and not proof of intent.
|
||||||
|
|
||||||
|
**Conclusion.** Label the deletion `OPEN`. Do not claim the behavior was obsolete, flaky, or intentionally replaced unless a maintainer or equivalent coverage proves it.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log --all --follow --format='%h %ad %s' --date=iso-strict -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
|
||||||
|
git show 7fd92a7^:src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
|
||||||
|
git show --no-ext-diff 7fd92a7 -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
|
||||||
|
rg -n 'quota|QuotaExceeded' src/test src/main
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `OPEN` test-coverage ambiguity. Restoring or replacing coverage requires change control; the historical file is evidence for intended behavior, not automatically the correct present test.
|
||||||
|
|
||||||
|
## Case 11: Rewritten Commits, Not Dead Feature Branches
|
||||||
|
|
||||||
|
**Observed on 2026-07-21.** `git fsck` reports six unreachable commits around the signed-download and disclosure fixes: `19ef4d3`, `5919210`, `65a29a4`, `84e6694`, `eb06021`, and `ba3f988`. Several share subjects, trees, parents, or near-identical changes with reachable `7307082` and `0d45f89`.
|
||||||
|
|
||||||
|
**Conclusion.** The evidence supports local amend/rewrite debris. It does not support a story about a reverted architecture or abandoned branch. The reachable history contains no commit whose subject is an explicit revert and only the main branch is present in the inspected repository.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git fsck --no-reflogs --unreachable --no-progress
|
||||||
|
git show -s --format='%H%nparents=%P%ntree=%T%nauthor=%ad%nsubject=%s%n' --date=iso-strict <unreachable-commit>
|
||||||
|
git branch -a -vv
|
||||||
|
git log --all --oneline --grep='^Revert'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `NOT EVIDENCE` for a rejected solution. If unreachable objects disappear after Git maintenance, this conclusion should rely on the recorded hashes here and reachable patches, not object availability.
|
||||||
|
|
||||||
|
## Case 12: Completed Plans Are Not Production Signoff
|
||||||
|
|
||||||
|
**Symptom.** `docs/video-clipping-service-implementation-plan.md` marks every repository-scoped milestone complete, including backup/restore validation and load-test signoff.
|
||||||
|
|
||||||
|
**What the docs actually say.** The same plan says real backup/restore and production-like load execution remain. `docs/load-test-signoff.md` says `Documentation complete` and `Execution evidence pending`; it explicitly says no checked-in end-to-end environment load evidence exists. The plan's recommended baseline also differs from current `pom.xml` and the current Fortune 500 modular-monolith objective.
|
||||||
|
|
||||||
|
**Wrong path.** Do not translate `[x]` into deployed, exercised, secure, scalable, or production-ready. A document, dashboard JSON, benchmark harness, or procedure proves only its own existence until execution evidence is attached.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '688,760p' docs/video-clipping-service-implementation-plan.md
|
||||||
|
sed -n '28,180p' docs/load-test-signoff.md
|
||||||
|
rg -n 'not prove|pending|not executed|still must|template' docs/backup-and-restore-validation.md docs/load-test-signoff.md docs/dlq-redrive-procedure.md
|
||||||
|
rg -n 'spring-boot-starter-parent|<java.version>' pom.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status.** `ACTIVE` evidence-labeling risk. Use current code/configuration and dated execution records as truth; route documentation corrections through change control.
|
||||||
|
|
||||||
|
## Investigation Record Template
|
||||||
|
|
||||||
|
Use this structure in a new incident note or pull request:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Symptom:
|
||||||
|
First known good / first known bad:
|
||||||
|
Scope and environment:
|
||||||
|
Evidence label: FIXED | ACTIVE | PARTIAL | HISTORICAL | OPEN | NOT EVIDENCE
|
||||||
|
Candidate mechanism:
|
||||||
|
Prediction before experiment:
|
||||||
|
Read-only or approved experiment:
|
||||||
|
Observed result:
|
||||||
|
Negative observations the mechanism also explains:
|
||||||
|
Historical analog and commit:
|
||||||
|
Rejected paths and why:
|
||||||
|
Behavior/config/security consequences:
|
||||||
|
Change-control classification:
|
||||||
|
Validation required before promotion:
|
||||||
|
Remaining uncertainty:
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not fill the root-cause field until one mechanism explains the positive and negative observations. Record a disconfirming result; it prevents the next engineer from repeating the dead end.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Primary evidence: reachable Git history through `97ba827`, current source/tests/configuration, planning and operations documents, current `.git/info/exclude`, Surefire reports, and dated ignored workspace artifacts, all inspected on 2026-07-21. Workspace artifacts and unreachable Git objects are volatile and are never load-bearing proof.
|
||||||
|
|
||||||
|
Re-verify the reachable incident sequence:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log --oneline --all --reverse | rg '7e8a214|9df9736|bd620fc|66e998e|7307082|0d45f89|8f4804c|3d961f2|1737d8b|5d889b0'
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify current storage, queue, cleanup, and segmentation contracts:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'materializeSourceObject|uploadGeneratedClip|RETRY_SCHEDULED|markTerminalFailure|cleanupLocalFiles|preserve-input-quality|force_key_frames' src/main src/test src/main/resources
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify local-model prohibitions and hermeticity risks without starting workers:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|strict-runtime|auto-start|fallback-to-heuristic' tools src/main src/test src/main/resources
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify volatile project artifacts and ignore rules:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git status --short --ignored
|
||||||
|
git ls-files input output tmp .venv-local-asset .venv-local-cv
|
||||||
|
sed -n '1,160p' .git/info/exclude
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify ambiguity around rewrites and the deleted test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git fsck --no-reflogs --unreachable --no-progress
|
||||||
|
git log --all --follow --name-status -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify documentation claims against their own caveats:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'All repository-scoped|Execution evidence pending|no checked-in evidence|Remaining work' docs
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,440 @@
|
||||||
|
---
|
||||||
|
name: video-editing-proof-and-analysis-toolkit
|
||||||
|
description: "Load this skill after symptom triage when a video-editing change or incident must be proved from first principles: state transitions, timeline or keyframe accuracy, FFmpeg filter/audio graphs, loudness and true peak, highlight-ranking precision/recall/normalized discounted cumulative gain (nDCG), local-model output versus fallback, queue retry/DLQ/idempotency, filesystem atomicity/path confinement, capacity/cost, reproducibility, or causal debugging. Use it to establish or refute a causal mechanism before accepting a benchmark, QA report, cinematic-quality claim, fallback, or production-readiness claim as evidence."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Proof And Analysis Toolkit
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Turn a claim into a prediction, a discriminating experiment, and an auditable conclusion. A **mechanism** is a causal account that explains the successful cases, failures, and negative controls with the same rules. A green test, an existing file, or a plausible story is not a mechanism.
|
||||||
|
|
||||||
|
Repository facts in this skill were verified on **2026-07-21**. Numeric baselines in `docs/` are historical workspace observations, not current acceptance thresholds.
|
||||||
|
|
||||||
|
## Use And Routing
|
||||||
|
|
||||||
|
Use this skill to derive what an observation should be and to distinguish competing causes. Use these siblings for the surrounding work:
|
||||||
|
|
||||||
|
| Need | Load instead or next |
|
||||||
|
|---|---|
|
||||||
|
| Classify, approve, implement, or promote a behavior change | `video-editing-change-control` |
|
||||||
|
| Triage a live symptom quickly | `video-editing-debugging-playbook` |
|
||||||
|
| Reconstruct an earlier incident or rejected fix | `video-editing-failure-archaeology` |
|
||||||
|
| Decide the complete acceptance suite or human review | `video-editing-validation-and-qa` |
|
||||||
|
| Interpret codec, color, audio, timing, or editing theory | `cinematic-media-engineering-reference` |
|
||||||
|
| Run approved local diagnostics | `video-editing-diagnostics-and-tooling` |
|
||||||
|
| Change flags or inspect effective configuration | `video-editing-config-and-flags` |
|
||||||
|
| Execute the hardest highlight-quality program | `video-editing-cinematic-highlights-campaign` |
|
||||||
|
|
||||||
|
Do **not** use this skill to authorize rendering, production mutation, DLQ redrive, dependency installation, or model acquisition. Analysis never routes around change control.
|
||||||
|
|
||||||
|
## Hard Safety Boundary
|
||||||
|
|
||||||
|
1. Work offline. Do not call external AI services or enable network access.
|
||||||
|
2. Use only pre-provisioned, licensed local models and licensed assets whose identity and checksum are recorded.
|
||||||
|
3. Do not invoke `tools/run_local_cv_worker.sh`, `tools/run_local_asset_worker.sh`, or application startup as an experiment. Current defaults can bootstrap runtimes, download packages/models, scan inputs, and render.
|
||||||
|
4. Reject placeholder silence and synthetic fallback tones as production assets. “A file exists” and “samples are nonzero” do not establish voice, music, or SFX validity.
|
||||||
|
5. Probe only existing approved media. Do not generate or render media without the approval gate in `video-editing-change-control`.
|
||||||
|
6. Never change a production-facing default to test a hypothesis. Use an isolated fixture and explicit test configuration after approval.
|
||||||
|
7. Keep diagnostic output in `target/` or an OS temporary directory. Never alter `input/`, `output/`, migrations, source media, or Git history.
|
||||||
|
8. Run Maven with `-o` during proof work so Maven fails on a missing cached dependency. Maven offline mode is not egress denial and does not prevent test code or subprocesses from networking; enforce network denial independently.
|
||||||
|
|
||||||
|
## The Proof Record
|
||||||
|
|
||||||
|
Write this record **before** running an experiment:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Claim:
|
||||||
|
Competing mechanisms (at least two):
|
||||||
|
Assumptions and controlled variables:
|
||||||
|
Predicted numeric/structural observation for each mechanism:
|
||||||
|
Positive control:
|
||||||
|
Negative control:
|
||||||
|
Command and input checksums:
|
||||||
|
Observed result and uncertainty:
|
||||||
|
Which mechanisms were falsified:
|
||||||
|
Adversarial refutation still required:
|
||||||
|
Decision: reject | retain as candidate | accept for stated scope
|
||||||
|
```
|
||||||
|
|
||||||
|
Apply these rules:
|
||||||
|
|
||||||
|
- Predict a number, interval, ordering, state tuple, or file set before execution. “It should improve” is invalid.
|
||||||
|
- Change one causal factor at a time. Use an **ablation**: remove one component while holding all others fixed.
|
||||||
|
- Preserve negative results. One mechanism must explain both the expected positive and the expected absence under the negative control.
|
||||||
|
- Separate **structural validity** (schema, file, command, state) from **semantic validity** (correct highlight, intelligible voice, appropriate sound) and **operational validity** (load, crash, recovery).
|
||||||
|
- Report the denominator, sample selection, hardware, binary/model checksums, and uncertainty. Never promote a single clip or a single warm run.
|
||||||
|
|
||||||
|
## Recipe 1: Prove State-Machine Invariants
|
||||||
|
|
||||||
|
**Question.** Can an entity reach only legal states, and do queue message state and job state remain consistent?
|
||||||
|
|
||||||
|
**Assumptions.** Define the entity, authoritative writer, transaction boundary, and terminal states. Current enums are `ClipJobStatus`, `EditProjectStatus`, and `HighlightProjectStatus`; enum membership alone does not enforce transitions.
|
||||||
|
|
||||||
|
**Derive before running.** Draw allowed edges and forbidden edges. For every operation predict `(before, event, after, durable artifacts)`. Require terminal-state monotonicity unless an explicit, audited recovery operation exists.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Inspect all writers, then run focused offline tests:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'ClipJobStatus\.|EditProjectStatus\.|HighlightProjectStatus\.|setStatus\(' src/main/java src/test/java
|
||||||
|
mvn -o -q -Dtest=DatabaseBackedClipJobQueueAdapterTest,HighlightDirectorFlowServiceTest,LocalDirectorSchedulerTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** A test that reaches the expected final state does not prove forbidden edges are rejected. Add transition-table tests through change control. Current highlight code writes `CREATED -> WAITING_FOR_DIRECTOR -> RENDERING -> RENDERED` and failures, while `ANALYZING`/`PLANNED` exist without a centralized transition guard; treat the model as partially asserted, not enforced.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Attempt stale replay, duplicate completion, failure after side effect, and terminal-to-active transition in an isolated test. Require the same rule to reject all forbidden sources, not controller-only validation.
|
||||||
|
|
||||||
|
**Worked history.** Before commit `1737d8b`, one processing exception could make the job `FAILED` while the DB queue still intended `PENDING` retry. The mechanism was split ownership of terminal policy. Moving terminal failure to the queue’s max-attempt decision explains both non-terminal retries and DLQ exhaustion.
|
||||||
|
|
||||||
|
## Recipe 2: Prove Timeline And Duration Arithmetic
|
||||||
|
|
||||||
|
**Question.** Does the planned timeline equal the rendered media timeline?
|
||||||
|
|
||||||
|
**Assumptions.** Fix the edit plan version, source identity, frame rate/timebase, audio sample rate, transition semantics, and output binary. Do not compare a stale plan to a newer render.
|
||||||
|
|
||||||
|
**Derive before running.** For decision `i`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_i = sourceEnd_i - sourceStart_i
|
||||||
|
rendered_i = source_i / playbackSpeed_i
|
||||||
|
planned_i = timelineEnd_i - timelineStart_i
|
||||||
|
timelineStart_1 = 0
|
||||||
|
timelineStart_i = timelineEnd_(i-1)
|
||||||
|
T_plan = timelineEnd_last
|
||||||
|
```
|
||||||
|
|
||||||
|
Current `EditPlanValidator` permits `|rendered_i - planned_i| <= 0.05 s`, contiguity error `<= 0.001 s`, final target error `<= 1.0 s`, and speed `[0.25, 4.0]`. These are implementation tolerances, not certified media-quality thresholds. Predict accumulated timing error from frame duration, audio sample period, transitions, and mux timebase before probing.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** For an existing approved plan/output pair:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PLAN=output/edit-projects/PROJECT/edit-plan.json
|
||||||
|
MEDIA=output/edit-projects/PROJECT/final.mp4
|
||||||
|
test -f "$PLAN" && test -f "$MEDIA"
|
||||||
|
ffprobe -v error -show_entries format=start_time,duration -show_entries stream=index,codec_type,time_base,start_time,duration,nb_frames -of json "$MEDIA"
|
||||||
|
rg -n 'EPSILON|TARGET_TOLERANCE_SECONDS|renderedDuration|plannedDuration' src/main/java/org/example/videoclips/editing/EditPlanValidator.java
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not infer media duration from `RenderManifest.durationSeconds`; the multi-clip and per-highlight render paths derive that field from the plan, while the project-level highlight flow sums ffprobe results. Probe the file used by the claim.
|
||||||
|
|
||||||
|
**Interpretation.** Distinguish plan arithmetic error, encoder/mux quantization, and transition overlap. A constant offset on every output implicates command/timebase behavior; error growing per segment implicates accumulation or rounding.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Include fractional frame rates, nonzero stream start time, audio shorter/longer than video, speeds at both limits, and a final partial segment.
|
||||||
|
|
||||||
|
**Worked history.** Current highlight QA sets `duration_matches_timeline=true` with the explanation that duration is based on the plan. That is asserted QA: it cannot detect a muxed output whose actual duration differs. The discriminating evidence is plan-derived `T_plan` versus ffprobe-derived stream/container durations.
|
||||||
|
|
||||||
|
## Recipe 3: Bound Keyframe And Segment Error
|
||||||
|
|
||||||
|
**Question.** Is a cut frame-accurate, or merely aligned to an existing keyframe?
|
||||||
|
|
||||||
|
**Assumptions.** `FolderFfmpegClipper` and `FfmpegVideoClipperAdapter` stream-copy in preservation/FAST paths; exact paths re-encode with forced keyframes. Stream copy preserves encoded packets and cannot create a keyframe at an arbitrary boundary.
|
||||||
|
|
||||||
|
**Derive before running.** Let requested boundary be `b`, surrounding keyframe timestamps `k_prev <= b <= k_next`, and maximum observed keyframe gap `G`. Predict the tool-specific chosen boundary from command shape; the empirical error must be compared with `|b-k_prev|` and `|k_next-b|`. Do not claim a universal direction without observing the muxer result. For forced-keyframe re-encode, predeclare a bound from output frame duration/timebase and then measure it.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Probe source keyframes and existing clips only:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SOURCE=/path/to/approved/source.mp4
|
||||||
|
CLIP=/path/to/existing/clip.mp4
|
||||||
|
ffprobe -v error -select_streams v:0 -skip_frame nokey -show_frames -show_entries frame=best_effort_timestamp_time,pkt_dts_time -of csv=p=0 "$SOURCE"
|
||||||
|
ffprobe -v error -show_entries format=start_time,duration -show_entries stream=time_base,start_time,duration -of json "$CLIP"
|
||||||
|
rg -n 'force_key_frames|segment_time|preserveInputQuality|AccuracyMode.EXACT' src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** Boundary error tracking source keyframe distance supports keyframe alignment. Error within a predeclared output-frame/timebase bound only in re-encode mode supports forced-keyframe accuracy. Codec names or clip-count metadata do not prove either.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Test long GOP, variable frame rate, B-frames, nonzero start time, and audio packet boundaries. Compare first/last decoded frame content, not container duration alone.
|
||||||
|
|
||||||
|
**Worked history.** The folder workflow explicitly retains a quality-versus-exactness trade-off: packaged `preserve-input-quality=true` uses `-c copy`; false uses libx264/AAC and forced keyframes. Calling both modes “exact” would contradict the mechanism.
|
||||||
|
|
||||||
|
## Recipe 4: Prove FFmpeg Filter And Audio Graphs
|
||||||
|
|
||||||
|
**Question.** Does every intended stream reach the output with the expected timebase, duration, gain, and label?
|
||||||
|
|
||||||
|
**Assumptions.** Freeze the exact command, input stream inventory, asset presence, channel layouts, and current editing properties. Treat optional input syntax separately from filter-label requirements.
|
||||||
|
|
||||||
|
**Derive before running.** Draw a directed graph: input stream -> trim/delay/rate -> gain -> sidechain -> mix -> loudness -> mapped output. For every label predict channel count, sample rate, start/end time, and whether absence is legal. Check video `setpts=PTS/speed` against audio `atempo=speed`.
|
||||||
|
|
||||||
|
Current renderer facts:
|
||||||
|
|
||||||
|
| Graph behavior | Current implementation consequence |
|
||||||
|
|---|---|
|
||||||
|
| Segment audio is optional at `-map 0:a?` | A video-only timeline can be created. |
|
||||||
|
| Mix graph starts with `[0:a]` | A video-only timeline makes the mix graph fail even when music/voice/SFX inputs exist. |
|
||||||
|
| Highlight audio-mix failure is caught | The renderer copies the post-timeline, potentially publishing without intended assets. |
|
||||||
|
| `amix duration=first` | Output audio length is governed by timeline audio, not longest asset. |
|
||||||
|
| Music+voice uses sidechain compression | Music is the compressed signal; voice is the detector and is also mixed. |
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Inspect command construction and probe an existing output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'audioMixCommand|sidechaincompress|amix=|loudnorm=|setpts=|atempo=|fallback.*copy' src/main/java/org/example/videoclips/editing
|
||||||
|
MEDIA=/path/to/existing/approved/final.mp4
|
||||||
|
ffprobe -v error -show_streams -show_format -of json "$MEDIA"
|
||||||
|
ffmpeg -hide_banner -nostats -v error -i "$MEDIA" -map 0:v:0 -f null - -map 0:a:0 -f null -
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** Command presence proves intent, not application. Require output streams plus measured signal. An audio-less result after a logged mix failure is explained by the catch-and-copy path; a successful graph with wrong loudness needs a different mechanism.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Use video-only source, mono/stereo assets, missing one optional asset, asset longer than timeline, and SFX beyond source-audio end. Never “fix” a graph by inserting prohibited silence.
|
||||||
|
|
||||||
|
**Worked history.** Commit `5d889b0` records “working version but not cinematic.” A valid FFmpeg graph explained renderability but not creative quality; later filters/assets/QA are hypotheses whose contribution still needs ablation and review.
|
||||||
|
|
||||||
|
## Recipe 5: Measure Loudness, Dynamics, And True Peak
|
||||||
|
|
||||||
|
**Question.** Does mastered audio meet a declared delivery envelope without clipping or masking speech?
|
||||||
|
|
||||||
|
**Definitions.** `LUFS-I` is integrated program loudness; `LRA` is loudness range; `dBTP` is inter-sample true peak. Sample peak from `volumedetect` is not true peak.
|
||||||
|
|
||||||
|
**Assumptions.** Freeze the exact encoded output, channel layout, measurement tool/version, delivery target, tolerance, and speech windows. Measure after the final lossy encode.
|
||||||
|
|
||||||
|
**Derive before running.** Declare target and tolerance through validation/change control. Packaged filter targets are `I=-16`, `TP=-1.5`, `LRA=11`; they are current defaults, not measured acceptance. Predict integrated loudness, maximum true peak, LRA, silence proportion, and voice/music delta for the fixture.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Measure an existing approved output; this decodes to a null sink and does not render a file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
MEDIA=/path/to/existing/approved/final.mp4
|
||||||
|
ffmpeg -hide_banner -nostats -i "$MEDIA" -map 0:a:0 -af 'loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json' -f null - 2>&1
|
||||||
|
ffmpeg -hide_banner -nostats -i "$MEDIA" -map 0:a:0 -af 'ebur128=peak=true' -f null - 2>&1
|
||||||
|
rg -n 'loudness-target-i|loudness-true-peak|loudness-range|clippingDetectCommand|volumedetect' src/main/resources/application.yml src/main/java/org/example/videoclips/editing
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** Compare measured `input_i`, `input_tp`, and `input_lra` to the preregistered envelope. Current QA’s `max_volume < -0.1 dBFS` clipping check is a sample-peak warning, not proof of the configured `-1.5 dBTP` limit. One-pass `loudnorm` filter presence is not measurement.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Include silence, isolated transient, codec round-trip, mono/stereo, and speech-over-music windows. A global LUFS pass can coexist with unintelligible voice; test local speech windows and human intelligibility separately.
|
||||||
|
|
||||||
|
**Worked history.** Commit `bb23ac8` added mastering filters. That establishes graph construction, not that every output hits targets, especially when highlight audio failure falls back to copying the timeline.
|
||||||
|
|
||||||
|
## Recipe 6: Prove Highlight Ranking
|
||||||
|
|
||||||
|
**Question.** Does the ranker find the moments reviewers label as highlights, in useful order?
|
||||||
|
|
||||||
|
**Assumptions.** Build a licensed, versioned evaluation set with independent timestamp labels and category strata. The repository currently has no certified creative golden set. `CinematicHighlightAnalyzer` uses metadata keyword classification, fixed windows, position roles, and clip-level quality scores repeated across every window; its tests prove bounds/order, not relevance.
|
||||||
|
|
||||||
|
**Derive before running.** Freeze temporal intersection-over-union threshold `tau`, top `K`, and split before scoring:
|
||||||
|
|
||||||
|
```text
|
||||||
|
IoU(candidate, truth) = overlap_seconds / union_seconds
|
||||||
|
precision@K = matched_predictions_in_top_K / K
|
||||||
|
recall@K = uniquely_matched_truth_windows / truth_window_count
|
||||||
|
DCG@K = sum((2^relevance_i - 1) / log2(i + 1)); NDCG@K = DCG@K / ideal_DCG@K
|
||||||
|
```
|
||||||
|
|
||||||
|
Use one-to-one matching so duplicate overlapping candidates cannot inflate recall. Report per category and macro average. Predict metric deltas and confidence intervals before each change; never select `tau` or `K` on the test set.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** First prove what current tests do and do not cover:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '1,220p' src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java
|
||||||
|
sed -n '1,180p' src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java
|
||||||
|
rg -n 'golden|ground.truth|precision|recall|NDCG|IoU' src/test docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Then evaluate exported candidate JSON against separately reviewed labels with a deterministic evaluator added only through change control. Record exact-match policy, ties, missing categories, and bootstrap seed.
|
||||||
|
|
||||||
|
**Ablations.** Hold candidates fixed and remove one feature family at a time: source position, visual scores, audio events, scene boundaries, semantic/local-model features. A useful feature must predict a preregistered metric change and survive category-stratified negatives.
|
||||||
|
|
||||||
|
**Interpretation.** High precision with low recall means conservative discovery; high recall with low precision burdens the director. NDCG distinguishes useful ordering from an unordered candidate set. Do not aggregate away a failing content category.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Include long clips where the 12-window cap changes step size, generic filenames, repeated scenes, quiet emotional moments, high-motion non-events, and near-duplicate windows.
|
||||||
|
|
||||||
|
**Worked history.** Category-aware planning (`5006dfd`) followed the “not cinematic” result, but history contains no controlled ranking comparison. Treat category boosts as candidates until labeled-set precision/recall/NDCG and ablations support them.
|
||||||
|
|
||||||
|
## Recipe 7: Distinguish Model Signal From Fallback
|
||||||
|
|
||||||
|
**Question.** Did a declared local model produce the semantic asset, or did the system emit a placeholder or host fallback?
|
||||||
|
|
||||||
|
**Assumptions.** Freeze the requested asset type/prompt, declared local model checksum, runtime, seed/config, and expected duration. Do not infer provenance from a filename or directory.
|
||||||
|
|
||||||
|
**Derive before running.** Predict all three layers: provenance `(model id, local path, checksum, runtime, seed/config)`, signal `(duration, sample rate, channels, RMS/silence, spectrum)`, and semantics `(speech transcript/intelligibility or licensed music/SFX rubric)`. Require all three. File existence is zero-layer proof.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Do not invoke the workers. Inspect current fallback paths and measure only an existing candidate asset:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'write_silence|write_fallback_tone|fallbackTone|writeSilence|speech-fallback|strict_runtime|strict-runtime' tools/local_asset_worker.py src/main/java src/main/resources
|
||||||
|
AUDIO=/path/to/existing/candidate.wav
|
||||||
|
ffprobe -v error -show_streams -show_format -of json "$AUDIO"
|
||||||
|
ffmpeg -hide_banner -nostats -i "$AUDIO" -af 'silencedetect=noise=-60dB:d=0.2,astats=metadata=0:reset=0' -f null - 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Historical artifacts may contain exact silence from the former Python voiceover fallback, a `110 Hz` low-level music tone, an `880 Hz` short SFX tone, Java-generated tones, or host `say`/`espeak` speech. Those success paths were removed from the 2026-07-21 working tree. Keep detecting the signatures in old outputs; do not use absence of a known signature as a provenance whitelist.
|
||||||
|
|
||||||
|
**Interpretation.** Silence proves failure. Nonzero samples disprove only digital silence; they do not prove speech or model generation. A stable simple tone matching a known path plus absent model provenance supports fallback. Real speech from a host tool is still not the approved embedded model.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Test a valid model asset, digital silence, known tone, corrupted WAV, unrelated audible audio, and host TTS. The mechanism must classify positives and negatives without relying on filename.
|
||||||
|
|
||||||
|
**Worked history.** On 2026-07-21 the workspace build passed while a clean archive failed `LocalAssetGenerationStageTest`; tests depended on an untracked virtual environment/host fallback. The working tree now makes that test deterministic and strict readiness fail closed. Environment-coupled success explains the old observations; only a new clean archive can confirm closure.
|
||||||
|
|
||||||
|
## Recipe 8: Prove Retry, DLQ, Idempotency, And Leases
|
||||||
|
|
||||||
|
**Question.** Does each logical job have bounded attempts and at-most-one durable effect despite crashes and redelivery?
|
||||||
|
|
||||||
|
**Assumptions.** Fix queue mode, `max-attempts`, retry backoff, visibility timeout, transaction boundary, idempotency key, and side effects. The DB-queue proof does not generalize to the memory queue or a future broker.
|
||||||
|
|
||||||
|
**Derive before running.** For configured maximum `M`, predict attempt sequence `1..M`; failures `1..M-1` yield message `PENDING`, job `QUEUED`, and one `RETRY_SCHEDULED`; failure `M` yields message `DLQ`, job `FAILED`, and terminal events. Predict side-effect cardinality by idempotency key. Let visibility timeout `V` exceed a measured runtime bound or provide lease extension; otherwise predict overlapping work after `V`.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Use existing mocks and read-only inspection:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -q -Dtest=DatabaseBackedClipJobQueueAdapterTest test
|
||||||
|
rg -n 'attemptCount|maxAttempts|visibilityTimeout|RETRY_SCHEDULED|markTerminalFailure|DLQ|Idempotency' src/main/java src/test/java src/main/resources
|
||||||
|
sed -n '1,260p' docs/dlq-redrive-procedure.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** State correctness does not prove effect idempotency. Compare object key, clip index, database uniqueness constraints, and event counts after duplicate delivery. Current queue claims and processing occur in one transaction method, but FFmpeg/object-storage effects are not rolled back with a database transaction.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Fail after upload/before row save, after row save/before completion, at `V-epsilon` and `V+epsilon`, and with two pollers. Do not redrive real DLQ rows as a test.
|
||||||
|
|
||||||
|
**Worked history.** Commit `1737d8b` fixed premature terminal job failure. It does not by itself prove exactly-once external effects; that requires idempotency and crash-boundary tests.
|
||||||
|
|
||||||
|
## Recipe 9: Prove Filesystem Atomicity And Path Confinement
|
||||||
|
|
||||||
|
**Question.** Can a reader observe a partial file, or can untrusted names escape the configured root?
|
||||||
|
|
||||||
|
**Assumptions.** Fix the configured root, filesystem/mount, writer/reader concurrency, crash point, and symlink policy. Lexical normalization and physical filesystem confinement are separate claims.
|
||||||
|
|
||||||
|
**Derive before running.** For publication require write-to-sibling-temp -> fsync as required -> atomic rename on the same filesystem. Predict reader observations as `{old, new}`, never partial. For confinement require validated token, `root.resolve(name).normalize()`, `startsWith(root)`, rejection of absolute paths, separators, and `..`; decide symlink policy explicitly.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Run current confinement tests and inspect publication sites:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -q -Dtest=FileSystemEditProjectStoreTest,FileSystemHighlightProjectStoreTest,EditPlanInboxScannerTest test
|
||||||
|
rg -n 'ATOMIC_MOVE|writeValue\(|writeString\(|Files.copy|startsWith\(|validateProjectId|validate.*File' src/main/java/org/example/videoclips/editing
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** Current stores validate lexical paths but write JSON directly to final files; path confinement tests do not prove crash-safe publication. Highlight source claiming and inbox archival use `ATOMIC_MOVE`, which can fail across filesystems rather than silently degrade. `startsWith` on normalized paths does not settle symlink traversal.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Test absolute paths, both separators, `..`, Unicode/confusable names if accepted, symlink inside root to outside, concurrent readers, process death mid-write, and cross-filesystem move. Keep these in JUnit temporary directories.
|
||||||
|
|
||||||
|
**Worked history.** Existing `project.json`/manifest files can claim rendered output while media is absent. That history shows why metadata publication and durable artifact publication must be one explicit invariant, not two independent existence checks.
|
||||||
|
|
||||||
|
## Recipe 10: Build Resource, Capacity, And Cost Models
|
||||||
|
|
||||||
|
**Question.** What arrival rate, concurrency, disk, memory, model residency, and cost can the service sustain?
|
||||||
|
|
||||||
|
**Assumptions.** Freeze workload mix, source durations/codecs, warm/cold model state, hardware, worker topology, storage/database limits, retention, retry rate, and dated prices. State every omitted cost.
|
||||||
|
|
||||||
|
**Derive before running.** Use measured distributions, not a single average:
|
||||||
|
|
||||||
|
```text
|
||||||
|
real_time_factor = processing_seconds / source_seconds
|
||||||
|
service_rate_per_worker = 1 / mean_job_seconds
|
||||||
|
required_concurrency ~= arrival_rate_per_second * target_system_time_seconds (Little's Law)
|
||||||
|
scratch_bytes >= concurrency * (source + intermediates + output) * safety_factor
|
||||||
|
compute_cost/source_min = worker_seconds/source_min / 3600 * worker_cost/hour
|
||||||
|
transfer_seconds/source_min = transferred_MiB/source_min / measured_MiB_per_second
|
||||||
|
```
|
||||||
|
|
||||||
|
Add fixed local-model memory once per worker process, peak temporary tensors per concurrent inference, FFmpeg threads, JVM heap, and database/object-storage limits. Predict saturation resource and queue-growth slope before load.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Inspect historical models without treating them as current:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -n '1,260p' docs/cost-model-by-video-minute.md
|
||||||
|
sed -n '1,220p' docs/ffmpeg-preset-benchmark-baseline.md
|
||||||
|
sed -n '1,220p' docs/object-storage-bandwidth-benchmark-baseline.md
|
||||||
|
sed -n '1,260p' docs/load-test-signoff.md
|
||||||
|
```
|
||||||
|
|
||||||
|
The checked-in cost harness hardcodes example prices and prior benchmark values. Replace assumptions with dated target-environment measurements only after an approved benchmark. Never claim scalability from the stub worker baseline or the documentation-only load signoff.
|
||||||
|
|
||||||
|
**Interpretation.** Validate predictions against p50/p95/p99 service time and resource saturation. If measured queue slope differs, the arrival rate, service-time distribution, concurrency constraint, or retry amplification is missing from the model.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Include cold model load, warm inference, mixed durations, 2x burst, slow storage, disk pressure, codec mix, retry amplification, and a 60-minute endurance run. A model must predict both steady state and backlog growth when `arrival_rate > capacity`.
|
||||||
|
|
||||||
|
**Worked history.** `docs/load-test-signoff.md` says execution evidence is pending. Existing local baselines explain planning inputs, not production API/worker throughput or cloud cost.
|
||||||
|
|
||||||
|
## Recipe 11: Prove Reproducibility With Checksums
|
||||||
|
|
||||||
|
**Question.** Can another approved macOS/Linux/VPS/cloud runtime reproduce the result from the same declared inputs?
|
||||||
|
|
||||||
|
**Assumptions.** Define whether the claim is byte reproducibility or metric-bounded reproducibility. Freeze source/config/model/asset checksums, toolchain, locale/timezone, device, and randomness.
|
||||||
|
|
||||||
|
**Derive before running.** Predict which bytes must be identical and which measurements may vary. Media encodes may differ across FFmpeg/library/hardware builds; in that case require identical inputs/config and bounded semantic/technical metrics, not a fabricated byte-equality promise.
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Capture source tree identity, tool versions, and existing artifact hashes without starting the service:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git rev-parse HEAD
|
||||||
|
java -version 2>&1
|
||||||
|
mvn -version
|
||||||
|
ffmpeg -version | sed -n '1,12p'
|
||||||
|
ffprobe -version | sed -n '1,5p'
|
||||||
|
git ls-files -s pom.xml src tools
|
||||||
|
ARTIFACT=/path/to/existing/approved/artifact
|
||||||
|
if command -v sha256sum >/dev/null 2>&1; then sha256sum "$ARTIFACT"; else shasum -a 256 "$ARTIFACT"; fi
|
||||||
|
```
|
||||||
|
|
||||||
|
For a local model record model-file checksum, license, runtime package lock, device, precision, seed, and inference parameters. A model name such as `musicgen-small` is not an immutable identity.
|
||||||
|
|
||||||
|
**Interpretation.** An identical artifact hash proves byte identity for that artifact only. A mismatch requires localization by input, intermediate, and toolchain hashes; it does not automatically prove semantic regression.
|
||||||
|
|
||||||
|
**Adversarial refutation.** Repeat from a clean archive with network disabled and no untracked virtual environments; compare macOS and Linux. Separate dependency-cache absence from test failure. Do not solve a clean failure by enabling downloads or committing a virtualenv.
|
||||||
|
|
||||||
|
**Worked history.** The workspace’s 216 tests passed while a clean archived checkout failed in local-asset generation. The untracked `.venv-local-asset`/host behavior is the discriminating variable, so “`mvn test` passed here” is not reproducible-build proof.
|
||||||
|
|
||||||
|
## Recipe 12: Perform Causal Debugging
|
||||||
|
|
||||||
|
**Question.** Which single mechanism explains all observations, including negatives?
|
||||||
|
|
||||||
|
**Assumptions.** Freeze the incident time window, build, inputs, configuration, environment, and observation reliability. Mark unknowns rather than silently filling them.
|
||||||
|
|
||||||
|
**Derive before running.** Build a causal graph from input identity -> configuration -> state -> external/local side effects -> persisted metadata -> published artifact -> measured quality. Mark observation points. For each candidate cause, write a prediction matrix before touching code:
|
||||||
|
|
||||||
|
| Experiment | Mechanism A predicts | Mechanism B predicts | Observation |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Positive control | number/state/signature | number/state/signature | pending |
|
||||||
|
| Negative control | absence/different value | same value | pending |
|
||||||
|
| One-factor ablation | directional delta | no delta | pending |
|
||||||
|
| Crash/boundary case | durable state/effects | different tuple | pending |
|
||||||
|
|
||||||
|
**Repo-safe experiment.** Locate the actual writers and historical change before selecting one minimal discriminator:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'objectKey|markTerminalFailure|duration_matches_timeline|audio_mastering_applied|fallback' src/main src/test
|
||||||
|
git log --oneline --all -- src/main/java/org/example/videoclips
|
||||||
|
git show --stat 7307082 1737d8b 5d889b0 9b56e89
|
||||||
|
```
|
||||||
|
|
||||||
|
**Interpretation.** Accept only a mechanism whose predictions match the positive, negative, and boundary cases. Correlation with a changed component is insufficient when another mechanism predicts the same observation.
|
||||||
|
|
||||||
|
Prefer the smallest experiment that makes predictions diverge. Reject explanations that merely restate the symptom. Preserve logs, checksums, commands, and negative results; do not patch before discrimination unless containing an incident.
|
||||||
|
|
||||||
|
**Worked examples.** Use the historical mechanism, not the headline:
|
||||||
|
|
||||||
|
| Symptom | Mechanism that explained positives and negatives | Remaining boundary |
|
||||||
|
|---|---|---|
|
||||||
|
| Signed download failed despite a clip row | Derived fake key differed from persisted `objectKey`; commit `7307082` made persisted key authoritative. | Production S3 end-to-end proof remains separate. |
|
||||||
|
| First transient error looked terminal | Processor finalized failure before queue retry policy; `1737d8b` moved terminal decision to exhaustion. | External side-effect idempotency still needs crash tests. |
|
||||||
|
| Renderer worked but was not cinematic | Renderability measured graph execution, not semantic highlight/edit quality. | No certified creative golden set or controlled ablation exists. |
|
||||||
|
| Workspace build passed, clean archive failed | Untracked runtime/host fallback affected local-asset tests. | Hermetic fake/runtime contract is still open. |
|
||||||
|
| Highlight QA passed duration/mastering checks | Some checks are constants or command-presence assertions rather than output measurements. | Probe-derived gates must be added through change control. |
|
||||||
|
|
||||||
|
**Adversarial refutation.** Before accepting a mechanism, assign a reviewer to construct the strongest counterexample and a test where the mechanism predicts no effect. If it cannot explain that negative, retain it as a candidate only.
|
||||||
|
|
||||||
|
## Promotion Checklist
|
||||||
|
|
||||||
|
- [ ] The proof record was written before execution.
|
||||||
|
- [ ] Inputs, commit, configuration, binaries, models, assets, licenses, and checksums are recorded.
|
||||||
|
- [ ] Commands ran offline and did not bootstrap, download, render without approval, mutate production, or use external AI.
|
||||||
|
- [ ] Structural, semantic, security, and operational claims are separated.
|
||||||
|
- [ ] Positive, negative, boundary, and failure controls passed.
|
||||||
|
- [ ] Ranking work reports denominators, category strata, matching policy, uncertainty, and ablations.
|
||||||
|
- [ ] Media work reports probe-derived duration, streams, loudness, true peak, silence, and human-review evidence where semantic judgment is required.
|
||||||
|
- [ ] One mechanism explains all observations, including negatives; an adversarial reviewer tried to falsify it.
|
||||||
|
- [ ] Unproven conclusions remain labeled `OPEN` or `CANDIDATE`.
|
||||||
|
- [ ] Any behavior/default/dependency/model/asset/render/production change proceeds through `video-editing-change-control` and the validation sibling.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Sources of record: current Java/Python implementation and tests; `application.yml`/`application.properties`; benchmark/runbook documents; Git commits `66e998e`, `7307082`, `1737d8b`, `5d889b0`, `5006dfd`, `bb23ac8`, and `9b56e89`; and the clean-checkout investigation recorded in `video-editing-failure-archaeology`.
|
||||||
|
|
||||||
|
Re-verify volatile facts before each use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git rev-parse HEAD
|
||||||
|
rg -n 'loudness-target-i|loudness-true-peak|loudness-range|strict-runtime|fallback-to-heuristic|max-attempts|visibility-timeout|preserve-input-quality' src/main/resources
|
||||||
|
rg -n 'duration_matches_timeline|audio_mastering_applied|write_silence|write_fallback_tone|fallbackTone|ATOMIC_MOVE|writeValue\(' src/main tools
|
||||||
|
rg -n 'precision|recall|NDCG|ground.truth|golden|Execution evidence pending' src/test docs
|
||||||
|
git show --stat 66e998e 7307082 1737d8b 5d889b0 5006dfd bb23ac8 9b56e89
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,285 @@
|
||||||
|
---
|
||||||
|
name: video-editing-research-frontier
|
||||||
|
description: Load when proposing, ranking, starting, evaluating, retiring, or positioning research that could advance this repository beyond its current cinematic-highlight baseline, especially offline local multimodal ranking, constrained directing, audiovisual QA, local voice/music/SFX generation, causal ablations, efficient model packaging, or restartable deterministic rendering. Do not load for ordinary defect repair, production hardening, or execution of the current highlight campaign.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Research Frontier
|
||||||
|
|
||||||
|
Use this runbook to turn an open technical question into a falsifiable, repo-local research program. Treat all landscape statements below as hypotheses, not claims about the external state of the art (SOTA). The repository does not contain a reviewed literature survey or competitive benchmark.
|
||||||
|
|
||||||
|
**Status date:** 2026-07-21.
|
||||||
|
|
||||||
|
## Choose The Correct Skill
|
||||||
|
|
||||||
|
| Need | Use instead |
|
||||||
|
|---|---|
|
||||||
|
| Fix a known defect, remove a prohibited fallback, make the clean build reproducible, or harden production | `video-editing-change-control` and the relevant engineering skill |
|
||||||
|
| Execute the hardest live objective with decision gates | `video-editing-cinematic-highlights-campaign` |
|
||||||
|
| Form hypotheses, pre-register predictions, assign adversarial review, or retire an idea | `video-editing-research-methodology` |
|
||||||
|
| Derive a score, prove an invariant, design an ablation, or calculate uncertainty | `video-editing-proof-and-analysis-toolkit` |
|
||||||
|
| Certify an output, add a golden, or define release evidence | `video-editing-validation-and-qa` |
|
||||||
|
| Make a paper, novelty, benchmark, or release claim | `video-editing-external-positioning` |
|
||||||
|
| Debug an observed failure | `video-editing-debugging-playbook` |
|
||||||
|
|
||||||
|
Do **not** use this skill to label missing production controls as research. Adding authentication, immutable model packaging, fail-closed startup, a clean build, render approval, or measured QA is engineering backlog. Research begins only when a controlled experiment compares mechanisms under those controls.
|
||||||
|
|
||||||
|
## Enforce The Research Perimeter
|
||||||
|
|
||||||
|
Apply these rules before any experiment:
|
||||||
|
|
||||||
|
- Run without network access. Never call an external AI service. Never let Maven, Python, a model library, or a worker download dependencies or weights.
|
||||||
|
- Provision every dependency and model before runtime through the approved artifact process. Record artifact name, exact version, SHA-256, license, approval, platform, and expected resource envelope.
|
||||||
|
- Use only footage, voices, music, sound effects, fonts, LUTs, and model outputs whose experiment and redistribution rights are recorded. Treat generated media as licensable artifacts, not automatically safe assets.
|
||||||
|
- Reject missing or invalid models and assets. Never substitute silence, tones, heuristic analysis, host `say`, host `espeak`, or an untracked local asset.
|
||||||
|
- Require explicit render approval. Keep every experiment flag off by default and leave production-facing defaults unchanged.
|
||||||
|
- Keep raw media and model prompts out of logs. Treat faces, voices, transcripts, filenames, and embeddings as sensitive data until the threat model says otherwise.
|
||||||
|
- Execute experiments in an isolated project/output root with quotas. Do not point a scheduler at production input or output directories.
|
||||||
|
- Route all source changes, model additions, data collection, and promotion through `video-editing-change-control`. A research result is not production approval.
|
||||||
|
|
||||||
|
The checked-in launchers violate the target perimeter if used as written: `tools/run_local_cv_worker.sh` can install Python packages and load the default `yolov8n.pt`; `tools/run_local_asset_worker.sh` can install packages; and `tools/local_asset_worker.py` calls `get_pretrained`. Do not run those launchers for research until an offline, pinned, checksum-verified mode exists.
|
||||||
|
|
||||||
|
## Separate Backlog From Research
|
||||||
|
|
||||||
|
Complete or explicitly gate this backlog before interpreting creative results:
|
||||||
|
|
||||||
|
| Known gap | Why it is not research | Required gate |
|
||||||
|
|---|---|---|
|
||||||
|
| Highlight rendering now defaults off and approval on, but approval is a bare file rather than authenticated evidence bound to source/plan/configuration digests | This is an authorization and provenance gap, not a hypothesis | Keep rendering disabled until digest-bound approval and recovery behavior are implemented and tested |
|
||||||
|
| `SourceVisualAnalyzer` can fall back to `HeuristicVisualAnalysisProvider` | Silent mechanism substitution invalidates attribution | Set fallback off and persist the exact analyzer identity |
|
||||||
|
| Generated audio still lacks certified semantic-fit and provenance evaluation | Placeholder rejection is implemented, but file audibility is not creative or licensing evidence | Add model/asset manifests and held-out scene-to-audio preference evaluation |
|
||||||
|
| No licensed, checksummed resident asset-model bundle is present | Packaging and provenance are engineering prerequisites | Provision and verify Piper, MusicGen, and AudioGen artifacts without runtime acquisition |
|
||||||
|
| Highlight QA measures only selected technical defects | Duration, black ranges, long silence, and sample peaks are probed, while loudness, true peak, A/V sync, freezes, raster text safety, semantic fit, and human correlation remain open | Extend the versioned probe portfolio before using certification labels |
|
||||||
|
| No licensed, versioned creative corpus or human-scored golden inventory is checked in | An experiment cannot generalize from anecdotal outputs | Approve a corpus manifest and annotation protocol; store media outside Git |
|
||||||
|
| Python audio dependencies are unpinned and model artifacts are not locked | Results cannot be reproduced | Build an immutable offline runtime with a Software Bill of Materials (SBOM) and checksums |
|
||||||
|
|
||||||
|
Verify the current guardrail-sensitive defaults without starting the application:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n "render-enabled:|require-director-approval:|fallback-to-heuristic:|auto-start:|strict-runtime:" src/main/resources/application.yml
|
||||||
|
rg -n "pip install|get_pretrained|YOLO\(|write_silence|write_fallback_tone|fallbackTone" tools src/main/java/org/example/videoclips/editing
|
||||||
|
```
|
||||||
|
|
||||||
|
## Establish One Evaluation Contract
|
||||||
|
|
||||||
|
Create one versioned corpus manifest and one immutable evaluation split before comparing ideas. Do not select examples after seeing results.
|
||||||
|
|
||||||
|
Define these terms once:
|
||||||
|
|
||||||
|
- **Temporal IoU:** intersection divided by union of a predicted and reference time range.
|
||||||
|
- **nDCG@3:** normalized discounted cumulative gain for the first three ranked candidates; it rewards relevant moments near the top.
|
||||||
|
- **Pairwise win rate:** fraction of blinded A/B judgments preferring the candidate over the baseline; ties count as one half.
|
||||||
|
- **Spearman rho:** rank correlation between an automated score and human ordering.
|
||||||
|
- **RTF:** real-time factor, processing seconds divided by source-media seconds; lower is faster.
|
||||||
|
- **Equivalence margin:** the largest quality loss accepted when testing that a faster method is practically no worse.
|
||||||
|
- **Confidence interval (CI):** an uncertainty interval computed by the pre-registered method; do not report a point estimate alone.
|
||||||
|
|
||||||
|
For every frontier item, write a dated experiment record with:
|
||||||
|
|
||||||
|
```text
|
||||||
|
question; mechanism; baseline commit; corpus-manifest hash; split hash
|
||||||
|
model/dependency hashes and licenses; hardware/OS; experiment flag
|
||||||
|
primary metric; prediction made before running; equivalence margin
|
||||||
|
negative controls; adversarial reviewer; stop/retire rule; raw artifact root
|
||||||
|
```
|
||||||
|
|
||||||
|
Use blinded review with multiple reviewers and report disagreement. Category, source identity, variant, and generation seed belong in the analysis table, not metric labels or normal application logs.
|
||||||
|
|
||||||
|
## Frontier Portfolio
|
||||||
|
|
||||||
|
Rank work in this order unless evidence changes the dependency graph.
|
||||||
|
|
||||||
|
| Rank | Program | Research question | Depends on |
|
||||||
|
|---:|---|---|---|
|
||||||
|
| 1 | Offline multimodal highlight ranking | Can local temporal evidence find human-valued moments better than positional windows? | Corpus, fail-closed local inference |
|
||||||
|
| 2 | Controllable cinematic planning | Can a constrained local director produce valid, intentional timelines rather than one-range templates? | Ranked moments, plan semantics |
|
||||||
|
| 3 | Human-correlated audiovisual QA | Can measurements predict reviewer judgment and block technical defects? | Scored outputs, real probes |
|
||||||
|
| 4 | Fail-closed local generative audio and voiceover | Can locally packaged models create licensed, intelligible, context-fit assets that pass gates? | Immutable model runtime, QA |
|
||||||
|
| 5 | Causal edit-component evaluation | Which effects actually cause preference changes, and for which categories? | Deterministic variants, blinded review |
|
||||||
|
| 6 | Quality-preserving local inference efficiency | Can cascades reduce runtime and memory without meaningful quality loss? | Stable quality metrics, hardware matrix |
|
||||||
|
| 7 | Restartable, reproducible rendering | Can content-addressed stages resume safely and reproduce equivalent outputs? | Manifested inputs, deterministic stage contracts |
|
||||||
|
|
||||||
|
## 1. Offline Multimodal Highlight Ranking
|
||||||
|
|
||||||
|
**Repo-grounded problem.** In the older edit-project flow, `CinematicHighlightAnalyzer` creates fixed eight-second windows, caps each clip at twelve candidates, classifies from project names and paths, and adds source-position bonuses. The single-source flow now has a safer deterministic baseline: `HighlightCandidateGenerator` persists shot-aligned or overlapping coverage windows, ranks them with source-level visual quality, scene score, duration fit, and FFmpeg non-silence, and downweights heuristic/fallback visual evidence. The visual scores are not temporally resolved, `unclassified_audio` is not semantic audio understanding, and there is no checked-in transcript or temporal audio-event classifier. The baseline therefore produces review candidates, not content-aware highlight discovery.
|
||||||
|
|
||||||
|
**Hypothesis.** A fully local, temporally aligned ranker using shot-level visual embeddings, speech/transcript evidence, audio events, motion, novelty, and narrative role will improve top-three relevance across car, food, family, and generic footage. This is a repo hypothesis, not an external SOTA claim.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse `ShotSceneSegmenter`, extracted frames/contact sheets, `SourceAudioAnalysis`, the `VisualAnalysisProvider` port, `HighlightCandidate`, persisted analysis JSON, and the tested `HighlightCandidateGenerator` as the frozen single-source baseline. Do not call that baseline a research result.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Approve a licensed corpus manifest; have blinded reviewers mark worthwhile ranges, category, narrative role, and 0-4 relevance. Freeze source-disjoint train/development/test splits and compute baseline nDCG@3 and temporal IoU.
|
||||||
|
2. Add a disabled experimental provider behind an explicit port. Feed per-shot features only from preprovisioned local models; persist timestamps, model hashes, feature schema, confidence, and missing-modality state. Fail if a required modality is missing.
|
||||||
|
3. Compare positional baseline, each single modality, the complete model, shuffled-timestamp control, and metadata-only control. Use the same candidate budget and report per-category CIs.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict at least `+0.10` absolute nDCG@3 and `+0.10` absolute best-match temporal IoU over the frozen baseline, with no category losing more than `0.03` nDCG@3. Lock these numbers in the experiment record before inference.
|
||||||
|
|
||||||
|
**You have a result when:** the held-out, source-disjoint test CI excludes zero for the primary improvement; shuffled timestamps materially reduce performance; and blinded reviewers prefer montages built from the new top candidates by at least 60% pairwise win rate. This proves a repo result, not SOTA.
|
||||||
|
|
||||||
|
**Retire or narrow when:** the CI includes zero after the pre-registered sample size, gains vanish on unseen sources, the metadata control matches the full model, or any required model lacks approved offline packaging or rights.
|
||||||
|
|
||||||
|
**Risks and obligations.** Control transcript leakage and repeated scenes; measure demographic/category error; encrypt or isolate embeddings; bound CPU/GPU memory and RTF; never send media to a remote endpoint.
|
||||||
|
|
||||||
|
## 2. Controllable Cinematic Planning
|
||||||
|
|
||||||
|
**Repo-grounded problem.** `HighlightDirectorFlowService` consumes a manually supplied and contract-validated `director/edit-plan.json`, converts each highlight to one `EditDecision`, schedules per-line voiceover from estimated reading time, spaces overlays evenly, and chooses mostly cut/fade transitions. The plan contains direction text, but the renderer still has a much smaller executable vocabulary.
|
||||||
|
|
||||||
|
**Hypothesis.** A local planner that combines a typed capability graph, hard timeline constraints, and a learned or search-based objective can produce more intentional and controllable edits than the current one-range template without allowing invalid plans.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse `HighlightDirectorPlan`, `EditPlan`, `EditPlanValidator`, `HighlightVisualEffectsStage`, render manifests, and FFmpeg rendering. Treat every renderer feature as unavailable until a contract test proves it executable.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Inventory the exact executable vocabulary: trims, speed bounds, transitions, overlays, audio cues, assets, and output profiles. Add round-trip contract fixtures for valid, invalid, and unsupported plans.
|
||||||
|
2. Define measurable planning controls: target duration, shot diversity, narrative roles, pacing curve, dialogue preservation, asset budget, effect density, and forbidden combinations. Predict all constraint values before rendering.
|
||||||
|
3. Generate blinded A/B plans from the frozen template and the experimental local planner over the same ranked moments and assets. Render only after approval; score validity, control adherence, repetition, story coherence, and preference.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict 100% hard-constraint validity, at least 95% requested-control adherence, and at least 60% pairwise preference over the template, with median rendered duration error no greater than 50 ms.
|
||||||
|
|
||||||
|
**You have a result when:** every held-out plan validates and renders, the preference CI is above 50%, changing one requested control changes its measured output while unrelated controls remain within pre-registered tolerances, and unsupported capabilities fail before rendering.
|
||||||
|
|
||||||
|
**Retire or narrow when:** preference is indistinguishable from the template, the planner depends on prompt wording rather than typed controls, renderer behavior cannot implement the plan, or constraint repairs erase the preference gain.
|
||||||
|
|
||||||
|
**Risks and obligations.** Prevent prompt/filename injection, fabricated factual voiceover, unsafe overlay text, excessive flash/cut rates, and unbounded plan complexity. Persist planner/model hash, seed, constraint decisions, repairs, and approval identity.
|
||||||
|
|
||||||
|
## 3. Human-Correlated Audiovisual QA
|
||||||
|
|
||||||
|
**Repo-grounded problem.** `HighlightFfmpegRenderer.buildQaReport` now probes encoded duration, black ranges, long silence, and sample peaks, while asset resolution, mastering-filter presence, and overlay plan bounds are structural checks. Neither renderer establishes integrated loudness, true peak, A/V sync, freeze detection, raster text safety, cinematic quality, or correlation with human judgment.
|
||||||
|
|
||||||
|
**Hypothesis.** A transparent scorecard combining technical defects, speech intelligibility, dialogue/music balance, cut/beat alignment, shot repetition, exposure/blur continuity, text safety, and artifact provenance will predict human accept/reject decisions well enough to gate review.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse `RenderQaReport`, `RenderQaCheck`, FFmpeg command capture, `RenderManifest`, source analysis, and the existing black/silence/peak probe patterns.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Define a 0-4 blinded human rubric with mandatory technical rejects and separate creative dimensions. Label real outputs plus deliberately injected black frames, silence, clipping, desynchronization, repetition, unsafe text, and missing provenance.
|
||||||
|
2. Implement each metric as a versioned probe that emits raw numbers, threshold source, evidence path, and failure semantics. Never infer success from a command string or planned value.
|
||||||
|
3. Fit thresholds on development data, freeze them, then report defect sensitivity/specificity, Spearman rho by dimension, calibration, and false-pass cases on held-out sources.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict at least 95% sensitivity to injected blocking defects, no more than 5% false passes for mandatory rejects, and Spearman `rho >= 0.65` between the composite score and overall human rank.
|
||||||
|
|
||||||
|
**You have a result when:** held-out results meet all three predictions, reviewer disagreement is reported, every blocking decision links to reproducible measurements, and removing any claimed useful metric causes the pre-registered degradation.
|
||||||
|
|
||||||
|
**Retire or narrow when:** correlation fails on a category, thresholds drift across render profiles, a proxy rewards visibly worse outputs, or human disagreement makes the target unidentifiable. Keep reliable technical probes even if the creative composite is retired.
|
||||||
|
|
||||||
|
**Risks and obligations.** Do not hide category-specific failures in an average. Avoid sensitive identifiers in metrics. Pin FFmpeg and probe versions. Treat automated creative scoring as review prioritization until change control explicitly promotes it.
|
||||||
|
|
||||||
|
## 4. Fail-Closed Local Generative Audio And Voiceover
|
||||||
|
|
||||||
|
**Repo-grounded problem.** The 2026-07-21 working tree removed silence/tone/host-speech success fallbacks, requires existing local AudioCraft model paths with offline resolution, deletes failed/inaudible output, and blocks rendering when requested assets are missing. The runtime still lacks a checked-in immutable licensed Piper/MusicGen/AudioGen bundle, semantic music/SFX fit measurement, speech intelligibility measurement, and artifact-level license attestations. Demonstrating locally generated production quality remains research.
|
||||||
|
|
||||||
|
**Hypothesis.** Preprovisioned local models plus artifact-level validation can generate script-faithful voiceover and context-fit music/SFX that reviewers prefer to source-audio-only edits without placeholders or unverifiable assets.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse typed `HighlightAssetRequest`, `ResolvedEditAsset`, category-aware libraries, audio cues, ducking/loudness filters, and render manifests.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Define an immutable model/voice registry with checksums, licenses, approved uses, voice consent, sample rates, languages, seeds, resource limits, and a zero-network startup test. Make missing entries block the experiment.
|
||||||
|
2. Add pre-mix asset gates: non-silence/non-tone detection, duration, clipping, script fidelity, pronunciation review, loudness, model provenance, and license completeness. Preserve rejected files as quarantined evidence, never render inputs.
|
||||||
|
3. Run approved, seed-controlled A/B tests for source-only, licensed-library, and local-generated variants. Score intelligibility, script faithfulness, semantic fit, distraction, mix balance, preference, generation RTF, and failure rate.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict zero placeholder false passes, at least 95% reviewer-rated script faithfulness for voiceover, at least 60% pairwise preference over source-only edits, and no unresolved provenance or license field.
|
||||||
|
|
||||||
|
**You have a result when:** all outputs are traceable to approved immutable artifacts, blocking gates catch every injected silence/tone/wrong-script control, preference clears its CI gate on held-out prompts, and an offline cold start neither attempts network access nor degrades to another mechanism.
|
||||||
|
|
||||||
|
**Retire or narrow when:** licensing blocks deployment, voice consent is absent, faithfulness misses target, generation is unstable across seeds, reviewers prefer source-only audio, or resource use violates the declared service envelope. Fall back only to an explicitly planned source-only edit, never a placeholder asset.
|
||||||
|
|
||||||
|
**Risks and obligations.** Threat-model voice impersonation, prompt injection, harmful content, model supply chain, copyrighted training/output concerns, GPU denial of service, and retention of generated voices. Record security and legal approval before promotion.
|
||||||
|
|
||||||
|
## 5. Causal Edit-Component Evaluation
|
||||||
|
|
||||||
|
**Repo-grounded problem.** The renderer applies coupled treatments such as contrast, saturation, sharpening, vignette, crop, transitions, overlays, music, SFX, and voiceover. Current tests prove command construction, not which component changes human judgment.
|
||||||
|
|
||||||
|
**Hypothesis.** Controlled counterfactual renders can identify category- and context-specific effects that cause preference changes, enabling a smaller and safer cinematic vocabulary.
|
||||||
|
|
||||||
|
**Concrete asset.** FFmpeg is a deterministic command backend; `EditPlan`, `HighlightVisualEffectsStage`, and manifests can generate paired variants from identical source decisions.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Freeze edit decisions and assets, then define one-factor removals and a small pre-registered interaction set. Hash all non-varied inputs and randomize blinded presentation order.
|
||||||
|
2. Render paired variants with identical codec settings. Verify duration, frames outside the treatment, and audio outside the treatment remain within declared equivalence tolerances.
|
||||||
|
3. Estimate per-category effect sizes and CIs; repeat the strongest result on new sources and assign an adversarial reviewer to search for confounding.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict that at least one component has an absolute pairwise preference effect of 10 percentage points or more in one category, while at least one currently available component is neutral or harmful.
|
||||||
|
|
||||||
|
**You have a result when:** the effect repeats on held-out sources, the negative control remains neutral, the changed mechanism explains all observed differences, and the effect survives correction for the pre-registered comparisons.
|
||||||
|
|
||||||
|
**Retire or narrow when:** effects reverse across sources without an explanatory moderator, equivalence checks show unintended differences, or reviewer blinding fails. Remove unsupported default treatments through change control; do not universalize a category-specific result.
|
||||||
|
|
||||||
|
**Risks and obligations.** Guard against photosensitive flash, abrupt loudness, reviewer fatigue, and source memorization. Store assignments and analysis code; do not store reviewer identity in application telemetry.
|
||||||
|
|
||||||
|
## 6. Quality-Preserving Local Inference Efficiency
|
||||||
|
|
||||||
|
**Repo-grounded problem.** Local CV is an HTTP worker loading YOLO; local audio generation loads Python/Torch/AudioCraft. The repository has no fixed model artifact set, cross-platform hardware matrix, per-stage memory limits, or quality-versus-runtime benchmark for macOS development, Linux/VPS production, and cloud deployment.
|
||||||
|
|
||||||
|
**Hypothesis.** An uncertainty-gated cascade using cheap temporal features first and expensive local models only for ambiguous segments can reduce RTF and peak memory while remaining within a pre-registered quality-equivalence margin.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse proxy media, thumbnails, shots, process timing logs, the visual provider boundary, local worker health contracts, and benchmark-harness conventions under `src/test/java/org/example/videoclips/perf`.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Declare reproducible macOS CPU, Linux/VPS CPU, and approved cloud CPU/GPU profiles; pin thread counts, model hashes, warm/cold state, corpus split, and power mode. Measure per-stage RTF, peak resident memory, throughput, failures, and quality.
|
||||||
|
2. Implement a disabled cascade that emits uncertainty and escalation reasons. Bound queue depth, concurrency, memory, timeout, and cancellation; never substitute a different analyzer after failure.
|
||||||
|
3. Sweep thresholds on development data, freeze one operating point per hardware profile, and run equivalence tests on held-out sources plus 2x load and 60-minute endurance tests.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict at least 30% lower median analysis RTF and 25% lower peak memory than always-expensive inference, with nDCG@3 loss no greater than `0.02` and no category loss greater than `0.03`.
|
||||||
|
|
||||||
|
**You have a result when:** both efficiency CIs clear their targets, quality stays inside the equivalence margins, memory does not grow continuously, and slowdown or failure does not cause unbounded work accumulation.
|
||||||
|
|
||||||
|
**Retire or narrow when:** quality equivalence fails, uncertainty is miscalibrated on unseen categories, GPU gains disappear after transfer/queue overhead, or separate artifacts create an unmaintainable supply chain. Keep platform-specific results platform-specific.
|
||||||
|
|
||||||
|
**Risks and obligations.** Enforce container/device least privilege, resource quotas, model license compatibility, SBOM coverage, predictable startup, and telemetry without high-cardinality media IDs.
|
||||||
|
|
||||||
|
## 7. Restartable, Reproducible Rendering
|
||||||
|
|
||||||
|
**Repo-grounded problem.** The highlight flow loops through assets and renders, then publishes final files and manifests. It has no checked-in content-addressed stage graph or failure-injection evidence for restart. Runtime timestamps, temporary prompt filenames, model randomness, host tools, and codecs can change bytes.
|
||||||
|
|
||||||
|
**Hypothesis.** A content-addressed stage contract with atomic publication can resume after failure, avoid repeated work, and reproduce exact bytes on one pinned platform plus semantically equivalent media across approved platforms.
|
||||||
|
|
||||||
|
**Concrete asset.** Reuse the folder contract, JSON stores, `RenderManifest`, recorded FFmpeg commands, stable project/highlight IDs, and deterministic edit decisions.
|
||||||
|
|
||||||
|
**First three steps.**
|
||||||
|
|
||||||
|
1. Define a stage key over normalized plan, source/asset/model/tool hashes, render profile, seed, and schema version. Separate immutable stage output from atomic publication; never include wall-clock time or absolute workspace paths in the key.
|
||||||
|
2. Add crash points before/after analysis, asset generation, segment render, mix, QA, and publication. Restart and record reused stages, recomputed stages, orphan cleanup, final status, and manifest lineage.
|
||||||
|
3. Compare clean and resumed runs on each pinned platform. Require exact hashes on the same platform/toolchain; define measured semantic tolerances for duration, frame/audio similarity, loudness, and synchronization across platforms.
|
||||||
|
|
||||||
|
**Prediction before running.** Predict at least 50% less recomputation after a failure beyond the midpoint, zero published partial finals, exact same-platform final hashes for deterministic assets, and all cross-platform measurements inside pre-registered tolerances.
|
||||||
|
|
||||||
|
**You have a result when:** every injected failure resumes or fails terminally without corrupting prior evidence, cache invalidation responds to every declared input change, same-platform hashes repeat, and cross-platform equivalence passes independent probes.
|
||||||
|
|
||||||
|
**Retire or narrow when:** hidden inputs repeatedly invalidate reproducibility, cache validation costs approach recomputation, nondeterministic models cannot expose seeds, or hardware codecs cannot meet exactness. Narrow exact reproducibility to pinned software encoding and retain semantic equivalence elsewhere.
|
||||||
|
|
||||||
|
**Risks and obligations.** Prevent cache poisoning, cross-tenant artifact reuse, path traversal, stale-license reuse, secret inclusion in keys, and disk exhaustion. Encrypt or isolate sensitive stage outputs and apply retention policy.
|
||||||
|
|
||||||
|
## Promote Or Retire
|
||||||
|
|
||||||
|
At the end of every experiment:
|
||||||
|
|
||||||
|
- Publish raw measurements, exclusions, failed predictions, negative controls, reviewer disagreement, licenses, hashes, and resource profiles.
|
||||||
|
- Ask the assigned adversarial reviewer to explain all positive and negative observations with one mechanism and attempt to refute it.
|
||||||
|
- Mark the idea `candidate`, `replicated`, `retired`, or `inconclusive`; never call it production-ready from one experiment.
|
||||||
|
- Require replication on held-out sources and at least the applicable macOS and Linux/VPS profiles before proposing a production change.
|
||||||
|
- Route adoption through `video-editing-change-control`, certification through `video-editing-validation-and-qa`, and any novelty/SOTA statement through `video-editing-external-positioning`.
|
||||||
|
- Preserve a retired idea's hypothesis, commands, evidence, and stop reason so the next engineer does not repeat it unchanged.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
This skill was derived from repository source, tests, configuration, plans, and Git history available on 2026-07-21. Predictions and thresholds are proposed falsifiable targets, not observed results. No external SOTA claims were verified.
|
||||||
|
|
||||||
|
Re-verify volatile facts with no network and no application startup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n "DEFAULT_CANDIDATE_SECONDS|MAX_CANDIDATES_PER_CLIP|source position|Project metadata" src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java
|
||||||
|
rg -n "analyzer.analyze|directorPromptGenerator.generate|CinematicHighlightAnalyzer" src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java
|
||||||
|
rg -n "List.of\(decision\)|step = targetDuration|transitionIn|transitionOut|missing_director_plan" src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java
|
||||||
|
rg -n "duration_matches_timeline|required_assets_resolved|text_overlays_safe|audio_mastering_applied" src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java
|
||||||
|
rg -n "blackDetectCommand|silenceDetectCommand|clippingDetectCommand" src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
|
||||||
|
rg -n "get_pretrained|write_silence|write_fallback_tone|say|espeak" tools/local_asset_worker.py src/main/java/org/example/videoclips/editing/LocalAssetSynthesizer.java
|
||||||
|
rg -n "pip install|YOLO\(|yolov8n.pt" tools/run_local_cv_worker.sh tools/run_local_asset_worker.sh tools/local_cv_worker.py
|
||||||
|
rg -n "^torch$|^audiocraft$|^soundfile$|^numpy$" tools/local_asset_requirements.txt
|
||||||
|
git log --oneline --all -- src/main/java/org/example/videoclips/editing tools docs/production-cinematic-highlight-editing-plan.md
|
||||||
|
mvn -o -Dtest=CinematicHighlightAnalyzerTest,HighlightDirectorFlowServiceTest,LocalCvVisualAnalysisProviderTest test
|
||||||
|
```
|
||||||
|
|
||||||
|
The final command passed locally on 2026-07-21 with eight tests. Offline Maven fails closed if required artifacts are absent; do not remove `-o` to make it pass.
|
||||||
|
|
@ -0,0 +1,366 @@
|
||||||
|
---
|
||||||
|
name: video-editing-research-methodology
|
||||||
|
description: "Load when turning an observation or hunch in this video-editing repository into a controlled experiment, preregistering numeric predictions, designing baselines/counterfactuals/ablations, assigning adversarial review, managing a disabled experiment flag, replicating offline on macOS and Linux, deciding whether a result is a candidate for adoption, or retiring an inconclusive or disproven idea without losing the negative evidence."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Research Methodology
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Turn a hunch into an auditable result without weakening production controls. Require one causal
|
||||||
|
**mechanism** (the rules by which a cause produces an outcome) to explain positive observations,
|
||||||
|
negative observations, boundary cases, and failures. Accept it only after an assigned reviewer tries
|
||||||
|
to refute it with a prediction that differs from competing explanations.
|
||||||
|
|
||||||
|
Use repository facts verified on **2026-07-21**. Treat numeric targets from planning documents or
|
||||||
|
sibling skills as candidate thresholds until `video-editing-change-control` approves and freezes them.
|
||||||
|
|
||||||
|
## Route The Work
|
||||||
|
|
||||||
|
| Need | Use this skill? | Exact sibling |
|
||||||
|
|---|---:|---|
|
||||||
|
| Form a hypothesis, preregister numbers, run a research lifecycle, or retire an idea | Yes | This skill |
|
||||||
|
| Select an open problem or make an external novelty claim | No | `video-editing-research-frontier` or `video-editing-external-positioning` |
|
||||||
|
| Derive a metric, invariant, causal discriminator, or uncertainty calculation | No | `video-editing-proof-and-analysis-toolkit` |
|
||||||
|
| Execute the current single-source cinematic-quality program | No | `video-editing-cinematic-highlights-campaign` |
|
||||||
|
| Define release evidence, goldens, or the acceptance portfolio | No | `video-editing-validation-and-qa` |
|
||||||
|
| Diagnose an unexplained live failure | No | `video-editing-debugging-playbook`, then `video-editing-failure-archaeology` |
|
||||||
|
| Change code, dependencies, configuration, defaults, runtime, or promotion status | No | `video-editing-change-control` |
|
||||||
|
| Look up an existing property or add a configuration axis | No | `video-editing-config-and-flags` |
|
||||||
|
|
||||||
|
Do not use this skill for routine implementation, incident triage, release certification, or current-campaign execution; use the exact sibling named in the table.
|
||||||
|
|
||||||
|
Do not label missing authentication, offline packaging, fail-closed behavior, reproducible builds,
|
||||||
|
measured QA, or deployment controls as research. Those are engineering obligations. Use research only
|
||||||
|
when two or more plausible mechanisms need a controlled comparison.
|
||||||
|
|
||||||
|
## Define The Terms Once
|
||||||
|
|
||||||
|
| Term | Definition in this project |
|
||||||
|
|---|---|
|
||||||
|
| Hypothesis | A falsifiable claim that names a mechanism and predicts measured outcomes before data collection. |
|
||||||
|
| Baseline | The frozen current or simplest approved method against which a candidate is compared. |
|
||||||
|
| Counterfactual | An otherwise equivalent case in which the proposed cause is absent or changed. |
|
||||||
|
| Ablation | A counterfactual made by removing exactly one candidate component while holding the rest fixed. |
|
||||||
|
| Negative control | A case where the mechanism predicts no effect; it detects leakage, confounding, and false positives. |
|
||||||
|
| Slice | A preregistered subgroup such as source category, speech presence, lighting, codec, platform, or failure mode. |
|
||||||
|
| Holdout | Data frozen before tuning and opened only for the confirmatory run. |
|
||||||
|
| Uncertainty | A stated interval or error bound around an estimate, produced by a preregistered method. |
|
||||||
|
| Replication | A repeat on independently provisioned runtime or unseen sources, not a rerun of the same warm process. |
|
||||||
|
| Adversarial reviewer | An independent person assigned before results to construct the strongest counterexample and try to falsify the mechanism. |
|
||||||
|
| Candidate | A replicated research result eligible for change-control review; it is not production-ready. |
|
||||||
|
| Retirement | A documented decision to stop an idea while preserving its prediction, evidence, and revisit condition. |
|
||||||
|
| nDCG | Normalized discounted cumulative gain, a ranking metric that rewards relevant candidates more when they appear earlier. |
|
||||||
|
|
||||||
|
## Enforce The Research Perimeter
|
||||||
|
|
||||||
|
Stop before experimentation when any item fails:
|
||||||
|
|
||||||
|
- [ ] Run with network access denied. Do not call external AI services, model hubs, package indexes,
|
||||||
|
remote telemetry, or loopback model services during certified experiments.
|
||||||
|
- [ ] Pre-provision every dependency and model. Record immutable version, SHA-256, source, license,
|
||||||
|
approved use, platform, runtime loader, and resource envelope. Never bootstrap or download at runtime.
|
||||||
|
- [ ] Use only rights-cleared footage, voices, music, sound effects, fonts, LUTs, and model outputs.
|
||||||
|
Record consent or voice rights where applicable. Generated output is not automatically licensed.
|
||||||
|
- [ ] Reject missing capabilities. Never replace requested voiceover, music, or SFX with silence,
|
||||||
|
tones, host TTS, generic media, another model, or another analysis mechanism.
|
||||||
|
- [ ] Obtain explicit render approval before FFmpeg creates a candidate render. Analysis permission is
|
||||||
|
not render permission.
|
||||||
|
- [ ] Leave every production-facing packaged default unchanged. Activate research only in an isolated,
|
||||||
|
approved environment with a dedicated experiment control whose safe default is off.
|
||||||
|
- [ ] Keep research inputs and results out of production queues, tenant data, normal application logs,
|
||||||
|
and uncontrolled metrics. Preserve privacy and retention classifications.
|
||||||
|
- [ ] Route any executable code, flag, schema, fixture, dependency, or behavior change through
|
||||||
|
`video-editing-change-control` before use.
|
||||||
|
|
||||||
|
Do not run `tools/run_local_cv_worker.sh`, `tools/run_local_asset_worker.sh`, or ordinary application
|
||||||
|
startup as an experiment. On the verified baseline, those paths can install packages, resolve model
|
||||||
|
weights, scan inputs, or render. Inspect first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
|
||||||
|
rg -n 'auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:' src/main/resources/application.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use The Research Lifecycle
|
||||||
|
|
||||||
|
Never skip a gate. A failed gate produces a negative result or retirement record, not an improvised
|
||||||
|
change to the method.
|
||||||
|
|
||||||
|
| Stage | Required artifact | Gate to advance |
|
||||||
|
|---:|---|---|
|
||||||
|
| 0. Observe | Observation record with evidence label | The symptom is reproducible or explicitly marked historical/open. |
|
||||||
|
| 1. Hypothesize | Preregistered experiment record | Mechanism, alternatives, numeric predictions, controls, slices, and stop rules are frozen. |
|
||||||
|
| 2. Isolate | Approved disabled experiment control and evidence location | Default is off; scope, owner, expiry, cleanup, and production isolation are verified. |
|
||||||
|
| 3. Calibrate | Frozen baseline on development data | Metric definitions, denominators, uncertainty method, and data exclusions are fixed. |
|
||||||
|
| 4. Discriminate | Positive, negative, boundary, counterfactual, and ablation results | Competing mechanisms make different predictions and the observations select among them. |
|
||||||
|
| 5. Replicate | Unseen-source and platform replication packet | Result repeats offline on supported macOS development and target Linux/VPS runtime. |
|
||||||
|
| 6. Refute | Signed adversarial review | Independent reviewer cannot produce an unexplained counterexample within the preregistered scope. |
|
||||||
|
| 7. Decide | Adoption candidate or retirement record | Evidence is complete; no result is silently left behind an expiring flag. |
|
||||||
|
| 8. Govern | Change-control decision | Adopt through normal gates or remove/disable and preserve the negative ledger. |
|
||||||
|
|
||||||
|
### Stage 0: Record The Observation
|
||||||
|
|
||||||
|
Distinguish an observation from an explanation. Record the exact source commit, input identity,
|
||||||
|
effective configuration, environment, and command that exposed it. Label a generated file, plan
|
||||||
|
checkbox, or hard-coded QA boolean `NOT EVIDENCE` when it cannot support the claim.
|
||||||
|
|
||||||
|
Before inventing a new explanation, search the established history:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log --oneline --decorate --all
|
||||||
|
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|fallback|silence|tone' docs src tools
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `video-editing-failure-archaeology` to determine whether the mechanism is already fixed, active,
|
||||||
|
partial, historical, or open. Do not rerun a retired path unchanged.
|
||||||
|
|
||||||
|
### Stage 1: Preregister A Numeric Hypothesis
|
||||||
|
|
||||||
|
Write and freeze the experiment record before inference, rendering, or opening the holdout. Make the
|
||||||
|
hypothesis predict numbers, not adjectives. Include expected direction, minimum effect, allowed
|
||||||
|
regression, denominator, interval method, and expected values for every negative control.
|
||||||
|
|
||||||
|
When no trustworthy baseline exists, run a labeled **exploratory baseline study** first. Use its data
|
||||||
|
only to design a later confirmatory study. Do not set a threshold from a dataset and claim success on
|
||||||
|
that same dataset.
|
||||||
|
|
||||||
|
For highlight work, name metrics such as nDCG@3, Recall@3 at a stated temporal IoU, false-positive
|
||||||
|
rate on no-highlight footage, per-category effects, and blinded preference. For audio/render work,
|
||||||
|
name probe-derived duration, loudness, true peak, silence/tone rejection, A/V synchronization,
|
||||||
|
technical failure rate, and blinded rubric dimensions. Get formulas and current campaign floors from
|
||||||
|
the proof toolkit and cinematic campaign; do not duplicate or silently revise them here.
|
||||||
|
|
||||||
|
### Stage 2: Isolate With A Safe Experiment Control
|
||||||
|
|
||||||
|
The verified repository has no dedicated experiment registry. Current scheduler, fallback, model,
|
||||||
|
and render properties are operational controls, not research assignment controls. Do not repurpose
|
||||||
|
them and do not treat `render-enabled` or `fallback-to-heuristic` as experiment flags.
|
||||||
|
|
||||||
|
Introduce any experiment control only through change control. Require this contract:
|
||||||
|
|
||||||
|
| Field | Required value |
|
||||||
|
|---|---|
|
||||||
|
| Identifier | Stable, descriptive, and unique; never a user or media identifier. |
|
||||||
|
| Owner | Named accountable team/person and independent reviewer. |
|
||||||
|
| Safe default | `false` in every packaged configuration and `matchIfMissing`; missing means disabled. |
|
||||||
|
| Scope | Approved fixtures/environments only; never implicit all-traffic assignment. |
|
||||||
|
| Start and expiry | UTC dates with a short expiry and automated or reviewed removal checkpoint. |
|
||||||
|
| Activation | Explicit test/experiment configuration outside production defaults. |
|
||||||
|
| Observability | Bounded, low-cardinality experiment ID; no source path, prompt, secret, or personal data. |
|
||||||
|
| Failure | Fail the experiment; never fall back to baseline and label the result candidate. |
|
||||||
|
| Removal | Delete or permanently disable after adoption/retirement; owner verifies no stale path remains. |
|
||||||
|
|
||||||
|
Do not create a speculative flag without a funded experiment, owner, decision date, and deletion
|
||||||
|
condition. A flag is not approval to render or bypass asset/model preflight.
|
||||||
|
|
||||||
|
### Stage 3: Freeze Baselines, Inputs, And Analysis
|
||||||
|
|
||||||
|
Use a source-disjoint development/holdout split. Compute a corpus-manifest SHA-256 and split SHA-256.
|
||||||
|
Freeze the baseline commit, candidate commit, toolchain, feature schema, random seeds, render plan,
|
||||||
|
models, assets, licenses, and analysis script before the confirmatory run.
|
||||||
|
|
||||||
|
| Design element | Required practice | Common invalid result |
|
||||||
|
|---|---|---|
|
||||||
|
| Baseline | Run the same inputs, candidate budget, render profile, and reviewer protocol. | Candidate gets more clips, assets, compute, or review time. |
|
||||||
|
| Counterfactual | Change only the claimed cause and verify output equivalence elsewhere. | Two renders differ in timing, encoding, or mix beyond the intended factor. |
|
||||||
|
| Ablation | Remove one component per variant; preregister multiple-comparison handling. | A kitchen-sink comparison cannot assign causality. |
|
||||||
|
| Negative control | Include shuffled timestamps, no-highlight footage, silence/tone, missing model, or unrelated audio as appropriate. | Positive-only examples hide leakage or fallback. |
|
||||||
|
| Slice | Report each preregistered category/platform/failure slice and its denominator. | Aggregate improvement hides a harmed category. |
|
||||||
|
| Uncertainty | Report confidence interval or error bound and method; include reviewer disagreement. | A point estimate or one clip is called a result. |
|
||||||
|
| Exclusion | Freeze objective exclusion rules before results; publish every exclusion count/reason. | Failed outputs disappear from the denominator. |
|
||||||
|
|
||||||
|
### Stage 4: Execute Once, Preserve Everything
|
||||||
|
|
||||||
|
Run only the exact approved commands in the record. Capture stdout/stderr, exit codes, timestamps,
|
||||||
|
resource measurements, raw per-item results, failed outputs, and hashes. Do not edit the hypothesis,
|
||||||
|
threshold, split, or exclusion rule after seeing results. Record deviations and classify the run
|
||||||
|
`invalid`; preregister a replacement instead of repairing history.
|
||||||
|
|
||||||
|
Add every failed prediction, neutral ablation, negative control, and platform difference to the
|
||||||
|
negative-result ledger. Preserve a false start even when a later variant works.
|
||||||
|
|
||||||
|
### Stage 5: Replicate Offline On macOS And Linux
|
||||||
|
|
||||||
|
Use the same committed source and immutable corpus, split, dependency, model, and asset manifests.
|
||||||
|
Provision them before execution through the approved artifact process. Deny egress for the whole
|
||||||
|
process tree. Record OS version, architecture, Java, Maven, Python runtime if applicable, FFmpeg,
|
||||||
|
ffprobe, CPU/GPU, memory, precision, thread/concurrency settings, seeds, and hashes.
|
||||||
|
|
||||||
|
Define replication before running as either:
|
||||||
|
|
||||||
|
- **Byte replication:** identical artifact SHA-256 is required; or
|
||||||
|
- **Metric-bounded replication:** deterministic bytes are not expected, so every preregistered metric
|
||||||
|
must remain within an approved tolerance and every categorical gate must agree.
|
||||||
|
|
||||||
|
Do not average platform disagreement away. Mark the result platform-specific or inconclusive when
|
||||||
|
one supported platform fails. A warm-cache repeat on one machine is not replication.
|
||||||
|
|
||||||
|
### Stage 6: Assign Adversarial Refutation
|
||||||
|
|
||||||
|
Assign the reviewer before results. The author, experiment operator, fixture annotator, and promoter
|
||||||
|
must not be the sole adversarial reviewer. Blind media variants and randomize order where practical.
|
||||||
|
|
||||||
|
Require the reviewer to:
|
||||||
|
|
||||||
|
1. State the strongest competing mechanism.
|
||||||
|
2. Identify one case where the proposed mechanism predicts an effect and the competitor does not.
|
||||||
|
3. Identify one negative case where the proposed mechanism predicts no effect.
|
||||||
|
4. Look for data leakage, source overlap, hidden fallbacks, changed budgets, reviewer unblinding,
|
||||||
|
selective exclusions, uncontrolled randomness, and platform-specific behavior.
|
||||||
|
5. Recompute a sample from raw data and verify hashes/licenses for a sample of inputs/models/assets.
|
||||||
|
6. Explain every positive, negative, boundary, and failed observation with the proposed mechanism.
|
||||||
|
7. Return `supported`, `refuted`, or `inconclusive`; never return “looks good.”
|
||||||
|
|
||||||
|
Refutation is not an approval gate substitute. A supported result still goes through change control,
|
||||||
|
validation, security, operations, licensing, and render approval.
|
||||||
|
|
||||||
|
### Stage 7: Adopt Or Retire
|
||||||
|
|
||||||
|
Classify the idea exactly once at the decision date:
|
||||||
|
|
||||||
|
| Status | Meaning | Required next action |
|
||||||
|
|---|---|---|
|
||||||
|
| `candidate` | First controlled result met its preregistered gate. | Replicate; do not promote. |
|
||||||
|
| `replicated-candidate` | Held-out and platform replication passed adversarial review. | Submit the eight-part decision packet to change control. |
|
||||||
|
| `inconclusive` | Evidence cannot distinguish mechanisms or uncertainty crosses the gate. | Retire temporarily or fund a newly preregistered discriminator. |
|
||||||
|
| `refuted` | A prediction failed or a counterexample broke the mechanism. | Retire; remove/disable the experiment path. |
|
||||||
|
| `retired` | Work stopped for evidence, cost, rights, safety, operability, or priority reasons. | Preserve the ledger, stop reason, and revisit condition. |
|
||||||
|
|
||||||
|
For an adoption candidate, provide: requirement; selected approach; alternatives; benefits/trade-offs;
|
||||||
|
operational consequences; security implications; verification; and revisit conditions. Then follow
|
||||||
|
`video-editing-change-control`. Research status never changes a production default.
|
||||||
|
|
||||||
|
## Use These Records
|
||||||
|
|
||||||
|
Keep each record in the approved review artifact or a repository docs-of-record location established
|
||||||
|
through change control. Do not invent a new production path or store secrets, proprietary model
|
||||||
|
weights, personal data, or unlicensed media in the record.
|
||||||
|
|
||||||
|
### Experiment Record Template
|
||||||
|
|
||||||
|
```text
|
||||||
|
Experiment ID / title / UTC date:
|
||||||
|
Owner / operator / independent adversarial reviewer:
|
||||||
|
Observation and evidence label:
|
||||||
|
Question:
|
||||||
|
Mechanism:
|
||||||
|
Strongest competing mechanisms:
|
||||||
|
Baseline commit and exact method:
|
||||||
|
Candidate commit and safe-disabled experiment control:
|
||||||
|
Flag owner / safe default=false / start / expiry / deletion condition:
|
||||||
|
Corpus-manifest hash / split hash / annotation version / licenses:
|
||||||
|
Dependency, binary, model, asset hashes and approved-use records:
|
||||||
|
macOS and Linux runtime/hardware/resource envelope:
|
||||||
|
Primary metric, denominator, formula, predicted baseline and candidate numbers:
|
||||||
|
Minimum effect / allowed regression / uncertainty method / alpha or interval:
|
||||||
|
Slice predictions and minimum sample counts:
|
||||||
|
Positive / boundary / negative controls and predicted numbers:
|
||||||
|
Counterfactuals / one-factor ablations / multiple-comparison method:
|
||||||
|
Randomization / blinding / seeds / exclusion rules:
|
||||||
|
Exact offline commands and egress-denial evidence:
|
||||||
|
Render approval reference, if rendering is required:
|
||||||
|
Stop rules / retirement rule / revisit condition:
|
||||||
|
Raw evidence location and retention/classification:
|
||||||
|
Observed results, deviations, failed predictions, and uncertainty (after run):
|
||||||
|
Replication outcome by platform (after run):
|
||||||
|
Disposition and change-control reference (after review):
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adversarial Review Template
|
||||||
|
|
||||||
|
```text
|
||||||
|
Reviewer / UTC date / independence conflicts:
|
||||||
|
Mechanism reviewed:
|
||||||
|
Strongest alternative:
|
||||||
|
Differentiating positive prediction:
|
||||||
|
Required negative prediction:
|
||||||
|
Counterexample attempted and exact command/data:
|
||||||
|
Leakage/fallback/blinding/exclusion/platform checks:
|
||||||
|
Raw result recomputation and hash/license sample:
|
||||||
|
Unexplained positive, negative, boundary, or failure observations:
|
||||||
|
Verdict: supported | refuted | inconclusive
|
||||||
|
Required follow-up and change-control blockers:
|
||||||
|
```
|
||||||
|
|
||||||
|
### Negative-Result Ledger Template
|
||||||
|
|
||||||
|
```text
|
||||||
|
Experiment ID / UTC date / commit:
|
||||||
|
Hypothesis and preregistered numeric prediction:
|
||||||
|
Observed number and uncertainty:
|
||||||
|
Failed slice/control/ablation/platform:
|
||||||
|
Evidence hashes and exact command:
|
||||||
|
Why the mechanism failed or remains unresolved:
|
||||||
|
Decision: refuted | inconclusive | retired
|
||||||
|
Experiment control removed/disabled by / date:
|
||||||
|
Do not retry unchanged because:
|
||||||
|
Revisit only when this falsifiable condition changes:
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stop And Revisit Discipline
|
||||||
|
|
||||||
|
Stop immediately for a rights gap, checksum mismatch, unexpected network attempt, automatic download,
|
||||||
|
external AI call, missing model, fallback, placeholder audio, unapproved render, path escape, sensitive
|
||||||
|
data leak, production-default drift, invalid blinding, split contamination, or corrupted evidence.
|
||||||
|
|
||||||
|
Stop the idea at the preregistered resource cap, sample size, failure-rate limit, effect threshold, or
|
||||||
|
expiry date. Do not extend until significance appears. Revisit a retired idea only when its recorded
|
||||||
|
condition changes, such as a newly licensed local model, a larger source-disjoint corpus, a corrected
|
||||||
|
measurement defect, a new discriminating mechanism, or a documented resource-envelope change.
|
||||||
|
|
||||||
|
## Learn From The Repository's Actual Origins
|
||||||
|
|
||||||
|
| Historical origin | Methodological rule it established |
|
||||||
|
|---|---|
|
||||||
|
| Real storage behavior arrived across commits beginning `7e8a214`; earlier state/metadata was not a durable media path. | Measure the real side effect and artifact existence. A successful state transition is not delivery evidence. |
|
||||||
|
| Commit `1737d8b` moved terminal failure ownership to the DB queue's retry/DLQ policy. | Put the decision at the layer that owns the mechanism; test transient, terminal, duplicate, and boundary attempts. |
|
||||||
|
| Commit `5d889b0` said “working version but not cinematic.” | Separate renderability from semantic selection and creative quality; require blinded baselines, rubrics, and ablations. |
|
||||||
|
| Category-aware planning and richer rendering followed in commits such as `5006dfd`, but no controlled causal comparison is recorded. | Treat richer analysis, effects, and assets as candidates until each contribution survives counterfactual testing. |
|
||||||
|
| Commit `9b56e89` added structural render QA; the current highlight path adds selected duration/black/silence/peak probes while other checks remain structural. | Diagnostics must distinguish measured outputs from plan/command assertions and predict reviewer-relevant failures. |
|
||||||
|
| Commit `97ba827` added local asset generation with runtime install/model resolution and silence/tone paths. | “Local” does not mean offline, licensed, immutable, or fail-closed; test missing and corrupted capabilities as negative controls. |
|
||||||
|
|
||||||
|
Use `video-editing-failure-archaeology` for the full chronicle. History suggests research questions;
|
||||||
|
it does not prove a current mechanism.
|
||||||
|
|
||||||
|
## Completion Checklist
|
||||||
|
|
||||||
|
- [ ] Observation, mechanism, alternatives, and numeric predictions were frozen before the run.
|
||||||
|
- [ ] Baseline, counterfactuals, one-factor ablations, negative controls, boundary cases, slices,
|
||||||
|
denominators, exclusions, and uncertainty were specified.
|
||||||
|
- [ ] Dependencies, models, assets, inputs, splits, licenses, approvals, and checksums are complete.
|
||||||
|
- [ ] The experiment was isolated behind an approved safe-disabled control with owner and expiry.
|
||||||
|
- [ ] No download, network, external AI, unlicensed asset, placeholder, unapproved render, fallback, or
|
||||||
|
production-default change occurred.
|
||||||
|
- [ ] The result replicated offline on supported macOS and target Linux/VPS, or is labeled otherwise.
|
||||||
|
- [ ] An independent reviewer attempted adversarial refutation and explained positive and negative cases.
|
||||||
|
- [ ] Failed predictions and neutral results are in the negative-result ledger.
|
||||||
|
- [ ] The idea became a replicated candidate routed through change control, or was documented and retired.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Facts and volatile repository observations were rechecked on **2026-07-21** against
|
||||||
|
`application.yml`, current source/tests, the six cited commits, and the completed sibling skills.
|
||||||
|
|
||||||
|
Re-verify the research perimeter and operational-control defaults:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
|
||||||
|
rg -n -C 3 'fallback-to-heuristic:|strict-runtime:|local-director:|highlight-scheduler:|render-enabled:|require-director-approval:' src/main/resources/application.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify historical claims and measurement implementations:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show -s --format='%h %cs %s' 7e8a214 1737d8b 5d889b0 5006dfd 9b56e89 97ba827
|
||||||
|
rg -n 'RETRY_SCHEDULED|markTerminalFailure|duration_matches_timeline' src/main src/test
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-verify the absence or presence of a dedicated experiment registry before documenting a flag:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n -i 'experiment|feature.?flag' pom.xml src/main src/test src/main/resources docs
|
||||||
|
```
|
||||||
|
|
||||||
|
If any command changes its result, update this skill and the authoritative sibling in the same
|
||||||
|
knowledge-only change. Do not reinterpret drift as approval to alter behavior.
|
||||||
|
|
@ -0,0 +1,345 @@
|
||||||
|
---
|
||||||
|
name: video-editing-run-and-operate
|
||||||
|
description: "Operate the video-editing service safely: choose and start exactly one of its REST clipping, folder segmentation, multi-clip cinematic edit, or single-source highlight workflows; perform preflight, ingest and approve work, locate artifacts, inspect health and queue state, quarantine failures, and stop or resume jobs. Load this skill for runtime commands, operator incidents, local-model readiness, output locations, scheduler behavior, or deployment-readiness questions."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Run and Operate Video Editing
|
||||||
|
|
||||||
|
Use this runbook from the repository root. It describes code verified on **2026-07-21**.
|
||||||
|
|
||||||
|
## Scope and safety boundary
|
||||||
|
|
||||||
|
Operate only an explicitly approved workflow. The checked-in `application.yml` still enables folder ingestion, editing, both editing schedulers, local worker auto-start, and heuristic fallback, while highlight rendering now defaults off and director approval defaults on. A plain `mvn spring-boot:run` or packaged-JAR start can therefore claim input and invoke the network-capable local-CV bootstrap path. The `cinematic-editing-local` profile explicitly enables the separate local-director auto-render path behind its approval flag.
|
||||||
|
|
||||||
|
`video-editing-change-control` is authoritative for the seven no-waiver prohibitions. Their operating consequences are:
|
||||||
|
|
||||||
|
- automatic dependency or model downloads;
|
||||||
|
- no network use for model/media inference or acquisition, including loopback RPC, and no external AI; use in-process or approved non-network IPC with pre-provisioned artifacts;
|
||||||
|
- unlicensed music, SFX, fonts, LUTs, models, or footage;
|
||||||
|
- silence, generated tones, or heuristic analysis presented as production assets or evidence;
|
||||||
|
- rendering without a reviewed plan and approval artifact;
|
||||||
|
- a production-facing default change outside change control.
|
||||||
|
|
||||||
|
**Current implementation warning:** strict asset readiness now fails startup and the Java/Python asset workers no longer accept silence, tones, or host speech as successful fallbacks. The repository still lacks resident Piper, MusicGen, and AudioGen model artifacts, and the standalone bootstrap script remains prohibited. Do not start the production profile until approved local paths are provisioned.
|
||||||
|
|
||||||
|
### When not to use this skill
|
||||||
|
|
||||||
|
| Need | Use instead |
|
||||||
|
|---|---|
|
||||||
|
| Change a flag or add a profile | `video-editing-config-and-flags` |
|
||||||
|
| Install tools or reconstruct the build | `video-editing-build-and-env` |
|
||||||
|
| Diagnose a failure after locating its state | `video-editing-debugging-playbook` |
|
||||||
|
| Decide whether an output passes | `video-editing-validation-and-qa` |
|
||||||
|
| Change runtime behavior or promote it | `video-editing-change-control` |
|
||||||
|
| Measure FFmpeg, queue, storage, or media behavior | `video-editing-diagnostics-and-tooling` |
|
||||||
|
|
||||||
|
## Definitions
|
||||||
|
|
||||||
|
| Term | Meaning here |
|
||||||
|
|---|---|
|
||||||
|
| Claim | An atomic move from a source directory to a working directory. The source name is not available for a second worker after the move. |
|
||||||
|
| Publish | Copy or upload a completed artifact to its externally consumed location. |
|
||||||
|
| Approval artifact | The current plain `approved.flag` scheduler signal. Its contents are not validated, so it is not production authorization. |
|
||||||
|
| Quarantine | Move failed input to `rejected`, or leave/rename it in `working` when that move fails. |
|
||||||
|
| Local model | Model weights provisioned before startup on the same runtime host; no request may fetch weights or call an external inference service. |
|
||||||
|
| Project store | A filesystem tree below `output/edit-projects` or `output/highlight-projects`; it is not PostgreSQL-backed. |
|
||||||
|
|
||||||
|
## Current system versus production target
|
||||||
|
|
||||||
|
| Concern | Current, verified | Required production target |
|
||||||
|
|---|---|---|
|
||||||
|
| Runtime packaging | Spring Boot JAR; no container or deployment manifest | One immutable platform-neutral application artifact plus signed platform-specific runtime/model/image bundles, each promoted without rebuild for the same target |
|
||||||
|
| REST metadata | Memory by default; `jpa` profile uses H2 in-memory | PostgreSQL with validated migrations, backup/restore, credentials, TLS, readiness, and persistent queue |
|
||||||
|
| REST media | Memory adapter emits example URLs and stub bytes; S3 adapter exists | Approved object storage with real multipart lifecycle, least privilege, encryption, retention, and integration evidence |
|
||||||
|
| Edit/highlight state | Local filesystem | Durable shared storage or an explicitly justified single-writer volume with recovery evidence |
|
||||||
|
| Highlight intelligence | Noncompliant loopback CV endpoint plus manual/external director-plan creation | All required director, vision, voice, music, and SFX models pre-provisioned; inference uses in-process or non-network IPC and fails closed |
|
||||||
|
| Asset failure | Non-strict readiness can log and continue, but synthesis now rejects missing/unlicensed models and inaudible output; requested render assets require license sidecars | Production startup must fail closed; add checksum, origin, allowed-use, and creative-quality gates beyond the current sidecar/audibility checks |
|
||||||
|
| Security | No Spring Security; Actuator exposes health, info, metrics, Prometheus | Authentication (AuthN), authorization (AuthZ), secured Actuator, restrictive network policy, audit trail, redaction, and threat-model gates |
|
||||||
|
|
||||||
|
There is no production profile, Open Container Initiative (OCI) image, Compose file, Kubernetes/Helm definition, or cloud deployment contract in this repository. Do not invent a deployment command. A Linux/VPS/cloud run is a **target**, not a certified procedure.
|
||||||
|
|
||||||
|
| Profile | Verified effect and operational status |
|
||||||
|
|---|---|
|
||||||
|
| no profile | Unsafe all-on file schedulers/editing defaults from `application.yml`; API uses memory/stub adapters. |
|
||||||
|
| `cinematic-editing-local` | Disables folder segmentation but enables local director auto-render with an approval gate; it does not disable the highlight scheduler inherited from base configuration. Do not use as a safe baseline. |
|
||||||
|
| `folder-scheduler-local` | Enables folder segmentation but does not disable inherited editing/highlight defaults. Do not use as an isolated folder lane. |
|
||||||
|
| `jpa` | Selects JPA and DB queue with H2 in PostgreSQL compatibility mode and stub processing. It is not a PostgreSQL production profile. |
|
||||||
|
|
||||||
|
## Preflight every run
|
||||||
|
|
||||||
|
Run read-only checks before approval. These commands neither start the service nor modify repository files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
java -version
|
||||||
|
mvn -version
|
||||||
|
ffmpeg -version
|
||||||
|
ffprobe -version
|
||||||
|
test -f pom.xml
|
||||||
|
test -f target/video-editing-1.0-SNAPSHOT.jar
|
||||||
|
```
|
||||||
|
|
||||||
|
The project requires Java 21. An offline Maven command fails instead of downloading missing dependencies:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mvn -o -DskipTests package
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not replace `-o` with an online build. Provision Maven artifacts through an approved, audited build process. Do not run either `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh`: their `auto` modes can create virtual environments, install packages, and load/download a YOLO model.
|
||||||
|
|
||||||
|
Before enabling any media workflow, check directories and binaries without starting it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -x "$(command -v ffmpeg)"
|
||||||
|
test -x "$(command -v ffprobe)"
|
||||||
|
find input -maxdepth 4 -type f -print 2>/dev/null
|
||||||
|
find output -maxdepth 4 -type f -print 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
Treat every listed input as work that a scheduler may claim. Confirm ownership, license, retention, and approval before continuing.
|
||||||
|
|
||||||
|
## Disabled-first baseline
|
||||||
|
|
||||||
|
Use this exact baseline for inspection-only startup. It binds to loopback, disables all file schedulers, editing, local worker bootstrap, rendering, fallbacks, and cleanup. `-o` prevents Maven dependency downloads.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
env \
|
||||||
|
SERVER_ADDRESS=127.0.0.1 \
|
||||||
|
FOLDER_SCHEDULER_ENABLED=false \
|
||||||
|
VIDEO_EDITING_ENABLED=false \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED=false \
|
||||||
|
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED=false \
|
||||||
|
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED=false \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=false \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=true \
|
||||||
|
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL=true \
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=false \
|
||||||
|
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
|
||||||
|
mvn -o spring-boot:run
|
||||||
|
```
|
||||||
|
|
||||||
|
For a prebuilt artifact, replace the last line with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
java -jar target/video-editing-1.0-SNAPSHOT.jar
|
||||||
|
```
|
||||||
|
|
||||||
|
Environment variables precede the command and override packaged values. Do not add `cinematic-editing-local` or `folder-scheduler-local`; those profiles enable work. Startup success only proves Spring context creation, not dependency connectivity: the repository, queue, and object-storage health indicators always report `UP` with adapter names and do not perform a real probe.
|
||||||
|
|
||||||
|
## Workflow 1: REST clipping API
|
||||||
|
|
||||||
|
### Current behavior
|
||||||
|
|
||||||
|
The API implements multipart-session metadata, clip jobs, cancellation, events, and signed URL responses. It is not a production upload service with default adapters: memory storage returns `storage.example` URLs, materializes stub content, and memory/stub state disappears on restart.
|
||||||
|
|
||||||
|
| Object | States |
|
||||||
|
|---|---|
|
||||||
|
| Asset | `PENDING_UPLOAD`, `UPLOADED`, `PROCESSING`, `READY`, `DELETED` |
|
||||||
|
| Upload | `OPEN`, `COMPLETED`, `ABORTED`, `EXPIRED` |
|
||||||
|
| Clip job | `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `CANCEL_REQUESTED`, `CANCELLED` |
|
||||||
|
| DB queue message | `PENDING`, `PROCESSING`, `COMPLETED`, `DLQ` |
|
||||||
|
|
||||||
|
### Start and exercise safely
|
||||||
|
|
||||||
|
Use the disabled-first baseline. It leaves API defaults at memory repository, memory storage, and stub processing. These are contract-demonstration adapters only; label all results non-production. Send `X-Tenant-Id` and stable `Idempotency-Key` values on create calls. Relevant routes are:
|
||||||
|
|
||||||
|
```text
|
||||||
|
POST /v1/video-assets
|
||||||
|
POST /v1/video-assets/{assetId}/uploads:complete
|
||||||
|
GET /v1/video-assets/{assetId}
|
||||||
|
GET /v1/video-assets/{assetId}/upload-session
|
||||||
|
POST /v1/video-assets/{assetId}/clip-jobs
|
||||||
|
GET /v1/video-assets/{assetId}/clip-jobs
|
||||||
|
GET /v1/video-assets/{assetId}/events
|
||||||
|
DELETE /v1/video-assets/{assetId}
|
||||||
|
GET /v1/clip-jobs/{jobId}
|
||||||
|
GET /v1/clip-jobs/{jobId}/clips
|
||||||
|
GET /v1/clip-jobs/{jobId}/events
|
||||||
|
POST /v1/clip-jobs/{jobId}:cancel
|
||||||
|
POST /v1/clips/{clipId}/download-url
|
||||||
|
```
|
||||||
|
|
||||||
|
A job publish on the memory queue invokes processing asynchronously in the same JVM. The database queue polls, claims with a visibility timeout, retries, and moves exhausted messages to `DLQ`. Cancellation is cooperative only before clip generation; do not claim that it interrupts FFmpeg.
|
||||||
|
|
||||||
|
### PostgreSQL and S3 gate
|
||||||
|
|
||||||
|
`application-jpa.properties` selects JPA plus the DB queue but configures H2, not PostgreSQL. PostgreSQL migrations `V1` through `V6` exist, but there is no production datasource profile or Testcontainer certification. The S3 adapter is present, but no approved environment, credentials contract, or end-to-end multipart validation is checked in. Treat both as candidates. Do not set `video-clipping.storage=s3`, `video-clipping.repository=jpa`, or `video-clipping.queue=db` in production until change control approves integration, recovery, security, and fault-injection evidence.
|
||||||
|
|
||||||
|
Scratch files appear under `tmp/ffmpeg-input`, `tmp/ffmpeg-output`, `tmp/in-memory-storage`, and `tmp/stub-output`. Successful processing deletes staged/generated worker files when `cleanup-local-files=true`; the scheduled cleanup also removes aged files when global cleanup is enabled.
|
||||||
|
|
||||||
|
## Workflow 2: folder segmentation
|
||||||
|
|
||||||
|
This scheduler turns one source video into numbered fixed-duration clips. It is independent of the REST API and edit project stores.
|
||||||
|
|
||||||
|
### Approved start
|
||||||
|
|
||||||
|
Obtain explicit approval for the exact source and output directories. Start only with all editing lanes off:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
env \
|
||||||
|
SERVER_ADDRESS=127.0.0.1 \
|
||||||
|
FOLDER_SCHEDULER_ENABLED=true \
|
||||||
|
VIDEO_EDITING_ENABLED=false \
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
|
||||||
|
mvn -o spring-boot:run
|
||||||
|
```
|
||||||
|
|
||||||
|
Defaults are `input/source`, `input/working`, `input/processed`, `input/rejected`, and `output/clips`, polled every five seconds with eight-second segments. Override paths through the documented `FOLDER_SCHEDULER_*` variables before approval when isolation is required.
|
||||||
|
|
||||||
|
### Ingest, claim, publish, quarantine
|
||||||
|
|
||||||
|
1. Copy a complete source as `name.mp4.part` in `input/source`.
|
||||||
|
2. Verify size and checksum outside the service.
|
||||||
|
3. Atomically rename it to a supported final extension only after approval. Accepted extensions are `mp4`, `mov`, `m4v`, `mkv`, `webm`, and `avi`; hidden files and `.tmp`, `.part`, `.download`, `.failed` are ignored.
|
||||||
|
4. Expect an atomic move to `input/working`.
|
||||||
|
5. Expect clips in `output/clips/<stem>[-N]/` and then the source in `input/processed`. The first output uses `<stem>`; if that output directory already exists, the clipper chooses `<stem>-1`, `<stem>-2`, and so on.
|
||||||
|
6. On validation/FFmpeg failure, expect the source in `input/rejected`. If quarantine movement fails, expect a `.failed` name in working.
|
||||||
|
|
||||||
|
The scheduler handles one naturally sorted candidate per poll. Output-directory collisions cause suffixing as described above. Source lifecycle collisions are different: moving a source to `working`, `processed`, or `rejected` refuses to overwrite an existing same-name file and fails/quarantines the attempt. `preserve-input-quality=true` uses stream copy where possible; false transcodes with the configured preset. Neither mode alone proves cinematic or delivery quality.
|
||||||
|
|
||||||
|
Restart resumes only files still visible as new candidates. A file stranded in `working` is not automatically reclaimed. Diagnose it, preserve evidence, and move it back to source only after proving that no process is active and that output collision handling is understood.
|
||||||
|
|
||||||
|
## Workflow 3: multi-clip cinematic edit
|
||||||
|
|
||||||
|
This lane accepts a folder of source clips, analyzes them, creates a filesystem project, imports a director plan, and can render `final.mp4`.
|
||||||
|
|
||||||
|
### Approved analysis start
|
||||||
|
|
||||||
|
Keep automatic rendering and asset/CV bootstrap disabled:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
env \
|
||||||
|
SERVER_ADDRESS=127.0.0.1 \
|
||||||
|
FOLDER_SCHEDULER_ENABLED=false \
|
||||||
|
VIDEO_EDITING_ENABLED=true \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED=true \
|
||||||
|
VIDEO_EDITING_HIGHLIGHT_SCHEDULER_ENABLED=false \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=false \
|
||||||
|
VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=true \
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START=false \
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=false \
|
||||||
|
VIDEO_CLIPPING_CLEANUP_ENABLED=false \
|
||||||
|
mvn -o spring-boot:run
|
||||||
|
```
|
||||||
|
|
||||||
|
Stage a complete project folder as `input/editing/source/<project>.tmp`, then atomically rename it to `<project>`. The scanner also groups loose root clips into `source-clips`, so do not leave unrelated media there. Temporary suffixes `.tmp`, `.part`, `.download`, and `.processing` are ignored.
|
||||||
|
|
||||||
|
Expected movement is `source/<project>` to `working/<project>` to `processed/<project>`. Failure moves it to `rejected/<project>` or creates a `.failed` marker/name when possible. The project tree is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/edit-projects/<project>/
|
||||||
|
project.json analysis.json
|
||||||
|
ai-director-prompt.md director-readme.md
|
||||||
|
thumbnails/ contact-sheets/
|
||||||
|
proxies/ audio/
|
||||||
|
assets/requests/ inbox/
|
||||||
|
rendered-clips/ render-work/
|
||||||
|
edit-plan.json render-manifest.json
|
||||||
|
qa-report.json final.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
Project states are `CREATED`, `ANALYZING`, `ANALYZED`, `WAITING_FOR_DIRECTOR`, `PLANNING`, `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`. Inspect state without modifying files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl --fail-with-body --show-error --silent --connect-timeout 2 --max-time 10 \
|
||||||
|
http://127.0.0.1:8080/v1/edit-projects/PROJECT_ID
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: HTTP 200 with the requested project. Exit 22 means an HTTP error such as unknown project; exit 7 means connection failure; exit 28 means a bound timeout. Preserve the response and branch to `video-editing-debugging-playbook`; do not retry without classifying it.
|
||||||
|
|
||||||
|
Place strict JSON at `inbox/edit-plan.json`. The inbox scanner validates and archives imports; rejected plans receive a rejected name and do not authorize render. Review source bounds, timeline, transitions, asset provenance, license, voice audibility, and all generated requests. **Do not call the unauthenticated `POST /v1/edit-projects/{projectId}:render` endpoint.** It has no approval check. It remains prohibited until authenticated authorization and auditable approval bound to source, plan, configuration, model, and asset digests are implemented and tested. Automatic render is also non-production until the same binding exists; a bare `inbox/approved.flag` is insufficient.
|
||||||
|
|
||||||
|
Do not render when any required asset is missing or was created by a placeholder path. A render writes segment files, `final.mp4`, `render-manifest.json`, and `qa-report.json`; the QA report is evidence to validate, not automatic acceptance.
|
||||||
|
|
||||||
|
## Workflow 4: single-source cinematic highlights
|
||||||
|
|
||||||
|
This is the strategically primary lane, but it is **not production-ready**. Current analysis calls a loopback local-CV HTTP endpoint, while director-plan authorship remains manual/external. No in-service local director model exists. Strict asset readiness now fails startup when the pre-provisioned Piper, MusicGen, and AudioGen runtime is incomplete, and placeholder success paths have been removed. No complete three-model bundle is checked in, so no current start command satisfies the production requirement of fully local, runtime-resident intelligence with fail-closed cinematic assets.
|
||||||
|
|
||||||
|
### No approved operation yet
|
||||||
|
|
||||||
|
Do not start this workflow as a certified or approved path. Its visual analysis requires the current loopback HTTP worker, and loopback RPC violates the no-network model/media rule. First replace that boundary with an in-process call or approved non-network IPC through `video-editing-change-control`; then validate the replacement through `video-editing-cinematic-highlights-campaign`. Use the read-only artifact inspection below for existing projects.
|
||||||
|
|
||||||
|
Stage one complete file as `.part`, then atomically rename it in `input/highlights/source`. Claim and quarantine rules match folder ingestion: source to working, then processed on success or rejected on failure. The source is also copied into its project.
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project>/
|
||||||
|
project.json manifest.json
|
||||||
|
source/ analysis/
|
||||||
|
director/director-brief.md director/director-prompt.md
|
||||||
|
director/edit-plan.json director/approved.flag
|
||||||
|
assets/ highlights/<highlight-id>/
|
||||||
|
render-manifest.json final.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
Analysis writes `analysis/source-analysis.json`, `ffprobe.json`, `scene-segments.json`, `audio-analysis.json`, `visual-analysis.json`, plus frames, a contact sheet, proxy, and waveform. The scheduler then calls `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Its scores are deterministic ranking hints from coarse source/shot/audio evidence, not proof of semantic importance. States are `CREATED`, `ANALYZING`, `WAITING_FOR_DIRECTOR`, `PLANNED`, `RENDERING`, `RENDERED`, and `FAILED`; a validated plan remains `PLANNED` while required local assets are pending.
|
||||||
|
|
||||||
|
Do not enable rendering until a change-controlled local director implementation, immutable local model inventory, license manifest, and fail-closed asset generation are verified. If running a non-production experiment after explicit approval, require `director/approved.flag`; the scanner selects only `WAITING_FOR_DIRECTOR` or `PLANNED` projects with `director/edit-plan.json` and no project-root `final.mp4`. It writes each `highlights/<id>/final.mp4`, then concatenates/copies them to project-root `final.mp4` and writes the root manifest. Plan rejection or render failure marks the project `FAILED`, which the scanner skips; investigate and repair through an explicit recovery procedure rather than changing the status by hand.
|
||||||
|
|
||||||
|
## Health, metrics, and logs
|
||||||
|
|
||||||
|
For a deliberately started inspection-only service, the following bounded API diagnostics use the service's loopback operator boundary; they do not call model workers:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for path in health info metrics prometheus; do
|
||||||
|
curl --fail-with-body --show-error --silent --connect-timeout 2 --max-time 10 \
|
||||||
|
"http://127.0.0.1:8080/actuator/$path" || exit $?
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: all four requests return HTTP 200. Exit 22 means an endpoint is unavailable/denied, exit 7 means no listener, and exit 28 means a timeout. Stop and route the symptom to `video-editing-debugging-playbook`; do not broaden Actuator exposure to make the probe pass.
|
||||||
|
|
||||||
|
The DB queue exports `video.clipping.queue.pending`, `.processing`, `.dlq`, and `.oldest.pending.age.seconds` only when `video-clipping.queue=db`. Editing observability is primarily structured-looking `event=...` log text, not metrics or traces. Search for workflow-specific events:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg 'event=(folder_|candidate_|processing_|local_director_|edit_|highlight_|local_asset_|local_cv_)' SERVICE_LOG
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not expose Actuator beyond loopback in the current service. Health `UP` is not proof that storage, repository, queue, models, or FFmpeg work.
|
||||||
|
|
||||||
|
## Stop, restart, resume, and cleanup
|
||||||
|
|
||||||
|
1. Stop ingestion first by terminating the single application process; flags are not dynamically reloadable.
|
||||||
|
2. Allow active FFmpeg work to finish when possible. The application has no verified drain/readiness protocol.
|
||||||
|
3. Record process logs, `project.json`, plans, manifests, QA reports, and working/rejected listings before changing files.
|
||||||
|
4. Classify every `working` item as active, safely resumable, or quarantined. The file schedulers do not generally reclaim working input after restart.
|
||||||
|
5. For REST memory adapters, expect all metadata/queue state to disappear. For DB queue, expired `PROCESSING` claims become claimable after the visibility timeout.
|
||||||
|
6. Never delete outputs, caches, or failed inputs during incident analysis.
|
||||||
|
|
||||||
|
Shared edit caches are `input/highlights/assets/{music,sfx,fonts,luts}` and `output/highlight-projects/_voiceover-cache`. Treat them as controlled artifact repositories: verify hash, provenance, license, model/version, and owner before use. The scheduled cleanup does not clean these project trees or shared caches. Global cleanup can delete REST source/clip records and scratch files; keep it disabled until retention and recovery are approved.
|
||||||
|
|
||||||
|
Use a new project ID or source filename for a clean retry. Reusing names can collide with processed, rejected, project, or output paths. Never resolve a collision by deleting evidence or enabling overwrite without change control.
|
||||||
|
|
||||||
|
## Production promotion checklist
|
||||||
|
|
||||||
|
- [ ] One workflow is selected; every other scheduler and renderer is explicitly disabled.
|
||||||
|
- [ ] Artifact was built offline/reproducibly and traced to a reviewed commit.
|
||||||
|
- [ ] All local models and dependencies are immutable, pre-provisioned, hashed, licensed, and network-independent.
|
||||||
|
- [ ] Missing/unhealthy models fail startup or the use case; no heuristic, silence, or tone fallback can publish.
|
||||||
|
- [ ] Director plan and render approval are authenticated, authorized, auditable, and tested.
|
||||||
|
- [ ] PostgreSQL/S3 or replacement adapters pass real integration, retry, concurrency, backup, restore, and retention tests.
|
||||||
|
- [ ] Filesystem state is durable and single-writer/shared-storage semantics are explicit.
|
||||||
|
- [ ] Health probes test real dependencies; logs, metrics, traces, SLOs, dashboards, and alerts have runbooks.
|
||||||
|
- [ ] Graceful shutdown, restart, duplicate claim, stuck working item, and DLQ recovery are exercised.
|
||||||
|
- [ ] Security, vulnerability, secret, image, and license gates pass with no unresolved critical/high findings.
|
||||||
|
- [ ] `video-editing-validation-and-qa` accepts objective media and human-review evidence.
|
||||||
|
- [ ] `video-editing-change-control` approves promotion. No checklist item may be waived silently.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
Ground truth: `application.yml`, application profiles, `VideoClippingProperties`, controllers, schedulers, project stores, queue/storage adapters, cleanup jobs, local worker scripts, renderers, health indicators, and Maven configuration, inspected 2026-07-21.
|
||||||
|
|
||||||
|
Re-verify drift with one-line commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n 'enabled:|auto-start:|auto-render|require-.*approval|fallback-to-heuristic|directory:' src/main/resources/application*.yml
|
||||||
|
rg -n '@(Get|Post|Put|Delete|Patch)Mapping|@RequestMapping' src/main/java/org/example/videoclips/api
|
||||||
|
rg -n 'ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
|
||||||
|
rg -n 'Files\.(move|copy)|StandardCopyOption|\.failed|rejected|processed|working' src/main/java/org/example/videoclips/{folder,editing}
|
||||||
|
rg -n 'pip install|download|from_pretrained|YOLO|silence|fallbackTone|write_fallback_tone' tools src/main/java/org/example/videoclips/editing
|
||||||
|
rg -n 'video\.clipping\.queue|Health\.up|management\.endpoints' src/main src/main/resources
|
||||||
|
rg -n '<java.version>|spring-boot-starter-parent|<artifactId>|<version>' pom.xml
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,305 @@
|
||||||
|
---
|
||||||
|
name: video-editing-validation-and-qa
|
||||||
|
description: Load this skill when defining acceptance criteria, adding or reviewing tests, validating API/database/local-model/media behavior, interpreting Maven or render QA results, creating fixtures or goldens, or deciding whether a video-editing change is fit for production promotion.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Video Editing Validation and QA
|
||||||
|
|
||||||
|
Use evidence to decide whether this service is correct, cinematic, and operable. Treat a green test run as one input, never as a release certificate.
|
||||||
|
|
||||||
|
## When not to use this skill
|
||||||
|
|
||||||
|
- For symptom-first incident triage, use `video-editing-debugging-playbook`.
|
||||||
|
- For commands that inspect a running system or media artifact, use `video-editing-diagnostics-and-tooling`.
|
||||||
|
- For change classification, approval, rollout, or rollback, use `video-editing-change-control`.
|
||||||
|
- For environment recreation and dependency provisioning, use `video-editing-build-and-env`.
|
||||||
|
- For runtime profiles and operator workflows, use `video-editing-run-and-operate`.
|
||||||
|
- For media theory and FFmpeg interpretation, use `cinematic-media-engineering-reference`.
|
||||||
|
- For the executable highlight-quality program, use `video-editing-cinematic-highlights-campaign`.
|
||||||
|
- For first-principles experiments and derivations, use `video-editing-proof-and-analysis-toolkit`.
|
||||||
|
|
||||||
|
## Non-negotiable release rule
|
||||||
|
|
||||||
|
As of 2026-07-21, do **not** describe this repository as production-ready, Fortune 500 reference-quality, or creatively certified. Evidence required for those claims is absent.
|
||||||
|
|
||||||
|
Fail a gate when evidence is missing, skipped, stale, environment-dependent, or only asserted. Never convert `unknown` to `pass`.
|
||||||
|
|
||||||
|
`video-editing-change-control` is authoritative for the seven no-waiver prohibitions; this table defines their validation evidence. `video-editing-config-and-flags` remains authoritative for defaults.
|
||||||
|
|
||||||
|
| Prohibited behavior | Required proof |
|
||||||
|
|---|---|
|
||||||
|
| Automatic dependency or model download | Run with network egress denied and pre-provisioned, checksum-pinned dependencies/models. Retain the egress policy and process logs. |
|
||||||
|
| External AI service access | Deny egress; verify no external endpoint is configured or called. |
|
||||||
|
| Unapproved network access | Prove model/media inference and acquisition use no network, including loopback; verify in-process or approved non-network IPC. Separately test every approved API/database/storage/telemetry connection for security, timeouts, and failure behavior. Maven/process offline flags alone are not egress denial. |
|
||||||
|
| Unlicensed music, SFX, fonts, LUTs, footage, or models | Require an asset/model manifest with source, license, version, checksum, and approved usage. Missing provenance blocks use. |
|
||||||
|
| Silence, tones, or synthetic stand-ins presented as finished assets | Detect and reject placeholders. Do not accept file existence as asset quality. |
|
||||||
|
| Unapproved rendering | Render only approved fixtures in isolated temporary directories until the change-control gate authorizes broader footage or an environment. |
|
||||||
|
| Production-facing default changes | Route through `video-editing-change-control`; test both old and proposed behavior and provide rollback evidence. |
|
||||||
|
|
||||||
|
The 2026-07-21 working tree removes silence/tone/host-speech fallbacks, restricts AudioCraft to existing local paths with offline flags, and makes strict readiness fail. A zero exit code or output WAV still does not prove semantic fit, license, model identity, loudness, or creative quality; retain final-media and provenance gates.
|
||||||
|
|
||||||
|
## Evidence tiers
|
||||||
|
|
||||||
|
Classify every claim with the highest tier actually earned.
|
||||||
|
|
||||||
|
| Tier | Name | Minimum evidence | Claims it permits |
|
||||||
|
|---|---|---|---|
|
||||||
|
| E0 | Assertion | Plan, prompt, comment, checkbox, or unexecuted command | Intent only |
|
||||||
|
| E1 | Static | Compiles; schema/config/command inspected; deterministic static check passes | Structural property only |
|
||||||
|
| E2 | Isolated | Deterministic unit, architecture, or adapter test with controlled collaborators | Tested component behavior |
|
||||||
|
| E3 | Integrated | Real local dependency or executable, representative fixture, machine-read output, no mock at the boundary under test | Integration behavior on the recorded platform |
|
||||||
|
| E4 | Clean system | Clean checkout, offline/reproducible build, real local models and media tools, source-to-output critical journey | System behavior for the recorded build and environment |
|
||||||
|
| E5 | Production-like | Security, load, failure injection, observability, container/deploy, and recovery evidence in a production-like environment | Readiness against declared SLOs |
|
||||||
|
| E6 | Certified | E5 plus independent review, signed evidence, licensed golden inventory, repeatability, and controlled promotion | Reference-quality claim for the tested scope |
|
||||||
|
|
||||||
|
Do not promote evidence across scopes. A macOS E3 FFmpeg result does not certify Linux/VPS, container, cloud storage, PostgreSQL, or a different FFmpeg/model build.
|
||||||
|
|
||||||
|
## Current evidence inventory
|
||||||
|
|
||||||
|
The following inventory was verified on 2026-07-21.
|
||||||
|
|
||||||
|
| Area | Current evidence | Honest status |
|
||||||
|
|---|---|---|
|
||||||
|
| Maven tests | A 2026-07-21 offline `mvn -q -o verify` run records 245 tests in 62 test classes, 0 failures, 0 errors, 0 skipped in the working tree | E2/E3 mix; not clean-checkout or no-egress certification |
|
||||||
|
| Coverage | `pom.xml` enforces 100% instruction, line, and branch coverage only for `org.example.videoclips.folder` during `verify` | No repository-wide threshold; no mutation testing |
|
||||||
|
| Real media tools | `FolderFfmpegIntegrationTest` and `CinematicEditingIntegrationTest` are conditional on local `ffmpeg` and `ffprobe` | Real FFmpeg coverage when available; conditional tests can otherwise skip |
|
||||||
|
| Clean build | A clean archived checkout has failed `mvn verify` at `LocalAssetGenerationStageTest` because success depends on untracked/runtime-local asset tooling or an audible speech fallback | Release blocker; source-alone reproducibility is unproven |
|
||||||
|
| API | 18 Spring Boot/MockMvc tests cover selected success, validation, and Problem Details paths | No published/validated OpenAPI, compatibility gate, authentication, or authorization evidence |
|
||||||
|
| Database | Flyway migrations and PostgreSQL runtime driver exist; normal tests use local/in-memory behavior and H2 can be selected | No real PostgreSQL Testcontainer repository/migration/concurrency evidence |
|
||||||
|
| Architecture | Package separation exists | No ArchUnit or Spring Modulith boundary test; no automated cycle/core-dependency gate |
|
||||||
|
| Legacy edit render | Real integration renders generated clips. `FfmpegEditRenderer` runs black-frame, long-silence, and audio-peak probes; unit tests cover their interpretation | Useful technical E3 evidence; the integration does not assert `qaReport.passed()` and creative quality is untested |
|
||||||
|
| Highlight render | Unit tests cover pieces; `HighlightDirectorFlowServiceTest` uses a mocked renderer and a dummy file | No real source-to-final highlight E2E |
|
||||||
|
| Highlight QA | `HighlightFfmpegRenderer.buildQaReport` probes output duration, black ranges, long silence, and sample peaks; it also checks assets, plan-level overlay bounds, and the configured mastering command. `HighlightDirectorFlowService` blocks project-final publication on failed `ERROR` checks | A fail-closed selected-defect gate, but no integrated-loudness/true-peak, A/V-sync, freeze, raster-safe-area, semantic-fit, or creative acceptance gate |
|
||||||
|
| Local models | Adapter/process tests exist | No pinned model inventory, offline no-egress proof, model quality baseline, license proof, or macOS/Linux/container execution matrix |
|
||||||
|
| Creative quality | Prompts and plans describe cinematic intent | No certified footage set, annotations, baseline, blinded rubric result, or accepted numeric threshold |
|
||||||
|
| Security | Input/error tests exist | No Spring Security, threat-model gate, auth tests, dependency/static/container/secret scans, or release vulnerability report |
|
||||||
|
| Performance | Benchmark harnesses and planning baselines exist under `src/test/java/.../perf` and `docs/` | No checked-in load generator or executed production-like load signoff |
|
||||||
|
| Resilience | Queue/cleanup unit tests and runbooks exist | No fault-injection, shutdown, retry, saturation, or dependency-outage certification |
|
||||||
|
| Container/deploy/CI | Operational planning documents exist | No CI definition, OCI image definition, deployment manifest, image scan, SBOM/signing gate, or automated promotion evidence |
|
||||||
|
|
||||||
|
Do not infer certification from tracked or generated artifacts. `docs/*baseline*.md`, dashboards, manifests, prompts, `qa-report.json`, and files under runtime input/output trees become evidence only when their production method, commit, environment, raw measurements, and reviewer are recorded.
|
||||||
|
|
||||||
|
The single-source highlight flow now uses `RenderQaReport.passed()` and failed `ERROR` checks to block project-final publication and mark the project failed. The legacy multi-clip renderer still treats its QA report as diagnostic output. In both paths, inspect warnings and remember that passing the implemented technical checks does not certify creative quality.
|
||||||
|
|
||||||
|
## Run the current checks safely
|
||||||
|
|
||||||
|
Do not allow Maven, Python, Hugging Face, Torch, FFmpeg inputs, or tests to use the network. Provision Java 21, Maven, FFmpeg/ffprobe, Python dependencies, and all models through an approved offline mechanism first.
|
||||||
|
|
||||||
|
Create a disposable copy of the committed tree, then run Maven offline. This command does not test uncommitted changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export MAVEN_REPO="${MAVEN_REPO:?set MAVEN_REPO to an approved pre-provisioned Maven repository}"
|
||||||
|
qa_root="$(mktemp -d)"
|
||||||
|
git archive --format=tar HEAD | tar -xf - -C "$qa_root"
|
||||||
|
(
|
||||||
|
cd "$qa_root"
|
||||||
|
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
|
||||||
|
mvn -o -Dmaven.repo.local="$MAVEN_REPO" verify
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Run this only inside an execution environment whose egress is denied; environment variables alone are not a network control. Expect the current clean checkout to fail until local-asset tests stop depending on workstation state. Record that failure rather than relaxing the test.
|
||||||
|
|
||||||
|
After Maven completes, reject skipped tests and summarize Surefire results:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
! rg -n 'skipped="[1-9][0-9]*"' "$qa_root"/target/surefire-reports/TEST-*.xml
|
||||||
|
perl -ne 'if (/<testsuite\b([^>]*)>/) {$a=$1; ($t)=$a=~/\btests="(\d+)"/; ($e)=$a=~/\berrors="(\d+)"/; ($f)=$a=~/\bfailures="(\d+)"/; ($s)=$a=~/\bskipped="(\d+)"/; $T+=$t; $E+=$e; $F+=$f; $S+=$s} END {print "tests=$T errors=$E failures=$F skipped=$S\n"}' "$qa_root"/target/surefire-reports/TEST-*.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not make `216` a permanent threshold. Test count can change; require all intended suites, zero failures/errors/skips, and explicit evidence for each risk.
|
||||||
|
|
||||||
|
## Apply the test portfolio
|
||||||
|
|
||||||
|
Use the smallest test that can falsify the claim, then add boundary evidence proportional to blast radius.
|
||||||
|
|
||||||
|
| Layer | Add it when | Required characteristics | Current gap |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Domain unit | A rule, score, state transition, interval, or value object changes | No Spring context; fixed clock/seed; boundary and negative cases; invalid states rejected | Coverage exists but is not mapped to every invariant |
|
||||||
|
| Architecture | Packages/modules or dependency direction change | Fail build on controller/domain, persistence/API, adapter/core, cycle, and cross-module violations | Entire layer absent |
|
||||||
|
| Web slice | Controller, DTO, validation, status, or Problem Details changes | Mock only application port; cover success, malformed input, validation, error mapping, bounded collections | Existing tests start full Spring context rather than slices |
|
||||||
|
| Persistence slice | Mapping, query, constraint, migration, transaction, or concurrency changes | Real PostgreSQL Testcontainer; migrate empty DB; test indexes/constraints, locking, pagination, UTC | Entire layer absent |
|
||||||
|
| Integration | A real adapter/tool contract changes | Exercise actual FFmpeg, ffprobe, local model, filesystem, PostgreSQL, storage, or queue boundary; parse result | FFmpeg only; other real boundaries absent |
|
||||||
|
| Contract | Public API, model-worker protocol, object-storage, or queue schema changes | Versioned producer/consumer contract; backward-compatibility gate; failure mapping | Entire layer absent |
|
||||||
|
| E2E | A critical journey or cross-stage contract changes | Start from approved input; use real intended adapters; assert persisted state, output, telemetry, and failure cleanup | Real highlight source-to-final path absent |
|
||||||
|
|
||||||
|
Use the user's suggested minimums as target gates until change control approves evidence-based replacements: 90% line coverage for domain/application logic, 80% overall line coverage, and 80% mutation coverage for critical paths. Report branch coverage as well. Never use coverage to waive missing boundary, failure, concurrency, security, or creative-quality tests.
|
||||||
|
|
||||||
|
Never use arbitrary sleeps for asynchronous acceptance. The current `VideoAssetControllerTest` contains bounded `Thread.sleep(50L)` polling; treat replacement with a deterministic executor or condition-based bounded wait as test debt, not a pattern to copy.
|
||||||
|
|
||||||
|
## Gate each behavior surface
|
||||||
|
|
||||||
|
### Build and dependency gate
|
||||||
|
|
||||||
|
Require all of the following before E4:
|
||||||
|
|
||||||
|
- Build from a clean checkout with one offline command on macOS and the target Linux runtime.
|
||||||
|
- Pin and centrally manage dependency/plugin versions; prove no snapshot, milestone, RC, deprecated, or unapproved override.
|
||||||
|
- Lock or otherwise make dependency resolution reproducible. Record JDK, Maven, OS, architecture, and artifact checksums.
|
||||||
|
- Produce dependency and Software Bill of Materials (SBOM) evidence. Resolve all critical/high vulnerabilities before release.
|
||||||
|
- Keep dependency/model acquisition out of application startup and tests.
|
||||||
|
|
||||||
|
### API gate
|
||||||
|
|
||||||
|
- Validate the OpenAPI document during the build and detect breaking changes.
|
||||||
|
- Cover every endpoint's success, validation, malformed input, authentication, authorization, not-found/conflict, and dependency-failure responses.
|
||||||
|
- Assert RFC 9457 media type and stable public fields, including a correlation/trace identifier for diagnosable failures.
|
||||||
|
- Prove persistence/internal exceptions, paths, keys, stack traces, and secrets never enter responses.
|
||||||
|
- Bound and test growing collections; test idempotency for retryable writes.
|
||||||
|
|
||||||
|
### PostgreSQL gate
|
||||||
|
|
||||||
|
- Start empty PostgreSQL and apply every Flyway migration in order.
|
||||||
|
- Test not-null, unique, foreign-key, and check constraints as database behavior.
|
||||||
|
- Test repository queries against PostgreSQL, including pagination, indexes, UTC timestamps, isolation, optimistic locking, and concurrent updates.
|
||||||
|
- Test rolling-deployment migration compatibility and the documented recovery path.
|
||||||
|
- Never certify production persistence from H2.
|
||||||
|
|
||||||
|
### Local-model gate
|
||||||
|
|
||||||
|
- Inventory each exact model/runtime with version, checksum, source, license, memory/disk needs, supported platform, and owner.
|
||||||
|
- Load and infer with egress denied. Fail if a model is absent or its checksum differs; do not download or silently select another model.
|
||||||
|
- Run representative deterministic or tolerance-bounded fixtures on macOS development and Linux/VPS/container targets.
|
||||||
|
- Measure task quality, inference latency distribution, warm-up, peak memory, CPU/GPU use, concurrency, timeout, crash recovery, and output determinism.
|
||||||
|
- Prove output is audible/non-placeholder and semantically matches the request. File existence and nonzero PCM samples are insufficient.
|
||||||
|
- Block final rendering when requested production music, SFX, or voiceover cannot be produced by an approved local model or licensed asset.
|
||||||
|
|
||||||
|
### Media technical gate
|
||||||
|
|
||||||
|
Parse actual `ffprobe`/FFmpeg output; never infer output properties from the plan or command string.
|
||||||
|
|
||||||
|
| Check | Current repository value or behavior | Promotion discipline |
|
||||||
|
|---|---|---|
|
||||||
|
| Container/codecs | Intended render profile names use MP4/H.264/AAC | Assert streams and decode the full output without errors |
|
||||||
|
| Geometry/rate | Defaults are 1920x1080 at 30 fps | Assert actual width, height, pixel format, frame rate, duration, and frame count against the approved profile |
|
||||||
|
| Audio | Defaults are 48 kHz and 192 kbit/s | Assert stream presence when required, sample rate, channels, duration, and A/V synchronization |
|
||||||
|
| Loudness | Mix command configures `I=-16`, `TP=-1.5`, `LRA=11` | Measure the completed file with loudness analysis; command presence is not evidence |
|
||||||
|
| Black frames | Legacy probe flags any `blackdetect=d=0.5:pic_th=0.98` range as an error | Preserve or revise only with baseline evidence and change control; distinguish intentional fades |
|
||||||
|
| Long silence | Legacy probe flags `silencedetect=noise=-45dB:d=2` as a warning | Require explicit creative disposition; placeholder or accidental silence is a failure |
|
||||||
|
| Clipping | Legacy probe flags peak at or above -0.1 dBFS as a warning | Require explicit disposition and reconcile with the -1.5 dB true-peak target |
|
||||||
|
| Timeline | Legacy in-memory tolerance is 0.05 seconds | Measure final output duration and per-cut alignment; do not compare two plan-derived values |
|
||||||
|
|
||||||
|
Establish baselines before setting thresholds for A/V sync, frozen frames, dropped/duplicate frames, encoding quality, scene-boundary accuracy, intelligibility, music/voice balance, and platform transcode survival. Label these thresholds `candidate` until approved.
|
||||||
|
|
||||||
|
### Highlight identification and cinematic gate
|
||||||
|
|
||||||
|
Build a licensed, diverse corpus spanning the supported categories, motion levels, lighting, speech/no-speech, shot lengths, codecs, resolutions, and negative footage with no worthwhile highlight.
|
||||||
|
|
||||||
|
Annotate event boundaries, must-include/must-exclude moments, narrative role, technical defects, and reviewer rationale. Pre-register and compute:
|
||||||
|
|
||||||
|
- Candidate recall and precision at the configured maximum highlight count.
|
||||||
|
- Temporal intersection-over-union and boundary error against annotations.
|
||||||
|
- Weak/repetitive selected duration and missed high-value duration.
|
||||||
|
- Diversity/redundancy across selected highlights.
|
||||||
|
- Category accuracy and confidence calibration where category drives the edit.
|
||||||
|
- Voiceover factual-grounding errors, text-overlay errors, and inappropriate music/SFX events.
|
||||||
|
|
||||||
|
Define a cinematic human-review rubric before evaluating a new approach. Blind and randomize baseline/candidate ordering; use multiple qualified reviewers; retain per-reviewer scores and disagreements. Cover story/hook, shot selection, pacing, continuity, visual treatment, sound design, music fit, voiceover quality and grounding, overlay legibility, emotional coherence, and publishability.
|
||||||
|
|
||||||
|
The repository provides no accepted media-specific score thresholds. Establish the baseline, predict target numbers before running, approve thresholds through change control, and then freeze a holdout set. Do not approve by watching only successful examples.
|
||||||
|
|
||||||
|
### Security gate
|
||||||
|
|
||||||
|
- Test authentication and authorization for every public/sensitive operation, including correct 401 versus 403 behavior.
|
||||||
|
- Run dependency, static, secret, container, and license scans in CI; release with no unresolved critical/high findings unless formally risk-accepted.
|
||||||
|
- Verify restrictive CORS, security headers, actuator isolation, input/path validation, archive/media parser limits, and least privilege.
|
||||||
|
- Test that logs, errors, manifests, prompts, and telemetry redact credentials, tokens, personal data, filesystem internals, and protected media content.
|
||||||
|
- Exercise the threat model's trust boundaries, especially uploaded media, FFmpeg/model subprocesses, filesystem paths, model/asset supply chain, database, storage, and operator-controlled plans.
|
||||||
|
|
||||||
|
### Performance and resilience gate
|
||||||
|
|
||||||
|
Treat the user's synchronous API baseline as a candidate until the actual workload is approved: at least 200 requests/second per instance, p95 below 200 ms, p99 below 500 ms, error rate below 0.1%, no continuous memory growth for 60 minutes, and graceful behavior at 2x expected peak. Flag regressions over 10% against the approved baseline.
|
||||||
|
|
||||||
|
Do not apply those latency numbers to asynchronous model inference or rendering. Define media-specific SLOs from source minutes, resolution, codec, model/hardware class, concurrency, queue delay, render factor, and output count.
|
||||||
|
|
||||||
|
- Record p50/p95/p99 latency, throughput, errors, CPU, memory, GC, threads, disk, database pool, queue age/depth, model memory, and FFmpeg process concurrency.
|
||||||
|
- Inject slow/unavailable PostgreSQL, storage, queue, model process, disk exhaustion, corrupt media, timeout, worker crash, and termination.
|
||||||
|
- Prove bounded timeouts/retries/concurrency, idempotency, cleanup, no duplicate/corrupt outputs, graceful shutdown, and recovery.
|
||||||
|
- Retain raw load input/output and telemetry. Planning baseline documents are not executed load evidence.
|
||||||
|
|
||||||
|
### Container, deployment, and operations gate
|
||||||
|
|
||||||
|
- Build one platform-neutral application artifact and signed Open Container Initiative (OCI)/runtime/model bundles per supported target. Promote each target bundle unchanged between environments; run non-root with least privilege and a read-only filesystem where practical.
|
||||||
|
- Scan the image and generate an SBOM/provenance record tied to the commit and model/asset checksums.
|
||||||
|
- Test liveness, readiness, startup/warm-up, graceful shutdown, resource limits, signals, temporary storage, and no hidden local persistent-state dependency.
|
||||||
|
- Validate deployment definitions and rolling upgrade/rollback with migrations and in-flight work.
|
||||||
|
- Prove logs, metrics, traces, health, dashboards, alerts, and runbooks correlate one critical journey without leaking protected data.
|
||||||
|
- Run smoke/acceptance on Linux/VPS and the intended cloud infrastructure. macOS evidence remains development-only.
|
||||||
|
|
||||||
|
## Certified and golden inventory
|
||||||
|
|
||||||
|
As of 2026-07-21:
|
||||||
|
|
||||||
|
- Certified creative goldens: **none**.
|
||||||
|
- Certified source-to-final highlight fixtures: **none**.
|
||||||
|
- Certified local-model artifacts: **none**.
|
||||||
|
- Certified PostgreSQL/container/deployment/load/security evidence: **none**.
|
||||||
|
- Technical generated fixtures: real FFmpeg tests generate synthetic video/audio under JUnit temporary directories; these are test inputs, not cinematic goldens.
|
||||||
|
- Runtime input/output trees and existing rendered files: disposable observations unless separately versioned, licensed, hashed, and approved.
|
||||||
|
|
||||||
|
Never silently bless an existing artifact as golden. Golden status requires an approved change with provenance, expected measurements, reviewer, scope, and invalidation conditions.
|
||||||
|
|
||||||
|
## Add a fixture or golden
|
||||||
|
|
||||||
|
Use this checklist:
|
||||||
|
|
||||||
|
1. State the requirement, failure mode, layer, platform scope, and exact claim the fixture can falsify.
|
||||||
|
2. Obtain legal approval; record source, license, consent/privacy classification, allowed distribution/use, and retention.
|
||||||
|
3. Minimize the fixture without removing the behavior. Keep secrets, customer media, and personal data out of the repository.
|
||||||
|
4. Record SHA-256, media/model metadata, generation recipe, deterministic seed/clock, and tool versions.
|
||||||
|
5. Store small approved fixtures under a purpose-specific `src/test/resources` path. Store large models/media only in an approved versioned artifact system; never auto-download them.
|
||||||
|
6. Store semantic expectations as structured data. Do not compare transcoded media byte-for-byte when decoded measurements are the contract.
|
||||||
|
7. Add positive, boundary, negative, malformed, missing-dependency, and cleanup assertions. Write only to JUnit temporary directories.
|
||||||
|
8. Run with egress denied on macOS and target Linux. Add container/cloud evidence when the claim includes them.
|
||||||
|
9. Route threshold, fixture, license, and behavior changes through `video-editing-change-control`.
|
||||||
|
10. Document owner, review date, expiry/invalidation trigger, and the command that reproduces the result.
|
||||||
|
|
||||||
|
## Record acceptance evidence
|
||||||
|
|
||||||
|
For every promoted change, retain:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Requirement and risk:
|
||||||
|
Commit and artifact digest:
|
||||||
|
Evidence tier and scope:
|
||||||
|
Exact offline command:
|
||||||
|
OS/architecture/JDK/Maven/FFmpeg/ffprobe versions:
|
||||||
|
Model and asset versions, checksums, and licenses:
|
||||||
|
Fixture/golden checksums:
|
||||||
|
Expected thresholds declared before execution:
|
||||||
|
Observed raw measurements and artifact locations:
|
||||||
|
Negative and failure-injection results:
|
||||||
|
Known limitations:
|
||||||
|
Independent reviewer and date:
|
||||||
|
Decision, rollout, rollback, and revisit condition:
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not accept screenshots, prose summaries, or a lone `passed=true` as the only evidence when raw machine-readable output exists.
|
||||||
|
|
||||||
|
## Fortune 500 promotion gate
|
||||||
|
|
||||||
|
Score each category from 0 (missing) to 4 (reference-quality, verified, reusable): architectural clarity, domain modeling, maintainability, security, test quality, API design, data integrity, resilience, observability, performance, cloud-native operation, CI/CD, developer experience, documentation, and operational readiness.
|
||||||
|
|
||||||
|
Require all of the following:
|
||||||
|
|
||||||
|
- No category below 3.
|
||||||
|
- Overall average at least 3.5.
|
||||||
|
- Security, data integrity, testing, and operational readiness each at least 3.
|
||||||
|
- Every score cites concrete code, tests, documentation, or automated evidence.
|
||||||
|
- Clean offline build, complete required checks, API contract, empty-database migrations, telemetry, load/resilience SLOs, secure container, deployment/rollback, ADRs, and absence of undocumented critical assumptions are all proven.
|
||||||
|
|
||||||
|
Every listed category is required for project qualification. An unscored category is 0; there is no category-exclusion or “not applicable” route. Route the final promotion through `video-editing-change-control`.
|
||||||
|
|
||||||
|
## Provenance and maintenance
|
||||||
|
|
||||||
|
This skill was verified against the repository on 2026-07-21. Re-run these one-line, read-only checks when the code or build changes:
|
||||||
|
|
||||||
|
- Test files: `find src/test/java -type f -name '*Test.java' | sort`
|
||||||
|
- Maven plugins/dependencies: `sed -n '1,240p' pom.xml`
|
||||||
|
- Surefire totals: `perl -ne 'if (/<testsuite\b([^>]*)>/) {$a=$1; ($t)=$a=~/\btests="(\d+)"/; ($e)=$a=~/\berrors="(\d+)"/; ($f)=$a=~/\bfailures="(\d+)"/; ($s)=$a=~/\bskipped="(\d+)"/; $T+=$t; $E+=$e; $F+=$f; $S+=$s} END {print "tests=$T errors=$E failures=$F skipped=$S\n"}' target/surefire-reports/TEST-*.xml`
|
||||||
|
- Conditional integration tests: `rg -n '@EnabledIf|ffmpeg|ffprobe' src/test/java/org/example/videoclips`
|
||||||
|
- Coverage scope: `rg -n 'jacoco|folder-coverage-check|COVEREDRATIO|minimum' pom.xml`
|
||||||
|
- Highlight versus legacy QA: `rg -n 'buildQaReport|runQaProbe|new RenderQaCheck' src/main/java/org/example/videoclips/editing/*Renderer.java`
|
||||||
|
- Placeholder/model-download paths: `rg -n 'fallbackTone|writeSilence|get_pretrained|strictRuntime|strict-runtime' src/main tools src/test`
|
||||||
|
- Enterprise gate artifacts: `find . -maxdepth 4 -type f \( -path './.github/*' -o -iname '*openapi*' -o -iname 'Dockerfile*' -o -iname 'compose*.yml' -o -iname '*testcontainer*' -o -iname '*archunit*' -o -iname '*helm*' -o -iname '*k8s*' \) -print`
|
||||||
|
- Current configuration values: `rg -n 'output-width|output-height|output-frame-rate|audio-sample-rate|audio-bitrate|loudness|strict-runtime|auto-start' src/main/resources src/test/resources`
|
||||||
|
|
@ -1208,9 +1208,9 @@ public class VideoClippingProperties {
|
||||||
@Min(1000)
|
@Min(1000)
|
||||||
private long pollIntervalMs = 5000;
|
private long pollIntervalMs = 5000;
|
||||||
|
|
||||||
private boolean renderEnabled = true;
|
private boolean renderEnabled = false;
|
||||||
|
|
||||||
private boolean requireDirectorApproval = false;
|
private boolean requireDirectorApproval = true;
|
||||||
|
|
||||||
private String approvalFileName = "approved.flag";
|
private String approvalFileName = "approved.flag";
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,53 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.StandardCopyOption;
|
||||||
|
import java.util.Optional;
|
||||||
|
|
||||||
|
final class AssetLicensePolicy {
|
||||||
|
|
||||||
|
static final String UNTRACKED = "untracked-local-asset";
|
||||||
|
|
||||||
|
private AssetLicensePolicy() {
|
||||||
|
}
|
||||||
|
|
||||||
|
static Path sidecar(Path asset) {
|
||||||
|
return asset.resolveSibling(asset.getFileName() + ".license.txt");
|
||||||
|
}
|
||||||
|
|
||||||
|
static Optional<String> read(Path asset) {
|
||||||
|
Path sidecar = sidecar(asset);
|
||||||
|
if (!Files.isRegularFile(sidecar)) {
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
String license = Files.readString(sidecar, StandardCharsets.UTF_8).strip();
|
||||||
|
return license.isBlank() || UNTRACKED.equals(license) ? Optional.empty() : Optional.of(license);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to read asset license sidecar: " + sidecar, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static boolean isLicensed(Path asset) {
|
||||||
|
return Files.isRegularFile(asset) && read(asset).isPresent();
|
||||||
|
}
|
||||||
|
|
||||||
|
static void copy(Path source, Path target) throws IOException {
|
||||||
|
String license = read(source)
|
||||||
|
.orElseThrow(() -> new IllegalStateException("Asset license sidecar is required: " + source));
|
||||||
|
Files.createDirectories(target.getParent());
|
||||||
|
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||||
|
Files.writeString(sidecar(target), license, StandardCharsets.UTF_8);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void recordGeneratedAsset(Path model, Path target) throws IOException {
|
||||||
|
String license = read(model)
|
||||||
|
.orElseThrow(() -> new IllegalStateException("Model license sidecar is required: " + model));
|
||||||
|
String provenance = "generated-from=" + model.toAbsolutePath().normalize() + System.lineSeparator()
|
||||||
|
+ "model-license=" + license + System.lineSeparator();
|
||||||
|
Files.writeString(sidecar(target), provenance, StandardCharsets.UTF_8);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -114,6 +114,8 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
|
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
|
||||||
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
|
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
|
||||||
}
|
}
|
||||||
|
Path audioDirectory = store.projectDirectory(projectId).resolve("audio");
|
||||||
|
requireRequestedAssets(plan, audioDirectory);
|
||||||
projectService.updateProject(projectId, EditProjectStatus.RENDERING, Path.of(project.inputDirectory()), null);
|
projectService.updateProject(projectId, EditProjectStatus.RENDERING, Path.of(project.inputDirectory()), null);
|
||||||
Path work = store.projectDirectory(projectId).resolve("render-work");
|
Path work = store.projectDirectory(projectId).resolve("render-work");
|
||||||
createDirectory(work);
|
createDirectory(work);
|
||||||
|
|
@ -138,7 +140,6 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
runAndRecord(overlayCommand(timeline, plan.overlays(), videoTimeline), commands);
|
runAndRecord(overlayCommand(timeline, plan.overlays(), videoTimeline), commands);
|
||||||
}
|
}
|
||||||
Path output = store.projectDirectory(projectId).resolve("final.mp4");
|
Path output = store.projectDirectory(projectId).resolve("final.mp4");
|
||||||
Path audioDirectory = store.projectDirectory(projectId).resolve("audio");
|
|
||||||
Path music = audioDirectory.resolve("music.wav");
|
Path music = audioDirectory.resolve("music.wav");
|
||||||
Path voiceover = audioDirectory.resolve("voiceover.wav");
|
Path voiceover = audioDirectory.resolve("voiceover.wav");
|
||||||
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
||||||
|
|
@ -146,9 +147,13 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
.filter(cue -> "sfx".equals(cue.type()))
|
.filter(cue -> "sfx".equals(cue.type()))
|
||||||
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
||||||
.toList();
|
.toList();
|
||||||
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !soundEffects.isEmpty()) {
|
AudioCue musicCue = plan.audioCues().stream()
|
||||||
runAndRecord(audioMixCommand(videoTimeline, Files.isRegularFile(music) ? music : null,
|
.filter(cue -> "music".equals(cue.type()))
|
||||||
Files.isRegularFile(voiceover) ? voiceover : null, soundEffects, output), commands);
|
.findFirst()
|
||||||
|
.orElse(null);
|
||||||
|
if (musicCue != null || !plan.voiceover().isEmpty() || !soundEffects.isEmpty()) {
|
||||||
|
runAndRecord(audioMixCommand(videoTimeline, musicCue == null ? null : music, musicCue,
|
||||||
|
plan.voiceover().isEmpty() ? null : voiceover, soundEffects, output), commands);
|
||||||
} else {
|
} else {
|
||||||
copy(videoTimeline, output);
|
copy(videoTimeline, output);
|
||||||
}
|
}
|
||||||
|
|
@ -289,10 +294,15 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
}
|
}
|
||||||
|
|
||||||
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, Path output) {
|
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, Path output) {
|
||||||
return audioMixCommand(timeline, music, voiceover, List.of(), output);
|
return audioMixCommand(timeline, music, null, voiceover, List.of(), output);
|
||||||
}
|
}
|
||||||
|
|
||||||
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
|
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
|
||||||
|
return audioMixCommand(timeline, music, null, voiceover, soundEffects, output);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> audioMixCommand(Path timeline, Path music, AudioCue musicCue, Path voiceover,
|
||||||
|
List<SfxInput> soundEffects, Path output) {
|
||||||
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||||
"-i", timeline.toString()));
|
"-i", timeline.toString()));
|
||||||
List<String> labels = new ArrayList<>(List.of("[0:a]"));
|
List<String> labels = new ArrayList<>(List.of("[0:a]"));
|
||||||
|
|
@ -302,7 +312,16 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
boolean hasVoiceover = voiceover != null;
|
boolean hasVoiceover = voiceover != null;
|
||||||
if (music != null) {
|
if (music != null) {
|
||||||
command.addAll(List.of("-i", music.toString()));
|
command.addAll(List.of("-i", music.toString()));
|
||||||
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
|
filters.append("[").append(input).append(":a]");
|
||||||
|
if (musicCue != null) {
|
||||||
|
double duration = musicCue.timelineEndSeconds() - musicCue.timelineStartSeconds();
|
||||||
|
long delayMillis = Math.round(musicCue.timelineStartSeconds() * 1000);
|
||||||
|
filters.append("atrim=duration=").append(duration)
|
||||||
|
.append(",asetpts=PTS-STARTPTS,volume=").append(musicCue.gainDb()).append("dB,adelay=")
|
||||||
|
.append(delayMillis).append("|").append(delayMillis).append("[music_raw];");
|
||||||
|
} else {
|
||||||
|
filters.append("volume=-12dB[music_raw];");
|
||||||
|
}
|
||||||
input++;
|
input++;
|
||||||
}
|
}
|
||||||
if (voiceover != null) {
|
if (voiceover != null) {
|
||||||
|
|
@ -355,12 +374,12 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
Path music = audioDirectory.resolve("music.wav");
|
Path music = audioDirectory.resolve("music.wav");
|
||||||
if (Files.isRegularFile(music)) {
|
if (Files.isRegularFile(music)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
||||||
"project-local", "project-audio"));
|
assetLicense(music), "project-audio"));
|
||||||
}
|
}
|
||||||
Path voiceover = audioDirectory.resolve("voiceover.wav");
|
Path voiceover = audioDirectory.resolve("voiceover.wav");
|
||||||
if (Files.isRegularFile(voiceover)) {
|
if (Files.isRegularFile(voiceover)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
|
||||||
"project-local", "project-audio"));
|
assetLicense(voiceover), "project-audio"));
|
||||||
}
|
}
|
||||||
for (AudioCue cue : plan.audioCues()) {
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
if (assetProvider != null) {
|
if (assetProvider != null) {
|
||||||
|
|
@ -372,13 +391,38 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
||||||
if (Files.isRegularFile(sfx)) {
|
if (Files.isRegularFile(sfx)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
||||||
"project-local", "project-audio"));
|
assetLicense(sfx), "project-audio"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return List.copyOf(assets);
|
return List.copyOf(assets);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void requireRequestedAssets(EditPlan plan, Path audioDirectory) {
|
||||||
|
List<String> missing = new ArrayList<>();
|
||||||
|
if (plan.audioCues().stream().anyMatch(cue -> "music".equals(cue.type()))
|
||||||
|
&& !AssetLicensePolicy.isLicensed(audioDirectory.resolve("music.wav"))) {
|
||||||
|
missing.add("music");
|
||||||
|
}
|
||||||
|
if (!plan.voiceover().isEmpty() && !AssetLicensePolicy.isLicensed(audioDirectory.resolve("voiceover.wav"))) {
|
||||||
|
missing.add("voiceover");
|
||||||
|
}
|
||||||
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
|
if ("sfx".equals(cue.type())
|
||||||
|
&& !AssetLicensePolicy.isLicensed(
|
||||||
|
audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"))) {
|
||||||
|
missing.add("sfx:" + cue.assetKey());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!missing.isEmpty()) {
|
||||||
|
throw new IllegalStateException("Requested render assets are missing: " + missing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String assetLicense(Path asset) {
|
||||||
|
return AssetLicensePolicy.read(asset).orElse(AssetLicensePolicy.UNTRACKED);
|
||||||
|
}
|
||||||
|
|
||||||
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||||
List<Path> renderedClips,
|
List<Path> renderedClips,
|
||||||
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||||
|
|
@ -428,6 +472,10 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
private RenderQaCheck requiredAssetsResolvedCheck(EditPlan plan, List<ResolvedEditAsset> assets) {
|
private RenderQaCheck requiredAssetsResolvedCheck(EditPlan plan, List<ResolvedEditAsset> assets) {
|
||||||
List<String> missing = new ArrayList<>();
|
List<String> missing = new ArrayList<>();
|
||||||
for (AudioCue cue : plan.audioCues()) {
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
|
if ("music".equals(cue.type()) && assets.stream()
|
||||||
|
.noneMatch(asset -> asset.type() == EditAssetType.MUSIC)) {
|
||||||
|
missing.add("music:" + cue.assetKey());
|
||||||
|
}
|
||||||
if ("sfx".equals(cue.type()) && assets.stream()
|
if ("sfx".equals(cue.type()) && assets.stream()
|
||||||
.noneMatch(asset -> asset.type() == EditAssetType.SFX && cue.assetKey().equals(asset.assetKey()))) {
|
.noneMatch(asset -> asset.type() == EditAssetType.SFX && cue.assetKey().equals(asset.assetKey()))) {
|
||||||
missing.add("sfx:" + cue.assetKey());
|
missing.add("sfx:" + cue.assetKey());
|
||||||
|
|
|
||||||
|
|
@ -69,20 +69,17 @@ public class HighlightAssetPreparationService {
|
||||||
if (!highlight.voiceover().isEmpty()) {
|
if (!highlight.voiceover().isEmpty()) {
|
||||||
Path script = voiceoverDirectory.resolve("voiceover-script.txt");
|
Path script = voiceoverDirectory.resolve("voiceover-script.txt");
|
||||||
writeString(script, String.join(System.lineSeparator(), highlight.voiceover()));
|
writeString(script, String.join(System.lineSeparator(), highlight.voiceover()));
|
||||||
Optional<ResolvedEditAsset> voiceover = assetProvider.list(EditAssetType.VOICEOVER, null).stream()
|
List<VoiceoverLine> lines = plannedVoiceoverLines(highlight);
|
||||||
.findFirst();
|
for (int index = 0; index < lines.size(); index++) {
|
||||||
if (voiceover.isPresent()) {
|
VoiceoverLine line = lines.get(index);
|
||||||
Path target = voiceoverDirectory.resolve("voiceover.wav");
|
String assetKey = "voiceover_%03d".formatted(index + 1);
|
||||||
copy(Path.of(voiceover.get().path()), target);
|
|
||||||
resolvedAssets.add(target.toString());
|
|
||||||
} else {
|
|
||||||
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "voiceover",
|
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "voiceover",
|
||||||
"voiceover_script", String.join(System.lineSeparator(), highlight.voiceover()),
|
assetKey, line.text(), voiceoverDirectory.resolve(assetKey + ".wav"),
|
||||||
voiceoverDirectory.resolve("voiceover.wav"), highlight.targetDurationSeconds(), false));
|
line.timelineEndSeconds() - line.timelineStartSeconds(), true));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
List<AudioCue> sfxCues = sfxCues(highlight);
|
List<AudioCue> sfxCues = plannedSfxCues(highlight);
|
||||||
for (AudioCue cue : sfxCues) {
|
for (AudioCue cue : sfxCues) {
|
||||||
Optional<ResolvedEditAsset> resolved = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
|
Optional<ResolvedEditAsset> resolved = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
|
||||||
cue.assetKey(), category, cue.timelineEndSeconds() - cue.timelineStartSeconds(), cue.notes()));
|
cue.assetKey(), category, cue.timelineEndSeconds() - cue.timelineStartSeconds(), cue.notes()));
|
||||||
|
|
@ -102,7 +99,10 @@ public class HighlightAssetPreparationService {
|
||||||
return new HighlightAssetPreparationResult(highlightDirectory, assetsDirectory, requestFiles, resolvedAssets);
|
return new HighlightAssetPreparationResult(highlightDirectory, assetsDirectory, requestFiles, resolvedAssets);
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<AudioCue> sfxCues(HighlightDirectorPlan.HighlightItem highlight) {
|
static List<AudioCue> plannedSfxCues(HighlightDirectorPlan.HighlightItem highlight) {
|
||||||
|
if (highlight.sfxDirection() == null || highlight.sfxDirection().isBlank()) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
double duration = Math.max(1.0, highlight.targetDurationSeconds());
|
double duration = Math.max(1.0, highlight.targetDurationSeconds());
|
||||||
List<AudioCue> cues = new ArrayList<>();
|
List<AudioCue> cues = new ArrayList<>();
|
||||||
if (highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
|
if (highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
|
||||||
|
|
@ -114,6 +114,40 @@ public class HighlightAssetPreparationService {
|
||||||
return cues;
|
return cues;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static List<VoiceoverLine> plannedVoiceoverLines(HighlightDirectorPlan.HighlightItem highlight) {
|
||||||
|
if (highlight.voiceover() == null || highlight.voiceover().isEmpty()) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
double targetDuration = Math.max(1.0, highlight.targetDurationSeconds());
|
||||||
|
List<Double> durations = highlight.voiceover().stream()
|
||||||
|
.map(HighlightAssetPreparationService::estimatedVoiceoverDuration)
|
||||||
|
.toList();
|
||||||
|
double speechDuration = durations.stream().mapToDouble(Double::doubleValue).sum();
|
||||||
|
if (speechDuration > targetDuration + 0.001) {
|
||||||
|
throw new IllegalArgumentException("Voiceover script exceeds the highlight timing budget");
|
||||||
|
}
|
||||||
|
double gap = (targetDuration - speechDuration) / (durations.size() + 1);
|
||||||
|
double cursor = gap;
|
||||||
|
List<VoiceoverLine> lines = new ArrayList<>();
|
||||||
|
for (int index = 0; index < durations.size(); index++) {
|
||||||
|
double duration = durations.get(index);
|
||||||
|
double start = roundSeconds(cursor);
|
||||||
|
double end = roundSeconds(Math.min(targetDuration, cursor + duration));
|
||||||
|
lines.add(new VoiceoverLine(highlight.voiceover().get(index), start, end, "cinematic_narration"));
|
||||||
|
cursor += duration + gap;
|
||||||
|
}
|
||||||
|
return List.copyOf(lines);
|
||||||
|
}
|
||||||
|
|
||||||
|
static double estimatedVoiceoverDuration(String text) {
|
||||||
|
long words = text == null || text.isBlank() ? 0 : text.trim().split("\\s+").length;
|
||||||
|
return Math.max(1.2, words / 2.5 + 0.4);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double roundSeconds(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
|
||||||
private String writeRequest(Path requestDirectory, String projectId, String highlightId, String type,
|
private String writeRequest(Path requestDirectory, String projectId, String highlightId, String type,
|
||||||
String assetKey, String notes, Path target, double durationSeconds, boolean blocking) {
|
String assetKey, String notes, Path target, double durationSeconds, boolean blocking) {
|
||||||
Path requestFile = requestDirectory.resolve(type + "-" + assetKey + ".md");
|
Path requestFile = requestDirectory.resolve(type + "-" + assetKey + ".md");
|
||||||
|
|
@ -159,8 +193,7 @@ public class HighlightAssetPreparationService {
|
||||||
|
|
||||||
private void copy(Path source, Path target) {
|
private void copy(Path source, Path target) {
|
||||||
try {
|
try {
|
||||||
Files.createDirectories(target.getParent());
|
AssetLicensePolicy.copy(source, target);
|
||||||
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
|
|
||||||
} catch (IOException ex) {
|
} catch (IOException ex) {
|
||||||
throw new IllegalStateException("Unable to copy highlight asset: " + source, ex);
|
throw new IllegalStateException("Unable to copy highlight asset: " + source, ex);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,345 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.time.Clock;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.LinkedHashSet;
|
||||||
|
import java.util.LinkedHashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Locale;
|
||||||
|
import java.util.Set;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightCandidateGenerator {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightCandidateGenerator.class);
|
||||||
|
private static final double DEFAULT_WINDOW_SECONDS = 12.0;
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
private final Clock clock;
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public HighlightCandidateGenerator(VideoClippingProperties properties, HighlightProjectStore store) {
|
||||||
|
this(properties, store, Clock.systemUTC());
|
||||||
|
}
|
||||||
|
|
||||||
|
HighlightCandidateGenerator(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
HighlightProjectStore store,
|
||||||
|
Clock clock
|
||||||
|
) {
|
||||||
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
|
this.store = store;
|
||||||
|
this.clock = clock;
|
||||||
|
}
|
||||||
|
|
||||||
|
public CinematicHighlightAnalysis generate(String projectId, HighlightSourceAnalysis analysis) {
|
||||||
|
if (!projectId.equals(analysis.projectId())) {
|
||||||
|
throw new IllegalArgumentException("Highlight source analysis belongs to a different project");
|
||||||
|
}
|
||||||
|
if (analysis.source() == null || analysis.source().durationSeconds() <= 0) {
|
||||||
|
throw new IllegalArgumentException("Highlight source duration must be greater than zero");
|
||||||
|
}
|
||||||
|
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
log.info("event=highlight_candidates_started project_id={} duration_seconds={} shot_segments={}",
|
||||||
|
projectId, analysis.source().durationSeconds(), safeShots(analysis).size());
|
||||||
|
|
||||||
|
CategoryDecision category = classify(analysis.visualAnalysis());
|
||||||
|
Map<RangeKey, ScoredWindow> uniqueRanges = new LinkedHashMap<>();
|
||||||
|
windows(analysis).stream().map(window -> score(window, analysis)).forEach(scored ->
|
||||||
|
uniqueRanges.merge(new RangeKey(scored.window().startSeconds(), scored.window().endSeconds()),
|
||||||
|
scored, (left, right) -> right.score() > left.score() ? right : left));
|
||||||
|
List<ScoredWindow> ranked = uniqueRanges.values().stream()
|
||||||
|
.sorted(Comparator.comparingDouble(ScoredWindow::score).reversed()
|
||||||
|
.thenComparingDouble(scored -> scored.window().startSeconds())
|
||||||
|
.thenComparingDouble(scored -> scored.window().endSeconds()))
|
||||||
|
.limit(properties.getMaxHighlightsPerSource())
|
||||||
|
.toList();
|
||||||
|
|
||||||
|
List<HighlightCandidate> candidates = new ArrayList<>();
|
||||||
|
for (int index = 0; index < ranked.size(); index++) {
|
||||||
|
ScoredWindow scored = ranked.get(index);
|
||||||
|
Window window = scored.window();
|
||||||
|
HighlightCandidate candidate = new HighlightCandidate(
|
||||||
|
"candidate_%03d".formatted(index + 1),
|
||||||
|
analysis.source().clipId(),
|
||||||
|
round(window.startSeconds()),
|
||||||
|
round(window.endSeconds()),
|
||||||
|
round(scored.score()),
|
||||||
|
role(window, analysis.source().durationSeconds()),
|
||||||
|
scored.reasons()
|
||||||
|
);
|
||||||
|
candidates.add(candidate);
|
||||||
|
log.info("event=highlight_candidate_scored project_id={} candidate_id={} start={} end={} score={} "
|
||||||
|
+ "reasons={}",
|
||||||
|
projectId, candidate.id(), candidate.sourceStartSeconds(), candidate.sourceEndSeconds(),
|
||||||
|
candidate.score(), candidate.reasons());
|
||||||
|
}
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = new CinematicHighlightAnalysis(
|
||||||
|
projectId,
|
||||||
|
category.category(),
|
||||||
|
category.confidence(),
|
||||||
|
category.reasons(),
|
||||||
|
List.copyOf(candidates),
|
||||||
|
Instant.now(clock)
|
||||||
|
);
|
||||||
|
store.writeJson(projectId, "analysis/category.json", result);
|
||||||
|
store.writeJson(projectId, "analysis/highlight-candidates.json", candidates.toArray(HighlightCandidate[]::new));
|
||||||
|
log.info("event=highlight_candidates_completed project_id={} count={} category={} "
|
||||||
|
+ "category_confidence={} elapsed_ms={}",
|
||||||
|
projectId, candidates.size(), result.category(), result.categoryConfidence(),
|
||||||
|
elapsedMillis(startedAt));
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private CategoryDecision classify(SourceVisualAnalysis visual) {
|
||||||
|
if (visual == null || !isIndependentVisualEvidence(visual)) {
|
||||||
|
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.25, List.of(
|
||||||
|
"Category-specific visual evidence is unavailable or came from metadata fallback.",
|
||||||
|
"Review frames and contact sheets before assigning category-specific creative direction."
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
List<VisualObjectLabel> labels = visual.objectLabels() == null ? List.of() : visual.objectLabels();
|
||||||
|
CategoryDecision best = null;
|
||||||
|
for (VisualObjectLabel label : labels) {
|
||||||
|
if (label == null || label.label() == null || label.confidence() < 0.55
|
||||||
|
|| "metadata_heuristic".equalsIgnoreCase(label.source())) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
ContentCategory category = categoryFor(label.label());
|
||||||
|
if (category == ContentCategory.GENERIC_VLOG) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
double confidence = round(Math.min(0.95, label.confidence()));
|
||||||
|
if (best == null || confidence > best.confidence()) {
|
||||||
|
best = new CategoryDecision(category, confidence, List.of(
|
||||||
|
"Local visual model label '%s' supports %s at confidence %.3f."
|
||||||
|
.formatted(label.label(), category.name().toLowerCase(Locale.ROOT), confidence),
|
||||||
|
"Confirm the category against representative frames before approving the edit plan."
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (best != null) {
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.35, List.of(
|
||||||
|
"Local visual analysis completed but produced no category-discriminating label above 0.55.",
|
||||||
|
"Use generic direction until a reviewer confirms the visible subject."
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isIndependentVisualEvidence(SourceVisualAnalysis visual) {
|
||||||
|
String method = visual.analysisMethod() == null ? "" : visual.analysisMethod().toLowerCase(Locale.ROOT);
|
||||||
|
return !method.isBlank() && !method.contains("fallback") && !method.contains("metadata");
|
||||||
|
}
|
||||||
|
|
||||||
|
private ContentCategory categoryFor(String rawLabel) {
|
||||||
|
String label = rawLabel.toLowerCase(Locale.ROOT).replace('_', ' ').trim();
|
||||||
|
if (containsAny(label, "car", "vehicle", "automobile", "truck", "motorcycle", "porsche")) {
|
||||||
|
return ContentCategory.CAR_VLOG;
|
||||||
|
}
|
||||||
|
if (containsAny(label, "food", "meal", "dish", "pizza", "sandwich", "cake", "kitchen")) {
|
||||||
|
return ContentCategory.FOOD_VLOG;
|
||||||
|
}
|
||||||
|
if (containsAny(label, "family", "child", "children", "baby", "birthday party")) {
|
||||||
|
return ContentCategory.FAMILY_VLOG;
|
||||||
|
}
|
||||||
|
return ContentCategory.GENERIC_VLOG;
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean containsAny(String value, String... candidates) {
|
||||||
|
for (String candidate : candidates) {
|
||||||
|
if (value.contains(candidate)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<Window> windows(HighlightSourceAnalysis analysis) {
|
||||||
|
double duration = analysis.source().durationSeconds();
|
||||||
|
List<ShotSegment> shots = safeShots(analysis).stream()
|
||||||
|
.filter(shot -> validShot(shot, duration))
|
||||||
|
.sorted(Comparator.comparingDouble(ShotSegment::startSeconds))
|
||||||
|
.toList();
|
||||||
|
if (shots.size() <= 1) {
|
||||||
|
String shotId = shots.isEmpty() ? "source_coverage" : shots.get(0).shotId();
|
||||||
|
return splitRange(0, duration, shotId, shots.isEmpty() ? 0 : shots.get(0).sceneScore(), true);
|
||||||
|
}
|
||||||
|
|
||||||
|
Set<Window> windows = new LinkedHashSet<>();
|
||||||
|
for (ShotSegment shot : shots) {
|
||||||
|
if (shot.durationSeconds() > properties.getHighlightMaxDurationSeconds()) {
|
||||||
|
windows.addAll(splitRange(shot.startSeconds(), shot.endSeconds(), shot.shotId(),
|
||||||
|
shot.sceneScore(), false));
|
||||||
|
} else {
|
||||||
|
windows.add(expandToMinimum(shot, duration));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return List.copyOf(windows);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<ShotSegment> safeShots(HighlightSourceAnalysis analysis) {
|
||||||
|
return analysis.shotSegments() == null ? List.of() : analysis.shotSegments();
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean validShot(ShotSegment shot, double sourceDuration) {
|
||||||
|
return shot != null && shot.startSeconds() >= 0 && shot.endSeconds() > shot.startSeconds()
|
||||||
|
&& shot.endSeconds() <= sourceDuration + 0.001;
|
||||||
|
}
|
||||||
|
|
||||||
|
private Window expandToMinimum(ShotSegment shot, double sourceDuration) {
|
||||||
|
double minimum = Math.min(properties.getHighlightMinDurationSeconds(), sourceDuration);
|
||||||
|
double start = shot.startSeconds();
|
||||||
|
double end = shot.endSeconds();
|
||||||
|
if (end - start < minimum) {
|
||||||
|
double center = clamp(shot.representativeTimestampSeconds(), start, end);
|
||||||
|
start = clamp(center - minimum / 2.0, 0, Math.max(0, sourceDuration - minimum));
|
||||||
|
end = Math.min(sourceDuration, start + minimum);
|
||||||
|
}
|
||||||
|
return new Window(round(start), round(end), shot.shotId(), shot.sceneScore(), false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<Window> splitRange(
|
||||||
|
double rangeStart,
|
||||||
|
double rangeEnd,
|
||||||
|
String shotId,
|
||||||
|
double sceneScore,
|
||||||
|
boolean coverageFallback
|
||||||
|
) {
|
||||||
|
double rangeDuration = rangeEnd - rangeStart;
|
||||||
|
double minimum = Math.min(properties.getHighlightMinDurationSeconds(), rangeDuration);
|
||||||
|
double target = Math.min(properties.getHighlightMaxDurationSeconds(),
|
||||||
|
Math.max(minimum, Math.min(DEFAULT_WINDOW_SECONDS, rangeDuration)));
|
||||||
|
if (rangeDuration <= target + 0.001) {
|
||||||
|
return List.of(new Window(round(rangeStart), round(rangeEnd), shotId, sceneScore, coverageFallback));
|
||||||
|
}
|
||||||
|
|
||||||
|
double step = Math.max(1.0, target * 0.75);
|
||||||
|
Set<Window> windows = new LinkedHashSet<>();
|
||||||
|
for (double start = rangeStart; start + target <= rangeEnd + 0.001; start += step) {
|
||||||
|
double boundedStart = Math.min(start, rangeEnd - target);
|
||||||
|
windows.add(new Window(round(boundedStart), round(boundedStart + target), shotId, sceneScore,
|
||||||
|
coverageFallback));
|
||||||
|
}
|
||||||
|
double finalStart = rangeEnd - target;
|
||||||
|
windows.add(new Window(round(finalStart), round(rangeEnd), shotId, sceneScore, coverageFallback));
|
||||||
|
return List.copyOf(windows);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ScoredWindow score(Window window, HighlightSourceAnalysis analysis) {
|
||||||
|
SourceVisualAnalysis visual = analysis.visualAnalysis();
|
||||||
|
double visualQuality = visual == null ? 0
|
||||||
|
: clamp(visual.blurScore()) * 0.30
|
||||||
|
+ clamp(visual.exposureScore()) * 0.20
|
||||||
|
+ clamp(visual.motionScore()) * 0.25
|
||||||
|
+ clamp(visual.compositionScore()) * 0.25;
|
||||||
|
double nonSilence = nonSilenceFraction(window, analysis.audioAnalysis());
|
||||||
|
double durationFit = durationFit(window.durationSeconds());
|
||||||
|
double sceneInterest = clamp(window.sceneScore());
|
||||||
|
double raw = 0.10 + visualQuality * 0.35 + nonSilence * 0.25 + sceneInterest * 0.20
|
||||||
|
+ durationFit * 0.10;
|
||||||
|
boolean weakVisualEvidence = visual == null || !isIndependentVisualEvidence(visual);
|
||||||
|
double score = clamp(raw * (weakVisualEvidence ? 0.72 : 1.0));
|
||||||
|
|
||||||
|
List<String> reasons = new ArrayList<>();
|
||||||
|
reasons.add(window.coverageFallback()
|
||||||
|
? "Coverage window created because analysis produced fewer than two valid shot boundaries."
|
||||||
|
: "Window is aligned to analyzed shot " + window.shotId() + ".");
|
||||||
|
reasons.add("Source-level visual quality score is %.3f; it is ranking context, not temporal proof."
|
||||||
|
.formatted(round(visualQuality)));
|
||||||
|
reasons.add("Audio is %.1f%% non-silent in this window according to FFmpeg silence detection."
|
||||||
|
.formatted(nonSilence * 100.0));
|
||||||
|
if (weakVisualEvidence) {
|
||||||
|
reasons.add("Visual evidence is heuristic or fallback-derived; inspect frames before selection.");
|
||||||
|
}
|
||||||
|
return new ScoredWindow(window, score, List.copyOf(reasons));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double nonSilenceFraction(Window window, SourceAudioAnalysis audio) {
|
||||||
|
if (audio == null || !audio.audioPresent() || audio.sections() == null || audio.sections().isEmpty()) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
double usable = 0;
|
||||||
|
for (AudioSection section : audio.sections()) {
|
||||||
|
if (section == null || "silence".equalsIgnoreCase(section.kind())
|
||||||
|
|| "missing_audio".equalsIgnoreCase(section.kind())) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
usable += overlap(window.startSeconds(), window.endSeconds(), section.startSeconds(), section.endSeconds());
|
||||||
|
}
|
||||||
|
return clamp(usable / window.durationSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private double overlap(double startA, double endA, double startB, double endB) {
|
||||||
|
return Math.max(0, Math.min(endA, endB) - Math.max(startA, startB));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double durationFit(double duration) {
|
||||||
|
double target = Math.min(properties.getHighlightMaxDurationSeconds(),
|
||||||
|
Math.max(properties.getHighlightMinDurationSeconds(), DEFAULT_WINDOW_SECONDS));
|
||||||
|
return clamp(1.0 - Math.abs(duration - target) / Math.max(target, 1.0));
|
||||||
|
}
|
||||||
|
|
||||||
|
private String role(Window window, double sourceDuration) {
|
||||||
|
double midpoint = (window.startSeconds() + window.endSeconds()) / 2.0;
|
||||||
|
if (midpoint <= sourceDuration * 0.25) {
|
||||||
|
return "opening_hook_candidate";
|
||||||
|
}
|
||||||
|
if (midpoint >= sourceDuration * 0.75) {
|
||||||
|
return "final_payoff_candidate";
|
||||||
|
}
|
||||||
|
return "story_build_candidate";
|
||||||
|
}
|
||||||
|
|
||||||
|
private double clamp(double value) {
|
||||||
|
return Math.max(0, Math.min(1, value));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double clamp(double value, double minimum, double maximum) {
|
||||||
|
return Math.max(minimum, Math.min(maximum, value));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double round(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private long elapsedMillis(long startedAt) {
|
||||||
|
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
private record Window(
|
||||||
|
double startSeconds,
|
||||||
|
double endSeconds,
|
||||||
|
String shotId,
|
||||||
|
double sceneScore,
|
||||||
|
boolean coverageFallback
|
||||||
|
) {
|
||||||
|
double durationSeconds() {
|
||||||
|
return endSeconds - startSeconds;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private record ScoredWindow(Window window, double score, List<String> reasons) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record RangeKey(double startSeconds, double endSeconds) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record CategoryDecision(ContentCategory category, double confidence, List<String> reasons) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -26,6 +26,7 @@ public class HighlightDirectorFlowService {
|
||||||
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
private final ObjectMapper objectMapper;
|
private final ObjectMapper objectMapper;
|
||||||
private final HighlightProjectStore store;
|
private final HighlightProjectStore store;
|
||||||
|
private final HighlightDirectorPlanValidator planValidator;
|
||||||
private final HighlightVisualEffectsStage visualEffectsStage;
|
private final HighlightVisualEffectsStage visualEffectsStage;
|
||||||
private final HighlightAssetPreparationService assetPreparationService;
|
private final HighlightAssetPreparationService assetPreparationService;
|
||||||
private final HighlightLocalAssetWorker assetWorker;
|
private final HighlightLocalAssetWorker assetWorker;
|
||||||
|
|
@ -33,6 +34,7 @@ public class HighlightDirectorFlowService {
|
||||||
|
|
||||||
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
|
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
|
HighlightDirectorPlanValidator planValidator,
|
||||||
HighlightVisualEffectsStage visualEffectsStage,
|
HighlightVisualEffectsStage visualEffectsStage,
|
||||||
HighlightAssetPreparationService assetPreparationService,
|
HighlightAssetPreparationService assetPreparationService,
|
||||||
HighlightLocalAssetWorker assetWorker,
|
HighlightLocalAssetWorker assetWorker,
|
||||||
|
|
@ -40,6 +42,7 @@ public class HighlightDirectorFlowService {
|
||||||
this.properties = properties.getEditing().getHighlightScheduler();
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
this.objectMapper = objectMapper;
|
this.objectMapper = objectMapper;
|
||||||
this.store = store;
|
this.store = store;
|
||||||
|
this.planValidator = planValidator;
|
||||||
this.visualEffectsStage = visualEffectsStage;
|
this.visualEffectsStage = visualEffectsStage;
|
||||||
this.assetPreparationService = assetPreparationService;
|
this.assetPreparationService = assetPreparationService;
|
||||||
this.assetWorker = assetWorker;
|
this.assetWorker = assetWorker;
|
||||||
|
|
@ -60,17 +63,20 @@ public class HighlightDirectorFlowService {
|
||||||
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
|
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
|
||||||
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
|
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
|
||||||
}
|
}
|
||||||
HighlightDirectorPlan plan = readPlan(planFile);
|
HighlightDirectorPlan plan;
|
||||||
if (plan.highlights().isEmpty()) {
|
try {
|
||||||
markFailed(project, "No highlights were provided by the director plan");
|
plan = planValidator.validate(projectId, readPlan(planFile));
|
||||||
throw new IllegalStateException("Director plan contains no highlights");
|
} catch (RuntimeException ex) {
|
||||||
|
markFailed(project, "Director plan rejected: " + ex.getMessage());
|
||||||
|
throw ex;
|
||||||
}
|
}
|
||||||
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
|
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
|
||||||
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
|
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
|
||||||
|
markStatus(project, HighlightProjectStatus.PLANNED, null);
|
||||||
List<Path> finalOutputs = new ArrayList<>();
|
List<Path> finalOutputs = new ArrayList<>();
|
||||||
List<String> renderedHighlightIds = new ArrayList<>();
|
List<String> renderedHighlightIds = new ArrayList<>();
|
||||||
ContentCategory category = contentCategory(plan.contentCategory());
|
ContentCategory category = contentCategory(plan.contentCategory());
|
||||||
int limit = Math.min(properties.getMaxHighlightsPerSource(), plan.highlights().size());
|
int limit = plan.highlights().size();
|
||||||
for (int index = 0; index < limit; index++) {
|
for (int index = 0; index < limit; index++) {
|
||||||
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
|
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
|
||||||
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||||
|
|
@ -94,7 +100,14 @@ public class HighlightDirectorFlowService {
|
||||||
return HighlightFlowResult.skipped(projectId, flowId, "assets_pending");
|
return HighlightFlowResult.skipped(projectId, flowId, "assets_pending");
|
||||||
}
|
}
|
||||||
markStatus(project, HighlightProjectStatus.RENDERING, null);
|
markStatus(project, HighlightProjectStatus.RENDERING, null);
|
||||||
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
|
HighlightFfmpegRenderer.HighlightRenderResult result;
|
||||||
|
try {
|
||||||
|
result = renderer.render(projectId, highlight, editPlan);
|
||||||
|
requireQaPassed(result);
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
markFailed(project, "Highlight render rejected: " + ex.getMessage());
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
finalOutputs.add(result.finalOutput());
|
finalOutputs.add(result.finalOutput());
|
||||||
renderedHighlightIds.add(result.highlightId());
|
renderedHighlightIds.add(result.highlightId());
|
||||||
}
|
}
|
||||||
|
|
@ -110,6 +123,19 @@ public class HighlightDirectorFlowService {
|
||||||
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
|
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void requireQaPassed(HighlightFfmpegRenderer.HighlightRenderResult result) {
|
||||||
|
if (result.qaReport() == null) {
|
||||||
|
throw new IllegalStateException("Highlight render did not produce a QA report");
|
||||||
|
}
|
||||||
|
List<String> blockingFailures = result.qaReport().checks().stream()
|
||||||
|
.filter(check -> !check.passed() && "ERROR".equals(check.severity()))
|
||||||
|
.map(RenderQaCheck::name)
|
||||||
|
.toList();
|
||||||
|
if (!result.qaReport().passed() || !blockingFailures.isEmpty()) {
|
||||||
|
throw new IllegalStateException("Highlight render failed blocking QA checks: " + blockingFailures);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private HighlightDirectorPlan readPlan(Path planFile) {
|
private HighlightDirectorPlan readPlan(Path planFile) {
|
||||||
try {
|
try {
|
||||||
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
|
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
|
||||||
|
|
@ -123,11 +149,8 @@ public class HighlightDirectorFlowService {
|
||||||
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
|
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
|
||||||
HighlightSourceAnalysis.class);
|
HighlightSourceAnalysis.class);
|
||||||
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
|
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
|
||||||
double requestedDuration = highlight.targetDurationSeconds() > 0 ? highlight.targetDurationSeconds()
|
double targetDuration = highlight.targetDurationSeconds();
|
||||||
: sourceDuration;
|
double playbackSpeed = sourceDuration / targetDuration;
|
||||||
double targetDuration = Math.max(properties.getHighlightMinDurationSeconds(),
|
|
||||||
Math.min(properties.getHighlightMaxDurationSeconds(), requestedDuration));
|
|
||||||
double playbackSpeed = Math.max(0.25, Math.min(4.0, sourceDuration / targetDuration));
|
|
||||||
String clipId = analysis.source().clipId();
|
String clipId = analysis.source().clipId();
|
||||||
EditDecision decision = new EditDecision(
|
EditDecision decision = new EditDecision(
|
||||||
clipId,
|
clipId,
|
||||||
|
|
@ -147,18 +170,9 @@ public class HighlightDirectorFlowService {
|
||||||
-10.0, highlight.musicDirection()));
|
-10.0, highlight.musicDirection()));
|
||||||
}
|
}
|
||||||
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
|
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
|
||||||
audioCues.add(new AudioCue("sfx", safeKey("sfx", highlight.storyPurpose(), highlight.highlightId()),
|
audioCues.addAll(HighlightAssetPreparationService.plannedSfxCues(highlight));
|
||||||
Math.max(0.0, targetDuration - 0.8), targetDuration, -6.0, highlight.sfxDirection()));
|
|
||||||
}
|
|
||||||
List<VoiceoverLine> voiceover = new ArrayList<>();
|
|
||||||
if (!highlight.voiceover().isEmpty()) {
|
|
||||||
double step = targetDuration / highlight.voiceover().size();
|
|
||||||
for (int i = 0; i < highlight.voiceover().size(); i++) {
|
|
||||||
double start = Math.min(targetDuration, i * step);
|
|
||||||
double end = Math.min(targetDuration, start + Math.max(1.0, step));
|
|
||||||
voiceover.add(new VoiceoverLine(highlight.voiceover().get(i), start, end, "cinematic_narration"));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
List<VoiceoverLine> voiceover = HighlightAssetPreparationService.plannedVoiceoverLines(highlight);
|
||||||
List<TextOverlay> overlays = new ArrayList<>();
|
List<TextOverlay> overlays = new ArrayList<>();
|
||||||
if (!highlight.overlays().isEmpty()) {
|
if (!highlight.overlays().isEmpty()) {
|
||||||
double step = targetDuration / highlight.overlays().size();
|
double step = targetDuration / highlight.overlays().size();
|
||||||
|
|
|
||||||
|
|
@ -27,13 +27,16 @@ public class HighlightDirectorPlanScanner {
|
||||||
|
|
||||||
private final VideoClippingProperties properties;
|
private final VideoClippingProperties properties;
|
||||||
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
|
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
|
||||||
|
private final HighlightProjectStore store;
|
||||||
private final HighlightDirectorFlowService flowService;
|
private final HighlightDirectorFlowService flowService;
|
||||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||||
private final AtomicLong scanSequence = new AtomicLong();
|
private final AtomicLong scanSequence = new AtomicLong();
|
||||||
|
|
||||||
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightDirectorFlowService flowService) {
|
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightProjectStore store,
|
||||||
|
HighlightDirectorFlowService flowService) {
|
||||||
this.properties = properties;
|
this.properties = properties;
|
||||||
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
|
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
|
||||||
|
this.store = store;
|
||||||
this.flowService = flowService;
|
this.flowService = flowService;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -78,10 +81,36 @@ public class HighlightDirectorPlanScanner {
|
||||||
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||||
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
|
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
|
||||||
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
|
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
|
||||||
|
.filter(this::isRetryableProject)
|
||||||
|
.filter(this::hasRequiredApproval)
|
||||||
.map(path -> path.getFileName().toString())
|
.map(path -> path.getFileName().toString())
|
||||||
.findFirst();
|
.findFirst();
|
||||||
} catch (IOException ex) {
|
} catch (IOException ex) {
|
||||||
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
|
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private boolean isRetryableProject(Path projectDirectory) {
|
||||||
|
String projectId = projectDirectory.getFileName().toString();
|
||||||
|
if (!Files.isRegularFile(projectDirectory.resolve("project.json"))) {
|
||||||
|
log.warn("event=highlight_render_project_skipped project_id={} reason=missing_project_metadata",
|
||||||
|
projectId);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
HighlightProjectStatus status = store.readJson(projectId, "project.json", HighlightProject.class).status();
|
||||||
|
return status == HighlightProjectStatus.WAITING_FOR_DIRECTOR
|
||||||
|
|| status == HighlightProjectStatus.PLANNED;
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
log.warn("event=highlight_render_project_skipped project_id={} reason=invalid_project_metadata "
|
||||||
|
+ "error_type={}", projectId, ex.getClass().getSimpleName());
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean hasRequiredApproval(Path projectDirectory) {
|
||||||
|
return !highlightScheduler.isRequireDirectorApproval()
|
||||||
|
|| Files.isRegularFile(projectDirectory.resolve("director")
|
||||||
|
.resolve(highlightScheduler.getApprovalFileName()));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,196 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.application.BadRequestException;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.function.Function;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightDirectorPlanValidator {
|
||||||
|
|
||||||
|
private static final Pattern SAFE_ID = Pattern.compile("[A-Za-z0-9][A-Za-z0-9._-]{0,127}");
|
||||||
|
private static final Set<String> STORY_PURPOSES = Set.of("opening_hook", "rising_energy", "hero_payoff");
|
||||||
|
private static final double EPSILON = 0.001;
|
||||||
|
private static final int MAX_TITLE_LENGTH = 120;
|
||||||
|
private static final int MAX_DIRECTION_LENGTH = 1_000;
|
||||||
|
private static final int MAX_VOICEOVER_LINE_LENGTH = 240;
|
||||||
|
private static final int MAX_OVERLAY_LENGTH = 80;
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
|
||||||
|
public HighlightDirectorPlanValidator(VideoClippingProperties properties, HighlightProjectStore store) {
|
||||||
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
|
this.store = store;
|
||||||
|
}
|
||||||
|
|
||||||
|
public HighlightDirectorPlan validate(String projectId, HighlightDirectorPlan plan) {
|
||||||
|
if (plan == null) {
|
||||||
|
reject("Highlight director plan is required");
|
||||||
|
}
|
||||||
|
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||||
|
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
|
||||||
|
HighlightSourceAnalysis.class);
|
||||||
|
if (!projectId.equals(plan.projectId())) {
|
||||||
|
reject("Director plan projectId must match the project");
|
||||||
|
}
|
||||||
|
if (!project.sourceVideoFileName().equals(plan.sourceVideoFileName())) {
|
||||||
|
reject("Director plan sourceVideoFileName must match the project source");
|
||||||
|
}
|
||||||
|
|
||||||
|
ContentCategory requestedCategory = parseCategory(plan.contentCategory());
|
||||||
|
Path categoryFile = store.analysisDirectory(projectId).resolve("category.json");
|
||||||
|
if (!Files.isRegularFile(categoryFile)) {
|
||||||
|
reject("Persisted category analysis is required before accepting a director plan");
|
||||||
|
}
|
||||||
|
CinematicHighlightAnalysis category = store.readJson(projectId, "analysis/category.json",
|
||||||
|
CinematicHighlightAnalysis.class);
|
||||||
|
if (requestedCategory != category.category()) {
|
||||||
|
reject("Director plan contentCategory must match the reviewed category analysis");
|
||||||
|
}
|
||||||
|
|
||||||
|
Path candidateFile = store.analysisDirectory(projectId).resolve("highlight-candidates.json");
|
||||||
|
if (!Files.isRegularFile(candidateFile)) {
|
||||||
|
reject("Persisted highlight candidates are required before accepting a director plan");
|
||||||
|
}
|
||||||
|
HighlightCandidate[] persisted = store.readJson(projectId, "analysis/highlight-candidates.json",
|
||||||
|
HighlightCandidate[].class);
|
||||||
|
Map<String, HighlightCandidate> candidates = Arrays.stream(persisted)
|
||||||
|
.collect(Collectors.toMap(HighlightCandidate::id, Function.identity(), (left, right) -> {
|
||||||
|
reject("Persisted highlight candidate IDs must be unique");
|
||||||
|
return left;
|
||||||
|
}));
|
||||||
|
|
||||||
|
if (plan.highlights() == null || plan.highlights().isEmpty()) {
|
||||||
|
reject("Director plan must contain at least one highlight");
|
||||||
|
}
|
||||||
|
if (plan.highlights().size() > properties.getMaxHighlightsPerSource()) {
|
||||||
|
reject("Director plan exceeds maxHighlightsPerSource=" + properties.getMaxHighlightsPerSource());
|
||||||
|
}
|
||||||
|
|
||||||
|
Set<String> highlightIds = new HashSet<>();
|
||||||
|
Set<String> candidateIds = new HashSet<>();
|
||||||
|
for (HighlightDirectorPlan.HighlightItem highlight : plan.highlights()) {
|
||||||
|
validateHighlight(highlight, analysis, candidates, highlightIds, candidateIds);
|
||||||
|
}
|
||||||
|
if (plan.summary() == null || plan.summary().isBlank() || plan.summary().length() > MAX_DIRECTION_LENGTH) {
|
||||||
|
reject("Director plan summary is required and must be 1000 characters or fewer");
|
||||||
|
}
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void validateHighlight(
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
HighlightSourceAnalysis analysis,
|
||||||
|
Map<String, HighlightCandidate> candidates,
|
||||||
|
Set<String> highlightIds,
|
||||||
|
Set<String> candidateIds
|
||||||
|
) {
|
||||||
|
if (highlight == null) {
|
||||||
|
reject("Director plan highlight item cannot be null");
|
||||||
|
}
|
||||||
|
if (!safeId(highlight.highlightId()) || !highlightIds.add(highlight.highlightId())) {
|
||||||
|
reject("Highlight IDs must be unique safe identifiers");
|
||||||
|
}
|
||||||
|
if (!safeId(highlight.candidateId()) || !candidateIds.add(highlight.candidateId())) {
|
||||||
|
reject("Candidate IDs must be unique safe identifiers in the director plan");
|
||||||
|
}
|
||||||
|
HighlightCandidate candidate = candidates.get(highlight.candidateId());
|
||||||
|
if (candidate == null) {
|
||||||
|
reject("Unknown highlight candidate: " + highlight.candidateId());
|
||||||
|
}
|
||||||
|
if (!analysis.source().clipId().equals(candidate.clipId())) {
|
||||||
|
reject("Highlight candidate does not belong to the analyzed source clip: " + candidate.id());
|
||||||
|
}
|
||||||
|
if (!finite(highlight.sourceStartSeconds()) || !finite(highlight.sourceEndSeconds())
|
||||||
|
|| highlight.sourceStartSeconds() < candidate.sourceStartSeconds() - EPSILON
|
||||||
|
|| highlight.sourceEndSeconds() > candidate.sourceEndSeconds() + EPSILON
|
||||||
|
|| highlight.sourceEndSeconds() <= highlight.sourceStartSeconds()
|
||||||
|
|| highlight.sourceEndSeconds() > analysis.source().durationSeconds() + EPSILON) {
|
||||||
|
reject("Highlight source range must be contained by candidate: " + candidate.id());
|
||||||
|
}
|
||||||
|
if (!finite(highlight.targetDurationSeconds())
|
||||||
|
|| highlight.targetDurationSeconds() < effectiveMinimumDuration() - EPSILON
|
||||||
|
|| highlight.targetDurationSeconds() > properties.getHighlightMaxDurationSeconds() + EPSILON) {
|
||||||
|
reject("Highlight target duration must be within the configured duration bounds");
|
||||||
|
}
|
||||||
|
double requiredPlaybackSpeed = (highlight.sourceEndSeconds() - highlight.sourceStartSeconds())
|
||||||
|
/ highlight.targetDurationSeconds();
|
||||||
|
if (requiredPlaybackSpeed < 0.25 - EPSILON || requiredPlaybackSpeed > 4.0 + EPSILON) {
|
||||||
|
reject("Highlight source range and target duration require playback speed outside 0.25..4.0");
|
||||||
|
}
|
||||||
|
requiredText(highlight.title(), MAX_TITLE_LENGTH, "Highlight title");
|
||||||
|
if (!STORY_PURPOSES.contains(highlight.storyPurpose())) {
|
||||||
|
reject("Unsupported storyPurpose: " + highlight.storyPurpose());
|
||||||
|
}
|
||||||
|
requiredText(highlight.visualTreatment(), MAX_DIRECTION_LENGTH, "Visual treatment");
|
||||||
|
requiredText(highlight.musicDirection(), MAX_DIRECTION_LENGTH, "Music direction");
|
||||||
|
requiredText(highlight.sfxDirection(), MAX_DIRECTION_LENGTH, "SFX direction");
|
||||||
|
requiredText(highlight.renderNotes(), MAX_DIRECTION_LENGTH, "Render notes");
|
||||||
|
if (highlight.voiceover() == null || highlight.voiceover().isEmpty()) {
|
||||||
|
reject("At least one grounded voiceover line is required");
|
||||||
|
}
|
||||||
|
for (String line : highlight.voiceover()) {
|
||||||
|
requiredText(line, MAX_VOICEOVER_LINE_LENGTH, "Voiceover line");
|
||||||
|
}
|
||||||
|
double voiceoverDuration = highlight.voiceover().stream()
|
||||||
|
.mapToDouble(HighlightAssetPreparationService::estimatedVoiceoverDuration)
|
||||||
|
.sum();
|
||||||
|
if (voiceoverDuration > highlight.targetDurationSeconds() + EPSILON) {
|
||||||
|
reject("Voiceover script exceeds the highlight timing budget");
|
||||||
|
}
|
||||||
|
if (highlight.overlays() == null) {
|
||||||
|
reject("Highlight overlays list is required");
|
||||||
|
}
|
||||||
|
for (String overlay : highlight.overlays()) {
|
||||||
|
requiredText(overlay, MAX_OVERLAY_LENGTH, "Overlay text");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private double effectiveMinimumDuration() {
|
||||||
|
return Math.min(properties.getHighlightMinDurationSeconds(), properties.getHighlightMaxDurationSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private ContentCategory parseCategory(String value) {
|
||||||
|
if (value == null || value.isBlank()) {
|
||||||
|
reject("Director plan contentCategory is required");
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return ContentCategory.valueOf(value.toUpperCase(Locale.ROOT));
|
||||||
|
} catch (IllegalArgumentException ex) {
|
||||||
|
reject("Unsupported director plan contentCategory: " + value);
|
||||||
|
return ContentCategory.GENERIC_VLOG;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean safeId(String value) {
|
||||||
|
return value != null && SAFE_ID.matcher(value).matches() && !value.contains("..");
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean finite(double value) {
|
||||||
|
return Double.isFinite(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void requiredText(String value, int maximumLength, String field) {
|
||||||
|
if (value == null || value.isBlank() || value.length() > maximumLength) {
|
||||||
|
reject(field + " is required and must be " + maximumLength + " characters or fewer");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void reject(String message) {
|
||||||
|
throw new BadRequestException(message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -66,6 +66,12 @@ public class HighlightDirectorPromptGenerator {
|
||||||
) {
|
) {
|
||||||
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
|
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
|
||||||
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
||||||
|
String evidenceStatus = cinematic == null || cinematic.categoryConfidence() < 0.55
|
||||||
|
? "review_required"
|
||||||
|
: "model_supported_review_required";
|
||||||
|
String categoryEvidence = cinematic == null
|
||||||
|
? "No persisted category analysis is available."
|
||||||
|
: "confidence=%s reasons=%s".formatted(cinematic.categoryConfidence(), cinematic.categoryReasons());
|
||||||
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
|
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
|
||||||
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
|
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
|
||||||
|
|
||||||
|
|
@ -97,10 +103,14 @@ public class HighlightDirectorPromptGenerator {
|
||||||
- Project ID: `%s`
|
- Project ID: `%s`
|
||||||
- Source file: `%s`
|
- Source file: `%s`
|
||||||
- Detected content category: `%s`
|
- Detected content category: `%s`
|
||||||
|
- Category evidence status: `%s`
|
||||||
|
- Category evidence: `%s`
|
||||||
- Final highlight duration must be content-driven, not fixed.
|
- Final highlight duration must be content-driven, not fixed.
|
||||||
- Keep the edit cinematic, premium, and grounded in the visible footage.
|
- Keep the edit cinematic, premium, and grounded in the visible footage.
|
||||||
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
|
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
|
||||||
- Do not invent clips, timestamps, or facts.
|
- Do not invent clips, timestamps, or facts.
|
||||||
|
- Treat candidate scores as ranking hints, not proof of cinematic quality.
|
||||||
|
- Inspect the referenced media before selecting any candidate or confirming the category.
|
||||||
- Do not render video.
|
- Do not render video.
|
||||||
- Do not modify source media or analysis artifacts.
|
- Do not modify source media or analysis artifacts.
|
||||||
- Return strict JSON only. No Markdown fences.
|
- Return strict JSON only. No Markdown fences.
|
||||||
|
|
@ -146,7 +156,7 @@ public class HighlightDirectorPromptGenerator {
|
||||||
"summary": "short rationale for the selected highlights"
|
"summary": "short rationale for the selected highlights"
|
||||||
}
|
}
|
||||||
""".formatted(
|
""".formatted(
|
||||||
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category,
|
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category, evidenceStatus, categoryEvidence,
|
||||||
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
|
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
|
||||||
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
|
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
|
||||||
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),
|
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),
|
||||||
|
|
|
||||||
|
|
@ -62,7 +62,14 @@ public class HighlightFfmpegRenderer {
|
||||||
Path projectDirectory = store.projectDirectory(projectId);
|
Path projectDirectory = store.projectDirectory(projectId);
|
||||||
Path source = projectDirectory.resolve("source").resolve(store.readJson(projectId, "project.json",
|
Path source = projectDirectory.resolve("source").resolve(store.readJson(projectId, "project.json",
|
||||||
HighlightProject.class).sourceVideoFileName());
|
HighlightProject.class).sourceVideoFileName());
|
||||||
|
HighlightSourceAnalysis sourceAnalysis = store.readJson(projectId, "analysis/source-analysis.json",
|
||||||
|
HighlightSourceAnalysis.class);
|
||||||
|
boolean sourceAudioPresent = sourceAnalysis.source() != null
|
||||||
|
&& sourceAnalysis.source().audioCodec() != null
|
||||||
|
&& !sourceAnalysis.source().audioCodec().isBlank();
|
||||||
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||||
|
Path audioDirectory = highlightDirectory.resolve("assets");
|
||||||
|
requireRequestedAssets(plan, audioDirectory);
|
||||||
Path work = highlightDirectory.resolve("render-work");
|
Path work = highlightDirectory.resolve("render-work");
|
||||||
createDirectory(work);
|
createDirectory(work);
|
||||||
createDirectory(highlightDirectory.resolve("rendered-clips"));
|
createDirectory(highlightDirectory.resolve("rendered-clips"));
|
||||||
|
|
@ -108,10 +115,9 @@ public class HighlightFfmpegRenderer {
|
||||||
|
|
||||||
Path output = highlightDirectory.resolve("final.mp4");
|
Path output = highlightDirectory.resolve("final.mp4");
|
||||||
Path preview = highlightDirectory.resolve("preview.mp4");
|
Path preview = highlightDirectory.resolve("preview.mp4");
|
||||||
Path audioDirectory = highlightDirectory.resolve("assets");
|
|
||||||
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
||||||
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||||
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
List<VoiceoverInput> voiceovers = voiceoverInputs(plan, audioDirectory);
|
||||||
List<SfxInput> sfx = plan.audioCues().stream()
|
List<SfxInput> sfx = plan.audioCues().stream()
|
||||||
.filter(cue -> "sfx".equals(cue.type()))
|
.filter(cue -> "sfx".equals(cue.type()))
|
||||||
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
||||||
|
|
@ -119,28 +125,18 @@ public class HighlightFfmpegRenderer {
|
||||||
log(projectId, highlight.highlightId(), "highlight_render_audio_started",
|
log(projectId, highlight.highlightId(), "highlight_render_audio_started",
|
||||||
"music", Boolean.toString(Files.isRegularFile(music)),
|
"music", Boolean.toString(Files.isRegularFile(music)),
|
||||||
"sfx", Integer.toString(sfx.size()),
|
"sfx", Integer.toString(sfx.size()),
|
||||||
"voiceover", Boolean.toString(Files.isRegularFile(voiceover)));
|
"voiceover_lines", Integer.toString(voiceovers.size()));
|
||||||
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) {
|
AudioCue musicCue = plan.audioCues().stream()
|
||||||
try {
|
.filter(cue -> "music".equals(cue.type()))
|
||||||
run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null,
|
.findFirst()
|
||||||
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands);
|
.orElse(null);
|
||||||
} catch (IllegalStateException ex) {
|
if (musicCue != null || !plan.voiceover().isEmpty() || !sfx.isEmpty()) {
|
||||||
log(projectId, highlight.highlightId(), "highlight_render_audio_failed",
|
run(audioMixCommand(postTimeline, musicCue == null ? null : music, musicCue,
|
||||||
"error", summarizeMessage(ex.getMessage()),
|
voiceovers, sfx, sourceAudioPresent, output), commands);
|
||||||
"fallback", "copy_post_timeline");
|
|
||||||
copy(postTimeline, output);
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
copy(postTimeline, output);
|
copy(postTimeline, output);
|
||||||
}
|
}
|
||||||
try {
|
|
||||||
run(previewCommand(output, preview), commands);
|
run(previewCommand(output, preview), commands);
|
||||||
} catch (IllegalStateException ex) {
|
|
||||||
log(projectId, highlight.highlightId(), "highlight_render_preview_failed",
|
|
||||||
"error", summarizeMessage(ex.getMessage()),
|
|
||||||
"fallback", "copy_final_output");
|
|
||||||
copy(output, preview);
|
|
||||||
}
|
|
||||||
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
||||||
.timelineEndSeconds();
|
.timelineEndSeconds();
|
||||||
RenderManifest manifest = new RenderManifest(projectId,
|
RenderManifest manifest = new RenderManifest(projectId,
|
||||||
|
|
@ -163,19 +159,23 @@ public class HighlightFfmpegRenderer {
|
||||||
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||||
if (Files.isRegularFile(music)) {
|
if (Files.isRegularFile(music)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
||||||
"project-local", "highlight-assets"));
|
assetLicense(music), "highlight-assets"));
|
||||||
}
|
}
|
||||||
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
List<VoiceoverInput> voiceovers = voiceoverInputs(plan, audioDirectory);
|
||||||
|
for (int index = 0; index < voiceovers.size(); index++) {
|
||||||
|
Path voiceover = voiceovers.get(index).path();
|
||||||
if (Files.isRegularFile(voiceover)) {
|
if (Files.isRegularFile(voiceover)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER,
|
||||||
"project-local", "highlight-assets"));
|
"voiceover_%03d".formatted(index + 1), voiceover.toString(),
|
||||||
|
assetLicense(voiceover), "highlight-assets"));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
for (AudioCue cue : plan.audioCues()) {
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
if ("sfx".equals(cue.type())) {
|
if ("sfx".equals(cue.type())) {
|
||||||
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
||||||
if (Files.isRegularFile(sfx)) {
|
if (Files.isRegularFile(sfx)) {
|
||||||
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
||||||
"project-local", "highlight-assets"));
|
assetLicense(sfx), "highlight-assets"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -236,24 +236,57 @@ public class HighlightFfmpegRenderer {
|
||||||
"-c:a", "copy", output.toString());
|
"-c:a", "copy", output.toString());
|
||||||
}
|
}
|
||||||
|
|
||||||
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
|
List<String> audioMixCommand(Path timeline, Path music, AudioCue musicCue, List<VoiceoverInput> voiceovers,
|
||||||
|
List<SfxInput> soundEffects, Path output) {
|
||||||
|
return audioMixCommand(timeline, music, musicCue, voiceovers, soundEffects, true, output);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> audioMixCommand(Path timeline, Path music, AudioCue musicCue, List<VoiceoverInput> voiceovers,
|
||||||
|
List<SfxInput> soundEffects, boolean sourceAudioPresent, Path output) {
|
||||||
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||||
"-i", timeline.toString()));
|
"-i", timeline.toString()));
|
||||||
List<String> labels = new ArrayList<>(List.of("[0:a]"));
|
List<String> labels = new ArrayList<>();
|
||||||
|
if (sourceAudioPresent) {
|
||||||
|
labels.add("[0:a]");
|
||||||
|
}
|
||||||
int input = 1;
|
int input = 1;
|
||||||
StringBuilder filters = new StringBuilder();
|
StringBuilder filters = new StringBuilder();
|
||||||
boolean hasMusic = music != null;
|
boolean hasMusic = music != null;
|
||||||
boolean hasVoiceover = voiceover != null;
|
boolean hasVoiceover = !voiceovers.isEmpty();
|
||||||
if (music != null) {
|
if (music != null) {
|
||||||
command.addAll(List.of("-i", music.toString()));
|
command.addAll(List.of("-i", music.toString()));
|
||||||
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
|
filters.append("[").append(input).append(":a]");
|
||||||
|
if (musicCue != null) {
|
||||||
|
double duration = musicCue.timelineEndSeconds() - musicCue.timelineStartSeconds();
|
||||||
|
long delayMillis = Math.round(musicCue.timelineStartSeconds() * 1000);
|
||||||
|
filters.append("atrim=duration=").append(duration)
|
||||||
|
.append(",asetpts=PTS-STARTPTS,volume=").append(musicCue.gainDb()).append("dB,adelay=")
|
||||||
|
.append(delayMillis).append("|").append(delayMillis).append("[music_raw];");
|
||||||
|
} else {
|
||||||
|
filters.append("volume=-12dB[music_raw];");
|
||||||
|
}
|
||||||
input++;
|
input++;
|
||||||
}
|
}
|
||||||
if (voiceover != null) {
|
List<String> voiceLabels = new ArrayList<>();
|
||||||
command.addAll(List.of("-i", voiceover.toString()));
|
for (int index = 0; index < voiceovers.size(); index++) {
|
||||||
filters.append("[").append(input).append(":a]volume=1.0[voice];");
|
VoiceoverInput voiceover = voiceovers.get(index);
|
||||||
|
VoiceoverLine line = voiceover.line();
|
||||||
|
String label = "voice_line_" + index;
|
||||||
|
double duration = line.timelineEndSeconds() - line.timelineStartSeconds();
|
||||||
|
long delayMillis = Math.round(line.timelineStartSeconds() * 1000);
|
||||||
|
command.addAll(List.of("-i", voiceover.path().toString()));
|
||||||
|
filters.append("[").append(input).append(":a]atrim=duration=").append(duration)
|
||||||
|
.append(",asetpts=PTS-STARTPTS,volume=1.0,adelay=")
|
||||||
|
.append(delayMillis).append("|").append(delayMillis).append("[").append(label).append("];");
|
||||||
|
voiceLabels.add("[" + label + "]");
|
||||||
input++;
|
input++;
|
||||||
}
|
}
|
||||||
|
if (voiceLabels.size() == 1) {
|
||||||
|
filters.append(voiceLabels.get(0)).append("anull[voice];");
|
||||||
|
} else if (voiceLabels.size() > 1) {
|
||||||
|
filters.append(String.join("", voiceLabels)).append("amix=inputs=").append(voiceLabels.size())
|
||||||
|
.append(":duration=longest:dropout_transition=0[voice];");
|
||||||
|
}
|
||||||
if (hasMusic && hasVoiceover) {
|
if (hasMusic && hasVoiceover) {
|
||||||
filters.append("[music_raw][voice]sidechaincompress=threshold=")
|
filters.append("[music_raw][voice]sidechaincompress=threshold=")
|
||||||
.append(properties.getMusicDuckingThreshold())
|
.append(properties.getMusicDuckingThreshold())
|
||||||
|
|
@ -283,17 +316,50 @@ public class HighlightFfmpegRenderer {
|
||||||
input++;
|
input++;
|
||||||
}
|
}
|
||||||
filters.append(String.join("", labels)).append("amix=inputs=").append(labels.size())
|
filters.append(String.join("", labels)).append("amix=inputs=").append(labels.size())
|
||||||
.append(":duration=first:dropout_transition=2,loudnorm=I=")
|
.append(":duration=").append(sourceAudioPresent ? "first" : "longest")
|
||||||
|
.append(":dropout_transition=2,loudnorm=I=")
|
||||||
.append(properties.getLoudnessTargetI())
|
.append(properties.getLoudnessTargetI())
|
||||||
.append(":TP=").append(properties.getLoudnessTruePeak())
|
.append(":TP=").append(properties.getLoudnessTruePeak())
|
||||||
.append(":LRA=").append(properties.getLoudnessRange())
|
.append(":LRA=").append(properties.getLoudnessRange());
|
||||||
.append("[a]");
|
if (!sourceAudioPresent) {
|
||||||
|
filters.append(",apad");
|
||||||
|
}
|
||||||
|
filters.append("[a]");
|
||||||
command.addAll(List.of("-filter_complex", filters.toString(), "-map", "0:v:0", "-map", "[a]",
|
command.addAll(List.of("-filter_complex", filters.toString(), "-map", "0:v:0", "-map", "[a]",
|
||||||
"-c:v", "copy", "-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
"-c:v", "copy", "-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
||||||
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()));
|
"-ar", Integer.toString(properties.getAudioSampleRate())));
|
||||||
|
if (!sourceAudioPresent) {
|
||||||
|
command.add("-shortest");
|
||||||
|
}
|
||||||
|
command.add(output.toString());
|
||||||
return List.copyOf(command);
|
return List.copyOf(command);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void requireRequestedAssets(EditPlan plan, Path audioDirectory) {
|
||||||
|
List<String> missing = new ArrayList<>();
|
||||||
|
if (plan.audioCues().stream().anyMatch(cue -> "music".equals(cue.type()))
|
||||||
|
&& !AssetLicensePolicy.isLicensed(audioDirectory.resolve("music").resolve("music.wav"))) {
|
||||||
|
missing.add("music");
|
||||||
|
}
|
||||||
|
for (int index = 0; index < plan.voiceover().size(); index++) {
|
||||||
|
Path voiceover = audioDirectory.resolve("voiceover")
|
||||||
|
.resolve("voiceover_%03d.wav".formatted(index + 1));
|
||||||
|
if (!AssetLicensePolicy.isLicensed(voiceover)) {
|
||||||
|
missing.add("voiceover:" + (index + 1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
|
if ("sfx".equals(cue.type())
|
||||||
|
&& !AssetLicensePolicy.isLicensed(
|
||||||
|
audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"))) {
|
||||||
|
missing.add("sfx:" + cue.assetKey());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!missing.isEmpty()) {
|
||||||
|
throw new IllegalStateException("Requested render assets are missing: " + missing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
private RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||||
List<Path> renderedClips,
|
List<Path> renderedClips,
|
||||||
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||||
|
|
@ -303,20 +369,150 @@ public class HighlightFfmpegRenderer {
|
||||||
checks.add(new RenderQaCheck("rendered_clips_exist",
|
checks.add(new RenderQaCheck("rendered_clips_exist",
|
||||||
renderedClips.stream().allMatch(Files::isRegularFile), "ERROR",
|
renderedClips.stream().allMatch(Files::isRegularFile), "ERROR",
|
||||||
"all rendered highlight clips exist"));
|
"all rendered highlight clips exist"));
|
||||||
checks.add(new RenderQaCheck("duration_matches_timeline", true, "ERROR",
|
checks.add(durationMatchesTimelineCheck(output, duration));
|
||||||
"highlight output duration is based on the edit plan"));
|
boolean requiredAssetsResolved = requestedAssetsResolved(plan, assets);
|
||||||
checks.add(new RenderQaCheck("required_assets_resolved", true, "WARNING",
|
checks.add(new RenderQaCheck("required_assets_resolved", requiredAssetsResolved, "ERROR",
|
||||||
assets.isEmpty() ? "no optional assets resolved" : "resolved assets: " + assets.size()));
|
requiredAssetsResolved ? "all requested assets resolved" : "one or more requested assets missing"));
|
||||||
checks.add(new RenderQaCheck("text_overlays_safe", true, "ERROR",
|
checks.add(overlaysSafeCheck(plan, duration));
|
||||||
"overlay timing was validated by the director plan"));
|
boolean audioRequested = !plan.audioCues().isEmpty() || !plan.voiceover().isEmpty();
|
||||||
checks.add(new RenderQaCheck("audio_mastering_applied", true, "WARNING",
|
boolean audioMasteringApplied = !audioRequested || commands.stream().flatMap(List::stream)
|
||||||
"optional asset audio mix used when available"));
|
.anyMatch(argument -> argument.contains("loudnorm"));
|
||||||
|
checks.add(new RenderQaCheck("audio_mastering_applied", audioMasteringApplied, "ERROR",
|
||||||
|
audioMasteringApplied ? "render command contains the required loudnorm filter"
|
||||||
|
: "requested audio render command is missing loudnorm"));
|
||||||
checks.add(new RenderQaCheck("ffmpeg_commands_completed", !commands.isEmpty(), "ERROR",
|
checks.add(new RenderQaCheck("ffmpeg_commands_completed", !commands.isEmpty(), "ERROR",
|
||||||
"recorded command count=" + commands.size()));
|
"recorded command count=" + commands.size()));
|
||||||
|
checks.add(runQaProbe("black_frames", blackDetectCommand(output), "ERROR",
|
||||||
|
result -> result.exitCode() == 0 && !result.output().contains("black_start"),
|
||||||
|
"blackdetect found one or more black frame ranges"));
|
||||||
|
checks.add(runQaProbe("long_silence", silenceDetectCommand(output), "WARNING",
|
||||||
|
result -> result.exitCode() == 0 && !result.output().contains("silence_start"),
|
||||||
|
"silencedetect found one or more silent ranges of at least two seconds"));
|
||||||
|
checks.add(runQaProbe("audio_clipping", clippingDetectCommand(output), "ERROR",
|
||||||
|
this::audioPeakIsSafe,
|
||||||
|
"audio peak is at or above -0.1 dBFS"));
|
||||||
boolean passed = checks.stream().noneMatch(check -> !check.passed() && "ERROR".equals(check.severity()));
|
boolean passed = checks.stream().noneMatch(check -> !check.passed() && "ERROR".equals(check.severity()));
|
||||||
return new RenderQaReport(projectId, passed, checks, Instant.now());
|
return new RenderQaReport(projectId, passed, checks, Instant.now());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private RenderQaCheck durationMatchesTimelineCheck(Path output, double expectedDuration) {
|
||||||
|
double tolerance = Math.max(0.1, 2.0 / properties.getOutputFrameRate());
|
||||||
|
try {
|
||||||
|
ProcessResult result = executor.execute(durationProbeCommand(output));
|
||||||
|
if (result.exitCode() != 0) {
|
||||||
|
return new RenderQaCheck("duration_matches_timeline", false, "ERROR",
|
||||||
|
"ffprobe exited with code " + result.exitCode());
|
||||||
|
}
|
||||||
|
double actualDuration = Double.parseDouble(result.output().trim());
|
||||||
|
boolean matches = Double.isFinite(actualDuration)
|
||||||
|
&& Math.abs(expectedDuration - actualDuration) <= tolerance;
|
||||||
|
return new RenderQaCheck("duration_matches_timeline", matches, "ERROR",
|
||||||
|
"expected=%s actual=%s tolerance=%s"
|
||||||
|
.formatted(expectedDuration, actualDuration, tolerance));
|
||||||
|
} catch (NumberFormatException ex) {
|
||||||
|
return new RenderQaCheck("duration_matches_timeline", false, "ERROR",
|
||||||
|
"ffprobe returned an invalid duration");
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return new RenderQaCheck("duration_matches_timeline", false, "ERROR", "ffprobe interrupted");
|
||||||
|
} catch (IOException ex) {
|
||||||
|
return new RenderQaCheck("duration_matches_timeline", false, "ERROR",
|
||||||
|
"ffprobe failed: " + ex.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private RenderQaCheck overlaysSafeCheck(EditPlan plan, double duration) {
|
||||||
|
List<String> unsafe = plan.overlays().stream()
|
||||||
|
.filter(overlay -> overlay.timelineStartSeconds() < 0
|
||||||
|
|| overlay.timelineEndSeconds() > duration
|
||||||
|
|| overlay.timelineStartSeconds() >= overlay.timelineEndSeconds()
|
||||||
|
|| !safeOverlayPlacements().contains(overlay.placement()))
|
||||||
|
.map(TextOverlay::text)
|
||||||
|
.toList();
|
||||||
|
return new RenderQaCheck("text_overlays_safe", unsafe.isEmpty(), "ERROR",
|
||||||
|
unsafe.isEmpty() ? "overlay timing and configured placement are within plan bounds"
|
||||||
|
: "unsafe overlays: " + unsafe);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<String> safeOverlayPlacements() {
|
||||||
|
return List.of("lower_left_safe", "upper_left_safe", "upper_right_safe", "center_safe",
|
||||||
|
"lower_center_safe");
|
||||||
|
}
|
||||||
|
|
||||||
|
private RenderQaCheck runQaProbe(String name, List<String> command, String severity,
|
||||||
|
java.util.function.Predicate<ProcessResult> passed,
|
||||||
|
String failureDetails) {
|
||||||
|
try {
|
||||||
|
ProcessResult result = executor.execute(command);
|
||||||
|
if (passed.test(result)) {
|
||||||
|
return new RenderQaCheck(name, true, severity, "probe passed");
|
||||||
|
}
|
||||||
|
return new RenderQaCheck(name, false, severity, result.exitCode() == 0
|
||||||
|
? failureDetails : "probe exited with code " + result.exitCode());
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return new RenderQaCheck(name, false, severity, "probe interrupted");
|
||||||
|
} catch (IOException ex) {
|
||||||
|
return new RenderQaCheck(name, false, severity, "probe failed: " + ex.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> durationProbeCommand(Path output) {
|
||||||
|
return List.of(properties.getFfprobeBinary(), "-v", "error", "-show_entries", "format=duration",
|
||||||
|
"-of", "default=noprint_wrappers=1:nokey=1", output.toString());
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> blackDetectCommand(Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-v", "info", "-i", output.toString(),
|
||||||
|
"-vf", "blackdetect=d=0.5:pic_th=0.98", "-an", "-f", "null", "-");
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> silenceDetectCommand(Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-v", "info", "-i", output.toString(),
|
||||||
|
"-af", "silencedetect=noise=-45dB:d=2", "-vn", "-f", "null", "-");
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> clippingDetectCommand(Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-v", "info", "-i", output.toString(),
|
||||||
|
"-af", "astats=metadata=1:reset=1", "-vn", "-f", "null", "-");
|
||||||
|
}
|
||||||
|
|
||||||
|
boolean audioPeakIsSafe(ProcessResult result) {
|
||||||
|
if (result.exitCode() != 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
List<Double> peaks = result.output().lines()
|
||||||
|
.filter(line -> line.contains("Peak level dB"))
|
||||||
|
.map(line -> line.substring(line.lastIndexOf(':') + 1).trim())
|
||||||
|
.map(value -> {
|
||||||
|
try {
|
||||||
|
return Double.parseDouble(value);
|
||||||
|
} catch (NumberFormatException ex) {
|
||||||
|
return Double.NaN;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.filter(Double::isFinite)
|
||||||
|
.toList();
|
||||||
|
return !peaks.isEmpty() && peaks.stream().noneMatch(peak -> peak >= -0.1);
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean requestedAssetsResolved(EditPlan plan, List<ResolvedEditAsset> assets) {
|
||||||
|
boolean musicResolved = plan.audioCues().stream().noneMatch(cue -> "music".equals(cue.type()))
|
||||||
|
|| assets.stream().anyMatch(asset -> asset.type() == EditAssetType.MUSIC);
|
||||||
|
long resolvedVoiceoverLines = assets.stream().filter(asset -> asset.type() == EditAssetType.VOICEOVER)
|
||||||
|
.count();
|
||||||
|
boolean voiceoverResolved = resolvedVoiceoverLines == plan.voiceover().size();
|
||||||
|
boolean sfxResolved = plan.audioCues().stream()
|
||||||
|
.filter(cue -> "sfx".equals(cue.type()))
|
||||||
|
.allMatch(cue -> assets.stream().anyMatch(asset -> asset.type() == EditAssetType.SFX
|
||||||
|
&& cue.assetKey().equals(asset.assetKey())));
|
||||||
|
return musicResolved && voiceoverResolved && sfxResolved;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String assetLicense(Path asset) {
|
||||||
|
return AssetLicensePolicy.read(asset).orElse(AssetLicensePolicy.UNTRACKED);
|
||||||
|
}
|
||||||
|
|
||||||
private String cinematicVisualFilter(String visualTreatment) {
|
private String cinematicVisualFilter(String visualTreatment) {
|
||||||
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||||
return "";
|
return "";
|
||||||
|
|
@ -326,6 +522,15 @@ public class HighlightFfmpegRenderer {
|
||||||
+ ",vignette=PI/7";
|
+ ",vignette=PI/7";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private List<VoiceoverInput> voiceoverInputs(EditPlan plan, Path audioDirectory) {
|
||||||
|
List<VoiceoverInput> inputs = new ArrayList<>();
|
||||||
|
for (int index = 0; index < plan.voiceover().size(); index++) {
|
||||||
|
inputs.add(new VoiceoverInput(audioDirectory.resolve("voiceover")
|
||||||
|
.resolve("voiceover_%03d.wav".formatted(index + 1)), plan.voiceover().get(index)));
|
||||||
|
}
|
||||||
|
return List.copyOf(inputs);
|
||||||
|
}
|
||||||
|
|
||||||
private String dynamicCropFilter(String visualTreatment) {
|
private String dynamicCropFilter(String visualTreatment) {
|
||||||
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||||
return "";
|
return "";
|
||||||
|
|
@ -471,6 +676,8 @@ public class HighlightFfmpegRenderer {
|
||||||
|
|
||||||
record SfxInput(Path path, AudioCue cue) { }
|
record SfxInput(Path path, AudioCue cue) { }
|
||||||
|
|
||||||
|
record VoiceoverInput(Path path, VoiceoverLine line) { }
|
||||||
|
|
||||||
@FunctionalInterface
|
@FunctionalInterface
|
||||||
interface ProcessExecutor {
|
interface ProcessExecutor {
|
||||||
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||||
|
|
|
||||||
|
|
@ -16,12 +16,16 @@ import java.util.ArrayList;
|
||||||
import java.util.Comparator;
|
import java.util.Comparator;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Optional;
|
import java.util.Optional;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
|
||||||
@Component
|
@Component
|
||||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
public class HighlightLocalAssetWorker {
|
public class HighlightLocalAssetWorker {
|
||||||
|
|
||||||
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
|
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
|
||||||
|
private static final Pattern SAFE_KEY = Pattern.compile("[A-Za-z0-9][A-Za-z0-9._-]{0,127}");
|
||||||
|
private static final Set<String> SUPPORTED_TYPES = Set.of("music", "sfx", "voiceover");
|
||||||
|
|
||||||
private final HighlightProjectStore store;
|
private final HighlightProjectStore store;
|
||||||
private final EditAssetProvider assetProvider;
|
private final EditAssetProvider assetProvider;
|
||||||
|
|
@ -52,6 +56,7 @@ public class HighlightLocalAssetWorker {
|
||||||
List<String> resolved = new ArrayList<>();
|
List<String> resolved = new ArrayList<>();
|
||||||
List<String> pending = new ArrayList<>();
|
List<String> pending = new ArrayList<>();
|
||||||
for (HighlightAssetRequest request : requests) {
|
for (HighlightAssetRequest request : requests) {
|
||||||
|
validateRequest(projectId, highlight, request, requestsDirectory);
|
||||||
log.info("event=highlight_asset_generation_requested project_id={} highlight_id={} type={} asset_key={} target={} blocking={} duration_seconds={} request={}",
|
log.info("event=highlight_asset_generation_requested project_id={} highlight_id={} type={} asset_key={} target={} blocking={} duration_seconds={} request={}",
|
||||||
projectId, highlight.highlightId(), request.type(), request.assetKey(), request.targetPath(),
|
projectId, highlight.highlightId(), request.type(), request.assetKey(), request.targetPath(),
|
||||||
request.blocking(), request.durationSeconds(), request.requestPath());
|
request.blocking(), request.durationSeconds(), request.requestPath());
|
||||||
|
|
@ -71,6 +76,54 @@ public class HighlightLocalAssetWorker {
|
||||||
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
|
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void validateRequest(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
HighlightAssetRequest request, Path requestsDirectory) {
|
||||||
|
if (request == null || !projectId.equals(request.projectId())
|
||||||
|
|| !highlight.highlightId().equals(request.highlightId())) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset request identity does not match the active project");
|
||||||
|
}
|
||||||
|
if (!SUPPORTED_TYPES.contains(request.type()) || request.assetKey() == null
|
||||||
|
|| !SAFE_KEY.matcher(request.assetKey()).matches() || request.assetKey().contains("..")) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset request type or asset key is invalid");
|
||||||
|
}
|
||||||
|
if (!Double.isFinite(request.durationSeconds()) || request.durationSeconds() <= 0
|
||||||
|
|| request.durationSeconds() > Math.max(1.0, highlight.targetDurationSeconds()) + 0.001) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset request duration is outside the highlight bounds");
|
||||||
|
}
|
||||||
|
if (request.notes() == null || request.notes().isBlank()) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset request notes are required");
|
||||||
|
}
|
||||||
|
|
||||||
|
Path assetsDirectory = requestsDirectory.getParent().toAbsolutePath().normalize();
|
||||||
|
Path expectedTarget = switch (request.type()) {
|
||||||
|
case "music" -> assetsDirectory.resolve("music/music.wav");
|
||||||
|
case "voiceover" -> assetsDirectory.resolve("voiceover").resolve(request.assetKey() + ".wav");
|
||||||
|
case "sfx" -> assetsDirectory.resolve("sfx").resolve(request.assetKey() + ".wav");
|
||||||
|
default -> throw new IllegalArgumentException("Unsupported highlight asset request type");
|
||||||
|
};
|
||||||
|
Path actualTarget = Path.of(request.targetPath()).toAbsolutePath().normalize();
|
||||||
|
Path expectedRequest = requestsDirectory.resolve(request.type() + "-" + request.assetKey() + ".md")
|
||||||
|
.toAbsolutePath().normalize();
|
||||||
|
Path actualRequest = Path.of(request.requestPath()).toAbsolutePath().normalize();
|
||||||
|
if (!actualTarget.equals(expectedTarget) || !actualRequest.equals(expectedRequest)) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset request paths do not match the project contract");
|
||||||
|
}
|
||||||
|
rejectSymlinkPath(store.projectDirectory(projectId).toAbsolutePath().normalize(), expectedTarget);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void rejectSymlinkPath(Path projectDirectory, Path target) {
|
||||||
|
if (!target.startsWith(projectDirectory)) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset target escapes the project directory");
|
||||||
|
}
|
||||||
|
Path cursor = projectDirectory;
|
||||||
|
for (Path part : projectDirectory.relativize(target)) {
|
||||||
|
cursor = cursor.resolve(part);
|
||||||
|
if (Files.isSymbolicLink(cursor)) {
|
||||||
|
throw new IllegalArgumentException("Highlight asset target contains a symbolic link");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private List<HighlightAssetRequest> readRequests(Path directory) {
|
private List<HighlightAssetRequest> readRequests(Path directory) {
|
||||||
try {
|
try {
|
||||||
try (var files = Files.list(directory)) {
|
try (var files = Files.list(directory)) {
|
||||||
|
|
@ -143,8 +196,7 @@ public class HighlightLocalAssetWorker {
|
||||||
|
|
||||||
private void copy(Path source, Path target) {
|
private void copy(Path source, Path target) {
|
||||||
try {
|
try {
|
||||||
Files.createDirectories(target.getParent());
|
AssetLicensePolicy.copy(source, target);
|
||||||
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
|
||||||
} catch (IOException ex) {
|
} catch (IOException ex) {
|
||||||
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
|
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,7 @@ public class HighlightSourceScheduler {
|
||||||
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
private final HighlightProjectStore store;
|
private final HighlightProjectStore store;
|
||||||
private final HighlightSourceAnalyzer analyzer;
|
private final HighlightSourceAnalyzer analyzer;
|
||||||
|
private final HighlightCandidateGenerator candidateGenerator;
|
||||||
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
||||||
private final Clock clock;
|
private final Clock clock;
|
||||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||||
|
|
@ -49,21 +50,24 @@ public class HighlightSourceScheduler {
|
||||||
VideoClippingProperties properties,
|
VideoClippingProperties properties,
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
HighlightSourceAnalyzer analyzer,
|
HighlightSourceAnalyzer analyzer,
|
||||||
|
HighlightCandidateGenerator candidateGenerator,
|
||||||
HighlightDirectorPromptGenerator directorPromptGenerator
|
HighlightDirectorPromptGenerator directorPromptGenerator
|
||||||
) {
|
) {
|
||||||
this(properties, store, analyzer, directorPromptGenerator, Clock.systemUTC());
|
this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, Clock.systemUTC());
|
||||||
}
|
}
|
||||||
|
|
||||||
HighlightSourceScheduler(
|
HighlightSourceScheduler(
|
||||||
VideoClippingProperties properties,
|
VideoClippingProperties properties,
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
HighlightSourceAnalyzer analyzer,
|
HighlightSourceAnalyzer analyzer,
|
||||||
|
HighlightCandidateGenerator candidateGenerator,
|
||||||
HighlightDirectorPromptGenerator directorPromptGenerator,
|
HighlightDirectorPromptGenerator directorPromptGenerator,
|
||||||
Clock clock
|
Clock clock
|
||||||
) {
|
) {
|
||||||
this.properties = properties.getEditing().getHighlightScheduler();
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
this.store = store;
|
this.store = store;
|
||||||
this.analyzer = analyzer;
|
this.analyzer = analyzer;
|
||||||
|
this.candidateGenerator = candidateGenerator;
|
||||||
this.directorPromptGenerator = directorPromptGenerator;
|
this.directorPromptGenerator = directorPromptGenerator;
|
||||||
this.clock = clock;
|
this.clock = clock;
|
||||||
}
|
}
|
||||||
|
|
@ -162,6 +166,11 @@ public class HighlightSourceScheduler {
|
||||||
+ "project_source_directory={}",
|
+ "project_source_directory={}",
|
||||||
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
|
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
|
||||||
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
|
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
|
||||||
|
CinematicHighlightAnalysis cinematic = candidateGenerator.generate(projectId, analysis);
|
||||||
|
log.info("event=highlight_flow_candidates_completed scan_id={} project_id={} count={} category={} "
|
||||||
|
+ "category_confidence={}",
|
||||||
|
scanId, projectId, cinematic.candidates().size(), cinematic.category(),
|
||||||
|
cinematic.categoryConfidence());
|
||||||
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
|
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
|
||||||
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
|
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
|
||||||
+ "processed_directory={}",
|
+ "processed_directory={}",
|
||||||
|
|
|
||||||
|
|
@ -61,20 +61,24 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
.filter(cue -> "music".equals(cue.type()))
|
.filter(cue -> "music".equals(cue.type()))
|
||||||
.findFirst();
|
.findFirst();
|
||||||
if (musicCue.isPresent()) {
|
if (musicCue.isPresent()) {
|
||||||
items.add(materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory, requestsDirectory));
|
AssetGenerationItem item = materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory,
|
||||||
|
requestsDirectory);
|
||||||
|
items.add(item);
|
||||||
|
readyForRender &= item.reused();
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!plan.voiceover().isEmpty()) {
|
if (!plan.voiceover().isEmpty()) {
|
||||||
items.add(materializeVoiceover(projectId, plan, projectAudioDirectory, requestsDirectory));
|
AssetGenerationItem item = materializeVoiceover(projectId, plan, projectAudioDirectory,
|
||||||
|
requestsDirectory);
|
||||||
|
items.add(item);
|
||||||
|
readyForRender &= item.reused();
|
||||||
}
|
}
|
||||||
|
|
||||||
for (AudioCue cue : plan.audioCues()) {
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
if ("sfx".equals(cue.type())) {
|
if ("sfx".equals(cue.type())) {
|
||||||
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
|
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
|
||||||
items.add(item);
|
items.add(item);
|
||||||
if (!item.reused()) {
|
readyForRender &= item.reused();
|
||||||
readyForRender = false;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -94,7 +98,8 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
"music", cacheKey + ".wav");
|
"music", cacheKey + ".wav");
|
||||||
Path projectTarget = projectAudioDirectory.resolve("music.wav");
|
Path projectTarget = projectAudioDirectory.resolve("music.wav");
|
||||||
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||||
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true,
|
||||||
|
List.of());
|
||||||
}
|
}
|
||||||
|
|
||||||
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
|
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
|
||||||
|
|
@ -105,7 +110,7 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
|
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
|
||||||
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
|
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
|
||||||
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
|
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
|
||||||
"voiceover lines=" + plan.voiceover().size(), requestsDirectory, plan.targetDurationSeconds(), false);
|
voiceoverText, requestsDirectory, plan.targetDurationSeconds(), true, plan.voiceover());
|
||||||
}
|
}
|
||||||
|
|
||||||
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
|
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
|
||||||
|
|
@ -117,15 +122,17 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
"sfx", cacheKey + ".wav");
|
"sfx", cacheKey + ".wav");
|
||||||
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
|
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
|
||||||
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||||
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true,
|
||||||
|
List.of());
|
||||||
}
|
}
|
||||||
|
|
||||||
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
|
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
|
||||||
Path cachePath, Path projectTarget, String notes,
|
Path cachePath, Path projectTarget, String notes,
|
||||||
Path requestsDirectory, double durationSeconds, boolean blocking) {
|
Path requestsDirectory, double durationSeconds, boolean blocking,
|
||||||
|
List<VoiceoverLine> voiceoverLines) {
|
||||||
try {
|
try {
|
||||||
createDirectory(cachePath.getParent());
|
createDirectory(cachePath.getParent());
|
||||||
if (Files.isRegularFile(cachePath)) {
|
if (AssetLicensePolicy.isLicensed(cachePath)) {
|
||||||
copy(cachePath, projectTarget);
|
copy(cachePath, projectTarget);
|
||||||
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||||
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
|
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
|
||||||
|
|
@ -137,8 +144,6 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
if (selected.isPresent()) {
|
if (selected.isPresent()) {
|
||||||
copy(Path.of(selected.get().path()), cachePath);
|
copy(Path.of(selected.get().path()), cachePath);
|
||||||
copy(Path.of(selected.get().path()), projectTarget);
|
copy(Path.of(selected.get().path()), projectTarget);
|
||||||
copyIfMissing(Path.of(selected.get().path()).resolveSibling(Path.of(selected.get().path()).getFileName()
|
|
||||||
+ ".license.txt"), projectTarget.resolveSibling(projectTarget.getFileName() + ".license.txt"));
|
|
||||||
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||||
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
|
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
|
||||||
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
|
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
|
||||||
|
|
@ -153,7 +158,8 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
|
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
|
||||||
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
||||||
requestFile.toString(), notes));
|
requestFile.toString(), notes));
|
||||||
Optional<Path> generated = generate(type, projectId, notes, durationSeconds, projectTarget);
|
Optional<Path> generated = generate(type, projectId, notes, durationSeconds, projectTarget,
|
||||||
|
voiceoverLines);
|
||||||
if (generated.isPresent()) {
|
if (generated.isPresent()) {
|
||||||
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={} strategy=local-worker",
|
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={} strategy=local-worker",
|
||||||
projectId, type, assetKey, cacheKey, generated.get(), projectTarget);
|
projectId, type, assetKey, cacheKey, generated.get(), projectTarget);
|
||||||
|
|
@ -175,7 +181,8 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
|
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
|
||||||
}
|
}
|
||||||
if ("voiceover".equals(type)) {
|
if ("voiceover".equals(type)) {
|
||||||
return provider.list(EditAssetType.VOICEOVER, null).stream().findFirst();
|
return provider.resolve(new EditAssetRequest(EditAssetType.VOICEOVER, assetKey, null,
|
||||||
|
durationSeconds, notes));
|
||||||
}
|
}
|
||||||
if ("sfx".equals(type)) {
|
if ("sfx".equals(type)) {
|
||||||
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
|
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
|
||||||
|
|
@ -189,15 +196,12 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
}
|
}
|
||||||
|
|
||||||
private Optional<Path> generate(String type, String projectId, String notes, double durationSeconds,
|
private Optional<Path> generate(String type, String projectId, String notes, double durationSeconds,
|
||||||
Path target) {
|
Path target, List<VoiceoverLine> voiceoverLines) {
|
||||||
if ("music".equals(type)) {
|
if ("music".equals(type)) {
|
||||||
return synthesizer.synthesizeMusic(projectId, notes, durationSeconds, target);
|
return synthesizer.synthesizeMusic(projectId, notes, durationSeconds, target);
|
||||||
}
|
}
|
||||||
if ("voiceover".equals(type)) {
|
if ("voiceover".equals(type)) {
|
||||||
List<VoiceoverLine> lines = notes == null || notes.isBlank()
|
return synthesizer.synthesizeVoiceover(projectId, voiceoverLines, target);
|
||||||
? List.of()
|
|
||||||
: List.of(new VoiceoverLine(notes, 0.0, Math.max(1.0, durationSeconds), "cinematic_narration"));
|
|
||||||
return synthesizer.synthesizeVoiceover(projectId, lines, target);
|
|
||||||
}
|
}
|
||||||
if ("sfx".equals(type)) {
|
if ("sfx".equals(type)) {
|
||||||
return synthesizer.synthesizeSfx(projectId, notes, durationSeconds, target);
|
return synthesizer.synthesizeSfx(projectId, notes, durationSeconds, target);
|
||||||
|
|
@ -256,14 +260,7 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
if (source.normalize().equals(target.normalize())) {
|
if (source.normalize().equals(target.normalize())) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
createDirectory(target.getParent());
|
AssetLicensePolicy.copy(source, target);
|
||||||
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
|
|
||||||
}
|
|
||||||
|
|
||||||
private void copyIfMissing(Path source, Path target) throws IOException {
|
|
||||||
if (Files.isRegularFile(source) && !Files.exists(target)) {
|
|
||||||
copy(source, target);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private void writeJson(Path path, Object value) {
|
private void writeJson(Path path, Object value) {
|
||||||
|
|
|
||||||
|
|
@ -8,15 +8,12 @@ import org.springframework.boot.ApplicationRunner;
|
||||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
import org.springframework.stereotype.Component;
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
import java.io.BufferedReader;
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
import java.io.InputStreamReader;
|
|
||||||
import java.nio.charset.StandardCharsets;
|
import java.nio.charset.StandardCharsets;
|
||||||
import java.nio.file.Files;
|
import java.nio.file.Files;
|
||||||
import java.nio.file.Path;
|
import java.nio.file.Path;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
|
||||||
|
|
||||||
@Component
|
@Component
|
||||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
|
@ -42,16 +39,13 @@ public class LocalAssetRuntimeVerifier implements ApplicationRunner {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
bootstrapEnvironment();
|
|
||||||
|
|
||||||
RuntimeReadiness readiness = verifyRuntime();
|
RuntimeReadiness readiness = verifyRuntime();
|
||||||
if (!readiness.ready()) {
|
if (!readiness.ready()) {
|
||||||
log.error("event=local_asset_runtime_check_failed missing={} voiceover_ready={} music_ready={} sfx_ready={}",
|
log.error("event=local_asset_runtime_check_failed missing={} voiceover_ready={} music_ready={} sfx_ready={}",
|
||||||
readiness.missing(), readiness.voiceoverReady(), readiness.musicReady(),
|
readiness.missing(), readiness.voiceoverReady(), readiness.musicReady(),
|
||||||
readiness.sfxReady());
|
readiness.sfxReady());
|
||||||
if (worker.isStrictRuntime()) {
|
if (worker.isStrictRuntime()) {
|
||||||
log.warn("event=local_asset_runtime_degraded strict_runtime_enabled=true missing={}",
|
throw new IllegalStateException("Strict local asset runtime is not ready: " + readiness.missing());
|
||||||
readiness.missing());
|
|
||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
@ -61,62 +55,13 @@ public class LocalAssetRuntimeVerifier implements ApplicationRunner {
|
||||||
worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
|
worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void bootstrapEnvironment() {
|
|
||||||
Path bootstrapScript = Path.of(worker.getBootstrapScript()).toAbsolutePath().normalize();
|
|
||||||
if (!Files.isRegularFile(bootstrapScript)) {
|
|
||||||
throw new IllegalStateException("Local asset bootstrap script does not exist: " + bootstrapScript);
|
|
||||||
}
|
|
||||||
log.info("event=local_asset_bootstrap_started script={} bootstrap_mode=auto preload_only=true python={} "
|
|
||||||
+ "piper_binary={} piper_model_path={}",
|
|
||||||
bootstrapScript, worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
|
|
||||||
ProcessBuilder builder = new ProcessBuilder("bash", bootstrapScript.toString())
|
|
||||||
.redirectErrorStream(true);
|
|
||||||
Map<String, String> environment = builder.environment();
|
|
||||||
environment.put("LOCAL_ASSET_BOOTSTRAP_MODE", "auto");
|
|
||||||
environment.put("LOCAL_ASSET_PRELOAD_ONLY", "true");
|
|
||||||
environment.put("LOCAL_ASSET_PIPER_BINARY", worker.getPiperBinary());
|
|
||||||
if (worker.getPiperModelPath() != null && !worker.getPiperModelPath().isBlank()) {
|
|
||||||
environment.put("LOCAL_ASSET_PIPER_MODEL_PATH", worker.getPiperModelPath());
|
|
||||||
}
|
|
||||||
long startedAt = System.nanoTime();
|
|
||||||
try {
|
|
||||||
Process process = builder.start();
|
|
||||||
try (BufferedReader reader = new BufferedReader(
|
|
||||||
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {
|
|
||||||
String line;
|
|
||||||
while ((line = reader.readLine()) != null) {
|
|
||||||
log.info("event=local_asset_bootstrap_output script={} message={}", bootstrapScript, line);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
int exitCode = process.waitFor();
|
|
||||||
if (exitCode != 0) {
|
|
||||||
log.error("event=local_asset_bootstrap_failed script={} exit_code={} reason=non_blocking",
|
|
||||||
bootstrapScript, exitCode);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
log.info("event=local_asset_bootstrap_completed script={} elapsed_ms={}", bootstrapScript,
|
|
||||||
elapsedMillis(startedAt));
|
|
||||||
} catch (IOException ex) {
|
|
||||||
log.error("event=local_asset_bootstrap_failed script={} error_type={} message={}", bootstrapScript,
|
|
||||||
ex.getClass().getSimpleName(), ex.getMessage());
|
|
||||||
throw new IllegalStateException("Unable to bootstrap local asset runtime", ex);
|
|
||||||
} catch (InterruptedException ex) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
log.error("event=local_asset_bootstrap_failed script={} error_type={} message={}", bootstrapScript,
|
|
||||||
ex.getClass().getSimpleName(), ex.getMessage());
|
|
||||||
throw new IllegalStateException("Interrupted while bootstrapping local asset runtime", ex);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private RuntimeReadiness verifyRuntime() {
|
private RuntimeReadiness verifyRuntime() {
|
||||||
List<String> missing = new ArrayList<>();
|
List<String> missing = new ArrayList<>();
|
||||||
boolean pythonReady = isPythonAudioStackReady();
|
boolean pythonPresent = Files.isRegularFile(Path.of(worker.getPythonBinary()));
|
||||||
if (!Files.isRegularFile(Path.of(worker.getPythonBinary()))) {
|
boolean audioStackReady = pythonPresent && isPythonAudioStackReady();
|
||||||
|
if (!pythonPresent) {
|
||||||
missing.add("python_binary:" + worker.getPythonBinary());
|
missing.add("python_binary:" + worker.getPythonBinary());
|
||||||
}
|
}
|
||||||
if (!Files.isRegularFile(Path.of(worker.getBootstrapScript()))) {
|
|
||||||
missing.add("bootstrap_script:" + worker.getBootstrapScript());
|
|
||||||
}
|
|
||||||
if (!Files.isRegularFile(Path.of(worker.getScript()))) {
|
if (!Files.isRegularFile(Path.of(worker.getScript()))) {
|
||||||
missing.add("worker_script:" + worker.getScript());
|
missing.add("worker_script:" + worker.getScript());
|
||||||
}
|
}
|
||||||
|
|
@ -127,15 +72,34 @@ public class LocalAssetRuntimeVerifier implements ApplicationRunner {
|
||||||
missing.add("piper_model_path:missing");
|
missing.add("piper_model_path:missing");
|
||||||
} else if (!Files.isRegularFile(Path.of(worker.getPiperModelPath()))) {
|
} else if (!Files.isRegularFile(Path.of(worker.getPiperModelPath()))) {
|
||||||
missing.add("piper_model_path:" + worker.getPiperModelPath());
|
missing.add("piper_model_path:" + worker.getPiperModelPath());
|
||||||
|
} else if (AssetLicensePolicy.read(Path.of(worker.getPiperModelPath())).isEmpty()) {
|
||||||
|
missing.add("piper_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getPiperModelPath())));
|
||||||
}
|
}
|
||||||
if (!pythonReady) {
|
boolean musicModelReady = isLocalModelPath(worker.getMusicModel());
|
||||||
|
if (!musicModelReady) {
|
||||||
|
missing.add("music_model_path:" + worker.getMusicModel());
|
||||||
|
} else if (AssetLicensePolicy.read(Path.of(worker.getMusicModel())).isEmpty()) {
|
||||||
|
missing.add("music_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getMusicModel())));
|
||||||
|
musicModelReady = false;
|
||||||
|
}
|
||||||
|
boolean sfxModelReady = isLocalModelPath(worker.getSfxModel());
|
||||||
|
if (!sfxModelReady) {
|
||||||
|
missing.add("sfx_model_path:" + worker.getSfxModel());
|
||||||
|
} else if (AssetLicensePolicy.read(Path.of(worker.getSfxModel())).isEmpty()) {
|
||||||
|
missing.add("sfx_model_license:" + AssetLicensePolicy.sidecar(Path.of(worker.getSfxModel())));
|
||||||
|
sfxModelReady = false;
|
||||||
|
}
|
||||||
|
if (!audioStackReady) {
|
||||||
missing.add("python_audio_stack:torch,audiocraft,soundfile,numpy");
|
missing.add("python_audio_stack:torch,audiocraft,soundfile,numpy");
|
||||||
}
|
}
|
||||||
|
boolean voiceoverReady = isVoiceoverReady(pythonPresent);
|
||||||
|
boolean musicReady = audioStackReady && musicModelReady;
|
||||||
|
boolean sfxReady = audioStackReady && sfxModelReady;
|
||||||
return new RuntimeReadiness(
|
return new RuntimeReadiness(
|
||||||
missing.isEmpty(),
|
voiceoverReady && musicReady && sfxReady,
|
||||||
isVoiceoverReady(pythonReady),
|
voiceoverReady,
|
||||||
pythonReady,
|
musicReady,
|
||||||
pythonReady,
|
sfxReady,
|
||||||
List.copyOf(missing)
|
List.copyOf(missing)
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
@ -147,7 +111,17 @@ public class LocalAssetRuntimeVerifier implements ApplicationRunner {
|
||||||
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
|
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
return isExecutablePresent(worker.getPiperBinary());
|
return Files.isRegularFile(Path.of(worker.getPiperModelPath()))
|
||||||
|
&& AssetLicensePolicy.read(Path.of(worker.getPiperModelPath())).isPresent()
|
||||||
|
&& isExecutablePresent(worker.getPiperBinary());
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isLocalModelPath(String value) {
|
||||||
|
if (value == null || value.isBlank()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
Path path = Path.of(value).normalize();
|
||||||
|
return Files.isRegularFile(path) || Files.isDirectory(path);
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean isPythonAudioStackReady() {
|
private boolean isPythonAudioStackReady() {
|
||||||
|
|
@ -187,10 +161,6 @@ public class LocalAssetRuntimeVerifier implements ApplicationRunner {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private long elapsedMillis(long startedAt) {
|
|
||||||
return (System.nanoTime() - startedAt) / 1_000_000;
|
|
||||||
}
|
|
||||||
|
|
||||||
private record RuntimeReadiness(
|
private record RuntimeReadiness(
|
||||||
boolean ready,
|
boolean ready,
|
||||||
boolean voiceoverReady,
|
boolean voiceoverReady,
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,6 @@ import java.io.IOException;
|
||||||
import java.nio.charset.StandardCharsets;
|
import java.nio.charset.StandardCharsets;
|
||||||
import java.nio.file.Files;
|
import java.nio.file.Files;
|
||||||
import java.nio.file.Path;
|
import java.nio.file.Path;
|
||||||
import java.nio.file.Files;
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Locale;
|
import java.util.Locale;
|
||||||
|
|
@ -45,28 +44,35 @@ public class LocalAssetSynthesizer {
|
||||||
}
|
}
|
||||||
String prompt = String.join(System.lineSeparator(),
|
String prompt = String.join(System.lineSeparator(),
|
||||||
lines.stream().map(VoiceoverLine::text).toList());
|
lines.stream().map(VoiceoverLine::text).toList());
|
||||||
return synthesize(projectId, "voiceover", prompt, target, Math.max(1.0, lines.size() * 2.5), true);
|
return synthesize(projectId, "voiceover", prompt, target, Math.max(1.0, lines.size() * 2.5));
|
||||||
}
|
}
|
||||||
|
|
||||||
public Optional<Path> synthesizeMusic(String projectId, String prompt, double durationSeconds, Path target) {
|
public Optional<Path> synthesizeMusic(String projectId, String prompt, double durationSeconds, Path target) {
|
||||||
return synthesize(projectId, "music", prompt, target, durationSeconds, false);
|
return synthesize(projectId, "music", prompt, target, durationSeconds);
|
||||||
}
|
}
|
||||||
|
|
||||||
public Optional<Path> synthesizeSfx(String projectId, String prompt, double durationSeconds, Path target) {
|
public Optional<Path> synthesizeSfx(String projectId, String prompt, double durationSeconds, Path target) {
|
||||||
return synthesize(projectId, "sfx", prompt, target, durationSeconds, false);
|
return synthesize(projectId, "sfx", prompt, target, durationSeconds);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Optional<Path> synthesize(String projectId, String kind, String prompt, Path target,
|
private Optional<Path> synthesize(String projectId, String kind, String prompt, Path target,
|
||||||
double durationSeconds, boolean allowSpeechFallback) {
|
double durationSeconds) {
|
||||||
if (prompt == null || prompt.isBlank()) {
|
if (prompt == null || prompt.isBlank()) {
|
||||||
return Optional.empty();
|
return Optional.empty();
|
||||||
}
|
}
|
||||||
|
Path promptFile = null;
|
||||||
try {
|
try {
|
||||||
String model = selectedModel(kind);
|
String model = selectedModel(kind);
|
||||||
|
if (model == null || model.isBlank() || !AssetLicensePolicy.isLicensed(Path.of(model))) {
|
||||||
|
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} "
|
||||||
|
+ "reason=model_license_missing",
|
||||||
|
projectId, kind, model, target);
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=local-worker",
|
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=local-worker",
|
||||||
projectId, kind, model, target, durationSeconds);
|
projectId, kind, model, target, durationSeconds);
|
||||||
Files.createDirectories(target.getParent());
|
Files.createDirectories(target.getParent());
|
||||||
Path promptFile = Files.createTempFile(target.getParent(),
|
promptFile = Files.createTempFile(target.getParent(),
|
||||||
sanitize(kind) + "-prompt-", ".txt");
|
sanitize(kind) + "-prompt-", ".txt");
|
||||||
Files.writeString(promptFile, prompt, StandardCharsets.UTF_8);
|
Files.writeString(promptFile, prompt, StandardCharsets.UTF_8);
|
||||||
List<String> command = new ArrayList<>();
|
List<String> command = new ArrayList<>();
|
||||||
|
|
@ -92,63 +98,30 @@ public class LocalAssetSynthesizer {
|
||||||
command.add(worker.getPiperBinary());
|
command.add(worker.getPiperBinary());
|
||||||
}
|
}
|
||||||
|
|
||||||
if (runWorker(command, target)) {
|
if (runWorker(command, target) && hasAudibleSignal(target)) {
|
||||||
|
AssetLicensePolicy.recordGeneratedAsset(Path.of(model), target);
|
||||||
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=local-worker",
|
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=local-worker",
|
||||||
projectId, kind, model, target);
|
projectId, kind, model, target);
|
||||||
return Optional.of(target);
|
return Optional.of(target);
|
||||||
}
|
}
|
||||||
|
deleteInvalidOutput(target);
|
||||||
if ("voiceover".equals(kind) && allowSpeechFallback) {
|
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=local_model_failed_or_inaudible",
|
||||||
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=speech-fallback",
|
|
||||||
projectId, kind, selectedVoiceoverFallback(), target, durationSeconds);
|
|
||||||
if (runSpeechFallback(prompt, target)) {
|
|
||||||
if (!hasAudibleSignal(target)) {
|
|
||||||
log.warn("event=local_asset_voiceover_silence_detected project_id={} target={} strategy=speech-fallback",
|
|
||||||
projectId, target);
|
|
||||||
} else {
|
|
||||||
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=speech-fallback",
|
|
||||||
projectId, kind, selectedVoiceoverFallback(), target);
|
|
||||||
return Optional.of(target);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if ("voiceover".equals(kind)) {
|
|
||||||
log.error("event=local_asset_voiceover_unavailable project_id={} target={} reason=no_audible_speech_generated",
|
|
||||||
projectId, target);
|
|
||||||
return Optional.empty();
|
|
||||||
}
|
|
||||||
|
|
||||||
if ("music".equals(kind)) {
|
|
||||||
if (worker.isStrictRuntime()) {
|
|
||||||
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
|
|
||||||
projectId, kind, model, target);
|
projectId, kind, model, target);
|
||||||
return Optional.empty();
|
return Optional.empty();
|
||||||
}
|
|
||||||
return fallbackTone(target, durationSeconds, 110.0, 0.02, projectId, kind);
|
|
||||||
}
|
|
||||||
if ("sfx".equals(kind)) {
|
|
||||||
if (worker.isStrictRuntime()) {
|
|
||||||
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
|
|
||||||
projectId, kind, model, target);
|
|
||||||
return Optional.empty();
|
|
||||||
}
|
|
||||||
return fallbackTone(target, Math.max(0.5, Math.min(1.0, durationSeconds)), 880.0, 0.12,
|
|
||||||
projectId, kind);
|
|
||||||
}
|
|
||||||
if (worker.isStrictRuntime()) {
|
|
||||||
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
|
|
||||||
projectId, kind, model, target);
|
|
||||||
return Optional.empty();
|
|
||||||
}
|
|
||||||
writeSilence(target, Math.max(1.0, durationSeconds));
|
|
||||||
log.warn("event=local_asset_generation_completed project_id={} type={} model=silence target={} strategy=silence",
|
|
||||||
projectId, kind, target);
|
|
||||||
return Optional.of(target);
|
|
||||||
} catch (IOException ex) {
|
} catch (IOException ex) {
|
||||||
|
deleteInvalidOutput(target);
|
||||||
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
|
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
|
||||||
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
|
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
return Optional.empty();
|
return Optional.empty();
|
||||||
|
} finally {
|
||||||
|
if (promptFile != null) {
|
||||||
|
try {
|
||||||
|
Files.deleteIfExists(promptFile);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
log.warn("event=local_asset_prompt_cleanup_failed path={} message={}", promptFile,
|
||||||
|
ex.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -175,83 +148,11 @@ public class LocalAssetSynthesizer {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean runSpeechFallback(String text, Path target) {
|
private void deleteInvalidOutput(Path target) {
|
||||||
List<List<String>> candidates = List.of(
|
|
||||||
List.of("say", "-o", target.toString(), text),
|
|
||||||
List.of("espeak", "-w", target.toString(), text)
|
|
||||||
);
|
|
||||||
for (List<String> command : candidates) {
|
|
||||||
if (command.isEmpty() || command.get(0).isBlank()) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (!isAvailable(command.get(0))) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
try {
|
try {
|
||||||
log.info("event=local_asset_voiceover_fallback_started tool={} target={}", command.get(0), target);
|
Files.deleteIfExists(target);
|
||||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
} catch (IOException ex) {
|
||||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
log.warn("event=local_asset_invalid_output_cleanup_failed target={} message={}", target, ex.getMessage());
|
||||||
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
|
|
||||||
log.info("event=local_asset_voiceover_fallback_completed tool={} target={}",
|
|
||||||
command.get(0), target);
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
if (!output.isBlank()) {
|
|
||||||
log.warn("event=local_asset_voiceover_fallback_failed target={} output={}", target, output);
|
|
||||||
}
|
|
||||||
} catch (IOException | InterruptedException ex) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
log.warn("event=local_asset_voiceover_fallback_failed target={} error_type={} message={}",
|
|
||||||
target, ex.getClass().getSimpleName(), ex.getMessage());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
private Optional<Path> fallbackTone(Path target, double durationSeconds, double frequency, double volume,
|
|
||||||
String projectId, String kind) {
|
|
||||||
try {
|
|
||||||
log.info("event=local_asset_generation_started project_id={} type={} model=ffmpeg-sine target={} duration_seconds={} strategy=fallback-tone",
|
|
||||||
projectId, kind, target, durationSeconds);
|
|
||||||
Files.createDirectories(target.getParent());
|
|
||||||
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
|
||||||
"-f", "lavfi", "-i", "sine=frequency=" + (int) frequency + ":sample_rate=48000",
|
|
||||||
"-t", Double.toString(durationSeconds), "-af", "volume=" + volume, "-c:a", "pcm_s16le",
|
|
||||||
target.toString());
|
|
||||||
Process process = processExecutor.execute(command);
|
|
||||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
|
||||||
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
|
|
||||||
log.info("event=local_asset_generation_completed project_id={} type={} model=ffmpeg-sine target={} strategy=fallback-tone",
|
|
||||||
projectId, kind, target);
|
|
||||||
return Optional.of(target);
|
|
||||||
}
|
|
||||||
if (!output.isBlank()) {
|
|
||||||
log.warn("event=local_asset_generation_failed project_id={} type={} target={} output={}",
|
|
||||||
projectId, kind, target, output);
|
|
||||||
}
|
|
||||||
return Optional.empty();
|
|
||||||
} catch (IOException | InterruptedException ex) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
|
|
||||||
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
|
|
||||||
return Optional.empty();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private void writeSilence(Path target, double durationSeconds) {
|
|
||||||
try {
|
|
||||||
log.info("event=local_asset_generation_started type=voiceover model=silence target={} duration_seconds={} strategy=silence",
|
|
||||||
target, durationSeconds);
|
|
||||||
Files.createDirectories(target.getParent());
|
|
||||||
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
|
||||||
"-f", "lavfi", "-i", "anullsrc=channel_layout=mono:sample_rate=24000",
|
|
||||||
"-t", Double.toString(durationSeconds), "-c:a", "pcm_s16le", target.toString());
|
|
||||||
Process process = processExecutor.execute(command);
|
|
||||||
process.waitFor();
|
|
||||||
log.info("event=local_asset_generation_completed type=voiceover model=silence target={} strategy=silence",
|
|
||||||
target);
|
|
||||||
} catch (IOException | InterruptedException ex) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -263,15 +164,11 @@ public class LocalAssetSynthesizer {
|
||||||
return worker.getSfxModel();
|
return worker.getSfxModel();
|
||||||
}
|
}
|
||||||
if ("voiceover".equals(kind)) {
|
if ("voiceover".equals(kind)) {
|
||||||
return worker.getPiperModelPath().isBlank() ? selectedVoiceoverFallback() : worker.getPiperModelPath();
|
return worker.getPiperModelPath();
|
||||||
}
|
}
|
||||||
return "unknown";
|
return "unknown";
|
||||||
}
|
}
|
||||||
|
|
||||||
private String selectedVoiceoverFallback() {
|
|
||||||
return "piper-fallback";
|
|
||||||
}
|
|
||||||
|
|
||||||
private boolean hasAudibleSignal(Path audioFile) {
|
private boolean hasAudibleSignal(Path audioFile) {
|
||||||
if (!Files.isRegularFile(audioFile)) {
|
if (!Files.isRegularFile(audioFile)) {
|
||||||
return false;
|
return false;
|
||||||
|
|
@ -294,17 +191,6 @@ public class LocalAssetSynthesizer {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean isAvailable(String executable) {
|
|
||||||
try {
|
|
||||||
Process process = new ProcessBuilder(executable, "--help").redirectErrorStream(true).start();
|
|
||||||
process.waitFor();
|
|
||||||
return true;
|
|
||||||
} catch (IOException | InterruptedException ex) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private String sanitize(String value) {
|
private String sanitize(String value) {
|
||||||
return value == null ? "asset" : value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9._-]+", "-");
|
return value == null ? "asset" : value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9._-]+", "-");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -37,7 +37,7 @@ public class LocalEditAssetLibrary implements EditAssetLibrary {
|
||||||
if (request.category() != null && request.category().equals(asset.category())) {
|
if (request.category() != null && request.category().equals(asset.category())) {
|
||||||
score += 100;
|
score += 100;
|
||||||
}
|
}
|
||||||
if (!"untracked-local-asset".equals(asset.license())) {
|
if (!AssetLicensePolicy.UNTRACKED.equals(asset.license())) {
|
||||||
score += 25;
|
score += 25;
|
||||||
}
|
}
|
||||||
Set<String> assetTags = asset.tags().stream()
|
Set<String> assetTags = asset.tags().stream()
|
||||||
|
|
|
||||||
|
|
@ -36,6 +36,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
|
||||||
}
|
}
|
||||||
return candidateRoots(request.type(), request.category()).stream()
|
return candidateRoots(request.type(), request.category()).stream()
|
||||||
.flatMap(root -> resolveInRoot(root, request).stream())
|
.flatMap(root -> resolveInRoot(root, request).stream())
|
||||||
|
.filter(asset -> AssetLicensePolicy.isLicensed(Path.of(asset.path())))
|
||||||
.findFirst();
|
.findFirst();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -47,6 +48,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
|
||||||
return candidateRoots(type, category).stream()
|
return candidateRoots(type, category).stream()
|
||||||
.filter(Files::isDirectory)
|
.filter(Files::isDirectory)
|
||||||
.flatMap(root -> listRoot(type, root).stream())
|
.flatMap(root -> listRoot(type, root).stream())
|
||||||
|
.filter(asset -> AssetLicensePolicy.isLicensed(Path.of(asset.path())))
|
||||||
.sorted(Comparator.comparing(ResolvedEditAsset::assetKey))
|
.sorted(Comparator.comparing(ResolvedEditAsset::assetKey))
|
||||||
.toList();
|
.toList();
|
||||||
}
|
}
|
||||||
|
|
@ -86,16 +88,7 @@ public class LocalEditAssetProvider implements EditAssetProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
private String license(Path path) {
|
private String license(Path path) {
|
||||||
Path sidecar = path.resolveSibling(path.getFileName() + ".license.txt");
|
return AssetLicensePolicy.read(path).orElse(AssetLicensePolicy.UNTRACKED);
|
||||||
if (!Files.isRegularFile(sidecar)) {
|
|
||||||
return "untracked-local-asset";
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
String license = Files.readString(sidecar).strip();
|
|
||||||
return license.isBlank() ? "untracked-local-asset" : license;
|
|
||||||
} catch (IOException ex) {
|
|
||||||
throw new IllegalStateException("Unable to read edit asset license sidecar: " + sidecar, ex);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<String> tags(Path path) {
|
private List<String> tags(Path path) {
|
||||||
|
|
|
||||||
|
|
@ -91,8 +91,8 @@ video-clipping:
|
||||||
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
|
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
|
||||||
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
|
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
|
||||||
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
|
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
|
||||||
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:true}
|
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:false}
|
||||||
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:false}
|
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:true}
|
||||||
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
||||||
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
||||||
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
|
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
|
||||||
|
|
|
||||||
|
|
@ -92,6 +92,8 @@ class VideoClippingPropertiesTest {
|
||||||
assertThat(properties.getEditing().getHighlightScheduler().getSourceDirectory())
|
assertThat(properties.getEditing().getHighlightScheduler().getSourceDirectory())
|
||||||
.isEqualTo("./input/highlights/source");
|
.isEqualTo("./input/highlights/source");
|
||||||
assertThat(properties.getEditing().getHighlightScheduler().getPollIntervalMs()).isEqualTo(5000);
|
assertThat(properties.getEditing().getHighlightScheduler().getPollIntervalMs()).isEqualTo(5000);
|
||||||
|
assertThat(properties.getEditing().getHighlightScheduler().isRenderEnabled()).isFalse();
|
||||||
|
assertThat(properties.getEditing().getHighlightScheduler().isRequireDirectorApproval()).isTrue();
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
|
|
||||||
|
class AssetLicensePolicyTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void copiesAssetAndPreservesLicenseEvidence() throws Exception {
|
||||||
|
Path source = tempDir.resolve("source.wav");
|
||||||
|
Files.writeString(source, "audio");
|
||||||
|
Files.writeString(AssetLicensePolicy.sidecar(source), "SPDX-License-Identifier: CC0-1.0\n");
|
||||||
|
Path target = tempDir.resolve("project/assets/music.wav");
|
||||||
|
|
||||||
|
AssetLicensePolicy.copy(source, target);
|
||||||
|
|
||||||
|
assertThat(Files.readString(target)).isEqualTo("audio");
|
||||||
|
assertThat(AssetLicensePolicy.read(target))
|
||||||
|
.contains("SPDX-License-Identifier: CC0-1.0");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsMissingAndUntrackedLicenseEvidence() throws Exception {
|
||||||
|
Path source = tempDir.resolve("source.wav");
|
||||||
|
Files.writeString(source, "audio");
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> AssetLicensePolicy.copy(source, tempDir.resolve("target.wav")))
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("license sidecar is required");
|
||||||
|
|
||||||
|
Files.writeString(AssetLicensePolicy.sidecar(source), AssetLicensePolicy.UNTRACKED);
|
||||||
|
|
||||||
|
assertThat(AssetLicensePolicy.isLicensed(source)).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void requiresModelFileBeforeRecordingGeneratedAsset() throws Exception {
|
||||||
|
Path missingModel = tempDir.resolve("missing-model.bin");
|
||||||
|
Files.writeString(AssetLicensePolicy.sidecar(missingModel), "Apache-2.0");
|
||||||
|
|
||||||
|
assertThat(AssetLicensePolicy.isLicensed(missingModel)).isFalse();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -135,12 +135,10 @@ class CinematicEditingIntegrationTest {
|
||||||
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
||||||
"cinematic grade", "integration sequence"))
|
"cinematic grade", "integration sequence"))
|
||||||
.toList();
|
.toList();
|
||||||
List<AudioCue> audioCues = List.of(new AudioCue("music", "cinematic-bed", 0.0,
|
|
||||||
decisions.get(decisions.size() - 1).timelineEndSeconds(), -12.0, "integration score"));
|
|
||||||
List<TextOverlay> overlays = List.of(new TextOverlay("Cinematic highlight", 0.0,
|
List<TextOverlay> overlays = List.of(new TextOverlay("Cinematic highlight", 0.0,
|
||||||
decisions.get(0).timelineEndSeconds(), "lower_center_safe", "fade_in", "integration title"));
|
decisions.get(0).timelineEndSeconds(), "lower_center_safe", "fade_in", "integration title"));
|
||||||
return new EditPlan(projectId, "category-aware-cinematic-highlights",
|
return new EditPlan(projectId, "category-aware-cinematic-highlights",
|
||||||
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, audioCues, List.of(),
|
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(),
|
||||||
overlays,
|
overlays,
|
||||||
"mp4-h264-aac-320p", "Generated integration edit");
|
"mp4-h264-aac-320p", "Generated integration edit");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -2,15 +2,22 @@ package org.example.videoclips.editing;
|
||||||
|
|
||||||
import org.example.videoclips.config.VideoClippingProperties;
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
import org.junit.jupiter.api.Test;
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
import java.nio.file.Path;
|
import java.nio.file.Path;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import static org.assertj.core.api.Assertions.assertThat;
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
import static org.mockito.Mockito.mock;
|
import static org.mockito.Mockito.mock;
|
||||||
|
|
||||||
class FfmpegEditRendererTest {
|
class FfmpegEditRendererTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void buildsNormalizedSegmentCommand() {
|
void buildsNormalizedSegmentCommand() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
|
@ -51,7 +58,7 @@ class FfmpegEditRendererTest {
|
||||||
Path.of("voiceover.wav"), Path.of("final.mp4"));
|
Path.of("voiceover.wav"), Path.of("final.mp4"));
|
||||||
|
|
||||||
assertThat(command).containsSubsequence("-i", "timeline.mp4", "-i", "music.wav", "-i", "voiceover.wav");
|
assertThat(command).containsSubsequence("-i", "timeline.mp4", "-i", "music.wav", "-i", "voiceover.wav");
|
||||||
assertThat(command).contains("[1:a]volume=0.25[music_raw];[2:a]volume=1.0[voice];"
|
assertThat(command).contains("[1:a]volume=-12dB[music_raw];[2:a]volume=1.0[voice];"
|
||||||
+ "[music_raw][voice]sidechaincompress=threshold=0.045:ratio=8.0:attack=20:release=250[music];"
|
+ "[music_raw][voice]sidechaincompress=threshold=0.045:ratio=8.0:attack=20:release=250[music];"
|
||||||
+ "[0:a][music][voice]amix=inputs=3:duration=first:dropout_transition=2,"
|
+ "[0:a][music][voice]amix=inputs=3:duration=first:dropout_transition=2,"
|
||||||
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
||||||
|
|
@ -67,7 +74,7 @@ class FfmpegEditRendererTest {
|
||||||
var command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), null,
|
var command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), null,
|
||||||
Path.of("final.mp4"));
|
Path.of("final.mp4"));
|
||||||
|
|
||||||
assertThat(command).contains("[1:a]volume=0.25[music_raw];[music_raw]anull[music];"
|
assertThat(command).contains("[1:a]volume=-12dB[music_raw];[music_raw]anull[music];"
|
||||||
+ "[0:a][music]amix=inputs=2:duration=first:dropout_transition=2,"
|
+ "[0:a][music]amix=inputs=2:duration=first:dropout_transition=2,"
|
||||||
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
||||||
assertThat(command).noneMatch(argument -> argument.contains("sidechaincompress"));
|
assertThat(command).noneMatch(argument -> argument.contains("sidechaincompress"));
|
||||||
|
|
@ -163,6 +170,45 @@ class FfmpegEditRendererTest {
|
||||||
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
+ "loudnorm=I=-16.0:TP=-1.5:LRA=11.0[a]");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void appliesMusicCueTimingAndGainToFinalMix() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
FfmpegEditRenderer renderer = new FfmpegEditRenderer(properties, mock(EditProjectStore.class),
|
||||||
|
mock(EditProjectService.class), mock(EditPlanValidator.class), command -> null);
|
||||||
|
AudioCue musicCue = new AudioCue("music", "scene-score", 2.0, 12.0, -7.5, "rising tension");
|
||||||
|
|
||||||
|
var command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), musicCue,
|
||||||
|
null, List.of(), Path.of("final.mp4"));
|
||||||
|
|
||||||
|
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
||||||
|
.contains("[1:a]atrim=duration=10.0,asetpts=PTS-STARTPTS,volume=-7.5dB,")
|
||||||
|
.contains("adelay=2000|2000[music_raw]"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsRenderWhenAnyRequestedCreativeAssetIsMissing() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
FfmpegEditRenderer renderer = new FfmpegEditRenderer(properties, mock(EditProjectStore.class),
|
||||||
|
mock(EditProjectService.class), mock(EditPlanValidator.class), command -> null);
|
||||||
|
EditPlan plan = planWithRequestedAudio();
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> renderer.requireRequestedAssets(plan, tempDir))
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("music")
|
||||||
|
.hasMessageContaining("voiceover")
|
||||||
|
.hasMessageContaining("sfx:impact");
|
||||||
|
|
||||||
|
Files.createDirectories(tempDir.resolve("sfx"));
|
||||||
|
Files.writeString(tempDir.resolve("music.wav"), "music");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover.wav"), "voice");
|
||||||
|
Files.writeString(tempDir.resolve("sfx/impact.wav"), "impact");
|
||||||
|
Files.writeString(tempDir.resolve("music.wav.license.txt"), "approved-music");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover.wav.license.txt"), "approved-voice");
|
||||||
|
Files.writeString(tempDir.resolve("sfx/impact.wav.license.txt"), "approved-sfx");
|
||||||
|
|
||||||
|
assertThatCode(() -> renderer.requireRequestedAssets(plan, tempDir)).doesNotThrowAnyException();
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void buildsOverlayCommandWithSafeDrawTextFilters() {
|
void buildsOverlayCommandWithSafeDrawTextFilters() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
|
@ -220,4 +266,15 @@ class FfmpegEditRendererTest {
|
||||||
assertThat(renderer.audioTempoFilter(0.25)).isEqualTo("atempo=0.5,atempo=0.5");
|
assertThat(renderer.audioTempoFilter(0.25)).isEqualTo("atempo=0.5,atempo=0.5");
|
||||||
assertThat(renderer.audioTempoFilter(1.5)).isEqualTo("atempo=1.5");
|
assertThat(renderer.audioTempoFilter(1.5)).isEqualTo("atempo=1.5");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private EditPlan planWithRequestedAudio() {
|
||||||
|
return new EditPlan("project", "cinematic", 10,
|
||||||
|
List.of(new EditDecision("clip", 0, 10, 0, 10, "cut", "cut", 1, "grade", "hero")),
|
||||||
|
List.of(
|
||||||
|
new AudioCue("music", "score", 0, 10, -8, "scene-fit score"),
|
||||||
|
new AudioCue("sfx", "impact", 2, 3, -6, "reveal")
|
||||||
|
),
|
||||||
|
List.of(new VoiceoverLine("Precision.", 1, 3, "controlled")),
|
||||||
|
"mp4-h264-aac-1080p", "summary");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -46,12 +46,25 @@ class HighlightAssetPreparationServiceTest {
|
||||||
assertThat(result.requestFiles()).isNotEmpty();
|
assertThat(result.requestFiles()).isNotEmpty();
|
||||||
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||||
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
|
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
|
||||||
|
assertThat(result.requestFiles()).anyMatch(path -> path.endsWith("voiceover-voiceover_001.md"));
|
||||||
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||||
.resolve("assets/requests"))) {
|
.resolve("assets/requests"))) {
|
||||||
assertThat(files.count()).isGreaterThan(0);
|
assertThat(files.count()).isGreaterThan(0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void usesStableSfxCueKeysSharedWithTheRendererPlan() {
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening Reveal", 0.0, 12.0, 12.0,
|
||||||
|
"opening_hook", "premium cinematic grade", "low cinematic pulse",
|
||||||
|
"whoosh on reveal", List.of(), List.of(), "hold the opening beat");
|
||||||
|
|
||||||
|
assertThat(HighlightAssetPreparationService.plannedSfxCues(highlight))
|
||||||
|
.extracting(AudioCue::assetKey)
|
||||||
|
.containsExactly("whoosh_soft", "impact_hit");
|
||||||
|
}
|
||||||
|
|
||||||
private VideoClippingProperties properties() {
|
private VideoClippingProperties properties() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,162 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Clock;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.time.ZoneOffset;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightCandidateGeneratorTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void persistsRankedCoverageWindowsAndMarksFallbackEvidenceForReview() {
|
||||||
|
Fixture fixture = fixture();
|
||||||
|
HighlightSourceAnalysis analysis = analysis(
|
||||||
|
42,
|
||||||
|
List.of(new ShotSegment("shot_0001", 0, 42, 42, 0, 21)),
|
||||||
|
new SourceAudioAnalysis("source", true, -30, -5, -35, 0.5, List.of(
|
||||||
|
new AudioSection("a1", "silence", 0, 12, 12),
|
||||||
|
new AudioSection("a2", "unclassified_audio", 12, 42, 30)
|
||||||
|
), null),
|
||||||
|
new SourceVisualAnalysis("source", 0.5, 0.5, 0.25, 0.75,
|
||||||
|
"unknown_without_face_detector",
|
||||||
|
List.of(new VisualObjectLabel("car", 0.82, "metadata_heuristic")),
|
||||||
|
List.of(), "local_cv_failed_fallback_metadata_thumbnail_scene_heuristic")
|
||||||
|
);
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
|
||||||
|
|
||||||
|
assertThat(result.category()).isEqualTo(ContentCategory.GENERIC_VLOG);
|
||||||
|
assertThat(result.categoryConfidence()).isEqualTo(0.25);
|
||||||
|
assertThat(result.categoryReasons()).anyMatch(reason -> reason.contains("metadata fallback"));
|
||||||
|
assertThat(result.candidates()).hasSize(3)
|
||||||
|
.allSatisfy(candidate -> {
|
||||||
|
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
|
||||||
|
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(42);
|
||||||
|
assertThat(candidate.sourceEndSeconds() - candidate.sourceStartSeconds()).isBetween(8.0, 35.0);
|
||||||
|
assertThat(candidate.reasons()).anyMatch(reason -> reason.contains("inspect frames"));
|
||||||
|
});
|
||||||
|
assertThat(result.candidates().get(0).sourceStartSeconds()).isGreaterThanOrEqualTo(9.0);
|
||||||
|
assertThat(Files.exists(fixture.store().analysisDirectory("project-1").resolve("category.json"))).isTrue();
|
||||||
|
assertThat(Files.exists(fixture.store().analysisDirectory("project-1")
|
||||||
|
.resolve("highlight-candidates.json"))).isTrue();
|
||||||
|
HighlightCandidate[] persisted = fixture.store().readJson("project-1",
|
||||||
|
"analysis/highlight-candidates.json", HighlightCandidate[].class);
|
||||||
|
assertThat(persisted).containsExactlyElementsOf(result.candidates());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void usesIndependentLocalModelLabelButStillRequiresReview() {
|
||||||
|
Fixture fixture = fixture();
|
||||||
|
HighlightSourceAnalysis analysis = analysis(
|
||||||
|
24,
|
||||||
|
List.of(
|
||||||
|
new ShotSegment("shot_0001", 0, 12, 12, 0.4, 6),
|
||||||
|
new ShotSegment("shot_0002", 12, 24, 12, 0.8, 18)
|
||||||
|
),
|
||||||
|
fullAudio(24),
|
||||||
|
new SourceVisualAnalysis("source", 0.8, 0.7, 0.75, 0.8, "none",
|
||||||
|
List.of(new VisualObjectLabel("car", 0.91, "yolo")),
|
||||||
|
List.of("frame.jpg"), "local_cv_yolo_clip")
|
||||||
|
);
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
|
||||||
|
|
||||||
|
assertThat(result.category()).isEqualTo(ContentCategory.CAR_VLOG);
|
||||||
|
assertThat(result.categoryConfidence()).isEqualTo(0.91);
|
||||||
|
assertThat(result.categoryReasons()).anyMatch(reason -> reason.contains("Confirm the category"));
|
||||||
|
assertThat(result.candidates()).extracting(HighlightCandidate::sourceStartSeconds)
|
||||||
|
.containsExactly(12.0, 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void expandsShortShotsWithoutLeavingSourceBounds() {
|
||||||
|
Fixture fixture = fixture();
|
||||||
|
HighlightSourceAnalysis analysis = analysis(
|
||||||
|
20,
|
||||||
|
List.of(
|
||||||
|
new ShotSegment("shot_0001", 0, 2, 2, 0.2, 1),
|
||||||
|
new ShotSegment("shot_0002", 18, 20, 2, 0.9, 19)
|
||||||
|
),
|
||||||
|
fullAudio(20),
|
||||||
|
new SourceVisualAnalysis("source", 0.7, 0.7, 0.7, 0.7, "unknown",
|
||||||
|
List.of(), List.of(), "local_cv")
|
||||||
|
);
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
|
||||||
|
|
||||||
|
assertThat(result.candidates()).hasSize(2)
|
||||||
|
.allSatisfy(candidate -> {
|
||||||
|
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
|
||||||
|
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(20);
|
||||||
|
assertThat(candidate.sourceEndSeconds() - candidate.sourceStartSeconds()).isEqualTo(8.0);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void removesDuplicateRangesCreatedByAdjacentShortShots() {
|
||||||
|
Fixture fixture = fixture();
|
||||||
|
HighlightSourceAnalysis analysis = analysis(
|
||||||
|
12,
|
||||||
|
List.of(
|
||||||
|
new ShotSegment("shot_0001", 0, 2, 2, 0.2, 1),
|
||||||
|
new ShotSegment("shot_0002", 2, 4, 2, 0.9, 3)
|
||||||
|
),
|
||||||
|
fullAudio(12),
|
||||||
|
new SourceVisualAnalysis("source", 0.7, 0.7, 0.7, 0.7, "unknown",
|
||||||
|
List.of(), List.of(), "local_cv")
|
||||||
|
);
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = fixture.generator().generate("project-1", analysis);
|
||||||
|
|
||||||
|
assertThat(result.candidates()).hasSize(1);
|
||||||
|
assertThat(result.candidates().get(0).sourceStartSeconds()).isZero();
|
||||||
|
assertThat(result.candidates().get(0).sourceEndSeconds()).isEqualTo(8.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Fixture fixture() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
|
||||||
|
properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
Instant now = Instant.parse("2026-07-21T10:00:00Z");
|
||||||
|
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.ANALYZING,
|
||||||
|
"source.mp4", store.projectDirectory("project-1").toString(), now, now, null);
|
||||||
|
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), now));
|
||||||
|
Clock clock = Clock.fixed(now, ZoneOffset.UTC);
|
||||||
|
return new Fixture(store, new HighlightCandidateGenerator(properties, store, clock));
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightSourceAnalysis analysis(
|
||||||
|
double duration,
|
||||||
|
List<ShotSegment> shots,
|
||||||
|
SourceAudioAnalysis audio,
|
||||||
|
SourceVisualAnalysis visual
|
||||||
|
) {
|
||||||
|
ClipAnalysis source = new ClipAnalysis("source", "source/source.mp4", duration, "h264", "aac",
|
||||||
|
1920, 1080, 30, List.of(), null, null, 0, 0, 0);
|
||||||
|
return new HighlightSourceAnalysis("project-1", "source.mp4", source, List.of(), null, null, null,
|
||||||
|
"analysis/scene-segments.json", shots, "analysis/audio-analysis.json", audio,
|
||||||
|
"analysis/visual-analysis.json", visual, Instant.parse("2026-07-21T10:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private SourceAudioAnalysis fullAudio(double duration) {
|
||||||
|
return new SourceAudioAnalysis("source", true, -18, -3, -35, 0.5,
|
||||||
|
List.of(new AudioSection("a1", "unclassified_audio", 0, duration, duration)), null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private record Fixture(FileSystemHighlightProjectStore store, HighlightCandidateGenerator generator) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -11,6 +11,7 @@ import java.time.Instant;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import static org.assertj.core.api.Assertions.assertThat;
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
import static org.mockito.ArgumentMatchers.any;
|
import static org.mockito.ArgumentMatchers.any;
|
||||||
import static org.mockito.ArgumentMatchers.anyString;
|
import static org.mockito.ArgumentMatchers.anyString;
|
||||||
import static org.mockito.Mockito.mock;
|
import static org.mockito.Mockito.mock;
|
||||||
|
|
@ -31,6 +32,7 @@ class HighlightDirectorFlowServiceTest {
|
||||||
HighlightProject project = project();
|
HighlightProject project = project();
|
||||||
store.createProject(project, manifest());
|
store.createProject(project, manifest());
|
||||||
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
||||||
|
writeDirectorEvidence(store);
|
||||||
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
||||||
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
||||||
createDummyFile(source);
|
createDummyFile(source);
|
||||||
|
|
@ -56,12 +58,19 @@ class HighlightDirectorFlowServiceTest {
|
||||||
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
|
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
|
||||||
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
|
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
|
||||||
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
||||||
|
new HighlightDirectorPlanValidator(properties, store),
|
||||||
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
||||||
|
|
||||||
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
||||||
|
|
||||||
assertThat(result.finalOutputs()).hasSize(1);
|
assertThat(result.finalOutputs()).hasSize(1);
|
||||||
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
|
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
|
||||||
|
EditPlan persistedPlan = store.readJson(project.id(), "highlights/highlight_001/edit-plan.json",
|
||||||
|
EditPlan.class);
|
||||||
|
assertThat(persistedPlan.audioCues().stream()
|
||||||
|
.filter(cue -> "sfx".equals(cue.type()))
|
||||||
|
.map(AudioCue::assetKey))
|
||||||
|
.containsExactly("whoosh_soft", "impact_hit");
|
||||||
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
|
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
|
||||||
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
|
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
|
||||||
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||||
|
|
@ -76,6 +85,7 @@ class HighlightDirectorFlowServiceTest {
|
||||||
HighlightProject project = project();
|
HighlightProject project = project();
|
||||||
store.createProject(project, manifest());
|
store.createProject(project, manifest());
|
||||||
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
||||||
|
writeDirectorEvidence(store);
|
||||||
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
||||||
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
||||||
createDummyFile(source);
|
createDummyFile(source);
|
||||||
|
|
@ -93,6 +103,7 @@ class HighlightDirectorFlowServiceTest {
|
||||||
List.of(), List.of("requests/voiceover.json")));
|
List.of(), List.of("requests/voiceover.json")));
|
||||||
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
|
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
|
||||||
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
||||||
|
new HighlightDirectorPlanValidator(properties, store),
|
||||||
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
||||||
|
|
||||||
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
||||||
|
|
@ -101,12 +112,54 @@ class HighlightDirectorFlowServiceTest {
|
||||||
assertThat(result.finalOutputs()).isEmpty();
|
assertThat(result.finalOutputs()).isEmpty();
|
||||||
verify(renderer, never()).render(anyString(), any(), any());
|
verify(renderer, never()).render(anyString(), any(), any());
|
||||||
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||||
.isEqualTo(HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
.isEqualTo(HighlightProjectStatus.PLANNED);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsRenderWhenBlockingQaCheckFails() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = project();
|
||||||
|
store.createProject(project, manifest());
|
||||||
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
||||||
|
writeDirectorEvidence(store);
|
||||||
|
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
||||||
|
createDummyFile(store.sourceDirectory(project.id()).resolve("source.mp4"));
|
||||||
|
|
||||||
|
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
|
||||||
|
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(
|
||||||
|
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
|
||||||
|
store.highlightsDirectory(project.id()).resolve("highlight_001"),
|
||||||
|
store.highlightsDirectory(project.id()).resolve("highlight_001/assets"),
|
||||||
|
List.of(), List.of()));
|
||||||
|
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
|
||||||
|
when(assetWorker.process(anyString(), any(), any())).thenReturn(
|
||||||
|
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
|
||||||
|
List.of(), List.of()));
|
||||||
|
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
|
||||||
|
Path failedOutput = store.highlightsDirectory(project.id()).resolve("highlight_001/final.mp4");
|
||||||
|
when(renderer.render(anyString(), any(), any())).thenReturn(
|
||||||
|
new HighlightFfmpegRenderer.HighlightRenderResult("highlight_001", failedOutput, failedOutput,
|
||||||
|
List.of(), null, new RenderQaReport(project.id(), false,
|
||||||
|
List.of(new RenderQaCheck("audio_clipping", false, "ERROR", "peak=0.0")), Instant.now())));
|
||||||
|
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
||||||
|
new HighlightDirectorPlanValidator(properties, store), mock(HighlightVisualEffectsStage.class),
|
||||||
|
assetPreparationService, assetWorker, renderer);
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> service.process(project.id(), 1))
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("audio_clipping");
|
||||||
|
|
||||||
|
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||||
|
.isEqualTo(HighlightProjectStatus.FAILED);
|
||||||
|
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).doesNotExist();
|
||||||
}
|
}
|
||||||
|
|
||||||
private VideoClippingProperties properties() {
|
private VideoClippingProperties properties() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
properties.getEditing().getHighlightScheduler().setRequireDirectorApproval(false);
|
||||||
return properties;
|
return properties;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -140,6 +193,16 @@ class HighlightDirectorFlowServiceTest {
|
||||||
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void writeDirectorEvidence(FileSystemHighlightProjectStore store) {
|
||||||
|
HighlightCandidate candidate = new HighlightCandidate("candidate_001", "source", 0, 12, 0.7,
|
||||||
|
"opening_hook_candidate", List.of("reviewed"));
|
||||||
|
store.writeJson("highlight-project-001", "analysis/highlight-candidates.json",
|
||||||
|
new HighlightCandidate[] { candidate });
|
||||||
|
store.writeJson("highlight-project-001", "analysis/category.json", new CinematicHighlightAnalysis(
|
||||||
|
"highlight-project-001", ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"),
|
||||||
|
List.of(candidate), Instant.parse("2026-07-11T08:00:00Z")));
|
||||||
|
}
|
||||||
|
|
||||||
private void createDummyFile(Path file) throws Exception {
|
private void createDummyFile(Path file) throws Exception {
|
||||||
Files.createDirectories(file.getParent());
|
Files.createDirectories(file.getParent());
|
||||||
Files.writeString(file, "dummy");
|
Files.writeString(file, "dummy");
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,77 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
|
||||||
|
class HighlightDirectorPlanScannerTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void skipsTerminalAndInProgressProjectsWhenSelectingWork() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
|
||||||
|
properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
createProject(store, "a-failed", HighlightProjectStatus.FAILED);
|
||||||
|
createProject(store, "b-rendering", HighlightProjectStatus.RENDERING);
|
||||||
|
createProject(store, "c-ready", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
||||||
|
approve(store, "c-ready");
|
||||||
|
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
|
||||||
|
properties, store, mock(HighlightDirectorFlowService.class));
|
||||||
|
|
||||||
|
assertThat(scanner.findNextRenderableProject()).contains("c-ready");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void leavesFailedProjectTerminalInsteadOfRetryingIt() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
|
||||||
|
properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
createProject(store, "failed", HighlightProjectStatus.FAILED);
|
||||||
|
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
|
||||||
|
properties, store, mock(HighlightDirectorFlowService.class));
|
||||||
|
|
||||||
|
assertThat(scanner.findNextRenderableProject()).isEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void unapprovedProjectDoesNotStarveLaterApprovedProject() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(
|
||||||
|
properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
createProject(store, "a-unapproved", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
||||||
|
createProject(store, "b-approved", HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
||||||
|
approve(store, "b-approved");
|
||||||
|
HighlightDirectorPlanScanner scanner = new HighlightDirectorPlanScanner(
|
||||||
|
properties, store, mock(HighlightDirectorFlowService.class));
|
||||||
|
|
||||||
|
assertThat(scanner.findNextRenderableProject()).contains("b-approved");
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createProject(FileSystemHighlightProjectStore store, String projectId,
|
||||||
|
HighlightProjectStatus status) throws Exception {
|
||||||
|
Instant now = Instant.parse("2026-07-21T10:00:00Z");
|
||||||
|
store.createProject(new HighlightProject(projectId, projectId, status, "source.mp4",
|
||||||
|
store.projectDirectory(projectId).toString(), now, now,
|
||||||
|
status == HighlightProjectStatus.FAILED ? "failed" : null),
|
||||||
|
new HighlightProjectManifest(projectId, "source.mp4", HighlightFolderContract.standard(), now));
|
||||||
|
Files.writeString(store.directorDirectory(projectId).resolve("edit-plan.json"), "{}");
|
||||||
|
}
|
||||||
|
|
||||||
|
private void approve(FileSystemHighlightProjectStore store, String projectId) throws Exception {
|
||||||
|
Files.writeString(store.directorDirectory(projectId).resolve("approved.flag"), "approved");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,124 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.application.BadRequestException;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.BeforeEach;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
|
|
||||||
|
class HighlightDirectorPlanValidatorTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
private VideoClippingProperties properties;
|
||||||
|
private FileSystemHighlightProjectStore store;
|
||||||
|
private HighlightDirectorPlanValidator validator;
|
||||||
|
|
||||||
|
@BeforeEach
|
||||||
|
void setUp() {
|
||||||
|
properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
Instant now = Instant.parse("2026-07-21T10:00:00Z");
|
||||||
|
store.createProject(new HighlightProject("project-1", "Project", HighlightProjectStatus.WAITING_FOR_DIRECTOR,
|
||||||
|
"source.mp4", store.projectDirectory("project-1").toString(), now, now, null),
|
||||||
|
new HighlightProjectManifest("project-1", "source.mp4", HighlightFolderContract.standard(), now));
|
||||||
|
ClipAnalysis source = new ClipAnalysis("source", "source.mp4", 30, "h264", "aac", 1920, 1080, 30,
|
||||||
|
List.of(), null, null, 0, 0, 0);
|
||||||
|
store.writeJson("project-1", "analysis/source-analysis.json", new HighlightSourceAnalysis(
|
||||||
|
"project-1", "source.mp4", source, List.of(), null, null, null,
|
||||||
|
"analysis/scene-segments.json", List.of(), "analysis/audio-analysis.json",
|
||||||
|
new SourceAudioAnalysis("source", true, -18, -3, -35, 0.5, List.of(), null),
|
||||||
|
"analysis/visual-analysis.json", new SourceVisualAnalysis("source", 0.5, 0.5, 0.5, 0.5,
|
||||||
|
"unknown", List.of(), List.of(), "metadata_thumbnail_scene_heuristic"), now));
|
||||||
|
HighlightCandidate candidate = candidate();
|
||||||
|
store.writeJson("project-1", "analysis/highlight-candidates.json",
|
||||||
|
new HighlightCandidate[] { candidate });
|
||||||
|
store.writeJson("project-1", "analysis/category.json", new CinematicHighlightAnalysis(
|
||||||
|
"project-1", ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"),
|
||||||
|
List.of(candidate), now));
|
||||||
|
validator = new HighlightDirectorPlanValidator(properties, store);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void acceptsPlanGroundedInPersistedProjectCategoryAndCandidate() {
|
||||||
|
assertThat(validator.validate("project-1", plan(item("candidate_001", 2, 14))))
|
||||||
|
.isEqualTo(plan(item("candidate_001", 2, 14)));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsUnknownCandidateAndRangeOutsideCandidate() {
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(item("candidate_999", 2, 14))))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("Unknown highlight candidate");
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(item("candidate_001", 1, 14))))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("contained by candidate");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsMismatchedCategoryAndMissingProductionDirections() {
|
||||||
|
HighlightDirectorPlan wrongCategory = new HighlightDirectorPlan("project-1", "source.mp4", "car_vlog",
|
||||||
|
List.of(item("candidate_001", 2, 14)), "summary");
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", wrongCategory))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("reviewed category analysis");
|
||||||
|
|
||||||
|
HighlightDirectorPlan.HighlightItem missingMusic = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening", 2, 14, 12,
|
||||||
|
"opening_hook", "cinematic grade", "", "whoosh at transition",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(missingMusic)))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("Music direction");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsUnsafeHighlightIdBeforeItCanBecomeAPath() {
|
||||||
|
HighlightDirectorPlan.HighlightItem unsafe = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"../outside", "candidate_001", "Opening", 2, 14, 12,
|
||||||
|
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(unsafe)))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("safe identifiers");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsTargetDurationThatRequiresUnsupportedPlaybackSpeed() {
|
||||||
|
HighlightDirectorPlan.HighlightItem tooSlow = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening", 2, 4, 12,
|
||||||
|
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(tooSlow)))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("playback speed outside 0.25..4.0");
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightCandidate candidate() {
|
||||||
|
return new HighlightCandidate("candidate_001", "source", 2, 14, 0.7,
|
||||||
|
"opening_hook_candidate", List.of("reviewed"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightDirectorPlan plan(HighlightDirectorPlan.HighlightItem item) {
|
||||||
|
return new HighlightDirectorPlan("project-1", "source.mp4", "generic_vlog", List.of(item), "summary");
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightDirectorPlan.HighlightItem item(String candidateId, double start, double end) {
|
||||||
|
return new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", candidateId, "Opening", start, end, 12,
|
||||||
|
"opening_hook", "cinematic grade", "scene-fit score", "whoosh at transition",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -37,6 +37,8 @@ class HighlightDirectorPromptGeneratorTest {
|
||||||
.contains("analysis/highlight-candidates.json")
|
.contains("analysis/highlight-candidates.json")
|
||||||
.contains("director/edit-plan.json")
|
.contains("director/edit-plan.json")
|
||||||
.contains("car_vlog")
|
.contains("car_vlog")
|
||||||
|
.contains("Category evidence status: `model_supported_review_required`")
|
||||||
|
.contains("Treat candidate scores as ranking hints")
|
||||||
.contains("candidate_001")
|
.contains("candidate_001")
|
||||||
.contains("premium, powerful, precise");
|
.contains("premium, powerful, precise");
|
||||||
assertThat(Files.readString(Path.of(files.briefPath())))
|
assertThat(Files.readString(Path.of(files.briefPath())))
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,137 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
|
||||||
|
class HighlightFfmpegRendererTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void appliesMusicCueTimingAndGain() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
AudioCue cue = new AudioCue("music", "local-score", 1.5, 9.5, -7, "match reveal");
|
||||||
|
|
||||||
|
List<String> command = renderer.audioMixCommand(Path.of("timeline.mp4"), Path.of("music.wav"), cue,
|
||||||
|
List.of(), List.of(), Path.of("final.mp4"));
|
||||||
|
|
||||||
|
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
||||||
|
.contains("[1:a]atrim=duration=8.0,asetpts=PTS-STARTPTS,volume=-7.0dB,")
|
||||||
|
.contains("adelay=1500|1500[music_raw]"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsMissingRequestedAssetsBeforeRendering() throws Exception {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
EditPlan plan = planWithRequestedAudio();
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> renderer.requireRequestedAssets(plan, tempDir))
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("music")
|
||||||
|
.hasMessageContaining("voiceover")
|
||||||
|
.hasMessageContaining("sfx:impact");
|
||||||
|
|
||||||
|
Files.createDirectories(tempDir.resolve("music"));
|
||||||
|
Files.createDirectories(tempDir.resolve("voiceover"));
|
||||||
|
Files.createDirectories(tempDir.resolve("sfx"));
|
||||||
|
Files.writeString(tempDir.resolve("music/music.wav"), "music");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover/voiceover_001.wav"), "voice");
|
||||||
|
Files.writeString(tempDir.resolve("sfx/impact.wav"), "impact");
|
||||||
|
Files.writeString(tempDir.resolve("music/music.wav.license.txt"), "approved-music");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover/voiceover_001.wav.license.txt"), "approved-voice");
|
||||||
|
Files.writeString(tempDir.resolve("sfx/impact.wav.license.txt"), "approved-sfx");
|
||||||
|
|
||||||
|
assertThatCode(() -> renderer.requireRequestedAssets(plan, tempDir)).doesNotThrowAnyException();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void alignsEachVoiceoverLineToItsPlannedTimelineRange() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
List<HighlightFfmpegRenderer.VoiceoverInput> voiceovers = List.of(
|
||||||
|
new HighlightFfmpegRenderer.VoiceoverInput(Path.of("line-1.wav"),
|
||||||
|
new VoiceoverLine("First.", 1.25, 2.75, "controlled")),
|
||||||
|
new HighlightFfmpegRenderer.VoiceoverInput(Path.of("line-2.wav"),
|
||||||
|
new VoiceoverLine("Second.", 5.0, 7.0, "controlled"))
|
||||||
|
);
|
||||||
|
|
||||||
|
List<String> command = renderer.audioMixCommand(Path.of("timeline.mp4"), null, null,
|
||||||
|
voiceovers, List.of(), Path.of("final.mp4"));
|
||||||
|
|
||||||
|
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
||||||
|
.contains("[1:a]atrim=duration=1.5")
|
||||||
|
.contains("adelay=1250|1250[voice_line_0]")
|
||||||
|
.contains("[2:a]atrim=duration=2.0")
|
||||||
|
.contains("adelay=5000|5000[voice_line_1]")
|
||||||
|
.contains("[voice_line_0][voice_line_1]amix=inputs=2:duration=longest"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void buildsMeasuredQaProbeCommands() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
|
||||||
|
assertThat(renderer.durationProbeCommand(Path.of("final.mp4")))
|
||||||
|
.containsSubsequence("-show_entries", "format=duration");
|
||||||
|
assertThat(renderer.blackDetectCommand(Path.of("final.mp4")))
|
||||||
|
.containsSubsequence("-vf", "blackdetect=d=0.5:pic_th=0.98");
|
||||||
|
assertThat(renderer.silenceDetectCommand(Path.of("final.mp4")))
|
||||||
|
.containsSubsequence("-af", "silencedetect=noise=-45dB:d=2");
|
||||||
|
assertThat(renderer.clippingDetectCommand(Path.of("final.mp4")))
|
||||||
|
.containsSubsequence("-af", "astats=metadata=1:reset=1");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void requiresMeasuredAudioPeakAndRejectsClipping() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
|
||||||
|
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
|
||||||
|
"[Parsed_astats] Peak level dB: -1.2"))).isTrue();
|
||||||
|
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
|
||||||
|
"[Parsed_astats] Peak level dB: 0.0"))).isFalse();
|
||||||
|
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(0,
|
||||||
|
"no peak measurement"))).isFalse();
|
||||||
|
assertThat(renderer.audioPeakIsSafe(new HighlightFfmpegRenderer.ProcessResult(1,
|
||||||
|
"failed"))).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void mixesGeneratedAudioWithoutReferencingMissingSourceAudio() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
AudioCue music = new AudioCue("music", "score", 0, 10, -8, "scene-fit score");
|
||||||
|
|
||||||
|
List<String> command = renderer.audioMixCommand(Path.of("video-only.mp4"), Path.of("music.wav"), music,
|
||||||
|
List.of(), List.of(), false, Path.of("final.mp4"));
|
||||||
|
|
||||||
|
assertThat(command).noneMatch(argument -> argument.contains("[0:a]"));
|
||||||
|
assertThat(command).anySatisfy(argument -> assertThat(argument)
|
||||||
|
.contains("amix=inputs=1:duration=longest")
|
||||||
|
.contains(",apad[a]"));
|
||||||
|
assertThat(command).contains("-shortest");
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightFfmpegRenderer renderer() {
|
||||||
|
return new HighlightFfmpegRenderer(new VideoClippingProperties(), mock(HighlightProjectStore.class),
|
||||||
|
mock(EditAssetProvider.class), mock(EditObservability.class), command -> null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private EditPlan planWithRequestedAudio() {
|
||||||
|
return new EditPlan("project", "cinematic", 10,
|
||||||
|
List.of(new EditDecision("clip", 0, 10, 0, 10, "cut", "cut", 1, "grade", "hero")),
|
||||||
|
List.of(
|
||||||
|
new AudioCue("music", "score", 0, 10, -8, "scene-fit score"),
|
||||||
|
new AudioCue("sfx", "impact", 2, 3, -6, "reveal")
|
||||||
|
),
|
||||||
|
List.of(new VoiceoverLine("Precision.", 1, 3, "controlled")),
|
||||||
|
"mp4-h264-aac-1080p", "summary");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -11,6 +11,7 @@ import java.time.Instant;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import static org.assertj.core.api.Assertions.assertThat;
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
import static org.mockito.Mockito.mock;
|
import static org.mockito.Mockito.mock;
|
||||||
import static org.mockito.Mockito.when;
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
|
@ -36,6 +37,7 @@ class HighlightLocalAssetWorkerTest {
|
||||||
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
|
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
|
||||||
Files.createDirectories(musicSource.getParent());
|
Files.createDirectories(musicSource.getParent());
|
||||||
Files.writeString(musicSource, "music");
|
Files.writeString(musicSource, "music");
|
||||||
|
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.license.txt"), "approved-music");
|
||||||
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
|
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
|
||||||
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
||||||
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
||||||
|
|
@ -62,4 +64,35 @@ class HighlightLocalAssetWorkerTest {
|
||||||
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
|
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
|
||||||
.exists();
|
.exists();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsTamperedAssetRequestTargetOutsideProject() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.PLANNED,
|
||||||
|
"source.mp4", store.projectDirectory("project-1").toString(),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
|
||||||
|
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
|
||||||
|
Path requests = store.highlightsDirectory("project-1").resolve("highlight_001/assets/requests");
|
||||||
|
Files.createDirectories(requests);
|
||||||
|
Path requestMarkdown = requests.resolve("music-music_bed.md");
|
||||||
|
Files.writeString(requestMarkdown, "request");
|
||||||
|
mapper.writeValue(requests.resolve("music-music_bed.json").toFile(), new HighlightAssetRequest(
|
||||||
|
"project-1", "highlight_001", "music", "music_bed",
|
||||||
|
tempDir.resolve("outside.wav").toString(), requestMarkdown.toString(),
|
||||||
|
"scene-fit score", 12, true));
|
||||||
|
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, mock(EditAssetProvider.class),
|
||||||
|
mock(EditAssetLibrary.class), mock(LocalAssetSynthesizer.class), mapper);
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening", 0, 12, 12, "opening_hook",
|
||||||
|
"grade", "score", "whoosh", List.of("Narration."), List.of(), "notes");
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> worker.process("project-1", highlight, ContentCategory.GENERIC_VLOG))
|
||||||
|
.isInstanceOf(IllegalArgumentException.class)
|
||||||
|
.hasMessageContaining("paths do not match");
|
||||||
|
assertThat(tempDir.resolve("outside.wav")).doesNotExist();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,7 @@ class HighlightSourceSchedulerTest {
|
||||||
private VideoClippingProperties properties;
|
private VideoClippingProperties properties;
|
||||||
private FileSystemHighlightProjectStore store;
|
private FileSystemHighlightProjectStore store;
|
||||||
private HighlightSourceAnalyzer analyzer;
|
private HighlightSourceAnalyzer analyzer;
|
||||||
|
private HighlightCandidateGenerator candidateGenerator;
|
||||||
private HighlightDirectorPromptGenerator promptGenerator;
|
private HighlightDirectorPromptGenerator promptGenerator;
|
||||||
|
|
||||||
@BeforeEach
|
@BeforeEach
|
||||||
|
|
@ -41,6 +42,7 @@ class HighlightSourceSchedulerTest {
|
||||||
new HighlightProjectDirectoryInitializer(properties).initialize();
|
new HighlightProjectDirectoryInitializer(properties).initialize();
|
||||||
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
analyzer = mock(HighlightSourceAnalyzer.class);
|
analyzer = mock(HighlightSourceAnalyzer.class);
|
||||||
|
candidateGenerator = mock(HighlightCandidateGenerator.class);
|
||||||
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
|
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -75,6 +77,7 @@ class HighlightSourceSchedulerTest {
|
||||||
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
|
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
|
||||||
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
|
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
|
||||||
verify(analyzer).analyze("1");
|
verify(analyzer).analyze("1");
|
||||||
|
verify(candidateGenerator).generate("1", analysis("1"));
|
||||||
verify(promptGenerator).generate("1");
|
verify(promptGenerator).generate("1");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -107,11 +110,19 @@ class HighlightSourceSchedulerTest {
|
||||||
|
|
||||||
private HighlightSourceScheduler scheduler() {
|
private HighlightSourceScheduler scheduler() {
|
||||||
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
|
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
|
||||||
when(analyzer.analyze("1")).thenReturn(analysis("1"));
|
stubAnalysis("1");
|
||||||
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
|
stubAnalysis("porsche");
|
||||||
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
|
stubAnalysis("porsche-1");
|
||||||
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
|
stubAnalysis("porsche-drive");
|
||||||
return new HighlightSourceScheduler(properties, store, analyzer, promptGenerator, clock);
|
return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator, clock);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void stubAnalysis(String projectId) {
|
||||||
|
HighlightSourceAnalysis sourceAnalysis = analysis(projectId);
|
||||||
|
when(analyzer.analyze(projectId)).thenReturn(sourceAnalysis);
|
||||||
|
when(candidateGenerator.generate(projectId, sourceAnalysis)).thenReturn(new CinematicHighlightAnalysis(
|
||||||
|
projectId, ContentCategory.GENERIC_VLOG, 0.25, List.of("review required"), List.of(),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z")));
|
||||||
}
|
}
|
||||||
|
|
||||||
private HighlightSourceAnalysis analysis(String projectId) {
|
private HighlightSourceAnalysis analysis(String projectId) {
|
||||||
|
|
|
||||||
|
|
@ -18,13 +18,17 @@ class LocalAssetGenerationStageTest {
|
||||||
Path tempDir;
|
Path tempDir;
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void reusesExistingSharedAssetsAndSynthesizesMissingVoiceover() throws Exception {
|
void blocksRenderWhenRequestedVoiceoverCannotBeGenerated() throws Exception {
|
||||||
VideoClippingProperties properties = properties();
|
VideoClippingProperties properties = properties();
|
||||||
Files.createDirectories(tempDir.resolve("shared/music"));
|
Files.createDirectories(tempDir.resolve("shared/music"));
|
||||||
Files.createDirectories(tempDir.resolve("shared/sfx"));
|
Files.createDirectories(tempDir.resolve("shared/sfx"));
|
||||||
Files.createDirectories(tempDir.resolve("voiceover-cache"));
|
Files.createDirectories(tempDir.resolve("voiceover-cache"));
|
||||||
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
|
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
|
||||||
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
|
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
|
||||||
|
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav.license.txt"), "approved-music");
|
||||||
|
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav.license.txt"), "approved-sfx");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover-cache/unrelated.wav"), "wrong narration");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover-cache/unrelated.wav.license.txt"), "approved-voice");
|
||||||
|
|
||||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||||
|
|
@ -53,16 +57,16 @@ class LocalAssetGenerationStageTest {
|
||||||
|
|
||||||
AssetGenerationResult result = stage.prepare("project", plan);
|
AssetGenerationResult result = stage.prepare("project", plan);
|
||||||
|
|
||||||
assertThat(result.readyForRender()).isTrue();
|
assertThat(result.readyForRender()).isFalse();
|
||||||
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
|
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
|
||||||
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
|
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
|
||||||
assertThat(tempDir.resolve("projects/project/audio/voiceover.wav")).exists();
|
assertThat(tempDir.resolve("projects/project/audio/voiceover.wav")).doesNotExist();
|
||||||
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
|
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
|
||||||
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
|
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void synthesizesMissingSfxAndKeepsProjectReadyForRender() throws Exception {
|
void blocksRenderWhenRequestedSfxCannotBeGenerated() throws Exception {
|
||||||
VideoClippingProperties properties = properties();
|
VideoClippingProperties properties = properties();
|
||||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||||
|
|
@ -84,8 +88,8 @@ class LocalAssetGenerationStageTest {
|
||||||
|
|
||||||
AssetGenerationResult result = stage.prepare("project", plan);
|
AssetGenerationResult result = stage.prepare("project", plan);
|
||||||
|
|
||||||
assertThat(result.readyForRender()).isTrue();
|
assertThat(result.readyForRender()).isFalse();
|
||||||
assertThat(tempDir.resolve("projects/project/audio/sfx/engine-hit.wav")).exists();
|
assertThat(tempDir.resolve("projects/project/audio/sfx/engine-hit.wav")).doesNotExist();
|
||||||
}
|
}
|
||||||
|
|
||||||
private VideoClippingProperties properties() {
|
private VideoClippingProperties properties() {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.springframework.boot.DefaultApplicationArguments;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
|
|
||||||
|
class LocalAssetRuntimeVerifierTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void strictRuntimeFailsClosedWhenResidentModelsAreMissing() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getLocalAssetWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getLocalAssetWorker().setStrictRuntime(true);
|
||||||
|
LocalAssetRuntimeVerifier verifier = new LocalAssetRuntimeVerifier(properties);
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> verifier.run(new DefaultApplicationArguments()))
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("Strict local asset runtime is not ready")
|
||||||
|
.hasMessageContaining("music_model_path")
|
||||||
|
.hasMessageContaining("sfx_model_path");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void disabledRuntimeDoesNotAttemptProvisioning() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getLocalAssetWorker().setAutoStart(false);
|
||||||
|
LocalAssetRuntimeVerifier verifier = new LocalAssetRuntimeVerifier(properties);
|
||||||
|
|
||||||
|
assertThatCode(() -> verifier.run(new DefaultApplicationArguments())).doesNotThrowAnyException();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -20,6 +20,7 @@ class LocalEditAssetLibraryTest {
|
||||||
Path base = Files.createDirectories(tempDir.resolve("music"));
|
Path base = Files.createDirectories(tempDir.resolve("music"));
|
||||||
Path car = Files.createDirectories(base.resolve("car_vlog"));
|
Path car = Files.createDirectories(base.resolve("car_vlog"));
|
||||||
Files.writeString(base.resolve("ambient.wav"), "base");
|
Files.writeString(base.resolve("ambient.wav"), "base");
|
||||||
|
Files.writeString(base.resolve("ambient.wav.license.txt"), "approved-ambient");
|
||||||
Files.writeString(base.resolve("ambient.wav.tags.txt"), "soft, calm");
|
Files.writeString(base.resolve("ambient.wav.tags.txt"), "soft, calm");
|
||||||
Files.writeString(car.resolve("premium-drive.wav"), "car");
|
Files.writeString(car.resolve("premium-drive.wav"), "car");
|
||||||
Files.writeString(car.resolve("premium-drive.wav.license.txt"), "licensed-local-track");
|
Files.writeString(car.resolve("premium-drive.wav.license.txt"), "licensed-local-track");
|
||||||
|
|
@ -42,6 +43,8 @@ class LocalEditAssetLibraryTest {
|
||||||
Path sfx = Files.createDirectories(tempDir.resolve("sfx"));
|
Path sfx = Files.createDirectories(tempDir.resolve("sfx"));
|
||||||
Files.writeString(sfx.resolve("alpha.wav"), "a");
|
Files.writeString(sfx.resolve("alpha.wav"), "a");
|
||||||
Files.writeString(sfx.resolve("beta.wav"), "b");
|
Files.writeString(sfx.resolve("beta.wav"), "b");
|
||||||
|
Files.writeString(sfx.resolve("alpha.wav.license.txt"), "approved-alpha");
|
||||||
|
Files.writeString(sfx.resolve("beta.wav.license.txt"), "approved-beta");
|
||||||
LocalEditAssetLibrary library = new LocalEditAssetLibrary(new LocalEditAssetProvider(properties));
|
LocalEditAssetLibrary library = new LocalEditAssetLibrary(new LocalEditAssetProvider(properties));
|
||||||
|
|
||||||
var selected = library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, "", 1));
|
var selected = library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, "", 1));
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,7 @@ class LocalEditAssetProviderTest {
|
||||||
VideoClippingProperties properties = properties();
|
VideoClippingProperties properties = properties();
|
||||||
Path sfx = Files.createDirectories(tempDir.resolve("sfx"));
|
Path sfx = Files.createDirectories(tempDir.resolve("sfx"));
|
||||||
Files.writeString(sfx.resolve("hit.wav"), "wav");
|
Files.writeString(sfx.resolve("hit.wav"), "wav");
|
||||||
|
Files.writeString(sfx.resolve("hit.wav.license.txt"), "approved-sfx");
|
||||||
Files.writeString(sfx.resolve("noise.txt"), "txt");
|
Files.writeString(sfx.resolve("noise.txt"), "txt");
|
||||||
|
|
||||||
LocalEditAssetProvider provider = new LocalEditAssetProvider(properties);
|
LocalEditAssetProvider provider = new LocalEditAssetProvider(properties);
|
||||||
|
|
@ -77,6 +78,9 @@ class LocalEditAssetProviderTest {
|
||||||
Files.writeString(tempDir.resolve("fonts/title.otf"), "font");
|
Files.writeString(tempDir.resolve("fonts/title.otf"), "font");
|
||||||
Files.writeString(tempDir.resolve("luts/warm.cube"), "lut");
|
Files.writeString(tempDir.resolve("luts/warm.cube"), "lut");
|
||||||
Files.writeString(tempDir.resolve("voiceover/line-001.mp3"), "voice");
|
Files.writeString(tempDir.resolve("voiceover/line-001.mp3"), "voice");
|
||||||
|
Files.writeString(tempDir.resolve("fonts/title.otf.license.txt"), "approved-font");
|
||||||
|
Files.writeString(tempDir.resolve("luts/warm.cube.license.txt"), "approved-lut");
|
||||||
|
Files.writeString(tempDir.resolve("voiceover/line-001.mp3.license.txt"), "approved-voice");
|
||||||
LocalEditAssetProvider provider = new LocalEditAssetProvider(properties);
|
LocalEditAssetProvider provider = new LocalEditAssetProvider(properties);
|
||||||
|
|
||||||
assertThat(provider.resolve(new EditAssetRequest(EditAssetType.FONT, "title", null, 0, null))).isPresent();
|
assertThat(provider.resolve(new EditAssetRequest(EditAssetType.FONT, "title", null, 0, null))).isPresent();
|
||||||
|
|
@ -85,6 +89,18 @@ class LocalEditAssetProviderTest {
|
||||||
.isPresent();
|
.isPresent();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void excludesAssetsWithoutLicenseEvidence() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
Path music = Files.createDirectories(tempDir.resolve("music"));
|
||||||
|
Files.writeString(music.resolve("unlicensed.wav"), "audio");
|
||||||
|
LocalEditAssetProvider provider = new LocalEditAssetProvider(properties);
|
||||||
|
|
||||||
|
assertThat(provider.resolve(new EditAssetRequest(
|
||||||
|
EditAssetType.MUSIC, "unlicensed", null, 10, "mood"))).isEmpty();
|
||||||
|
assertThat(provider.list(EditAssetType.MUSIC, null)).isEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
private VideoClippingProperties properties() {
|
private VideoClippingProperties properties() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import math
|
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
|
|
@ -43,19 +42,16 @@ def main() -> int:
|
||||||
if args.kind == "voiceover":
|
if args.kind == "voiceover":
|
||||||
if synthesize_voiceover(prompt, output, args.piper_binary, args.model):
|
if synthesize_voiceover(prompt, output, args.piper_binary, args.model):
|
||||||
return 0
|
return 0
|
||||||
write_silence(output, max(1.0, args.duration))
|
return 2
|
||||||
return 0
|
|
||||||
|
|
||||||
if args.kind == "music":
|
if args.kind == "music":
|
||||||
if synthesize_audiocraft("music", prompt, output, args.duration, args.model):
|
if synthesize_audiocraft("music", prompt, output, args.duration, args.model):
|
||||||
return 0
|
return 0
|
||||||
write_fallback_tone(output, max(4.0, args.duration), 110.0, 0.02)
|
return 2
|
||||||
return 0
|
|
||||||
|
|
||||||
if synthesize_audiocraft("sfx", prompt, output, args.duration, args.model):
|
if synthesize_audiocraft("sfx", prompt, output, args.duration, args.model):
|
||||||
return 0
|
return 0
|
||||||
write_fallback_tone(output, max(0.5, min(1.0, args.duration)), 880.0, 0.12)
|
return 2
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
def synthesize_voiceover(prompt: str, output: Path, piper_binary: str, model: str) -> bool:
|
def synthesize_voiceover(prompt: str, output: Path, piper_binary: str, model: str) -> bool:
|
||||||
|
|
@ -74,27 +70,20 @@ def synthesize_voiceover(prompt: str, output: Path, piper_binary: str, model: st
|
||||||
except Exception:
|
except Exception:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
say = shutil.which("say")
|
|
||||||
if say:
|
|
||||||
process = subprocess.run([say, "-o", str(output), prompt], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
|
||||||
return process.returncode == 0 and output.is_file()
|
|
||||||
|
|
||||||
espeak = shutil.which("espeak")
|
|
||||||
if espeak:
|
|
||||||
process = subprocess.run([espeak, "-w", str(output), prompt], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
|
||||||
return process.returncode == 0 and output.is_file()
|
|
||||||
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def synthesize_audiocraft(kind: str, prompt: str, output: Path, duration: float, model_name: str) -> bool:
|
def synthesize_audiocraft(kind: str, prompt: str, output: Path, duration: float, model_name: str) -> bool:
|
||||||
if torch is None:
|
model_path = Path(model_name).expanduser() if model_name else None
|
||||||
|
if torch is None or model_path is None or not model_path.exists():
|
||||||
return False
|
return False
|
||||||
|
os.environ.setdefault("HF_HUB_OFFLINE", "1")
|
||||||
|
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
|
||||||
try:
|
try:
|
||||||
if kind == "music":
|
if kind == "music":
|
||||||
from audiocraft.models import MusicGen # type: ignore
|
from audiocraft.models import MusicGen # type: ignore
|
||||||
|
|
||||||
model = MusicGen.get_pretrained(normalize_model_name(model_name, "musicgen-small"))
|
model = MusicGen.get_pretrained(str(model_path.resolve()))
|
||||||
model.set_generation_params(duration=max(1.0, float(duration)))
|
model.set_generation_params(duration=max(1.0, float(duration)))
|
||||||
audio = model.generate([prompt])[0]
|
audio = model.generate([prompt])[0]
|
||||||
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
|
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
|
||||||
|
|
@ -102,7 +91,7 @@ def synthesize_audiocraft(kind: str, prompt: str, output: Path, duration: float,
|
||||||
|
|
||||||
from audiocraft.models import AudioGen # type: ignore
|
from audiocraft.models import AudioGen # type: ignore
|
||||||
|
|
||||||
model = AudioGen.get_pretrained(normalize_model_name(model_name, "audiogen-medium"))
|
model = AudioGen.get_pretrained(str(model_path.resolve()))
|
||||||
model.set_generation_params(duration=max(0.5, float(duration)))
|
model.set_generation_params(duration=max(0.5, float(duration)))
|
||||||
audio = model.generate([prompt])[0]
|
audio = model.generate([prompt])[0]
|
||||||
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
|
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
|
||||||
|
|
@ -110,14 +99,6 @@ def synthesize_audiocraft(kind: str, prompt: str, output: Path, duration: float,
|
||||||
except Exception:
|
except Exception:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def normalize_model_name(model_name: str, default_name: str) -> str:
|
|
||||||
value = model_name.strip() if model_name else default_name
|
|
||||||
if "/" in value:
|
|
||||||
return value
|
|
||||||
return f"facebook/{value}"
|
|
||||||
|
|
||||||
|
|
||||||
def write_tensor_audio(audio, output: Path, sample_rate: int) -> None:
|
def write_tensor_audio(audio, output: Path, sample_rate: int) -> None:
|
||||||
if np is None:
|
if np is None:
|
||||||
raise RuntimeError("numpy is required to write model-generated audio")
|
raise RuntimeError("numpy is required to write model-generated audio")
|
||||||
|
|
@ -136,26 +117,6 @@ def write_tensor_audio(audio, output: Path, sample_rate: int) -> None:
|
||||||
write_wave_file(output, audio, sample_rate)
|
write_wave_file(output, audio, sample_rate)
|
||||||
|
|
||||||
|
|
||||||
def write_fallback_tone(output: Path, duration: float, frequency: float, volume: float) -> None:
|
|
||||||
sample_rate = 48000
|
|
||||||
total_samples = int(sample_rate * duration)
|
|
||||||
frames = array(
|
|
||||||
"h",
|
|
||||||
[
|
|
||||||
int(max(-1.0, min(1.0, math.sin(2.0 * math.pi * frequency * (i / sample_rate))) * volume) * 32767.0)
|
|
||||||
for i in range(total_samples)
|
|
||||||
],
|
|
||||||
)
|
|
||||||
write_wave_file(output, frames, sample_rate, channels=1)
|
|
||||||
|
|
||||||
|
|
||||||
def write_silence(output: Path, duration: float) -> None:
|
|
||||||
sample_rate = 24000
|
|
||||||
total_samples = int(sample_rate * duration)
|
|
||||||
frames = array("h", [0] * total_samples)
|
|
||||||
write_wave_file(output, frames, sample_rate, channels=1)
|
|
||||||
|
|
||||||
|
|
||||||
def write_wave_file(output: Path, waveform, sample_rate: int, channels: int | None = None) -> None:
|
def write_wave_file(output: Path, waveform, sample_rate: int, channels: int | None = None) -> None:
|
||||||
if hasattr(waveform, "ndim") and np is not None:
|
if hasattr(waveform, "ndim") and np is not None:
|
||||||
waveform = np.asarray(waveform)
|
waveform = np.asarray(waveform)
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue