477 lines
33 KiB
Markdown
477 lines
33 KiB
Markdown
---
|
|
name: video-editing-cinematic-highlights-campaign
|
|
description: "Load this skill when planning, implementing, debugging, or promoting the single-source cinematic highlight campaign: local-only highlight identification and editing with production visuals, music, SFX, and voiceover; especially for candidate generation, an offline director, model packaging, director-plan safety, measured media QA, resumability, source-to-final tests, or macOS/Linux parity."
|
|
---
|
|
|
|
# Cinematic Highlights Campaign
|
|
|
|
Status verified against the repository on **2026-07-21**. Treat every path or component marked
|
|
`[PROPOSED]` as work that does not exist yet. Introduce it only through
|
|
`video-editing-change-control`.
|
|
|
|
## Use this campaign correctly
|
|
|
|
Use this skill to drive the hardest live problem: turn one source video into correctly selected,
|
|
high-cinematic-quality highlights with production visuals, music, sound effects (SFX), and
|
|
voiceover, using only pre-provisioned local models resident in the service runtime.
|
|
|
|
Do **not** use it for:
|
|
|
|
| Need | Use instead |
|
|
|---|---|
|
|
| A routine, already-understood patch | `video-editing-change-control` |
|
|
| A symptom with no established root cause | `video-editing-debugging-playbook` |
|
|
| Media theory or FFmpeg fundamentals | `cinematic-media-engineering-reference` |
|
|
| General test evidence or existing test inventory | `video-editing-validation-and-qa` |
|
|
| Existing measurement commands and artifact inspection | `video-editing-diagnostics-and-tooling` |
|
|
| Post-triage causal proof or first-principles derivation | `video-editing-proof-and-analysis-toolkit` |
|
|
| Environment setup or normal operation | `video-editing-build-and-env` or `video-editing-run-and-operate` |
|
|
|
|
Never use this campaign to route around approval, licensing, security, or change control.
|
|
|
|
## Terms and hard constraints
|
|
|
|
| Term | Meaning here |
|
|
|---|---|
|
|
| Certified fixture | Rights-cleared source media plus immutable annotations, checksum, provenance, and an assigned development or locked-holdout split. |
|
|
| Temporal IoU (tIoU) | Intersection duration divided by union duration for a predicted and annotated highlight interval. |
|
|
| Recall@3 | Fraction of annotated highlights matched by one of the top three predictions at the stated tIoU. |
|
|
| Macro-F1 | Unweighted mean of per-category F1 scores, so a large category cannot hide a weak one. |
|
|
| nDCG@3 | Normalized discounted cumulative gain for the top three ranked candidates; higher relevance near rank one receives more credit. |
|
|
| SLO | Service-level objective: an approved target for a measured service-level indicator over a stated workload/window. |
|
|
| SBOM | Software Bill of Materials: an inventory of shipped software components and their identities. |
|
|
| Fail closed | Stop before rendering or promotion when required evidence, models, assets, approval, or QA is missing. |
|
|
| Local model | A model whose weights, loader, tokenizer, and runtime dependencies are pre-provisioned in the deployed artifact or mounted runtime bundle; loading it performs no network operation. |
|
|
| Creative rubric | A blinded 0-4 human score for story, selection, pacing, visual craft, sound design, voiceover, and factual grounding. |
|
|
|
|
The no-waiver rules are authoritative in `video-editing-change-control`; these are their campaign consequences:
|
|
|
|
- Make no automatic dependency or model downloads. Do not call external AI services.
|
|
- Make no network connection during the certified offline run, including loopback and model
|
|
resolution. The current CV worker uses loopback HTTP; replace that transport under change control
|
|
before certification.
|
|
- Use only assets with recorded licenses and permitted production use.
|
|
- Produce no placeholder silence, tones, generic assets, or unrequested fallback render.
|
|
- Render only after explicit approval. Do not change packaged production-facing defaults.
|
|
- Do not treat a successful FFmpeg exit or a JSON field set to `true` as quality evidence.
|
|
- Keep every experiment isolated behind an explicit, disabled-by-default campaign control. Promotion
|
|
still follows `video-editing-change-control`.
|
|
|
|
## Current truth: do not design from the plan documents
|
|
|
|
| Repository fact | Evidence and consequence |
|
|
|---|---|
|
|
| `HighlightSourceScheduler` calls `HighlightCandidateGenerator` after source analysis and before `HighlightDirectorPromptGenerator`. | The single-source flow now persists `analysis/category.json` and `analysis/highlight-candidates.json`; missing files mean candidate generation failed or the project predates this change. |
|
|
| `HighlightCandidateGenerator` is a deterministic baseline; `CinematicHighlightAnalyzer` remains the older multi-clip ranker. | The single-source generator ranks shot/coverage windows from source-level visual scores, FFmpeg silence sections, scene score, duration, and position. It explicitly downweights heuristic/fallback visual evidence; do not call it semantic highlight understanding. |
|
|
| No local director executes the generated highlight prompt. | `director/director-brief.md` tells an operator to use Codex, Claude, or another agent; rendering begins only when `director/edit-plan.json` appears. |
|
|
| The highlight plan has a dedicated `HighlightDirectorPlanValidator`; it does not use the multi-clip `EditPlanValidator`. | It requires persisted category/candidates, safe unique IDs, candidate-contained time ranges, configured duration bounds, and nonblank visual/music/SFX/voiceover direction before asset work. |
|
|
| Plan-derived `highlightId` and mutable asset-request paths are validated. | `HighlightLocalAssetWorker` requires exact project/highlight identity, supported types, safe keys, bounded duration, contract-exact request/target paths, project containment, and no symlink component before materialization. Adjacent nonblank asset-license sidecars are required and preserved; authenticity, checksum, origin, allowed-use validation, and request-file integrity signatures remain open. |
|
|
| Standalone bootstrap scripts remain network-capable. | Both worker shell scripts run `pip install` in `auto` mode; the CV script defaults to `yolov8n.pt`. `LocalAssetRuntimeVerifier` no longer invokes the asset bootstrap, but operators must still not run either launcher in a certified environment. |
|
|
| Strict asset readiness now fails startup. | The 2026-07-21 working tree requires pre-provisioned Piper, MusicGen, and AudioGen paths and throws when strict readiness is incomplete. No such three-model bundle is present in this repository. |
|
|
| Placeholder asset success was removed. | The Python and Java asset workers now return failure and delete invalid output instead of emitting OS speech, silence, or tones. Missing requested music, SFX, or voiceover blocks rendering. |
|
|
| Audio and preview failures now fail the highlight render. | The renderer no longer copies the pre-mix timeline or final output after those failures. Prepared SFX keys and rendered SFX keys are shared. |
|
|
| Highlight QA now includes selected media probes but is not a creative-quality gate. | Final duration, black ranges, long silence, and sample peaks are probed. Integrated loudness, true peak, A/V sync, freeze detection, raster safe area, asset semantics, and human creative acceptance remain open. |
|
|
| The source-to-final test is incomplete. | `HighlightDirectorFlowServiceTest` uses a mocked renderer and dummy files. `CinematicEditingIntegrationTest` covers the separate multi-clip flow. |
|
|
| Processing is not stage-resumable. | The scanner skips only projects with root `final.mp4`; there is no per-stage journal, input digest, lease, or verified checkpoint. |
|
|
|
|
Reconfirm those claims before opening a campaign change:
|
|
|
|
```bash
|
|
rg -n "HighlightCandidateGenerator|category.json|highlight-candidates.json" \
|
|
src/main/java/org/example/videoclips/editing/{HighlightSourceScheduler,HighlightCandidateGenerator,HighlightDirectorPromptGenerator}.java
|
|
rg -n "planValidator|HighlightDirectorPlanValidator|highlightId\(\)|targetPath\(\)" \
|
|
src/main/java/org/example/videoclips/editing/{HighlightDirectorFlowService,HighlightDirectorPlanValidator,HighlightLocalAssetWorker}.java
|
|
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded|duration_matches_timeline" \
|
|
tools src/main/java/org/example/videoclips/editing
|
|
```
|
|
|
|
Expected on the 2026-07-21 working tree: matches in all three commands; explicit scheduler wiring to
|
|
`HighlightCandidateGenerator`; no placeholder-success implementation in the asset synthesizers; network-capable
|
|
launcher matches remain non-zero. Request target strings remain in the code but are now checked against exact
|
|
project-local paths before use. If that changes, update this campaign before proceeding.
|
|
|
|
## Campaign gate map
|
|
|
|
Do not start a later phase until the preceding gate is green.
|
|
|
|
| Phase | Deliverable | Numeric gate |
|
|
|---:|---|---|
|
|
| 0 | Frozen offline baseline | 245 tests in 62 test classes; the full offline Maven suite has 0 failures, 0 errors, and 0 skips |
|
|
| 1 | Certified fixture and annotation set | 4 current categories represented; 2 independent annotations per source; 100% checksums/licenses/splits present |
|
|
| 2 | Single-source category and candidate wiring | Macro-F1 and Recall@3 thresholds calibrated and recorded; every processed fixture emits exactly 1 category file and 1 candidate file |
|
|
| 3 | Resident local director and safe plan | 0 network operations; 100% schema-valid plans; all adversarial plans rejected before filesystem/render work |
|
|
| 4 | Fail-closed model and asset runtime | 100% manifest/checksum/license/canary pass; any one missing capability prevents readiness and render |
|
|
| 5 | Production render semantics | Every requested effect/audio/VO action either has execution evidence or fails the render; 0 silent degradations |
|
|
| 6 | Measured technical and creative QA | 0 ERROR checks; creative average >=3.5/4 and no dimension <3; paired-preference lower 95% bound >50% |
|
|
| 7 | Resumability and real source-to-final test | 1 final per requested highlight, 0 duplicate side effects, and exact recovery after every injected stage interruption |
|
|
| 8 | Offline platform, security, performance, operations | macOS and Linux/VPS outputs pass identical gates; 0 attempted network calls; 0 path escapes; workload SLOs met |
|
|
| 9 | Controlled promotion | All change-control evidence present; human reference-architecture rubric has no score <3 and average >=3.5 |
|
|
|
|
## Phase 0: freeze an offline baseline
|
|
|
|
1. Work from the repository root. Record, but do not mutate, source and tool state:
|
|
|
|
```bash
|
|
git status --short
|
|
git rev-parse HEAD
|
|
java -version
|
|
mvn -version
|
|
ffmpeg -version | head -n 1
|
|
ffprobe -version | head -n 1
|
|
printf 'test_methods='; rg -n '^\s*@Test\b' src/test/java | wc -l
|
|
```
|
|
|
|
2. Run the current tests that do not invoke the network-capable model/asset bootstrap paths:
|
|
|
|
```bash
|
|
mvn -q -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
|
|
```
|
|
|
|
Gate 0 expectations for the current working tree: Maven exits `0`. Treat generated test media as
|
|
disposable, not certified evidence. A 2026-07-21 `mvn -q -o verify` run executed all 245 tests in 62 test classes with
|
|
zero failures, errors, or skips and passed the bound JaCoCo gate. Maven offline mode does not prevent
|
|
subprocess networking, so call this offline dependency-resolution evidence, not no-egress certification.
|
|
|
|
- If offline Maven reports a missing artifact, branch to `video-editing-build-and-env`; pre-provision a
|
|
locked artifact repository or build cache. Do not enable network.
|
|
- If a test fails because local voice/TTS behavior differs, branch to Phase 4. The current test suite
|
|
can consume an untracked local environment and is not yet clean-checkout reproducible.
|
|
- If the worktree is dirty, preserve unrelated changes. Record their paths and isolate campaign
|
|
evidence; never reset them.
|
|
|
|
## Phase 1: create certified fixtures and annotations
|
|
|
|
The repository has no certified creative golden inventory. Do not promote existing `input/` or
|
|
`output/` media to goldens by assumption.
|
|
|
|
Under change control, add `[PROPOSED]` fixture metadata and small rights-cleared media, or an
|
|
authenticated internal fixture-bundle mount whose immutable digest is recorded. The metadata must
|
|
contain: fixture ID, SHA-256, byte size, duration, codecs, category, rights/license ID, permitted use,
|
|
provenance, split, and annotation version. Keep weight files and large/proprietary footage out of Git.
|
|
|
|
For each source, collect two independent annotations containing category, positive highlight
|
|
intervals, excluded intervals, story role, visible facts allowed in voiceover, unsafe/blurred frames,
|
|
speech intervals, and music/SFX intent. An adjudicator resolves disagreement without seeing model
|
|
output. Lock the holdout split before tuning.
|
|
|
|
Starting floor (a campaign minimum, not a SOTA claim): at least 12 sources, at least 3 each for
|
|
`car_vlog`, `food_vlog`, `family_vlog`, and `generic_vlog`; at least one no-audio and one low-quality
|
|
negative source; 2 annotators per source. Record a larger statistically powered sample before making
|
|
external quality claims.
|
|
|
|
Gate 1 `[PROPOSED]` command; implement a repository script only through change control:
|
|
|
|
```bash
|
|
test -x tools/validate-highlight-fixtures && tools/validate-highlight-fixtures --offline --strict
|
|
```
|
|
|
|
Expected: exit `0`, `sources>=12`, `categories=4`, `annotators_per_source>=2`,
|
|
`missing_checksums=0`, `missing_licenses=0`, `split_overlap=0`. If the script is absent, stop: do not
|
|
replace it with visual inspection. If rights are unclear, quarantine the fixture and branch to legal/
|
|
asset governance.
|
|
|
|
## Phase 2: wire category and candidate generation
|
|
|
|
Implement a single-source application use case between `HighlightSourceAnalyzer.analyze` and prompt
|
|
generation. It must consume persisted source, scene, audio, and visual analysis; classify using actual
|
|
local evidence; generate shot-boundary-aligned candidates; and atomically write exactly:
|
|
|
|
- `analysis/category.json`
|
|
- `analysis/highlight-candidates.json`
|
|
|
|
Do not merely copy the metadata-keyword classifier from the multi-clip path. First define a
|
|
deterministic heuristic baseline, then compare local learned scorers against it. Candidate IDs must be
|
|
stable for the same source/config/model digests. Durations must honor the configured 8-35 second
|
|
range; at most the configured top 3 proceed to planning. Uncertain classification is a review state,
|
|
not permission to render generic content.
|
|
|
|
Before running, predict category confusion, candidate Recall@3 at `tIoU>=0.5`, and ranking nDCG@3 on
|
|
development fixtures. Record predicted numbers, then run once. Set final thresholds from baseline plus
|
|
confidence intervals in an ADR; until that ADR exists, use the campaign floors: macro-F1 `>=0.85`,
|
|
Recall@3 `>=0.80`, nDCG@3 `>=0.80`, and invalid/out-of-range candidates `=0`. These are candidate
|
|
promotion thresholds, not current results.
|
|
|
|
Gate 2 `[PROPOSED]`:
|
|
|
|
```bash
|
|
mvn -o -Dtest=HighlightSourceCandidatePipelineTest,HighlightCandidateEvaluationTest test
|
|
test -x tools/evaluate-highlight-selection && \
|
|
tools/evaluate-highlight-selection --fixtures certified --split holdout --offline
|
|
```
|
|
|
|
Expected: both exit `0`; one category and one candidate artifact per source; the four numeric floors
|
|
above pass. If category passes but Recall@3 fails, branch to segmentation/fusion experiments. If Recall
|
|
passes but nDCG fails, keep candidates and work only on ranking. If all offline metrics pass but human
|
|
selected moments remain weak, inspect annotation agreement before changing the model.
|
|
|
|
## Phase 3: implement a resident local director and validate plans
|
|
|
|
Replace the operator-mediated external-agent step with an explicit application port and a local
|
|
adapter. The adapter must use a pre-provisioned model path, constrained JSON decoding, fixed schema
|
|
version, bounded context/output, deterministic seed where supported, timeout, cancellation, and no
|
|
socket or model-hub resolution. Preserve human approval; the model may propose but never approve.
|
|
|
|
Validate `HighlightDirectorPlan` before creating storyboard, asset requests, or directories:
|
|
|
|
| Required validation | Reject when |
|
|
|---|---|
|
|
| Identity | Project/source mismatch; unknown candidate; duplicate/unsafe highlight ID |
|
|
| Numeric safety | NaN/infinity; negative/reversed/out-of-source time; target outside configured range |
|
|
| Structural bounds | Zero highlights; more than configured maximum; empty/oversized strings or arrays |
|
|
| Semantics | Unknown category, story role, effect, transition, render profile, voice, or asset type |
|
|
| Grounding | Voiceover asserts a fact absent from certified visible/transcript evidence |
|
|
| Path safety | Absolute path, `..`, separator, symlink escape, or caller-supplied target path |
|
|
| Asset closure | A requested music/SFX/voice/font/LUT capability has no licensed local resolution |
|
|
|
|
Derive target paths server-side from validated IDs. Resolve against the project root, normalize,
|
|
verify containment, reject symlinks at trust boundaries, and create through the store. Make the JSON
|
|
schema version explicit and validate it during the Maven build.
|
|
|
|
Gate 3 `[PROPOSED]`:
|
|
|
|
```bash
|
|
mvn -o -Dtest=LocalHighlightDirectorContractTest,HighlightDirectorPlanValidatorTest,HighlightPathSafetyTest test
|
|
```
|
|
|
|
Expected: exit `0`; valid plans accepted `=100%`; malformed, unknown-field, timestamp, NaN/infinity,
|
|
duplicate-ID, `../`, absolute-path, backslash, symlink, oversized-text, unknown-effect, and missing-asset
|
|
cases rejected `=100%`; filesystem writes after a rejected plan `=0`; network operations `=0`.
|
|
|
|
If constrained decoding still emits invalid JSON, do not repair it silently: record the failure and
|
|
stop. If factual grounding cannot be established, omit the line or route to human review.
|
|
|
|
## Phase 4: package and preflight all local capabilities
|
|
|
|
Add `[PROPOSED]` typed configuration for a runtime model root and a signed/immutable model manifest;
|
|
do not overload the current free-form model-name properties. For every CV, ASR, director, TTS, music,
|
|
and SFX model record: purpose, relative path, exact bytes, SHA-256, loader/runtime version, tokenizer/
|
|
config paths, license identifier and license-text digest, approved use, attribution, memory estimate,
|
|
and a deterministic canary input/output predicate.
|
|
|
|
Pre-provision Python wheels and weights in the build/deployment supply chain. Use locked versions and
|
|
hashes. Production startup must never run `pip`, `YOLO("yolov8n.pt")`, or
|
|
`get_pretrained("facebook/...")`. A readiness preflight must verify regular-file containment,
|
|
checksums, licenses, loader compatibility, capacity, and one inference canary for every enabled
|
|
capability. Any failure makes readiness false and blocks project claims and rendering.
|
|
|
|
Gate 4 `[PROPOSED]`:
|
|
|
|
```bash
|
|
test -x tools/verify-local-model-bundle && \
|
|
tools/verify-local-model-bundle --manifest "${VIDEO_EDITING_MODEL_MANIFEST:?set manifest}" --offline --strict
|
|
mvn -o -Dtest=LocalModelManifestTest,OfflineCapabilityPreflightTest,FailClosedAssetGenerationTest test
|
|
```
|
|
|
|
Expected: `models_checked>=6`, `checksum_mismatches=0`, `license_failures=0`, `canary_failures=0`,
|
|
`network_attempts=0`; Maven exits `0`. Re-run after corrupting one byte in a disposable test copy:
|
|
expected preflight exit non-zero, readiness false, renders started `=0`. If a license is absent or
|
|
ambiguous, the model/asset is unavailable, not experimental.
|
|
|
|
Remove production paths to `say`, `espeak`, tones, and silence. Keep negative tests proving these
|
|
cannot satisfy a required asset. Do not edit packaged defaults during the campaign; activate new
|
|
components only with explicit environment configuration in approved environments.
|
|
|
|
## Phase 5: make render instructions real
|
|
|
|
Implement an allowlisted renderer capability registry. A plan may name only behavior the renderer
|
|
can prove it executed. Today, any non-empty visual treatment maps to the same crop/grade and fades are
|
|
per-segment. Audio and preview failures now propagate; keep regression tests on that fail-closed behavior.
|
|
Do not relabel the remaining generic visual behaviors as cinematic.
|
|
|
|
| Plane | Required semantics and evidence |
|
|
|---|---|
|
|
| Selection/edit | Multiple decisions when the story requires them; source/timeline arithmetic verified; true inter-shot transitions rather than independent fades |
|
|
| Visual | Explicit color transform, stabilization, reframing, speed ramp, overlay/font, and transition parameters; command/filter evidence plus output measurements |
|
|
| Music | Licensed/generated asset digest; deliberate in/out points, trim/loop policy, gain envelope, and final mix contribution |
|
|
| SFX | One existing local asset per cue; sample-accurate or measured timing; no missing-path attempt |
|
|
| Voiceover | One grounded script version, approved local voice/model, line timing/alignment, intelligibility measurement, and ducking evidence |
|
|
| Master | Two-pass or otherwise measured loudness normalization; output codec, geometry, cadence, duration, and channel layout probed from the file |
|
|
|
|
Make every requested production asset blocking. On FFmpeg, preview, or mix failure, mark the attempt
|
|
failed and publish no `final.mp4`. Write to a temporary name and atomically promote only after QA.
|
|
Record tool versions, complete argv, exit codes, input/output hashes, models/assets/licenses, timing,
|
|
and effective config in `render-manifest.json`.
|
|
|
|
Gate 5 `[PROPOSED]`:
|
|
|
|
```bash
|
|
mvn -o -Dtest=HighlightRendererIntegrationTest,HighlightAudioMixIntegrationTest,HighlightEffectContractTest test
|
|
```
|
|
|
|
Expected: exit `0`; requested actions `= executed actions`; unresolved actions `=0`; silent
|
|
degradations `=0`; failure injection for each FFmpeg stage publishes final files `=0`. If a filter is
|
|
unsupported on macOS or Linux, do not substitute one; branch to Phase 8 and either standardize FFmpeg
|
|
or remove the capability through change control.
|
|
|
|
## Phase 6: replace asserted QA with observed QA
|
|
|
|
Each QA check must include expected value, observed value, unit, tool/algorithm version, command or
|
|
method, artifact path/digest, severity, and pass/fail. Probe the actual final file. At minimum measure:
|
|
stream presence, duration delta, 1920x1080 geometry, 30 fps cadence, H.264/AAC codecs, 48 kHz audio,
|
|
black/freeze spans, silence spans, clipping/true peak, integrated loudness, overlay bounds, asset
|
|
contribution, voice intelligibility/alignment, and requested-effect execution.
|
|
|
|
Use the current configured mastering targets as initial technical gates: integrated loudness
|
|
`-16 LUFS +/-1`, true peak `<=-1.5 dBTP`, duration error `<=0.25 s`, missing requested streams/assets
|
|
`=0`, and ERROR checks `=0`. Calibrate black/silence/freeze tolerances by content and annotations;
|
|
do not apply a universal threshold that rejects intentional black or quiet moments.
|
|
|
|
Operator spot-check commands (diagnostic only, not a replacement for automated parsing):
|
|
|
|
```bash
|
|
FINAL="${FINAL:?set final mp4}"
|
|
ffprobe -v error -show_entries format=duration:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$FINAL"
|
|
ffmpeg -hide_banner -i "$FINAL" -vf "blackdetect=d=0.25:pix_th=0.10" -af "silencedetect=n=-45dB:d=1.0" -f null -
|
|
ffmpeg -hide_banner -nostats -i "$FINAL" -filter_complex "ebur128=peak=true" -f null -
|
|
```
|
|
|
|
Gate 6 technical command is `[PROPOSED]`; implement the parser through change control rather than
|
|
copying console text into `qa-report.json`:
|
|
|
|
```bash
|
|
test -x tools/verify-highlight-render && \
|
|
tools/verify-highlight-render --final "$FINAL" --manifest "${MANIFEST:?set render manifest}" --strict
|
|
```
|
|
|
|
Expected: exit `0`, `error_checks=0`, `missing_requested_assets=0`, `duration_error_seconds<=0.25`,
|
|
`integrated_lufs` in `[-17,-15]`, and `true_peak_dbtp<=-1.5`. If the tool is absent, Gate 6 is red.
|
|
|
|
Run blinded creative review against the frozen deterministic baseline. Use at least 3 reviewers per
|
|
output; score the seven rubric dimensions 0-4; randomize A/B order; retain disagreements. Gate:
|
|
average `>=3.5`, every dimension median `>=3`, factual errors `=0`, and the 95% confidence interval
|
|
lower bound for preference over baseline `>50%`. If technical QA passes but creative QA fails, do not
|
|
tune FFmpeg blindly: branch according to reviewer tags (selection -> Phase 2, story/VO -> Phase 3,
|
|
craft/audio -> Phase 5).
|
|
|
|
## Phase 7: add resumability and a real source-to-final test
|
|
|
|
Add a durable stage journal with state, attempt, lease owner/expiry, started/completed timestamps,
|
|
code/config/model/input digests, output paths/digests, and failure classification. Required stages:
|
|
claim, analyze, classify, rank, direct, validate, approve, prepare-assets, render, QA, publish. Commit
|
|
each checkpoint atomically. Resume only when all input digests match; otherwise invalidate that stage
|
|
and every dependent stage. Never infer completion from file existence alone.
|
|
|
|
Build a real source-to-final test using a tiny rights-cleared fixture and pre-provisioned test models/
|
|
assets. Do not mock the analyzer, director, asset generator, renderer, FFmpeg, or FFprobe. Test restart
|
|
after every stage, duplicate scans, stale leases, corrupt checkpoints, changed model digest, invalid
|
|
plan, missing asset, FFmpeg failure, and crash between temporary output and atomic publish.
|
|
|
|
Gate 7 `[PROPOSED]`:
|
|
|
|
```bash
|
|
mvn -o -Dtest=HighlightSourceToFinalIT,HighlightWorkflowResumeIT,HighlightWorkflowFailureIT test
|
|
mvn -o verify
|
|
```
|
|
|
|
Expected: exit `0`; final outputs equal requested approved highlights; duplicate renders/uploads `=0`;
|
|
published partial files `=0`; each injected interruption resumes at the first invalid/incomplete stage;
|
|
changing any input digest recomputes all dependent stages. If the test uses a mock renderer or dummy
|
|
text as MP4, it does not satisfy this gate.
|
|
|
|
## Phase 8: prove offline parity, security, performance, and operations
|
|
|
|
Run the same immutable fixture bundle, model bundle, service artifact, config digest, and FFmpeg build
|
|
on macOS development and Linux/VPS. Cloud deployment may mount the same pre-provisioned bundle but
|
|
must not fetch it at application startup. Container work is `[PROPOSED]`; the repository has no current
|
|
container definition, so do not claim `--network none` validation until one is approved.
|
|
|
|
Define the representative workload before measuring: source minutes, resolution/codecs, highlight
|
|
count, concurrent jobs, hardware/accelerator, warm/cold state, and storage. Record per-stage p50/p95/
|
|
p99, real-time factor (wall seconds/source seconds), CPU, accelerator, RSS, disk peak, queue depth,
|
|
failures, and output bytes. Establish SLOs from measured capacity and business demand; never import the
|
|
200-RPS synchronous-API example into a media pipeline.
|
|
|
|
Gate 8 `[PROPOSED]`:
|
|
|
|
```bash
|
|
test -x tools/run-offline-highlight-certification && \
|
|
tools/run-offline-highlight-certification --fixtures certified --models "${VIDEO_EDITING_MODEL_MANIFEST:?}" --strict
|
|
mvn -o -Dtest=HighlightOfflineSecurityIT,HighlightCapacityIT,HighlightGracefulShutdownIT test
|
|
mvn -o verify
|
|
```
|
|
|
|
Expected on both platforms: technical and creative Gate 6 pass; `network_attempts=0`; path escapes
|
|
and symlink escapes `=0`; checksum/license failures `=0`; orphan child processes `=0`; corrupt or
|
|
partial finals `=0`; measured p95 and resource peaks remain within the approved workload SLO. Compare
|
|
semantic decisions and QA outcomes, not bit-identical encoded video unless the codec/toolchain is
|
|
fully standardized. If platform results diverge, capture tool/model/config digests and stop promotion.
|
|
|
|
Security review must threat-model malicious media, decompression/resource exhaustion, hostile model
|
|
output, path traversal/symlinks, command injection, poisoned weights, unlicensed assets, prompt
|
|
injection in transcripts/metadata, denial of service, and sensitive media/log leakage. FFmpeg commands
|
|
remain argv lists, never shell strings. Run workers with least privilege, bounded CPU/memory/time/disk,
|
|
read-only model mounts, project-scoped writable storage, and no network capability.
|
|
|
|
Operations must expose stage duration/failure, queue age/depth, active jobs, model readiness,
|
|
candidate counts, approval waits, render real-time factor, QA failures, disk pressure, and worker
|
|
restarts without source filenames, prompt text, or project IDs as metric labels. Define alerts,
|
|
retention, cancellation, graceful shutdown, recovery, and model rollback before production.
|
|
|
|
## Ranked solution menu
|
|
|
|
Choose with measured evidence, not model popularity.
|
|
|
|
| Rank | Candidate | Use when | Required derivation and disproof |
|
|
|---:|---|---|---|
|
|
| 1 | Local multimodal director over deterministic shot/audio candidates | Resident model fits capacity and materially improves grounded story decisions | Predict memory, latency, Recall@3-to-plan loss, grounding error, and creative preference. Disprove with locked holdout, ablation without visual/audio inputs, adversarial transcripts, and local-only trace. |
|
|
| 2 | Local feature models + deterministic ranker + smaller constrained text director | A full multimodal director is too slow or weakly grounded | Derive feature normalization, fusion weights, calibration, context budget, and failure states. Ablate each feature; require one mechanism to explain positive and negative fixtures. |
|
|
| 3 | Local learned temporal scorer + template/constrained planner | Director variance blocks schema safety but selection can be learned | Prove scorer improves Recall/nDCG and templates improve repeatability without lowering blinded creative scores. Reject if category styles collapse to generic edits. |
|
|
| 4 | Deterministic heuristics and fixed renderer | Establish the baseline and retain an emergency diagnostic comparator only | Specify every weight/rule and sensitivity. Never promote as “high cinematic quality” without independently passing all creative gates. |
|
|
|
|
For any chosen option, the decision record must state: requirement; selected approach; alternatives;
|
|
benefits/trade-offs; operational consequences; security implications; verification; revisit condition.
|
|
No new database, queue, service, framework, or model is justified merely by this campaign.
|
|
|
|
## Known wrong paths: fence these off
|
|
|
|
- Do not use filename/project-style keywords as proof of visual content category.
|
|
- Do not ask an external agent to write the plan and call the workflow local-only.
|
|
- Do not let model libraries resolve names such as `yolov8n.pt` or `facebook/musicgen-small` at runtime.
|
|
- Do not interpret a non-strict readiness log as certification. Strict mode now fails startup, but its
|
|
sidecar and file-presence checks do not establish provenance authenticity, checksums, allowed use, or quality.
|
|
- Do not accept silence, sine tones, OS `say`/`espeak`, missing optional assets, or source-audio-only
|
|
copies when the approved plan requests production audio.
|
|
- Do not use `EditPlanValidator` unchanged for highlight projects; it reads the multi-clip store and
|
|
analysis contract. Reuse validation concepts, not the wrong persistence contract.
|
|
- Do not resolve untrusted plan IDs or request target paths directly with `Path.resolve`/`Path.of`.
|
|
- Do not add more prompt adjectives to fix missing analysis, weak selection, or unsupported effects.
|
|
- Do not call a fixed 4% crop a dynamic crop, per-segment fades a crossfade, or hard-coded `true` QA.
|
|
- Do not tune on the locked holdout, promote unlabeled output, or judge cinematic quality by eye.
|
|
- Do not retry permanent validation/model/license failures; do not resume from file existence alone.
|
|
- Do not alter packaged production defaults to expose campaign behavior.
|
|
|
|
## Phase 9: promotion protocol
|
|
|
|
Route every implementation and final promotion through `video-editing-change-control`. Promotion
|
|
requires one evidence bundle containing baseline commit/tool digests; fixture/annotation version and
|
|
rights; model/dependency SBOM, hashes, licenses, and signatures; predictions made before experiments;
|
|
selection/director/renderer/QA results including negatives and ablations; offline network-denial proof;
|
|
macOS/Linux parity; performance/capacity; threat model; recovery/graceful-shutdown evidence; human
|
|
review data; runbook/alerts; rollback; and ADRs.
|
|
|
|
The project is not campaign-complete until every gate passes on the locked holdout, the Fortune 500
|
|
human-review rubric has no category below 3 and average at least 3.5, security/testing/operational
|
|
readiness are each at least 3, and no critical behavior relies on an undocumented assumption. A model
|
|
upgrade restarts Phases 4, 6, 7, and 8. A schema/renderer change restarts Phases 3, 5, 6, 7, and 8.
|
|
|
|
## Provenance and maintenance
|
|
|
|
Primary code: `HighlightSourceScheduler`, `HighlightSourceAnalyzer`,
|
|
`HighlightDirectorPromptGenerator`, `HighlightDirectorPlanScanner`, `HighlightDirectorFlowService`,
|
|
`HighlightDirectorPlan`, `HighlightLocalAssetWorker`, `LocalAssetSynthesizer`,
|
|
`LocalAssetRuntimeVerifier`, `HighlightFfmpegRenderer`, `FileSystemHighlightProjectStore`, and the two
|
|
worker scripts. Planning documents are context, not implementation truth.
|
|
|
|
Re-verify volatile facts in one line each:
|
|
|
|
```bash
|
|
rg -n '^\s*@Test\b' src/test/java | wc -l
|
|
rg -n "category.json|highlight-candidates.json" src/main/java/org/example/videoclips/editing
|
|
rg -n "pip install|get_pretrained|write_silence|fallbackTone|strict_runtime_degraded" tools src/main/java/org/example/videoclips/editing
|
|
rg -n "duration_matches_timeline|required_assets_resolved|audio_mastering_applied" src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java
|
|
mvn -o -Dtest=HighlightSourceSchedulerTest,HighlightDirectorPromptGeneratorTest,HighlightDirectorFlowServiceTest test
|
|
```
|
|
|
|
Update the dated truth table, baseline counts, thresholds/ADR links, fixture inventory, model manifest
|
|
contract, and platform/SLO evidence whenever those commands or campaign decisions change.
|