diff --git a/.claude/memory/ai-handover.md b/.claude/memory/ai-handover.md new file mode 100644 index 0000000..7a5d89d --- /dev/null +++ b/.claude/memory/ai-handover.md @@ -0,0 +1,65 @@ +# AI Handover: Local Cinematic Highlight PoC + +Last updated: 2026-07-23 (Europe/Berlin). **This file goes stale — verify before trusting it.** + +## FIRST: check for staleness (prevents the 2026-07-23 stale-session incident) + +This handover was once resumed from an old commit while the repo had advanced 2 days on another +branch/agent, wasting a whole session re-deriving already-built work. Do NOT let that happen: + +```bash +git rev-parse --short HEAD # compare to "Recorded HEAD" below +git log -8 --oneline --decorate +git status --short # uncommitted work may be the real current state +``` + +- **Recorded HEAD when this was written: `8b68116`**, plus uncommitted working-tree changes (portrait + + audio-mix + unbounded-duration fixes, see below). +- If HEAD is newer than `8b68116`, or there are unfamiliar commits/uncommitted files, **this file is behind + the repo**. The authoritative live state is: the git log, `git status`, and + `docs/cinematic-highlight-poc-plan.md` (milestone log at its tail). Trust those over this file. +- If a `claude agents` background session is still "running" for this repo, only one agent should mutate the + repo at a time (renders/commits/state). Confirm the others are stopped. + +## What exists NOW (built, not aspirational) + +The full local single-source cinematic-highlight pipeline is implemented and working end-to-end: +analysis → candidates/category → (manual) director plan → local asset generation → cinematic FFmpeg render +→ QA. Models are provisioned and proven (Piper voice, MusicGen music, AudioLDM2 SFX) — see +[[local-model-runtime-intel-mac]]. The user authorized model downloads on 2026-07-21 (this **reversed** the +earlier no-download stance; that reversal is real and current). + +To run source → final render, follow [[highlight-e2e-render-runbook]] — do not re-derive it. + +## Fixes applied 2026-07-23 (whole service, not just PoC) — uncommitted in working tree + +1. **Portrait/orientation**: `FfmpegClipInspector` now reads rotation side-data/`rotate` tag and stores the + EFFECTIVE (display) width/height. `HighlightFfmpegRenderer.outputGeometry()` renders portrait sources to a + portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait phone footage being + stretched into landscape. +2. **Audio mix**: source audio (`[0:a]`) is now ducked (-20 dB, or -24 with narration) so the generated score + leads instead of being buried; montage music cue raised to -9 dB. In `HighlightFfmpegRenderer.audioMixCommand` + + the montage cue in `HighlightDirectorFlowService`. +3. **Duration constraint removed**: `HighlightDirectorPlanValidator` no longer enforces min/max highlight + duration (only positive/finite + sane playback speed + candidate containment). Config defaults relaxed + (min 2 / max 3600 as soft candidate hints only). A highlight can be as long as the story needs. Montage path + was already unconstrained. + +`mvn -o verify`: 254 tests, 0 failures (JaCoCo gate passes). 6 new tests cover the above. NOT yet committed — +the user has not asked to commit. + +## Still open / next levers (from docs/cinematic-highlight-poc-plan.md Phase 5) + +- Source footage quality is the ceiling (e.g. 576p phone clips upscale soft). Biggest lever. +- No resident local **director** — plans are hand-authored (`montage.json`). P5.2/P5.5. +- Bigger/continuous music (musicgen-medium is slow on CPU) — P5.3. +- In-shot `zoompan` push-in, beat-synced cut lengths — P5.4. +- Licensing: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL → non-commercial. Production needs + commercially-licensed models — P5.7. +- Production hardening (security, CI/CD, containers, PostgreSQL, no-egress cert) deferred until quality passes. + +## Non-negotiables still in force (except downloads, which the user authorized) + +No external AI services in the media path; no unlicensed assets shipped as production; no placeholder +silence/tones as success; no rendering without explicit approval; record license/provenance for every model. +Manual approval is fine for the PoC. Do not commit unless the user asks. diff --git a/.claude/memory/current-state.md b/.claude/memory/current-state.md new file mode 100644 index 0000000..f7c898d --- /dev/null +++ b/.claude/memory/current-state.md @@ -0,0 +1,30 @@ +# Video Editing Service: Current State + +Last verified: 2026-07-23 (Europe/Berlin). **Verify against git before trusting — see ai-handover.md.** + +## Snapshot +- Branch `feature/imrovements`; recorded HEAD `8b68116` + uncommitted working-tree fixes (2026-07-23). +- `mvn -o verify`: **254 tests, 0 failures**, JaCoCo gate passes. +- The full local single-source cinematic-highlight pipeline is BUILT and works end-to-end: + analysis → candidates/category → manual `montage.json` director plan → local audio generation + (Piper voice, MusicGen music, AudioLDM2 SFX) → cinematic FFmpeg render (grade, grain, overlays, + multi-cut, slow-mo, orientation-aware geometry) → QA. +- Models provisioned + proven locally; user authorized downloads 2026-07-21 (reversed the no-download stance). + +## Authoritative sources (do not duplicate them here) +- Live code/state: `git log`, `git status`. +- Roadmap + milestone log: `docs/cinematic-highlight-poc-plan.md`. +- How to render e2e: memory `highlight-e2e-render-runbook`. +- Model runtime facts: memory `local-model-runtime-intel-mac`. +- Handover + staleness check: `.claude/memory/ai-handover.md`. + +## 2026-07-23 fixes (whole service) +- Orientation-aware: portrait sources render portrait (rotation read in `FfmpegClipInspector`, + geometry chosen in `HighlightFfmpegRenderer.outputGeometry`), letterbox landscape-only. +- Audio mix: source audio ducked under the generated score (score now leads). +- Highlight duration no longer capped (validator + config); a highlight can be as long as needed. + +## Known limits / next (Phase 5 in poc-plan) +Source footage quality is the ceiling; no resident local director (plans are manual); bigger/continuous +music pending; licensing of local models is non-commercial (CC-BY-NC / AGPL); production hardening deferred +until the creative-quality gate passes. Highlight rendering still defaults OFF; approval still required. diff --git a/docs/cinematic-highlight-poc-plan.md b/docs/cinematic-highlight-poc-plan.md index 0a7e0df..9d6230a 100644 --- a/docs/cinematic-highlight-poc-plan.md +++ b/docs/cinematic-highlight-poc-plan.md @@ -144,21 +144,63 @@ better music, NO voiceover, bolder finish. Done so far (commit 842f2a7): Verified: mvn 247/0; final −16.5 LUFS, TP −2.7 dBTP, 1080p, no VO. Deliverable: `output/localpoc/highlight-projects/dji_21230510013241_0091_d/final.mp4`. +### 2026-07-23 — Second source (bowling strike) + orientation/audio/duration fixes +- New source `bowling_strike.mp4` (portrait phone footage) rendered e2e via montage; confirmed the DJI + Phase-5 finding that **source footage is the ceiling**. +- **Service-wide fixes applied** (not just PoC), all with tests, `mvn -o verify` 254/0: + - **Orientation-aware rendering.** `FfmpegClipInspector` now reads display rotation (side-data + `rotate` + tag) and stores effective width/height; `HighlightFfmpegRenderer.outputGeometry()` renders portrait + sources to a portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait footage + being stretched into landscape. + - **Audio mix.** Source audio (`[0:a]`) ducked to −20 dB (−24 with narration) so the generated score leads + instead of being buried; montage music cue raised to −9 dB. + - **Highlight duration constraint removed.** Validator no longer enforces min/max (only positive/finite + + sane speed + candidate containment); config bounds relaxed to soft candidate hints (min 2 / max 3600). + A highlight can be as long as the story needs. +- E2e render flow captured in memory `highlight-e2e-render-runbook` (token-saver). + ### Phase 5 — Next steps (roadmap as of 2026-07-22, after montage mode) To reach genuinely cinematic output (Gate B): - [ ] P5.1 **Better source footage** (biggest lever; source is the real ceiling — parked car / industrial lot). -- [ ] P5.2 **Automate shot selection** — generate the montage shot list from analysis (scene cuts + CV - quality/variety) instead of the current hand-authored `montage.json`. +- [x] P5.2 **Automate shot selection — Tier 1 (DONE 2026-07-23).** `HighlightMontageDirector` composes + `director/montage.json` from measured motion (YDIF) + audio-energy (RMS) curves: setup → continuous + action/tension → slow-mo payoff on the audio climax → resolution button, camera-whip tail trimmed. Wired + into the scheduler behind `auto-director-enabled` (localpoc on). Verified: auto cut ≈ hand cut on bowling. - [ ] P5.3 **Better music** — musicgen-medium/large (slow) or licensed production track; one produced score. - [ ] P5.4 Polish: proper `zoompan` in-shot push-in; beat-synced cut lengths; better/optional voice. To make it a real system: -- [ ] P5.5 **Resident local director** writes the edit/montage plan (handover Step 3; today plans are manual). +- [x] P5.5 **Resident local director (DONE 2026-07-23).** Two tiers, both offline + auto: Tier-1 + `HighlightMontageDirector` (measured motion+audio → shot list) and Tier-2 `HighlightVisionDirector` + + `tools/vision_caption.py` (moondream2 captions the payoff → semantic overlay "STRIKE" + scene music). + Wired into the scheduler (`auto-director-enabled`, `vision-director-enabled`; localpoc on). Plans are no + longer manual. See `docs/cinematic-quality-rules.md` R9. To reach production (deferred until quality passes): - [ ] P5.6 Formal Gate B human review (rubric in cinematic-highlight-acceptance-review.md). - [ ] P5.7 **Licensing blocker**: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL are non-commercial/ copyleft -> production needs commercially-licensed models/assets. - [ ] P5.8 The hardening backlog below (security, CI/CD, containers, PostgreSQL, no-egress cert, repro build). +### Director review of the bowling portrait cut (2026-07-23) — general craft gaps +Frame-by-frame + audio review. Verdict: technically clean (correct portrait, -15.5 LUFS, TP -2.8, no defects, +score leads) but NOT yet top-tier cinematic. Craft gaps, all fixable as GENERAL pipeline features: +- [x] P5.9 **Adaptive exposure (DONE 2026-07-23).** `probeSourceLuma` measures source frames (signalstats + YAVG); `exposureNormalizationFilter` maps the measured mean to a target band (source-driven gamma); the + grade is now exposure-preserving (black-lift, gamma≥1, brightness≥0, soft vignette). Bowling final went + 63→~100 mean luma with a measured gamma of 1.098. General rules captured in `docs/cinematic-quality-rules.md`. +- [x] P5.10 **Life in held frames (DONE 2026-07-23).** `probeSegmentMotion` (YDIF) measures per-shot motion; + `pushInAmount` maps low motion → stronger in-shot `zoompan` push-in, high motion → gentle (floor keeps a + little life). Applied before `setpts` so slow-mo survives. Bowling: static opening got 0.117 push, active + celebration 0.084. See `docs/cinematic-quality-rules.md` R5. +- [ ] P5.11 **Speed-ramp + crossfades.** The cut into the slow-mo hero is an abrupt speed change and beats are + hard cuts. Ease into slow-mo and add short crossfades on beat boundaries (renderer transition upgrade). +- [ ] P5.12 **Bolder, animated overlay synced to the music hit.** Current overlay is small plain white text; + make it larger/bolder with a scale/fade punch timed to the payoff (general overlay upgrade). +- [ ] P5.13 **Music climax sync.** Align the generated score's peak to the hero beat's timeline position + (pass the payoff timestamp into music prompting/trimming). Ties to P5.3. +- [ ] P5.14 **Show the actual action.** The edit never shows the pins fall (source is on the person). General: + shot-selection (P5.2) should locate the true impact via audio-transient + motion, and source coverage + should include it. + ## Deferred (until output-quality gate passes) Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed ops, digest-bound authenticated approval. Recorded, not deleted. diff --git a/docs/cinematic-quality-rules.md b/docs/cinematic-quality-rules.md new file mode 100644 index 0000000..1e6f74e --- /dev/null +++ b/docs/cinematic-quality-rules.md @@ -0,0 +1,69 @@ +# Cinematic Highlight Quality Rules (source-adaptive) + +Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle: +**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant +tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next. + +Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x). + +## R1 — Exposure: normalize to a target, never crush ✅ +- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma. + `HighlightFfmpegRenderer.probeSourceLuma`. +- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band + (~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down. +- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1, + brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63). +- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098. + +## R2 — Orientation: respect the source, never distort ✅ +- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H. +- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39 + letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch. + +## R3 — Audio balance: the story audio must lead ✅ +- **Rule:** the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with + narration); with narration, voice leads via side-chain ducking. No element is silently buried. +- **Verify:** integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present. + +## R4 — Duration: length follows the story ✅ +- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive + duration + sane playback speed + candidate containment). + +## R5 — No dead air: motion in held shots ✅ +- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source + range. `HighlightFfmpegRenderer.probeSegmentMotion`. +- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a + gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives. +- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active + celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s). + +## R6 — Transitions: ease, don't jerk ⏳ (P5.11) +- Ramp speed into a slow-motion beat (don't hard-switch playback speed); crossfade on beat boundaries where + the cut isn't meant to be a hard cut. + +## R7 — Overlays: bold, animated, synced ⏳ (P5.12) +- Title/label text scaled to the frame, with an entrance (scale/fade) timed to the musical/edit accent — not + small static text dropped on screen. + +## R8 — Music sync: climax on the payoff ⏳ (P5.13) +- **Measure:** the payoff beat's timeline position. +- **Adapt:** prompt/trim the score so its peak lands on the payoff, not wherever the generator happened to put it. + +## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳ +- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve + (YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup → + continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax → + resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled` + (on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut. +- **Tier 2 (semantic VLM director, DONE 2026-07-23):** `HighlightVisionDirector` + `tools/vision_caption.py` + run a local vision-language model (moondream2, offline) to caption the payoff frame, then AUGMENT the Tier-1 + montage with a semantic overlay and a scene-informed music direction. On bowling it read the celebration and + produced the overlay "STRIKE" and a scene-accurate music prompt — automatically. Enabled by + `highlight-scheduler.vision-director-enabled` (localpoc on); ~25s/frame on CPU; fails soft (Tier-1 stands). + moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models). +- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can. + +--- +Rules R1–R5 are live in `HighlightFfmpegRenderer` / `FfmpegClipInspector` / `HighlightDirectorPlanValidator` +and apply to **every** project automatically. R6–R9 are the next implementation targets; each must likewise be +driven by a source measurement, never a per-video constant. diff --git a/src/main/java/org/example/videoclips/config/VideoClippingProperties.java b/src/main/java/org/example/videoclips/config/VideoClippingProperties.java index ca84750..5e9ef28 100644 --- a/src/main/java/org/example/videoclips/config/VideoClippingProperties.java +++ b/src/main/java/org/example/videoclips/config/VideoClippingProperties.java @@ -1212,6 +1212,12 @@ public class VideoClippingProperties { private boolean requireDirectorApproval = true; + /** When true, the Tier-1 automatic director composes director/montage.json after analysis. */ + private boolean autoDirectorEnabled = false; + + /** When true, the Tier-2 vision director augments the montage with a semantic overlay + music. */ + private boolean visionDirectorEnabled = false; + private String approvalFileName = "approved.flag"; @Min(1) @@ -1287,6 +1293,22 @@ public class VideoClippingProperties { this.requireDirectorApproval = requireDirectorApproval; } + public boolean isAutoDirectorEnabled() { + return autoDirectorEnabled; + } + + public void setAutoDirectorEnabled(boolean autoDirectorEnabled) { + this.autoDirectorEnabled = autoDirectorEnabled; + } + + public boolean isVisionDirectorEnabled() { + return visionDirectorEnabled; + } + + public void setVisionDirectorEnabled(boolean visionDirectorEnabled) { + this.visionDirectorEnabled = visionDirectorEnabled; + } + public String getApprovalFileName() { return approvalFileName; } diff --git a/src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java b/src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java index 131a6cb..9bf9f99 100644 --- a/src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java +++ b/src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java @@ -41,6 +41,8 @@ public class FfmpegClipInspector { "-v", "error", "-show_entries", "format=duration", "-show_entries", "stream=codec_type,codec_name,width,height,r_frame_rate", + "-show_entries", "stream_side_data=rotation", + "-show_entries", "stream_tags=rotate", "-of", "json", clip.toString() ); @@ -72,6 +74,14 @@ public class FfmpegClipInspector { if (duration <= 0 || width <= 0 || height <= 0) { throw new IllegalStateException("ffprobe did not report valid clip duration and dimensions"); } + // Respect display rotation: portrait phone footage is often stored as landscape pixels with a + // +/-90 degree rotation flag. Store the EFFECTIVE (display) dimensions so every downstream + // consumer (analysis, prompt, aspect ratio, render geometry) sees the true orientation. + if (Math.floorMod(readRotationDegrees(video), 180) == 90) { + int swap = width; + width = height; + height = swap; + } JsonNode audio = firstStream(root, "audio"); return new ClipAnalysis( clipId(clip), @@ -94,6 +104,31 @@ public class FfmpegClipInspector { } } + /** + * Reads the video display rotation in degrees. Modern containers expose it as a Display Matrix + * side data entry ({@code side_data_list[].rotation}); older files use the {@code tags.rotate} string. + * Returns 0 when no rotation is present. + */ + private int readRotationDegrees(JsonNode video) { + JsonNode sideDataList = video.path("side_data_list"); + if (sideDataList.isArray()) { + for (JsonNode sideData : sideDataList) { + if (sideData.has("rotation")) { + return sideData.path("rotation").asInt(0); + } + } + } + String rotateTag = video.path("tags").path("rotate").asText("").trim(); + if (!rotateTag.isEmpty()) { + try { + return Integer.parseInt(rotateTag); + } catch (NumberFormatException ignored) { + return 0; + } + } + return 0; + } + private JsonNode firstStream(JsonNode root, String codecType) { for (JsonNode stream : root.path("streams")) { if (codecType.equals(stream.path("codec_type").asText())) { diff --git a/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java b/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java index a99f78b..db70a1d 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java +++ b/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java @@ -175,7 +175,7 @@ public class HighlightDirectorFlowService { List audioCues = new ArrayList<>(); if (montage.musicDirection() != null && !montage.musicDirection().isBlank()) { - audioCues.add(new AudioCue("music", safeKey("music", montage.musicDirection()), 0.0, total, -14.0, + audioCues.add(new AudioCue("music", safeKey("music", montage.musicDirection()), 0.0, total, -9.0, montage.musicDirection())); } List overlays = new ArrayList<>(); diff --git a/src/main/java/org/example/videoclips/editing/HighlightDirectorPlanValidator.java b/src/main/java/org/example/videoclips/editing/HighlightDirectorPlanValidator.java index 31d3f9e..d05d756 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightDirectorPlanValidator.java +++ b/src/main/java/org/example/videoclips/editing/HighlightDirectorPlanValidator.java @@ -122,10 +122,12 @@ public class HighlightDirectorPlanValidator { || highlight.sourceEndSeconds() > analysis.source().durationSeconds() + EPSILON) { reject("Highlight source range must be contained by candidate: " + candidate.id()); } - if (!finite(highlight.targetDurationSeconds()) - || highlight.targetDurationSeconds() < effectiveMinimumDuration() - EPSILON - || highlight.targetDurationSeconds() > properties.getHighlightMaxDurationSeconds() + EPSILON) { - reject("Highlight target duration must be within the configured duration bounds"); + // A highlight may be as long as the story needs — there is no fixed min/max duration. The only + // hard requirements are that the target duration is a positive, finite number (checked here), that + // the source range is contained by the candidate and the source (checked above), and that the + // resulting playback speed stays in a sane range (checked below). + if (!finite(highlight.targetDurationSeconds()) || highlight.targetDurationSeconds() <= EPSILON) { + reject("Highlight target duration must be a positive number of seconds"); } double requiredPlaybackSpeed = (highlight.sourceEndSeconds() - highlight.sourceStartSeconds()) / highlight.targetDurationSeconds(); @@ -166,10 +168,6 @@ public class HighlightDirectorPlanValidator { } } - private double effectiveMinimumDuration() { - return Math.min(properties.getHighlightMinDurationSeconds(), properties.getHighlightMaxDurationSeconds()); - } - private ContentCategory parseCategory(String value) { if (value == null || value.isBlank()) { reject("Director plan contentCategory is required"); diff --git a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java index 31804ee..d88eebf 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java +++ b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java @@ -84,6 +84,8 @@ public class HighlightFfmpegRenderer { new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(), plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0) ); + int[] geometry = outputGeometry(sourceAnalysis); + String exposureFilter = exposureNormalizationFilter(probeSourceLuma(source.toString())); List> commands = new ArrayList<>(); List segments = new ArrayList<>(); for (int index = 0; index < plan.decisions().size(); index++) { @@ -93,8 +95,10 @@ public class HighlightFfmpegRenderer { "segment", Integer.toString(index + 1), "source_start", Double.toString(decision.sourceStartSeconds()), "source_end", Double.toString(decision.sourceEndSeconds())); - run(segmentCommand(source.toString(), decision, plan.style(), index, plan.decisions().size(), output), - commands); + double pushIn = pushInAmount(probeSegmentMotion(source.toString(), + decision.sourceStartSeconds(), decision.sourceEndSeconds())); + run(segmentCommand(source.toString(), decision, plan.style(), index, plan.decisions().size(), + geometry[0], geometry[1], exposureFilter, pushIn, output), commands); segments.add(output); log(projectId, highlight.highlightId(), "highlight_render_segment_completed", "segment", Integer.toString(index + 1)); @@ -184,11 +188,39 @@ public class HighlightFfmpegRenderer { return List.copyOf(assets); } + /** + * Chooses render geometry from the source's effective orientation. Portrait sources render to a portrait + * frame (and skip the widescreen letterbox); landscape sources keep the configured widescreen frame. The + * configured output dimensions define the long/short edges regardless of orientation. + */ + int[] outputGeometry(HighlightSourceAnalysis analysis) { + int longEdge = Math.max(properties.getOutputWidth(), properties.getOutputHeight()); + int shortEdge = Math.min(properties.getOutputWidth(), properties.getOutputHeight()); + boolean portrait = analysis != null && analysis.source() != null + && analysis.source().width() > 0 + && analysis.source().height() > analysis.source().width(); + return portrait ? new int[]{shortEdge, longEdge} : new int[]{longEdge, shortEdge}; + } + List segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount, Path output) { + return segmentCommand(source, decision, style, cutIndex, cutCount, + properties.getOutputWidth(), properties.getOutputHeight(), output); + } + + List segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount, + int w, int h, Path output) { + return segmentCommand(source, decision, style, cutIndex, cutCount, w, h, "", output); + } + + List segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount, + int w, int h, String exposureFilter, Path output) { + return segmentCommand(source, decision, style, cutIndex, cutCount, w, h, exposureFilter, 0.0, output); + } + + List segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount, + int w, int h, String exposureFilter, double pushIn, Path output) { double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed(); - int w = properties.getOutputWidth(); - int h = properties.getOutputHeight(); boolean styled = !(decision.visualTreatment() == null || decision.visualTreatment().isBlank() || "none".equalsIgnoreCase(decision.visualTreatment())); StringBuilder filter = new StringBuilder(); @@ -200,16 +232,33 @@ public class HighlightFfmpegRenderer { if (zoom <= 0) { zoom = 1.16 + Math.min(cutIndex, 3) * 0.12; } - filter.append("crop=iw/%.4f:ih/%.4f,scale=%d:%d,format=yuv420p".formatted(zoom, zoom, w, h)); + if (pushIn > 0.001) { + // R5 motion-adaptive in-shot push-in: animate the zoom across the shot so held/low-motion + // frames are never dead air. MUST precede setpts so slow-motion is preserved. Frame count is + // estimated from the source span; min() caps the zoom so any estimate error can't overshoot. + int frames = Math.max(2, (int) Math.round( + (decision.sourceEndSeconds() - decision.sourceStartSeconds()) + * properties.getOutputFrameRate())); + filter.append(("scale=%d:%d,zoompan=z='min(%.4f+%.4f*on/%d\\,%.4f)':d=1" + + ":x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':s=%dx%d,format=yuv420p") + .formatted(w, h, zoom, pushIn, frames - 1, zoom + pushIn, w, h)); + } else { + filter.append("crop=iw/%.4f:ih/%.4f,scale=%d:%d,format=yuv420p".formatted(zoom, zoom, w, h)); + } } else { filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(w, h) + "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(w, h)); } + if (exposureFilter != null && !exposureFilter.isBlank()) { + filter.append(exposureFilter); // adaptive per-source exposure normalization, before the grade + } filter.append(",setpts=PTS/").append(decision.playbackSpeed()); filter.append(cinematicVisualFilter(decision.visualTreatment(), style)); if (styled) { filter.append(",noise=alls=6:allf=t"); // subtle film grain - filter.append(letterboxFilter()); // 2.39:1 cinematic bars + if (w > h) { + filter.append(letterboxFilter()); // 2.39:1 cinematic bars (landscape only) + } } double fadeDuration = Math.min(0.5, outputDuration / 2); if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) { @@ -236,6 +285,103 @@ public class HighlightFfmpegRenderer { )); } + /** + * Assesses the actual frames of the source and returns a DYNAMIC exposure-normalization filter that lifts + * (or lowers) the picture toward a target average brightness. It adapts per source: a dark clip is + * brightened, a well-exposed clip is left alone — nothing is hard-coded to one video. Returns "" when the + * source cannot be measured or is already within the target band. + */ + String exposureNormalizationFilter(double sourceMeanLuma) { + if (sourceMeanLuma <= 0) { + return ""; // could not measure -> make no assumption + } + double target = 120.0; // target mean luma (of 255) for a bright, readable subject + if (Math.abs(sourceMeanLuma - target) <= 8.0) { + return ""; // already well exposed -> leave it + } + double s = Math.min(240.0, Math.max(16.0, sourceMeanLuma)) / 255.0; + double t = target / 255.0; + // out = in^(1/gamma); pick gamma so the measured mean maps to the target mean. gamma>1 brightens. + double gamma = Math.min(2.5, Math.max(0.5, Math.log(s) / Math.log(t))); + return ",eq=gamma=" + String.format(Locale.ROOT, "%.3f", gamma); + } + + /** + * Measures the source's mean luma (YAVG, 0-255) by sampling frames across the clip with ffmpeg signalstats. + * Bounded (2 fps, first 60s) so it stays cheap. Returns -1 when it cannot be determined, in which case the + * caller makes no exposure adjustment. + */ + double probeSourceLuma(String source) { + List command = List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", + "-i", source, "-an", "-vf", "fps=2,signalstats,metadata=print:key=lavfi.signalstats.YAVG", + "-t", "60", "-f", "null", "-"); + try { + Process process = new ProcessBuilder(command).redirectErrorStream(true).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + process.waitFor(); + java.util.regex.Matcher matcher = YAVG_TOKEN.matcher(out); + double sum = 0; + int count = 0; + while (matcher.find()) { + sum += Double.parseDouble(matcher.group(1)); + count++; + } + return count == 0 ? -1 : sum / count; + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + return -1; + } catch (IOException | RuntimeException ex) { + return -1; + } + } + + /** + * R5: converts a measured per-shot motion (mean YDIF) into an in-shot push-in amount. Low-motion / held + * shots get a stronger push-in so they never read as dead air; active shots get only a gentle one. A + * failed/unknown measurement (-1) yields a mild default push so there is always some life. + */ + double pushInAmount(double motionYdif) { + double gentle = 0.05; + if (motionYdif < 0) { + return gentle; // could not measure -> a mild default push + } + double lowMotion = 2.5; + double highMotion = 9.0; + double normalized = Math.min(1.0, Math.max(0.0, (motionYdif - lowMotion) / (highMotion - lowMotion))); + double maxPush = 0.12; + double floor = 0.03; + return Math.max(floor, maxPush * (1.0 - normalized)); + } + + /** + * Measures per-shot motion as the mean temporal luma difference (YDIF via signalstats) over the shot's + * source range. Higher = more motion. Returns -1 when it cannot be determined. + */ + double probeSegmentMotion(String source, double startSeconds, double endSeconds) { + List command = List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", + "-ss", Double.toString(Math.max(0.0, startSeconds)), "-to", Double.toString(endSeconds), + "-i", source, "-an", "-vf", "signalstats,metadata=print:key=lavfi.signalstats.YDIF", + "-f", "null", "-"); + try { + Process process = new ProcessBuilder(command).redirectErrorStream(true).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + process.waitFor(); + java.util.regex.Matcher matcher = YDIF_TOKEN.matcher(out); + double sum = 0; + int count = 0; + while (matcher.find()) { + sum += Double.parseDouble(matcher.group(1)); + count++; + } + return count == 0 ? -1 : sum / count; + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + return -1; + } catch (IOException | RuntimeException ex) { + return -1; + } + } + List overlayCommand(Path input, List overlays, Path output) { return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(), "-vf", overlayFilter(overlays), @@ -268,13 +414,21 @@ public class HighlightFfmpegRenderer { List command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", timeline.toString())); List labels = new ArrayList<>(); - if (sourceAudioPresent) { - labels.add("[0:a]"); - } int input = 1; StringBuilder filters = new StringBuilder(); boolean hasMusic = music != null; boolean hasVoiceover = !voiceovers.isEmpty(); + if (sourceAudioPresent) { + // Source audio is a background bed. When a generated score is present it must sit clearly + // under the music (and further under any narration), not drown it. Attenuate accordingly. + double sourceBedDb = !hasMusic ? 0.0 : (hasVoiceover ? -24.0 : -20.0); + if (sourceBedDb == 0.0) { + labels.add("[0:a]"); + } else { + filters.append("[0:a]volume=").append(sourceBedDb).append("dB[src_bed];"); + labels.add("[src_bed]"); + } + } if (music != null) { command.addAll(List.of("-i", music.toString())); filters.append("[").append(input).append(":a]"); @@ -562,29 +716,33 @@ public class HighlightFfmpegRenderer { // Beat-specific filmic grade. All variants share the teal-shadow / warm-highlight ("teal-orange") // language but shift contrast, warmth and vignette to match the story beat encoded in the plan // style key (safeKey: style__). Unknown/rising -> balanced base grade. + // EXPOSURE-PRESERVING grades. The stylistic look (teal-orange, contrast, vignette) must not crush + // overall brightness: a fixed dark grade turned a well-exposed source (mean luma ~111/255) into ~63. + // Every variant now lifts the black point (curves 0/0.06), keeps gamma >= 1 and brightness >= 0, and + // uses a soft vignette, so the stylistic look never crushes overall exposure. String beat = style == null ? "" : style.toLowerCase(Locale.ROOT); if (beat.contains("opening")) { // Opening hook: calmer, cooler, softer contrast to invite the viewer in. - return ",curves=preset=linear_contrast" + return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'" + ",colorbalance=rs=-0.05:bs=0.08:rm=0.01:bm=-0.01:rh=0.03:bh=-0.03" - + ",eq=contrast=1.08:saturation=1.08:gamma=0.98:brightness=-0.008" + + ",eq=contrast=1.05:saturation=1.08:gamma=1.04:brightness=0.015" + ",unsharp=5:5:0.3:3:3:0.15" - + ",vignette=PI/7"; + + ",vignette=PI/10"; } if (beat.contains("hero")) { - // Hero payoff: richest grade, stronger S-curve, warmer highlights, deeper vignette. - return ",curves=preset=strong_contrast" + // Hero payoff: richest grade, warmer highlights, but shadows lifted so the subject stays visible. + return ",curves=preset=medium_contrast,curves=all='0/0.06 1/1'" + ",colorbalance=rs=-0.03:bs=0.05:rm=0.01:bm=-0.01:rh=0.09:bh=-0.07" - + ",eq=contrast=1.16:saturation=1.16:gamma=0.96:brightness=-0.015" + + ",eq=contrast=1.08:saturation=1.16:gamma=1.06:brightness=0.02" + ",unsharp=5:5:0.5:3:3:0.25" - + ",vignette=PI/5.5"; + + ",vignette=PI/9"; } // Rising energy / default: balanced base grade. - return ",curves=preset=medium_contrast" + return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'" + ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05" - + ",eq=contrast=1.12:saturation=1.12:gamma=0.97:brightness=-0.012" + + ",eq=contrast=1.07:saturation=1.12:gamma=1.05:brightness=0.018" + ",unsharp=5:5:0.4:3:3:0.2" - + ",vignette=PI/6"; + + ",vignette=PI/9"; } private List voiceoverInputs(EditPlan plan, Path audioDirectory) { @@ -621,6 +779,10 @@ public class HighlightFfmpegRenderer { private static final java.util.regex.Pattern ZOOM_TOKEN = java.util.regex.Pattern.compile("zoom=([0-9]+(?:\\.[0-9]+)?)"); + private static final java.util.regex.Pattern YAVG_TOKEN = + java.util.regex.Pattern.compile("YAVG=([0-9]+(?:\\.[0-9]+)?)"); + private static final java.util.regex.Pattern YDIF_TOKEN = + java.util.regex.Pattern.compile("YDIF=([0-9]+(?:\\.[0-9]+)?)"); // 2.39:1 cinematic letterbox: crop the center band, then pad back to frame with black bars. private String letterboxFilter() { diff --git a/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java b/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java new file mode 100644 index 0000000..b85c61f --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java @@ -0,0 +1,239 @@ +package org.example.videoclips.editing; + +import org.example.videoclips.config.VideoClippingProperties; +import org.springframework.stereotype.Component; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; + +/** + * Tier-1 automatic director. It inspects MEASURED signals from the source — per-window motion (temporal luma + * difference) and audio energy (RMS) — and composes a story-structured montage from them: + * + *
    + *
  1. setup (before the action);
  2. + *
  3. a continuous action + tension shot (release/roll/watch — never chopped);
  4. + *
  5. a short realization shot;
  6. + *
  7. a slow-motion payoff centered on the audio climax;
  8. + *
  9. a resolution button after the climax;
  10. + *
+ * + *

while trimming a high-motion tail (e.g. a camera whip). Every decision is derived from a measurement of + * this source — nothing is hard-coded to one video. Semantic captions/overlays and emotional nuance (e.g. + * "he wasn't sure it was a strike") are deliberately NOT attempted here; that is the Tier-2 vision-language + * director's job. This tier gives a strong, deterministic, offline baseline cut. + */ +@Component +public class HighlightMontageDirector { + + static final double WINDOW_SECONDS = 0.5; + + private final VideoClippingProperties.Editing properties; + + public HighlightMontageDirector(VideoClippingProperties properties) { + this.properties = properties.getEditing(); + } + + /** Measure the source and compose an automatic montage plan. */ + public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds) { + double[] motion = probeCurve(source.toString(), true, durationSeconds); + double[] audio = probeCurve(source.toString(), false, durationSeconds); + return composeMontage(projectId, sourceFileName, motion, audio, WINDOW_SECONDS, durationSeconds); + } + + /** + * Pure, deterministic shot composition from the measured curves. Package-visible for unit testing without + * ffmpeg. {@code motion[i]} and {@code audio[i]} cover the window starting at {@code i * window} seconds. + */ + MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio, + double window, double duration) { + int n = Math.min(motion.length, audio.length); + if (n < 4 || duration <= 0) { + return straightCut(projectId, sourceFileName, duration); + } + double[] m = smooth(motion, n); + double[] a = smooth(audio, n); + + // Climax = loudest sustained audio in the central band (skip the intro and the messy tail). + int climaxIdx = argMax(a, (int) Math.floor(0.25 * n), (int) Math.ceil(0.80 * n)); + double climaxTime = climaxIdx * window; + + // Action = the biggest motion spike before the climax; the action shot leads in ~1.5s before it so + // the release is included and the release->roll->watch stays continuous. + int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)), + Math.max(2, (int) Math.floor((climaxTime - 1.0) / window))); + double actionTime = actionIdx * window; + + // Trim a high-motion tail (camera whip): the first strong post-climax motion rise well above the + // celebration's own motion level. + double baseMotion = mean(m, 0, Math.max(1, climaxIdx)); + double tailThreshold = baseMotion * 2.4; + double tailStart = duration; + for (int i = climaxIdx + 4; i < n; i++) { + if (m[i] > tailThreshold) { + tailStart = i * window; + break; + } + } + double endLimit = Math.min(Math.min(duration, tailStart), climaxTime + 4.5); + + List shots = new ArrayList<>(); + addShot(shots, actionTime - 3.2, actionTime - 1.5, 1.03, 1.0, duration); // setup + addShot(shots, actionTime - 1.5, climaxTime - 0.6, 1.04, 1.0, duration); // action + tension + addShot(shots, climaxTime - 0.6, climaxTime + 0.2, 1.06, 1.0, duration); // realization + double payoffEnd = Math.min(climaxTime + 2.2, endLimit - 0.9); + addShot(shots, climaxTime + 0.2, payoffEnd, 1.05, 0.7, duration); // slow-mo payoff + addShot(shots, payoffEnd, endLimit, 1.04, 0.9, duration); // resolution button + + if (shots.isEmpty()) { + return straightCut(projectId, sourceFileName, duration); + } + return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), List.of(), shots); + } + + private MontagePlan straightCut(String projectId, String sourceFileName, double duration) { + double d = Math.max(0.5, Math.min(duration <= 0 ? 6.0 : duration, 8.0)); + return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), List.of(), + List.of(new MontagePlan.Shot(0.0, d, 1.03, 1.0))); + } + + private void addShot(List shots, double srcStart, double srcEnd, + double zoom, double speed, double duration) { + double s = clamp(srcStart, 0.0, duration); + double e = clamp(srcEnd, 0.0, duration); + double span = e - s; + if (span < 0.4) { + return; // skip degenerate shots + } + shots.add(new MontagePlan.Shot(round(s), round(span / speed), zoom, speed)); + } + + private String genericMusic() { + return "cinematic build, quiet tense strings rising to a triumphant orchestral and percussion hit at " + + "the climax, then a short warm resolve, modern trailer film score, no vocals"; + } + + // --- measurement --------------------------------------------------------------------------------------- + + /** + * Measures a per-window curve over the source: motion (signalstats YDIF, temporal luma diff) when + * {@code motion} is true, else audio energy (astats RMS in dB). Bins samples into {@code WINDOW_SECONDS} + * buckets. Missing buckets get a floor value. Returns an empty array when nothing could be measured. + */ + double[] probeCurve(String source, boolean motion, double duration) { + int size = Math.max(1, (int) Math.ceil(duration / WINDOW_SECONDS)); + List command = motion + ? List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-an", + "-vf", "signalstats,metadata=print:key=lavfi.signalstats.YDIF", "-f", "null", "-") + : List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-vn", + "-af", "astats=metadata=1:reset=1,ametadata=print:key=lavfi.astats.Overall.RMS_level", + "-f", "null", "-"); + String token = motion ? "YDIF" : "RMS_level"; + double floor = motion ? 0.0 : -90.0; + double[] sum = new double[size]; + int[] count = new int[size]; + try { + Process process = new ProcessBuilder(command).redirectErrorStream(true).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + process.waitFor(); + double lastPts = -1; + for (String line : out.split("\\R")) { + int pts = line.indexOf("pts_time:"); + if (pts >= 0) { + lastPts = parseTrailingNumber(line.substring(pts + "pts_time:".length())); + continue; + } + int tok = line.indexOf(token + "="); + if (tok >= 0 && lastPts >= 0) { + double value = parseTrailingNumber(line.substring(tok + token.length() + 1)); + int idx = Math.min(size - 1, (int) Math.floor(lastPts / WINDOW_SECONDS)); + if (idx >= 0 && !Double.isNaN(value)) { + sum[idx] += value; + count[idx]++; + } + } + } + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + return new double[0]; + } catch (IOException | RuntimeException ex) { + return new double[0]; + } + double[] curve = new double[size]; + boolean any = false; + for (int i = 0; i < size; i++) { + curve[i] = count[i] == 0 ? floor : sum[i] / count[i]; + any |= count[i] > 0; + } + return any ? curve : new double[0]; + } + + private static double parseTrailingNumber(String text) { + int i = 0; + StringBuilder sb = new StringBuilder(); + String trimmed = text.trim(); + while (i < trimmed.length()) { + char c = trimmed.charAt(i); + if (Character.isDigit(c) || c == '.' || c == '-' || c == '+' || c == 'e' || c == 'E') { + sb.append(c); + i++; + } else { + break; + } + } + try { + return sb.length() == 0 ? Double.NaN : Double.parseDouble(sb.toString()); + } catch (NumberFormatException ex) { + return Double.NaN; + } + } + + // --- small numeric helpers ---------------------------------------------------------------------------- + + private static double[] smooth(double[] values, int n) { + double[] out = new double[n]; + for (int i = 0; i < n; i++) { + double s = 0; + int c = 0; + for (int j = Math.max(0, i - 1); j <= Math.min(n - 1, i + 1); j++) { + s += values[j]; + c++; + } + out[i] = s / c; + } + return out; + } + + private static int argMax(double[] values, int from, int to) { + int lo = Math.max(0, from); + int hi = Math.min(values.length, Math.max(lo + 1, to)); + int best = lo; + for (int i = lo; i < hi; i++) { + if (values[i] > values[best]) { + best = i; + } + } + return best; + } + + private static double mean(double[] values, int from, int to) { + int lo = Math.max(0, from); + int hi = Math.min(values.length, Math.max(lo + 1, to)); + double s = 0; + for (int i = lo; i < hi; i++) { + s += values[i]; + } + return s / (hi - lo); + } + + private static double clamp(double v, double lo, double hi) { + return Math.max(lo, Math.min(hi, v)); + } + + private static double round(double v) { + return Math.round(v * 1000.0) / 1000.0; + } +} diff --git a/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java b/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java index aa082d4..fb34333 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java +++ b/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java @@ -41,6 +41,8 @@ public class HighlightSourceScheduler { private final HighlightSourceAnalyzer analyzer; private final HighlightCandidateGenerator candidateGenerator; private final HighlightDirectorPromptGenerator directorPromptGenerator; + private final HighlightMontageDirector montageDirector; + private final HighlightVisionDirector visionDirector; private final Clock clock; private final AtomicBoolean scanning = new AtomicBoolean(false); private final AtomicLong scanSequence = new AtomicLong(); @@ -51,9 +53,12 @@ public class HighlightSourceScheduler { HighlightProjectStore store, HighlightSourceAnalyzer analyzer, HighlightCandidateGenerator candidateGenerator, - HighlightDirectorPromptGenerator directorPromptGenerator + HighlightDirectorPromptGenerator directorPromptGenerator, + HighlightMontageDirector montageDirector, + HighlightVisionDirector visionDirector ) { - this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, Clock.systemUTC()); + this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, montageDirector, + visionDirector, Clock.systemUTC()); } HighlightSourceScheduler( @@ -62,6 +67,8 @@ public class HighlightSourceScheduler { HighlightSourceAnalyzer analyzer, HighlightCandidateGenerator candidateGenerator, HighlightDirectorPromptGenerator directorPromptGenerator, + HighlightMontageDirector montageDirector, + HighlightVisionDirector visionDirector, Clock clock ) { this.properties = properties.getEditing().getHighlightScheduler(); @@ -69,6 +76,8 @@ public class HighlightSourceScheduler { this.analyzer = analyzer; this.candidateGenerator = candidateGenerator; this.directorPromptGenerator = directorPromptGenerator; + this.montageDirector = montageDirector; + this.visionDirector = visionDirector; this.clock = clock; } @@ -179,6 +188,11 @@ public class HighlightSourceScheduler { log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}", scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"), store.directorDirectory(projectId).resolve("director-brief.md")); + if (properties.isAutoDirectorEnabled()) { + autoDirect(projectId, project.sourceVideoFileName(), + store.sourceDirectory(projectId).resolve(workingFile.getFileName()), + analysis, cinematic, scanId); + } log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} " + "analysis_file={} elapsed_ms={}", scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json", @@ -209,6 +223,54 @@ public class HighlightSourceScheduler { } } + private void autoDirect(String projectId, String sourceFileName, Path sourcePath, + HighlightSourceAnalysis analysis, CinematicHighlightAnalysis cinematic, long scanId) { + try { + double duration = analysis.source() == null ? 0.0 : analysis.source().durationSeconds(); + MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration); + if (properties.isVisionDirectorEnabled()) { + montage = visionDirector.augment(montage, sourcePath, + store.directorDirectory(projectId).resolve("vision-work")); + } + store.writeJson(projectId, "director/montage.json", montage); + // Minimal edit-plan.json so the render scanner selects the project; montage.json takes precedence. + store.writeJson(projectId, "director/edit-plan.json", + fallbackEditPlan(projectId, sourceFileName, cinematic, duration)); + log.info("event=highlight_auto_director_completed scan_id={} project_id={} shots={} source_duration={}", + scanId, projectId, montage.shots().size(), duration); + } catch (RuntimeException ex) { + log.warn("event=highlight_auto_director_failed scan_id={} project_id={} error_type={} message={}", + scanId, projectId, ex.getClass().getSimpleName(), ex.getMessage()); + } + } + + private HighlightDirectorPlan fallbackEditPlan(String projectId, String sourceFileName, + CinematicHighlightAnalysis cinematic, double duration) { + String category = cinematic.category() == null ? "generic_vlog" + : cinematic.category().name().toLowerCase(java.util.Locale.ROOT); + String candidateId; + double start; + double end; + if (cinematic.candidates() != null && !cinematic.candidates().isEmpty()) { + HighlightCandidate candidate = cinematic.candidates().get(0); + candidateId = candidate.id(); + start = candidate.sourceStartSeconds(); + end = candidate.sourceEndSeconds(); + } else { + candidateId = "candidate_001"; + start = 0.0; + end = Math.max(1.0, Math.min(duration <= 0 ? 8.0 : duration, 8.0)); + } + double span = Math.max(1.0, end - start); + HighlightDirectorPlan.HighlightItem item = new HighlightDirectorPlan.HighlightItem( + "highlight_001", candidateId, "Highlight", start, end, span, + "hero_payoff", "cinematic hero grade", "cinematic score", "", + java.util.List.of(), java.util.List.of(), + "Auto-generated fallback; the adjacent montage.json takes precedence."); + return new HighlightDirectorPlan(projectId, sourceFileName, category, java.util.List.of(item), + "Auto-director fallback edit-plan; montage.json drives the render."); + } + private void copySourceToProject(Path source, Path target) { try { Files.copy(source, target, StandardCopyOption.COPY_ATTRIBUTES); diff --git a/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java b/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java new file mode 100644 index 0000000..0a09b72 --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java @@ -0,0 +1,179 @@ +package org.example.videoclips.editing; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.example.videoclips.config.VideoClippingProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; + +/** + * Tier-2 semantic director. It samples the payoff frame of a Tier-1 {@link MontagePlan}, asks a local + * vision-language model (moondream2, via {@code tools/vision_caption.py}) what the moment is and what its + * mood is, and AUGMENTS the plan with a semantic overlay caption and a scene-informed music direction — the + * nuance measurement alone cannot reach. + * + *

It never fails the pipeline: on any error (model missing, worker failure, timeout) it returns the input + * plan unchanged, so the Tier-1 cut always stands. + */ +@Component +public class HighlightVisionDirector { + + private static final Logger log = LoggerFactory.getLogger(HighlightVisionDirector.class); + private static final String SCRIPT = "tools/vision_caption.py"; + private static final long TIMEOUT_SECONDS = 600; + + private final VideoClippingProperties.Editing.LocalAssetWorker worker; + private final String ffmpegBinary; + private final ObjectMapper objectMapper; + + public HighlightVisionDirector(VideoClippingProperties properties, ObjectMapper objectMapper) { + this.worker = properties.getEditing().getLocalAssetWorker(); + this.ffmpegBinary = properties.getEditing().getFfmpegBinary(); + this.objectMapper = objectMapper; + } + + /** Returns a plan augmented with a semantic overlay + scene-informed music, or the input plan on failure. */ + public MontagePlan augment(MontagePlan plan, Path source, Path workDir) { + try { + int payoff = payoffShotIndex(plan); + if (payoff < 0) { + return plan; + } + MontagePlan.Shot shot = plan.shots().get(payoff); + double sourceMidpoint = shot.sourceStartSeconds() + shot.durationSeconds() * shot.speed() / 2.0; + + Files.createDirectories(workDir); + Path frame = workDir.resolve("vision-payoff.jpg"); + if (!extractFrame(source, sourceMidpoint, frame)) { + return plan; + } + + List captions = caption(workDir, frame); + String label = captions.stream().filter(c -> "label".equals(c.id())).map(Caption::answer) + .findFirst().orElse(""); + String scene = captions.stream().filter(c -> "scene".equals(c.id())).map(Caption::answer) + .findFirst().orElse(""); + + String overlayText = toOverlayText(label); + List overlays = new ArrayList<>( + plan.overlays() == null ? List.of() : plan.overlays()); + if (!overlayText.isBlank()) { + double[] span = payoffTimeline(plan, payoff); + overlays.add(new MontagePlan.Overlay(overlayText, span[0] + 0.2, + Math.max(span[0] + 0.8, span[1] - 0.2), "lower_center_safe")); + } + String music = scene.isBlank() ? plan.musicDirection() + : "cinematic film score for this scene: " + scene + + " Build quiet tension to a triumphant climax hit, then a short warm resolve, no vocals."; + + log.info("event=highlight_vision_director_completed overlay=\"{}\" scene_len={}", + overlayText, scene.length()); + return new MontagePlan(plan.projectId(), plan.sourceVideoFileName(), plan.grade(), music, + plan.voiceover(), overlays, plan.shots()); + } catch (RuntimeException | IOException | InterruptedException ex) { + if (ex instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } + log.warn("event=highlight_vision_director_failed error_type={} message={}", + ex.getClass().getSimpleName(), ex.getMessage()); + return plan; + } + } + + /** The strongest slow-motion shot is the payoff; -1 if there is none. */ + static int payoffShotIndex(MontagePlan plan) { + int best = -1; + double slowest = 1.0; + List shots = plan.shots(); + for (int i = 0; i < shots.size(); i++) { + if (shots.get(i).speed() < slowest) { + slowest = shots.get(i).speed(); + best = i; + } + } + return best; + } + + static double[] payoffTimeline(MontagePlan plan, int index) { + double start = 0; + for (int i = 0; i < index; i++) { + start += plan.shots().get(i).durationSeconds(); + } + return new double[]{start, start + plan.shots().get(index).durationSeconds()}; + } + + /** + * Cleans a model answer into a short, bold overlay: strips punctuation, keeps up to three words, upper-cases. + * Returns "" when the answer is empty or unusably long/gibberish. + */ + static String toOverlayText(String answer) { + if (answer == null) { + return ""; + } + String cleaned = answer.replaceAll("[\"'.!?,:;]", " ").trim(); + if (cleaned.isBlank()) { + return ""; + } + String[] words = cleaned.split("\\s+"); + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < Math.min(3, words.length); i++) { + if (words[i].length() > 18) { + continue; + } + if (sb.length() > 0) { + sb.append(' '); + } + sb.append(words[i]); + } + return sb.toString().toUpperCase(Locale.ROOT).trim(); + } + + private boolean extractFrame(Path source, double atSeconds, Path output) throws IOException, InterruptedException { + List command = List.of(ffmpegBinary, "-hide_banner", "-loglevel", "error", + "-y", "-ss", Double.toString(Math.max(0.0, atSeconds)), "-i", source.toString(), + "-frames:v", "1", output.toString()); + Process process = new ProcessBuilder(command).redirectErrorStream(true).start(); + process.getInputStream().readAllBytes(); + return process.waitFor() == 0 && Files.isRegularFile(output); + } + + private List caption(Path workDir, Path frame) throws IOException, InterruptedException { + List manifest = List.of( + new Manifest("label", frame.toAbsolutePath().toString(), + "In one to three words, what is the exciting achievement or action in this moment?"), + new Manifest("scene", frame.toAbsolutePath().toString(), + "Describe the scene and mood in one short sentence for a film score composer.")); + Path manifestFile = workDir.resolve("vision-manifest.json"); + Path outputFile = workDir.resolve("vision-captions.json"); + objectMapper.writeValue(manifestFile.toFile(), manifest); + + List command = List.of(worker.getPythonBinary(), SCRIPT, + "--manifest", manifestFile.toString(), "--output", outputFile.toString()); + Process process = new ProcessBuilder(command).redirectErrorStream(true).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + boolean finished = process.waitFor(TIMEOUT_SECONDS, java.util.concurrent.TimeUnit.SECONDS); + if (!finished) { + process.destroyForcibly(); + throw new IllegalStateException("vision caption worker timed out"); + } + if (process.exitValue() != 0 || !Files.isRegularFile(outputFile)) { + throw new IllegalStateException("vision caption worker failed: " + out.strip()); + } + Caption[] captions = objectMapper.readValue(outputFile.toFile(), Caption[].class); + return List.of(captions); + } + + record Manifest(String id, String image, String question) { + } + + record Caption(String id, String answer) { + } +} diff --git a/src/main/resources/application-localpoc.yml b/src/main/resources/application-localpoc.yml index 428d185..79ba43d 100644 --- a/src/main/resources/application-localpoc.yml +++ b/src/main/resources/application-localpoc.yml @@ -59,9 +59,14 @@ video-clipping: rejected-directory: ./input/localpoc/highlights/rejected render-enabled: false require-director-approval: true + # Tier-1 automatic director: compose director/montage.json from measured motion + audio after analysis. + auto-director-enabled: true + # Tier-2 vision director: augment the montage with a semantic overlay + scene-informed music (moondream2). + vision-director-enabled: true max-highlights-per-source: 3 - highlight-min-duration-seconds: 8 - highlight-max-duration-seconds: 35 + # No fixed highlight length — a highlight can be as long as the story needs (soft candidate hints only). + highlight-min-duration-seconds: 2 + highlight-max-duration-seconds: 3600 logging: level: diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index 9bff3bb..758a0da 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -95,8 +95,11 @@ video-clipping: require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:true} approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag} max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3} - highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8} - highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35} + # Highlights are not restricted to a fixed length: a highlight may be as long as the story needs. + # These bounds are only soft hints for coverage-window candidate generation; the director plan + # validator no longer rejects a highlight on duration (only positive/finite duration + sane speed). + highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:2} + highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:3600} logging: level: diff --git a/src/test/java/org/example/videoclips/editing/FfmpegClipInspectorTest.java b/src/test/java/org/example/videoclips/editing/FfmpegClipInspectorTest.java index ced9780..45194ee 100644 --- a/src/test/java/org/example/videoclips/editing/FfmpegClipInspectorTest.java +++ b/src/test/java/org/example/videoclips/editing/FfmpegClipInspectorTest.java @@ -121,6 +121,67 @@ class FfmpegClipInspectorTest { } } + @Test + void swapsDimensionsForRotatedPortraitSideData() { + AtomicReference> command = new AtomicReference<>(); + FfmpegClipInspector inspector = inspector(arguments -> { + command.set(arguments); + return new FfmpegClipInspector.ProcessResult(0, """ + { + "streams": [ + { + "codec_type": "video", + "codec_name": "h264", + "width": 1024, + "height": 576, + "r_frame_rate": "30/1", + "side_data_list": [ { "rotation": -90 } ] + } + ], + "format": { "duration": "16.0" } + } + """); + }); + + ClipAnalysis analysis = inspector.inspect(Path.of("portrait.mp4")); + + // Effective (display) orientation is portrait: stored landscape pixels are swapped. + assertThat(analysis.width()).isEqualTo(576); + assertThat(analysis.height()).isEqualTo(1024); + assertThat(command.get()).contains("stream_side_data=rotation", "stream_tags=rotate"); + } + + @Test + void swapsDimensionsForRotateTagAndIgnoresHalfTurn() { + FfmpegClipInspector inspector = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """ + { + "streams": [ { + "codec_type": "video", "codec_name": "h264", + "width": 1080, "height": 1920, "r_frame_rate": "30/1", + "tags": { "rotate": "90" } + } ], + "format": { "duration": "5.0" } + } + """)); + ClipAnalysis rotated = inspector.inspect(Path.of("tag.mp4")); + assertThat(rotated.width()).isEqualTo(1920); + assertThat(rotated.height()).isEqualTo(1080); + + FfmpegClipInspector upsideDown = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """ + { + "streams": [ { + "codec_type": "video", "codec_name": "h264", + "width": 1920, "height": 1080, "r_frame_rate": "30/1", + "side_data_list": [ { "rotation": 180 } ] + } ], + "format": { "duration": "5.0" } + } + """)); + ClipAnalysis flipped = upsideDown.inspect(Path.of("flip.mp4")); + assertThat(flipped.width()).isEqualTo(1920); + assertThat(flipped.height()).isEqualTo(1080); + } + private FfmpegClipInspector inspector(FfmpegClipInspector.ProcessExecutor executor) { return new FfmpegClipInspector(new VideoClippingProperties(), new ObjectMapper(), executor); } diff --git a/src/test/java/org/example/videoclips/editing/HighlightDirectorPlanValidatorTest.java b/src/test/java/org/example/videoclips/editing/HighlightDirectorPlanValidatorTest.java index aeee2b4..95306df 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightDirectorPlanValidatorTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightDirectorPlanValidatorTest.java @@ -106,6 +106,31 @@ class HighlightDirectorPlanValidatorTest { .hasMessageContaining("playback speed outside 0.25..4.0"); } + @Test + void acceptsHighlightLongerThanTheFormerFixedMaximum() { + // Source span 2..14 (12s) stretched to a 40s target (0.3x slow-mo) would have been rejected by the + // old 35s cap. Highlights may now be as long as the story needs (speed must still be sane). + HighlightDirectorPlan.HighlightItem longHighlight = new HighlightDirectorPlan.HighlightItem( + "highlight_001", "candidate_001", "Epic", 2, 14, 40, + "hero_payoff", "cinematic grade", "scene-fit score", "impact", + List.of("Grounded narration."), List.of(), "reviewed treatment"); + + assertThat(validator.validate("project-1", plan(longHighlight))) + .isEqualTo(plan(longHighlight)); + } + + @Test + void rejectsNonPositiveTargetDuration() { + HighlightDirectorPlan.HighlightItem zero = new HighlightDirectorPlan.HighlightItem( + "highlight_001", "candidate_001", "Opening", 2, 14, 0, + "opening_hook", "cinematic grade", "scene-fit score", "impact", + List.of("Grounded narration."), List.of(), "reviewed treatment"); + + assertThatThrownBy(() -> validator.validate("project-1", plan(zero))) + .isInstanceOf(BadRequestException.class) + .hasMessageContaining("positive number of seconds"); + } + private HighlightCandidate candidate() { return new HighlightCandidate("candidate_001", "source", 2, 14, 0.7, "opening_hook_candidate", List.of("reviewed")); diff --git a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java index c64d07f..84a55ed 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java @@ -88,9 +88,11 @@ class HighlightFfmpegRendererTest { String hero = String.join(" ", renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); - assertThat(opening).contains("curves=preset=linear_contrast").contains("vignette=PI/7"); - assertThat(rising).contains("curves=preset=medium_contrast").contains("eq=contrast=1.12"); - assertThat(hero).contains("curves=preset=strong_contrast").contains("eq=contrast=1.16"); + assertThat(opening).contains("curves=preset=linear_contrast").contains("vignette=PI/10"); + assertThat(rising).contains("curves=preset=linear_contrast").contains("eq=contrast=1.07"); + assertThat(hero).contains("curves=preset=medium_contrast").contains("eq=contrast=1.08"); + // Exposure-preserving: every beat lifts the black point and never pushes gamma < 1 or brightness < 0. + assertThat(hero).contains("curves=all='0/0.06 1/1'").contains("gamma=1.06").contains("brightness=0.02"); assertThat(opening).isNotEqualTo(hero); } @@ -172,6 +174,89 @@ class HighlightFfmpegRendererTest { assertThat(command).contains("-shortest"); } + @Test + void choosesPortraitGeometryForPortraitSourceAndLandscapeOtherwise() { + HighlightFfmpegRenderer renderer = renderer(); // configured 1920x1080 + + assertThat(renderer.outputGeometry(analysisWithDimensions(576, 1024))) + .containsExactly(1080, 1920); + assertThat(renderer.outputGeometry(analysisWithDimensions(1920, 1080))) + .containsExactly(1920, 1080); + assertThat(renderer.outputGeometry(null)).containsExactly(1920, 1080); + } + + @Test + void portraitSegmentSkipsWidescreenLetterbox() { + HighlightFfmpegRenderer renderer = renderer(); + EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "fade-in", "cut", 1.0, + "hero cinematic", "montage"); + + String landscape = String.join(" ", + renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1920, 1080, + Path.of("l.mp4"))); + String portrait = String.join(" ", + renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, + Path.of("p.mp4"))); + + assertThat(landscape).contains("scale=1920:1080").contains("crop=1920:"); // 2.39 letterbox present + assertThat(portrait).contains("scale=1080:1920").doesNotContain("crop=1080:"); // no letterbox + } + + @Test + void exposureNormalizationAdaptsToMeasuredSourceBrightness() { + HighlightFfmpegRenderer renderer = renderer(); + + // A dark source is brightened (gamma > 1). + String dark = renderer.exposureNormalizationFilter(60); + assertThat(dark).startsWith(",eq=gamma="); + assertThat(Double.parseDouble(dark.replace(",eq=gamma=", ""))).isGreaterThan(1.0); + + // A well-exposed source is left alone. + assertThat(renderer.exposureNormalizationFilter(120)).isEmpty(); + + // An over-bright source is darkened (gamma < 1). + String bright = renderer.exposureNormalizationFilter(215); + assertThat(Double.parseDouble(bright.replace(",eq=gamma=", ""))).isLessThan(1.0); + + // Unmeasurable source -> no adjustment. + assertThat(renderer.exposureNormalizationFilter(-1)).isEmpty(); + } + + @Test + void pushInAmountIsStrongerForLowMotionShots() { + HighlightFfmpegRenderer renderer = renderer(); + + double staticShot = renderer.pushInAmount(2.6); // near-static (held) + double activeShot = renderer.pushInAmount(8.0); // lots of motion + + assertThat(staticShot).isGreaterThan(activeShot); + assertThat(staticShot).isBetween(0.03, 0.12); + assertThat(activeShot).isGreaterThanOrEqualTo(0.03); // floor: always a little life + assertThat(renderer.pushInAmount(-1)).isGreaterThan(0.0); // unknown -> mild default push + } + + @Test + void appliesMotionAdaptivePushInOnlyWhenRequested() { + HighlightFfmpegRenderer renderer = renderer(); + EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "cut", "cut", 1.0, "hero cinematic", + "montage"); + + String withPush = String.join(" ", renderer.segmentCommand("s.mp4", decision, + "style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.10, Path.of("p.mp4"))); + String noPush = String.join(" ", renderer.segmentCommand("s.mp4", decision, + "style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.0, Path.of("n.mp4"))); + + assertThat(withPush).contains("zoompan=z="); + assertThat(noPush).doesNotContain("zoompan").contains("crop=iw/"); // static crop preserved at 0 + } + + private HighlightSourceAnalysis analysisWithDimensions(int width, int height) { + ClipAnalysis source = new ClipAnalysis("clip", "s.mp4", 16, "h264", "aac", width, height, 30, + List.of(), null, null, 0, 0, 0); + return new HighlightSourceAnalysis("p", "s.mp4", source, List.of(), null, null, null, null, + List.of(), null, null, null, null, null); + } + private HighlightFfmpegRenderer renderer() { return new HighlightFfmpegRenderer(new VideoClippingProperties(), mock(HighlightProjectStore.class), mock(EditAssetProvider.class), mock(EditObservability.class), command -> null); diff --git a/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java b/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java new file mode 100644 index 0000000..6775b0c --- /dev/null +++ b/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java @@ -0,0 +1,63 @@ +package org.example.videoclips.editing; + +import org.example.videoclips.config.VideoClippingProperties; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class HighlightMontageDirectorTest { + + private final HighlightMontageDirector director = new HighlightMontageDirector(new VideoClippingProperties()); + + @Test + void composesAStoryStructureFromMeasuredMotionAndAudio() { + // 16s at 0.5s windows = 32 buckets. Motion: baseline with a release spike at ~4s, a celebration bump + // at ~9-11s, and a strong camera-whip tail from ~12.5s. Audio: climax (celebration) peaking at ~8s; + // loud tail spikes after ~13.5s that must be ignored (outside the central band). + int n = 32; + double[] motion = new double[n]; + double[] audio = new double[n]; + for (int i = 0; i < n; i++) { + double t = i * 0.5; + motion[i] = 2.0; + if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0; // release spike + if (t >= 9.0 && t <= 11.0) motion[i] = 4.0; // celebration motion + if (t >= 12.5) motion[i] = 9.0; // camera-whip tail + audio[i] = -30.0; + if (t >= 6.5 && t <= 9.5) audio[i] = -12.0; // celebration band + if (Math.abs(t - 8.0) < 0.6) audio[i] = -8.0; // climax peak + if (t >= 13.5) audio[i] = -10.0; // loud aftermath (must be ignored) + } + + MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 16.0); + + // A real multi-beat structure, not one straight shot. + assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4); + + // Exactly one strong slow-motion payoff, and it sits on the audio climax (~8s). + var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList(); + assertThat(slowMo).hasSize(1); + assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(7.5, 8.6); + + // There is a continuous action/tension shot (long, not chopped). + double longestSpan = plan.shots().stream() + .mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0); + assertThat(longestSpan).isGreaterThan(2.0); + + // The camera-whip tail (>=12.5s) is trimmed: no shot reads source beyond it. + double maxSourceEnd = plan.shots().stream() + .mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0); + assertThat(maxSourceEnd).isLessThanOrEqualTo(13.0); + + assertThat(plan.grade()).isEqualTo("hero"); + assertThat(plan.musicDirection()).isNotBlank(); + } + + @Test + void fallsBackToAStraightCutForVeryShortSources() { + MontagePlan plan = director.composeMontage("p", "src.mp4", new double[]{1, 1}, new double[]{-30, -30}, + 0.5, 1.0); + assertThat(plan.shots()).hasSize(1); + assertThat(plan.shots().get(0).sourceStartSeconds()).isEqualTo(0.0); + } +} diff --git a/src/test/java/org/example/videoclips/editing/HighlightSourceSchedulerTest.java b/src/test/java/org/example/videoclips/editing/HighlightSourceSchedulerTest.java index 55adfe1..05b2b6b 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightSourceSchedulerTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightSourceSchedulerTest.java @@ -114,7 +114,9 @@ class HighlightSourceSchedulerTest { stubAnalysis("porsche"); stubAnalysis("porsche-1"); stubAnalysis("porsche-drive"); - return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator, clock); + return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator, + new HighlightMontageDirector(properties), + new HighlightVisionDirector(properties, new com.fasterxml.jackson.databind.ObjectMapper()), clock); } private void stubAnalysis(String projectId) { diff --git a/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java b/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java new file mode 100644 index 0000000..805cd37 --- /dev/null +++ b/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java @@ -0,0 +1,42 @@ +package org.example.videoclips.editing; + +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +class HighlightVisionDirectorTest { + + @Test + void cleansModelAnswersIntoShortBoldOverlays() { + assertThat(HighlightVisionDirector.toOverlayText("A strike!")).isEqualTo("A STRIKE"); + assertThat(HighlightVisionDirector.toOverlayText("Bowling strike celebration moment")) + .isEqualTo("BOWLING STRIKE CELEBRATION"); // capped at 3 words + assertThat(HighlightVisionDirector.toOverlayText(" ")).isEmpty(); + assertThat(HighlightVisionDirector.toOverlayText(null)).isEmpty(); + } + + @Test + void findsThePayoffShotAndItsTimelineWindow() { + MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(), List.of( + new MontagePlan.Shot(0.0, 1.5, 1.03, 1.0), + new MontagePlan.Shot(2.0, 2.0, 1.04, 1.0), + new MontagePlan.Shot(8.0, 3.0, 1.05, 0.7), // payoff (slowest) + new MontagePlan.Shot(11.0, 1.2, 1.04, 0.9))); + + int payoff = HighlightVisionDirector.payoffShotIndex(plan); + assertThat(payoff).isEqualTo(2); + + double[] window = HighlightVisionDirector.payoffTimeline(plan, payoff); + assertThat(window[0]).isEqualTo(3.5); // 1.5 + 2.0 + assertThat(window[1]).isEqualTo(6.5); // + 3.0 + } + + @Test + void hasNoPayoffWhenNothingIsSlowMotion() { + MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(), + List.of(new MontagePlan.Shot(0.0, 2.0, 1.03, 1.0))); + assertThat(HighlightVisionDirector.payoffShotIndex(plan)).isEqualTo(-1); + } +} diff --git a/tools/vision_caption.py b/tools/vision_caption.py new file mode 100644 index 0000000..905fff5 --- /dev/null +++ b/tools/vision_caption.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""Tier-2 vision captioner for the highlight director. + +Loads a local vision-language model (moondream2) ONCE and answers a question for each frame listed in a +manifest, writing the answers to a JSON file. Runs fully offline from the local Hugging Face cache — it +performs no network access (HF_HUB_OFFLINE is forced on). + +Manifest JSON: [{"id": "...", "image": "/abs/path.jpg", "question": "..."}] +Output JSON: [{"id": "...", "answer": "..."}] +""" +from __future__ import annotations + +import argparse +import json +import os +import sys + + +def main() -> int: + parser = argparse.ArgumentParser(description="Local VLM frame captioner (moondream2)") + parser.add_argument("--manifest", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--model-id", default="vikhyatk/moondream2") + parser.add_argument("--revision", default="2024-08-26") + args = parser.parse_args() + + # Hard offline: never reach the network. The model must already be in the local cache. + os.environ["HF_HUB_OFFLINE"] = "1" + os.environ["TRANSFORMERS_OFFLINE"] = "1" + os.environ.setdefault("HF_HUB_DISABLE_XET", "1") + + try: + from PIL import Image + from transformers import AutoModelForCausalLM, AutoTokenizer + except Exception as exc: # pragma: no cover - dependency guard + print(f"vision_caption: missing dependency: {exc}", file=sys.stderr) + return 3 + + with open(args.manifest, "r", encoding="utf-8") as handle: + items = json.load(handle) + + try: + model = AutoModelForCausalLM.from_pretrained( + args.model_id, revision=args.revision, trust_remote_code=True, local_files_only=True) + tokenizer = AutoTokenizer.from_pretrained( + args.model_id, revision=args.revision, local_files_only=True) + model.eval() + except Exception as exc: + print(f"vision_caption: unable to load local model: {exc}", file=sys.stderr) + return 2 + + results = [] + for item in items: + try: + image = Image.open(item["image"]).convert("RGB") + encoded = model.encode_image(image) + answer = model.answer_question(encoded, item["question"], tokenizer).strip() + except Exception as exc: # keep going; a single bad frame must not fail the batch + print(f"vision_caption: frame {item.get('id')} failed: {exc}", file=sys.stderr) + answer = "" + results.append({"id": item.get("id"), "answer": answer}) + + os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True) + with open(args.output, "w", encoding="utf-8") as handle: + json.dump(results, handle, ensure_ascii=False, indent=2) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())