From d89a28e08444807283d1a578fb5127666a6f86f2 Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Sat, 25 Jul 2026 18:37:38 +0200 Subject: [PATCH] refactor(director): decisive moment = measurement proposes, vision model judges Choosing WHICH moment is the highlight is a question of meaning, not motion or loudness, and there is no generic rule in measurement alone: on the real bowling clip a camera turn-away has the highest motion AND is louder than the celebration. Positional bands / thresholds only move which video breaks. So responsibilities are now split, with zero per-video constants: - HighlightMontageDirector.candidatePeaks: measurement PROPOSES the intensity (motion+audio) local maxima, strongest-first, min-separated. No opinion on which is the highlight; no band, no threshold. - HighlightVisionDirector.rankDecisiveMoment: the vision model JUDGES each candidate by highlight-worthiness (a celebration/goal outranks a loud turn-away or an "about to..." build; anticipation is not the payoff). Highest score wins; falls back to the strongest peak only if the model declines. - composeMontageAt: builds the action segment (measured onset -> chosen peak -> measured resolution that sweeps in the outcome+reaction) around the choice. - MomentChooser interface makes the judge a drop-in: a stronger local VLM plugs in with no director changes. Removes the previous positional-band / semantic-weight heuristics. Honest, verified ceiling (documented in R15): the judge is only as good as its eyes. moondream2 perceives some actions (soccer: "kicking a soccer ball" -> the goal is chosen correctly) but not others -- on distant portrait bowling footage it describes every frame as "standing"/"walking" and never sees the celebration (a posture prompt collapsed to a constant "Standing still"). When it can't discriminate, candidates tie and it falls back to the loudest peak. This is a model-capability limit, not a design flaw; the fix is a stronger VLM (drop-in). mvn verify: 293 tests green. Co-Authored-By: Claude Opus 4.8 --- docs/cinematic-quality-rules.md | 38 ++++- .../editing/HighlightMontageDirector.java | 148 ++++++++++++------ .../editing/HighlightSourceScheduler.java | 19 ++- .../editing/HighlightVisionDirector.java | 72 ++++++++- .../editing/HighlightMontageDirectorTest.java | 145 +++++++---------- .../editing/HighlightVisionDirectorTest.java | 18 +++ 6 files changed, 293 insertions(+), 147 deletions(-) diff --git a/docs/cinematic-quality-rules.md b/docs/cinematic-quality-rules.md index 385122c..9818e2a 100644 --- a/docs/cinematic-quality-rules.md +++ b/docs/cinematic-quality-rules.md @@ -134,10 +134,42 @@ Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5 - **Honest limits:** **YOLOv8 is AGPL-3.0 → non-commercial** (matches the repo's CV stance); per-frame detection is CPU-slow; on tiny/distant subjects detection is unreliable. +## R15 — Decisive moment: measurement proposes, the vision model judges ✅ (with a hard VLM ceiling) +A highlight is an **action unit** — a start, a decisive peak, and an outcome/reaction — and choosing *which* +moment is the highlight is a question of **meaning**, not of motion or loudness. **There is no generic rule in +measurement alone:** a camera turn-away or a loud aftermath routinely out-scores a quiet celebration on both +motion and audio (measured on the real bowling clip: the turn-around has the clip's highest motion *and* is +louder than the celebration). So the design separates the two responsibilities — no positional bands, no +per-video thresholds, ever: + +- **Measurement PROPOSES (`HighlightMontageDirector.candidatePeaks`):** the local maxima of intensity + (normalised motion + audio), strongest first, min-separated. Every real event becomes a candidate — a strike, + a celebration, a turn-away, a goal, an anticipation — with **no opinion** about which is the highlight. +- **The vision model JUDGES (`HighlightVisionDirector.rankDecisiveMoment`):** it captions each candidate and + scores **highlight-worthiness by meaning** — a celebration or a scored goal outranks a loud turn-away or an + "about to…" build (anticipation is explicitly *not* the payoff). The highest-scoring candidate wins. +- **Fallback:** if the model declines/fails, the strongest-intensity peak is used (a measured last resort). +- **Segment (`composeMontageAt`):** around the chosen peak, the **onset** (measured motion rising into it, + build-capped) and the **resolution** (measured motion settling after it, sweeping in the outcome + reaction — + the pins falling *and* the celebration; the ball crossing the line *and* settling in the net) are built. +- **Honest labelling (R9/C):** a dense read of the *shown* segment names it and never asserts an action the + segment doesn't contain — an anticipatory cut becomes a teaser question, not a false "KICK". + +**Honest ceiling (verified, not theoretical):** the judge is only as good as its eyes. The local **moondream2** +model reliably perceives some actions (soccer: *"kicking a soccer ball"* → the goal is chosen correctly) but +**cannot** perceive others — on the distant, portrait bowling clip it describes every frame as *"standing"* / +*"walking"* / *"a bowling alley"* and never sees the arms-raised celebration, regardless of prompt (a +posture-focused prompt collapsed to a constant *"Standing still"*). When the model can't discriminate, all +candidates tie and it falls back to the loudest peak (the turn-away). This is a **model-capability limit, not a +design flaw** — the fix is a stronger local VLM, which is a **drop-in**: the judge is a clean interface +(`MomentChooser` / `rankDecisiveMoment`) with no director changes required. Forcing the weak case with more +heuristics is prohibited — that is the hack this rule exists to avoid. + --- -Rules R1–R14 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightBeatSync` / -`HighlightSubjectTracker` and apply to **every** project automatically (R13/R14 behind opt-in flags). Each is -driven by a source measurement or a global cinematic standard, never a per-video constant. +Rules R1–R15 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightMontageDirector` +/ `HighlightVisionDirector` / `HighlightBeatSync` / `HighlightSubjectTracker` and apply to **every** project +automatically (R13/R14 behind opt-in flags). Each is driven by a source measurement or a global cinematic +standard, never a per-video constant. ## Still missing for "cinematic" (researched gap — not yet implemented) Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work: diff --git a/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java b/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java index 3702b90..2029017 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java +++ b/src/main/java/org/example/videoclips/editing/HighlightMontageDirector.java @@ -30,6 +30,12 @@ import java.util.List; public class HighlightMontageDirector { static final double WINDOW_SECONDS = 0.5; + /** How many candidate decisive moments to propose for the vision judge to rank. */ + static final int MAX_CANDIDATES = 5; + /** Minimum spacing (seconds) between proposed candidates, so they are distinct moments. Kept short so a + * reaction close on the heels of the action (a celebration right after the strike) stays a separate + * candidate the vision judge can score, rather than being merged into the louder neighbouring peak. */ + static final double MIN_CANDIDATE_SEPARATION = 1.0; private final VideoClippingProperties.Editing properties; @@ -37,80 +43,134 @@ public class HighlightMontageDirector { this.properties = properties.getEditing(); } - /** Measure the source and compose an automatic montage plan. */ + /** Measure the source and compose a montage, choosing the decisive moment by measured intensity alone. */ public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds) { return direct(projectId, sourceFileName, source, durationSeconds, null); } /** - * Compose a montage plan from measured signals, optionally guided by a Tier-2 semantic curve (per-window - * "highlight-worthiness" from vision captions). When present, the semantic signal helps choose the payoff. + * Measure the source, propose candidate decisive moments (intensity peaks), and let {@code chooser} — the + * Tier-2 vision judge — pick which candidate is the actual highlight. This is the generic rule: measurement + * never decides the moment by position or threshold (a camera whip or a loud aftermath can out-score a + * quiet celebration); it only PROPOSES, and the model JUDGES by meaning. When no chooser is supplied, or it + * declines (NaN), the highest-intensity candidate is used as a measured fallback. */ public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds, - double[] semantic) { + MomentChooser chooser) { double[] motion = probeCurve(source.toString(), true, durationSeconds); double[] audio = probeCurve(source.toString(), false, durationSeconds); - return composeMontage(projectId, sourceFileName, motion, audio, semantic, WINDOW_SECONDS, durationSeconds); + List candidates = candidatePeaks(motion, audio, WINDOW_SECONDS, durationSeconds); + double climax = Double.NaN; + if (chooser != null && !candidates.isEmpty()) { + climax = chooser.choose(List.copyOf(candidates)); + } + if (!Double.isFinite(climax)) { + climax = candidates.isEmpty() ? 0.0 : candidates.get(0); // measured fallback: strongest peak + } + return composeMontageAt(projectId, sourceFileName, motion, audio, climax, WINDOW_SECONDS, durationSeconds); + } + + /** Chooses which candidate moment (seconds) is the real highlight; returns NaN to defer to measurement. */ + @FunctionalInterface + public interface MomentChooser { + double choose(List candidateTimeSeconds); + } + + /** + * Proposes candidate decisive moments as the local maxima of measured intensity (motion + audio), + * strongest first, min-separated. GENERIC: no positional band, no per-video threshold — every real + * event (a strike, a celebration, a turn-away, a goal, an anticipation) becomes a candidate, and the + * vision judge decides which is the highlight. {@code motion[i]}/{@code audio[i]} cover the window at + * {@code i * window} seconds. + */ + List candidatePeaks(double[] motion, double[] audio, double window, double duration) { + int n = Math.min(motion.length, audio.length); + if (n < 4 || duration <= 0) { + return List.of(); + } + double[] mN = normalize(smooth(motion, n), n); + double[] aN = normalize(smooth(audio, n), n); + double[] intensity = new double[n]; + double globalMax = 0; + for (int i = 0; i < n; i++) { + intensity[i] = aN[i] + mN[i]; + globalMax = Math.max(globalMax, intensity[i]); + } + if (globalMax <= 1e-9) { + return List.of(); + } + int neigh = Math.max(1, (int) Math.round(1.2 / window)); // local-max over +/-1.2s + double threshold = 0.35 * globalMax; // ignore minor bumps + List peaks = new ArrayList<>(); + for (int i = 1; i < n - 1; i++) { // skip degenerate first/last window + if (intensity[i] < threshold) { + continue; + } + boolean isMax = true; + for (int j = Math.max(0, i - neigh); j <= Math.min(n - 1, i + neigh); j++) { + if (intensity[j] > intensity[i]) { + isMax = false; + break; + } + } + if (isMax) { + peaks.add(i); + } + } + peaks.sort((x, y) -> Double.compare(intensity[y], intensity[x])); // strongest first + List chosen = new ArrayList<>(); + for (int idx : peaks) { + double t = idx * window; + boolean tooClose = chosen.stream().anyMatch(c -> Math.abs(c - t) < MIN_CANDIDATE_SEPARATION); + if (!tooClose) { + chosen.add(t); + } + if (chosen.size() >= MAX_CANDIDATES) { + break; + } + } + return chosen; } MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio, double window, double duration) { - return composeMontage(projectId, sourceFileName, motion, audio, null, window, duration); + // Measured path (no vision judge): build around the strongest intensity peak. + List candidates = candidatePeaks(motion, audio, window, duration); + double climax = candidates.isEmpty() ? 0.0 : candidates.get(0); + return composeMontageAt(projectId, sourceFileName, motion, audio, climax, window, duration); } /** - * Pure, deterministic shot composition from the measured curves. Package-visible for unit testing without - * ffmpeg. {@code motion[i]} and {@code audio[i]} cover the window starting at {@code i * window} seconds. + * Builds the ACTION SEGMENT (start -> peak -> outcome) around a chosen climax time. Pure and deterministic; + * package-visible for unit testing without ffmpeg. The onset (start) and resolution (end) are MEASURED from + * motion around the given peak — the peak itself is chosen by the caller (the vision judge). */ - MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio, - double[] semantic, double window, double duration) { + MontagePlan composeMontageAt(String projectId, String sourceFileName, double[] motion, double[] audio, + double climaxTime, double window, double duration) { int n = Math.min(motion.length, audio.length); if (n < 4 || duration <= 0) { return straightCut(projectId, sourceFileName, duration); } double[] m = smooth(motion, n); - double[] a = smooth(audio, n); + int climaxIdx = Math.max(1, Math.min(n - 1, (int) Math.round(climaxTime / window))); + climaxTime = climaxIdx * window; - // Localize the DECISIVE MOMENT. The old code searched only the central 25-80% band on audio(+semantic), - // which structurally EXCLUDED a payoff in the last fifth of a clip (a late kick/goal) and could land on - // the anticipation rather than the action. Now: search almost the whole clip and localize primarily by - // MOTION — the decisive action is a motion event — with audio an equal partner and any semantic curve a - // weak tie-breaker. Motion catches pure-action payoffs (a kick); audio catches reaction payoffs (a - // celebration cheer). This is the generic fix for "the highlight cut off before the actual action". - double[] mN = normalize(m, n); - double[] aN = normalize(a, n); - boolean haveSemantic = semantic != null && semantic.length >= n; - double[] sN = haveSemantic ? normalize(semantic, n) : null; - double[] payoffSignal = new double[n]; - for (int i = 0; i < n; i++) { - payoffSignal[i] = aN[i] + 0.6 * mN[i] + (haveSemantic ? 0.9 * sN[i] : 0.0); - } - // Exclude a late CAMERA WHIP (a brief motion spike far above the clip's own motion level — the hand-held - // pan after the action) from both the climax search and the window. The baseline is the 40th-percentile - // motion (robust to the action spikes themselves), so widening the search to nearly the whole clip can't - // land the payoff on the messy aftermath — while a genuine late action (well below whip level) stays in. + // Onset: the biggest motion spike before the peak (the release/approach); the action shot leads in. + int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)), + Math.max(2, (int) Math.floor((climaxTime - 1.0) / window))); + double actionTime = actionIdx * window; + + // Resolution: extend past the peak through the outcome AND the immediate reaction (the pins falling and + // the celebration; the ball settling in the net), stopping when motion settles back to baseline for a + // sustained stretch, at a camera whip, or at a cap. This is MEASURED, not positional. double whipLevel = Math.max(1e-6, percentile(m, n, 0.4) * 4.0); int whipCutoff = n; - for (int i = (int) Math.floor(0.55 * n); i < n; i++) { + for (int i = climaxIdx + 2; i < n; i++) { if (m[i] > whipLevel) { whipCutoff = i; break; } } - int lo = (int) Math.floor(0.08 * n); - int searchHi = Math.min((int) Math.ceil(0.95 * n), whipCutoff); - int climaxIdx = argMax(payoffSignal, lo, Math.max(lo + 1, searchHi)); - double climaxTime = climaxIdx * window; - - // Action = the biggest motion spike before the climax; the action shot leads in ~1.5s before it so - // the release is included and the release->roll->watch stays continuous. - int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)), - Math.max(2, (int) Math.floor((climaxTime - 1.0) / window))); - double actionTime = actionIdx * window; - - // Resolution: the highlight must show the OUTCOME, not cut off at the action. Extend past the climax - // while motion stays elevated (the action and its result, e.g. the ball rolling away), and stop when - // motion SETTLES back near baseline for a sustained stretch, at the whip, or at a cap. double baseMotion = mean(m, 0, Math.max(1, climaxIdx)); double settleLevel = baseMotion * 1.4; double maxResolution = climaxTime + 6.0; diff --git a/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java b/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java index 4d050ae..8b9c003 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java +++ b/src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java @@ -228,15 +228,18 @@ public class HighlightSourceScheduler { HighlightSourceAnalysis analysis, CinematicHighlightAnalysis cinematic, long scanId) { try { double duration = analysis.source() == null ? 0.0 : analysis.source().durationSeconds(); - // Localize the decisive moment from MEASURED motion + audio (robust, whole-clip). A sparse whole-clip - // VLM pass used to mislocalize the payoff onto the anticipation, so the VLM is now used only to - // GROUND the overlay in the decisive window (below) — never to pick the moment. - MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration); + Path visionWork = store.directorDirectory(projectId).resolve("vision-work"); + // GENERIC rule: measurement PROPOSES candidate decisive moments (intensity peaks); the vision judge + // PICKS which is the real highlight by meaning (a celebration/goal beats a loud turn-away). No + // positional band, no per-video threshold. Falls back to the strongest peak if the model declines. + HighlightMontageDirector.MomentChooser chooser = properties.isVisionDirectorEnabled() + ? candidates -> visionDirector.rankDecisiveMoment(sourcePath, candidates, visionWork) + : null; + MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration, chooser); if (properties.isVisionDirectorEnabled()) { - // Dense pass over the decisive window (payoff shot through the outcome) with the honesty rule: - // never assert an action the shown window doesn't contain (a "KICK" over a cut before the kick). - montage = visionDirector.groundOverlay(montage, sourcePath, - store.directorDirectory(projectId).resolve("vision-work")); + // Dense pass over the chosen decisive window (payoff shot through the outcome) with the honesty + // rule: never assert an action the shown window doesn't contain (a "KICK" over a cut before it). + montage = visionDirector.groundOverlay(montage, sourcePath, visionWork); } store.writeJson(projectId, "director/montage.json", montage); // Minimal edit-plan.json so the render scanner selects the project; montage.json takes precedence. diff --git a/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java b/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java index e62ebb4..1edc8ce 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java +++ b/src/main/java/org/example/videoclips/editing/HighlightVisionDirector.java @@ -29,6 +29,14 @@ public class HighlightVisionDirector { private static final Logger log = LoggerFactory.getLogger(HighlightVisionDirector.class); private static final String SCRIPT = "tools/vision_caption.py"; private static final long TIMEOUT_SECONDS = 600; + /** + * A descriptive, open-ended question. Small VLMs give far more discriminative answers to this than to a + * terse posture question (an example-laden "arms raised / mid-throw / standing" prompt collapses to a + * constant "Standing still" on distant footage). The description feeds both the highlight-worthiness score + * and the honest-overlay anticipation check. + */ + private static final String DESCRIPTIVE_QUESTION = + "In one sentence, describe what the person is doing with their body right now."; private final VideoClippingProperties.Editing.LocalAssetWorker worker; private final String ffmpegBinary; @@ -115,8 +123,7 @@ public class HighlightVisionDirector { Path frame = workDir.resolve("payoff-" + i + ".jpg"); if (extractFrame(source, t, frame)) { String img = frame.toAbsolutePath().toString(); - manifest.add(new Manifest("d" + i, img, - "In one sentence, describe what the person is doing with their body right now.")); + manifest.add(new Manifest("d" + i, img, DESCRIPTIVE_QUESTION)); manifest.add(new Manifest("l" + i, img, "In one to three words, what is the exciting action in this moment?")); manifest.add(new Manifest("q" + i, img, @@ -166,6 +173,64 @@ public class HighlightVisionDirector { } } + /** + * The Tier-2 vision JUDGE. Given candidate decisive moments proposed by measurement, it captions each and + * returns the time of the most highlight-worthy one — a celebration or a scored goal beats a loud + * camera turn-away or an anticipatory build, because "is this THE highlight?" is a question of meaning, not + * of motion or loudness. Returns NaN on any failure so the director falls back to the strongest peak. + */ + public double rankDecisiveMoment(Path source, List candidateTimes, Path workDir) { + if (candidateTimes == null || candidateTimes.isEmpty()) { + return Double.NaN; + } + try { + Files.createDirectories(workDir); + List manifest = new ArrayList<>(); + List ids = new ArrayList<>(); + for (int i = 0; i < candidateTimes.size(); i++) { + Path frame = workDir.resolve("cand-" + i + ".jpg"); + if (extractFrame(source, candidateTimes.get(i), frame)) { + manifest.add(new Manifest("d" + i, frame.toAbsolutePath().toString(), DESCRIPTIVE_QUESTION)); + ids.add(i); + } + } + if (ids.isEmpty()) { + return Double.NaN; + } + List caps = runCaptioner(workDir, manifest); + int best = -1; + double bestScore = -1; + for (int i : ids) { + double score = highlightWorthiness(answerFor(caps, "d" + i)); + if (score > bestScore) { // strict >: ties keep the stronger-intensity (earlier) candidate + bestScore = score; + best = i; + } + } + if (best < 0) { + return Double.NaN; + } + log.info("event=highlight_vision_moment_ranked candidates={} chosen_time={} score={}", + ids.size(), candidateTimes.get(best), bestScore); + return candidateTimes.get(best); + } catch (RuntimeException | IOException | InterruptedException ex) { + if (ex instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } + log.warn("event=highlight_vision_director_failed stage=rank error_type={} message={}", + ex.getClass().getSimpleName(), ex.getMessage()); + return Double.NaN; + } + } + + /** Highlight-worthiness of a described moment: anticipation is NOT the payoff, so it scores low. */ + static double highlightWorthiness(String description) { + if (isAnticipatory(description)) { + return 0.15; // "about to kick" / "walking up to the ball" is the build-up, never the highlight + } + return semanticScore(description); + } + /** True when a description reads as anticipation ("about to…") rather than a shown/ongoing action. */ static boolean isAnticipatory(String description) { if (description == null || description.isBlank()) { @@ -228,7 +293,8 @@ public class HighlightVisionDirector { String[] payoff = {"celebrat", "cheer", "victor", "triumph", "raising", "arms up", "arms rais", "excit", "happy", "applau", "fist", "champion", "thumbs", "dancing", "jumping"}; String[] action = {"throw", "roll", "release", "swing", "kick", "shoot", "spin", "sliding"}; - String[] idle = {"stand", "wait", "empty", "prepare", "background", "looking", "watching", "sitting"}; + String[] idle = {"stand", "wait", "empty", "prepare", "background", "looking", "watching", "sitting", + "walk", "turning", "turns", "turned", "away", "leaving", "adjust"}; if (containsAny(c, payoff)) { return 1.0; } diff --git a/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java b/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java index 0413c7a..e2bb6aa 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightMontageDirectorTest.java @@ -3,6 +3,8 @@ package org.example.videoclips.editing; import org.example.videoclips.config.VideoClippingProperties; import org.junit.jupiter.api.Test; +import java.util.List; + import static org.assertj.core.api.Assertions.assertThat; class HighlightMontageDirectorTest { @@ -10,144 +12,104 @@ class HighlightMontageDirectorTest { private final HighlightMontageDirector director = new HighlightMontageDirector(new VideoClippingProperties()); @Test - void composesAStoryStructureFromMeasuredMotionAndAudio() { - // 16s at 0.5s windows = 32 buckets. Motion: baseline with a release spike at ~4s, a celebration bump - // at ~9-11s, and a strong camera-whip tail from ~12.5s. Audio: climax (celebration) peaking at ~8s; - // loud tail spikes after ~13.5s that must be ignored (outside the central band). - int n = 32; + void proposesEveryRealMomentAsACandidateWithoutBias() { + // 64s source: an early goal (~8s) AND a late, HIGHER-intensity aftermath (~55s: loud + high motion). + // Measurement must propose BOTH as candidates — it does not get to decide which is the highlight + // (that is the vision judge's job). Note the aftermath ranks first: measurement alone would pick it. + int n = 128; double[] motion = new double[n]; double[] audio = new double[n]; for (int i = 0; i < n; i++) { double t = i * 0.5; motion[i] = 2.0; - if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0; // release spike - if (t >= 9.0 && t <= 11.0) motion[i] = 4.0; // celebration motion - if (t >= 12.5 && t <= 13.0) motion[i] = 14.0; // brief camera-whip (far above action motion) + if (t >= 7.0 && t <= 10.0) motion[i] = 7.0; // the goal + if (t >= 54.0 && t <= 56.0) motion[i] = 8.0; // the aftermath (higher motion) audio[i] = -30.0; - if (t >= 6.5 && t <= 9.5) audio[i] = -12.0; // celebration band - if (Math.abs(t - 8.0) < 0.6) audio[i] = -8.0; // climax peak - if (t >= 13.5) audio[i] = -10.0; // loud aftermath (must be ignored) + if (t >= 8.0 && t <= 11.0) audio[i] = -10.0; // goal reaction + if (t >= 54.0 && t <= 56.0) audio[i] = -12.0; // aftermath (loud) } - MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 16.0); + List candidates = director.candidatePeaks(motion, audio, 0.5, 64.0); - // A real multi-beat structure, not one straight shot. - assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4); + assertThat(candidates).anySatisfy(t -> assertThat(t).isBetween(6.0, 11.0)); // the goal is a candidate + assertThat(candidates).anySatisfy(t -> assertThat(t).isBetween(53.0, 57.0)); // the aftermath too + // Distinct moments (min-separated), strongest first -> the aftermath, which a naive pick would take. + assertThat(candidates.get(0)).isBetween(53.0, 57.0); + } - // Exactly one strong slow-motion payoff, and it sits on the audio climax (~8s). - var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList(); - assertThat(slowMo).hasSize(1); - assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(7.5, 8.6); + @Test + void buildsTheActionSegmentAroundTheCHOSENPeakAndSweepsInTheOutcome() { + // The judge chose the early goal (~8s), NOT the louder late aftermath. Given that choice, the segment + // must place the slow-mo payoff on the goal and extend the window through its outcome (ball settling). + int n = 128; + double[] motion = new double[n]; + double[] audio = new double[n]; + for (int i = 0; i < n; i++) { + double t = i * 0.5; + motion[i] = 2.0; + if (t >= 7.0 && t <= 10.0) motion[i] = 7.0; // the goal (peak) + if (t > 10.0 && t <= 13.0) motion[i] = 4.0; // the ball settling in the net (outcome) + if (t >= 54.0 && t <= 56.0) motion[i] = 8.0; // aftermath elsewhere (not chosen) + audio[i] = -30.0; + if (t >= 8.0 && t <= 11.0) audio[i] = -10.0; + } - // There is a continuous action/tension shot (long, not chopped). - double longestSpan = plan.shots().stream() - .mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0); - assertThat(longestSpan).isGreaterThan(2.0); + MontagePlan plan = director.composeMontageAt("p", "src.mp4", motion, audio, 8.5, 0.5, 64.0); - // The camera-whip tail (>=12.5s) is trimmed: no shot reads source beyond it. + double payoff = payoffSourceStart(plan); + assertThat(payoff).isBetween(7.5, 10.0); // slow-mo on the goal double maxSourceEnd = plan.shots().stream() .mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0); - assertThat(maxSourceEnd).isLessThanOrEqualTo(13.0); - + assertThat(maxSourceEnd).isGreaterThan(11.0); // outcome included + assertThat(maxSourceEnd).isLessThan(20.0); // not the far aftermath + assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4); assertThat(plan.grade()).isEqualTo("hero"); assertThat(plan.musicDirection()).isNotBlank(); } @Test - void selectsALateDecisiveMomentAndIncludesItsOutcome() { - // 64s source: nothing happens for most of it; the decisive ACTION (a kick) is at ~53s — in the last - // fifth, which the old central-band search structurally excluded — and its OUTCOME (the ball rolling - // away) runs to ~58s. The director must select the late action AND extend the window to the outcome. + void measuredFallbackUsesTheStrongestPeakWhenThereIsNoJudge() { + // With no vision judge, composeMontage builds around the strongest intensity peak (~45s here). int n = 128; double[] motion = new double[n]; double[] audio = new double[n]; for (int i = 0; i < n; i++) { double t = i * 0.5; motion[i] = 2.0; - if (t >= 52.0 && t <= 55.0) motion[i] = 7.0; // the kick (a motion event) - if (t > 55.0 && t <= 58.0) motion[i] = 4.0; // the ball rolling away (the outcome) + if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0; + if (t >= 44.0 && t <= 46.0) motion[i] = 4.0; audio[i] = -30.0; - if (t >= 53.0 && t <= 56.0) audio[i] = -10.0; // the kick / kids reacting + if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // strongest (loud + motion) } MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 64.0); - // The payoff lands on the late kick (~52-55s), not the long anticipation before it. - double payoff = payoffSourceStart(plan); - assertThat(payoff).isBetween(51.0, 56.0); - // The window reaches into the outcome (ball rolling, ~56-58s) instead of cutting off at the kick. - double maxSourceEnd = plan.shots().stream() - .mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0); - assertThat(maxSourceEnd).isGreaterThan(56.0); + assertThat(payoffSourceStart(plan)).isBetween(44.0, 46.0); } @Test void capsThePreClimaxBuildOnLongSourcesSoNoSingleShotRunsAway() { - // 64s source: action spike early (~5s), climax far away (~45s). Without the cap the action/tension - // shot would stretch ~38s (one giant shot); the cap must keep it bounded and the montage tight. + // Peak far from the pre-peak action spike; the build shot must be bounded by montage-max-build-seconds. int n = 128; double[] motion = new double[n]; double[] audio = new double[n]; for (int i = 0; i < n; i++) { double t = i * 0.5; motion[i] = 2.0; - if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0; // early action spike - if (t >= 44.0 && t <= 46.0) motion[i] = 4.0; // celebration motion near the climax + if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0; // early action spike + if (t >= 44.0 && t <= 46.0) motion[i] = 4.0; audio[i] = -30.0; - if (t >= 43.0 && t <= 46.5) audio[i] = -12.0; // celebration band - if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // climax peak ~45s + if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // strongest peak ~45s } - MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 64.0); + MontagePlan plan = director.composeMontageAt("p", "src.mp4", motion, audio, 45.0, 0.5, 64.0); - // No single shot's source span may exceed the configured build cap (6s default) plus a small margin. double longestSpan = plan.shots().stream() .mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0); assertThat(longestSpan).isLessThanOrEqualTo(6.5); - // The whole highlight stays tight (a handful of short shots around the climax), not tens of seconds. double total = plan.shots().stream().mapToDouble(MontagePlan.Shot::durationSeconds).sum(); assertThat(total).isLessThanOrEqualTo(16.0); - // The payoff still lands on the real climax (~45s), so the cap didn't move the story. - var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList(); - assertThat(slowMo).hasSize(1); - assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(44.0, 46.0); - } - - @Test - void semanticCurveMovesThePayoffToTheMeaningfulMoment() { - // Audio is slightly LOUDER at t=5 (a bang) than at t=8 (the celebration). Motion has a release spike. - int n = 32; - double[] motion = new double[n]; - double[] audio = new double[n]; - for (int i = 0; i < n; i++) { - double t = i * 0.5; - motion[i] = 2.0; - if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0; - audio[i] = -30.0; - if (Math.abs(t - 5.0) < 0.7) audio[i] = -8.0; // loudest (a wide bang), but not the highlight - if (Math.abs(t - 8.0) < 0.7) audio[i] = -14.0; // celebration, quieter - } - // Semantics say the celebration (t~8) is the highlight. - double[] semantic = new double[n]; - for (int i = 0; i < n; i++) { - double t = i * 0.5; - semantic[i] = 0.2; - if (t >= 7.5 && t <= 8.5) semantic[i] = 1.0; - } - - double payoffWithoutSemantics = payoffSourceStart( - director.composeMontage("p", "s.mp4", motion, audio, 0.5, 16.0)); - double payoffWithSemantics = payoffSourceStart( - director.composeMontage("p", "s.mp4", motion, audio, semantic, 0.5, 16.0)); - - // Without semantics the payoff sits on the loud bang (~5s); with them it moves onto the celebration (~8s). - assertThat(payoffWithoutSemantics).isLessThan(6.5); - assertThat(payoffWithSemantics).isGreaterThan(7.5); - } - - private double payoffSourceStart(MontagePlan plan) { - return plan.shots().stream().filter(s -> s.speed() < 0.8).findFirst() - .map(MontagePlan.Shot::sourceStartSeconds).orElse(-1.0); + assertThat(payoffSourceStart(plan)).isBetween(44.0, 46.0); } @Test @@ -157,4 +119,9 @@ class HighlightMontageDirectorTest { assertThat(plan.shots()).hasSize(1); assertThat(plan.shots().get(0).sourceStartSeconds()).isEqualTo(0.0); } + + private double payoffSourceStart(MontagePlan plan) { + return plan.shots().stream().filter(s -> s.speed() < 0.8).findFirst() + .map(MontagePlan.Shot::sourceStartSeconds).orElse(-1.0); + } } diff --git a/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java b/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java index b2692f4..64e1b59 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightVisionDirectorTest.java @@ -47,6 +47,24 @@ class HighlightVisionDirectorTest { assertThat(HighlightVisionDirector.semanticScore("Standing and waiting")).isEqualTo(0.2); assertThat(HighlightVisionDirector.semanticScore("A quiet room")).isEqualTo(0.4); // neutral assertThat(HighlightVisionDirector.semanticScore("")).isEqualTo(0.4); + // A subject turning/walking away is idle, not a highlight. + assertThat(HighlightVisionDirector.semanticScore("The person is turning away from the lane")).isEqualTo(0.2); + assertThat(HighlightVisionDirector.semanticScore("walking back to the seats")).isEqualTo(0.2); + } + + @Test + void theJudgeRanksTheHighlightAboveLoudOrHighMotionNonHighlights() { + // This is the generic rule: a celebration or scored goal outranks a loud turn-away or an anticipatory + // build, because highlight-worthiness is a question of MEANING, not of motion/loudness. + double celebration = HighlightVisionDirector.highlightWorthiness("raising his arms in celebration"); + double goal = HighlightVisionDirector.highlightWorthiness("kicking the ball into the net"); + double turnAway = HighlightVisionDirector.highlightWorthiness("turning away and walking back"); + double anticipation = HighlightVisionDirector.highlightWorthiness("about to kick the ball"); + + assertThat(celebration).isGreaterThan(turnAway); + assertThat(goal).isGreaterThan(turnAway); + assertThat(celebration).isGreaterThan(anticipation); // anticipation is the build-up, not the payoff + assertThat(goal).isGreaterThan(anticipation); } @Test