refactor(director): decisive moment = measurement proposes, vision model judges
Choosing WHICH moment is the highlight is a question of meaning, not motion or loudness, and there is no generic rule in measurement alone: on the real bowling clip a camera turn-away has the highest motion AND is louder than the celebration. Positional bands / thresholds only move which video breaks. So responsibilities are now split, with zero per-video constants: - HighlightMontageDirector.candidatePeaks: measurement PROPOSES the intensity (motion+audio) local maxima, strongest-first, min-separated. No opinion on which is the highlight; no band, no threshold. - HighlightVisionDirector.rankDecisiveMoment: the vision model JUDGES each candidate by highlight-worthiness (a celebration/goal outranks a loud turn-away or an "about to..." build; anticipation is not the payoff). Highest score wins; falls back to the strongest peak only if the model declines. - composeMontageAt: builds the action segment (measured onset -> chosen peak -> measured resolution that sweeps in the outcome+reaction) around the choice. - MomentChooser interface makes the judge a drop-in: a stronger local VLM plugs in with no director changes. Removes the previous positional-band / semantic-weight heuristics. Honest, verified ceiling (documented in R15): the judge is only as good as its eyes. moondream2 perceives some actions (soccer: "kicking a soccer ball" -> the goal is chosen correctly) but not others -- on distant portrait bowling footage it describes every frame as "standing"/"walking" and never sees the celebration (a posture prompt collapsed to a constant "Standing still"). When it can't discriminate, candidates tie and it falls back to the loudest peak. This is a model-capability limit, not a design flaw; the fix is a stronger VLM (drop-in). mvn verify: 293 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JSLMPR
parent
8b00b41d7c
commit
d89a28e084
|
|
@ -134,10 +134,42 @@ Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5
|
|||
- **Honest limits:** **YOLOv8 is AGPL-3.0 → non-commercial** (matches the repo's CV stance); per-frame detection
|
||||
is CPU-slow; on tiny/distant subjects detection is unreliable.
|
||||
|
||||
## R15 — Decisive moment: measurement proposes, the vision model judges ✅ (with a hard VLM ceiling)
|
||||
A highlight is an **action unit** — a start, a decisive peak, and an outcome/reaction — and choosing *which*
|
||||
moment is the highlight is a question of **meaning**, not of motion or loudness. **There is no generic rule in
|
||||
measurement alone:** a camera turn-away or a loud aftermath routinely out-scores a quiet celebration on both
|
||||
motion and audio (measured on the real bowling clip: the turn-around has the clip's highest motion *and* is
|
||||
louder than the celebration). So the design separates the two responsibilities — no positional bands, no
|
||||
per-video thresholds, ever:
|
||||
|
||||
- **Measurement PROPOSES (`HighlightMontageDirector.candidatePeaks`):** the local maxima of intensity
|
||||
(normalised motion + audio), strongest first, min-separated. Every real event becomes a candidate — a strike,
|
||||
a celebration, a turn-away, a goal, an anticipation — with **no opinion** about which is the highlight.
|
||||
- **The vision model JUDGES (`HighlightVisionDirector.rankDecisiveMoment`):** it captions each candidate and
|
||||
scores **highlight-worthiness by meaning** — a celebration or a scored goal outranks a loud turn-away or an
|
||||
"about to…" build (anticipation is explicitly *not* the payoff). The highest-scoring candidate wins.
|
||||
- **Fallback:** if the model declines/fails, the strongest-intensity peak is used (a measured last resort).
|
||||
- **Segment (`composeMontageAt`):** around the chosen peak, the **onset** (measured motion rising into it,
|
||||
build-capped) and the **resolution** (measured motion settling after it, sweeping in the outcome + reaction —
|
||||
the pins falling *and* the celebration; the ball crossing the line *and* settling in the net) are built.
|
||||
- **Honest labelling (R9/C):** a dense read of the *shown* segment names it and never asserts an action the
|
||||
segment doesn't contain — an anticipatory cut becomes a teaser question, not a false "KICK".
|
||||
|
||||
**Honest ceiling (verified, not theoretical):** the judge is only as good as its eyes. The local **moondream2**
|
||||
model reliably perceives some actions (soccer: *"kicking a soccer ball"* → the goal is chosen correctly) but
|
||||
**cannot** perceive others — on the distant, portrait bowling clip it describes every frame as *"standing"* /
|
||||
*"walking"* / *"a bowling alley"* and never sees the arms-raised celebration, regardless of prompt (a
|
||||
posture-focused prompt collapsed to a constant *"Standing still"*). When the model can't discriminate, all
|
||||
candidates tie and it falls back to the loudest peak (the turn-away). This is a **model-capability limit, not a
|
||||
design flaw** — the fix is a stronger local VLM, which is a **drop-in**: the judge is a clean interface
|
||||
(`MomentChooser` / `rankDecisiveMoment`) with no director changes required. Forcing the weak case with more
|
||||
heuristics is prohibited — that is the hack this rule exists to avoid.
|
||||
|
||||
---
|
||||
Rules R1–R14 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightBeatSync` /
|
||||
`HighlightSubjectTracker` and apply to **every** project automatically (R13/R14 behind opt-in flags). Each is
|
||||
driven by a source measurement or a global cinematic standard, never a per-video constant.
|
||||
Rules R1–R15 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightMontageDirector`
|
||||
/ `HighlightVisionDirector` / `HighlightBeatSync` / `HighlightSubjectTracker` and apply to **every** project
|
||||
automatically (R13/R14 behind opt-in flags). Each is driven by a source measurement or a global cinematic
|
||||
standard, never a per-video constant.
|
||||
|
||||
## Still missing for "cinematic" (researched gap — not yet implemented)
|
||||
Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work:
|
||||
|
|
|
|||
|
|
@ -30,6 +30,12 @@ import java.util.List;
|
|||
public class HighlightMontageDirector {
|
||||
|
||||
static final double WINDOW_SECONDS = 0.5;
|
||||
/** How many candidate decisive moments to propose for the vision judge to rank. */
|
||||
static final int MAX_CANDIDATES = 5;
|
||||
/** Minimum spacing (seconds) between proposed candidates, so they are distinct moments. Kept short so a
|
||||
* reaction close on the heels of the action (a celebration right after the strike) stays a separate
|
||||
* candidate the vision judge can score, rather than being merged into the louder neighbouring peak. */
|
||||
static final double MIN_CANDIDATE_SEPARATION = 1.0;
|
||||
|
||||
private final VideoClippingProperties.Editing properties;
|
||||
|
||||
|
|
@ -37,80 +43,134 @@ public class HighlightMontageDirector {
|
|||
this.properties = properties.getEditing();
|
||||
}
|
||||
|
||||
/** Measure the source and compose an automatic montage plan. */
|
||||
/** Measure the source and compose a montage, choosing the decisive moment by measured intensity alone. */
|
||||
public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds) {
|
||||
return direct(projectId, sourceFileName, source, durationSeconds, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compose a montage plan from measured signals, optionally guided by a Tier-2 semantic curve (per-window
|
||||
* "highlight-worthiness" from vision captions). When present, the semantic signal helps choose the payoff.
|
||||
* Measure the source, propose candidate decisive moments (intensity peaks), and let {@code chooser} — the
|
||||
* Tier-2 vision judge — pick which candidate is the actual highlight. This is the generic rule: measurement
|
||||
* never decides the moment by position or threshold (a camera whip or a loud aftermath can out-score a
|
||||
* quiet celebration); it only PROPOSES, and the model JUDGES by meaning. When no chooser is supplied, or it
|
||||
* declines (NaN), the highest-intensity candidate is used as a measured fallback.
|
||||
*/
|
||||
public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds,
|
||||
double[] semantic) {
|
||||
MomentChooser chooser) {
|
||||
double[] motion = probeCurve(source.toString(), true, durationSeconds);
|
||||
double[] audio = probeCurve(source.toString(), false, durationSeconds);
|
||||
return composeMontage(projectId, sourceFileName, motion, audio, semantic, WINDOW_SECONDS, durationSeconds);
|
||||
List<Double> candidates = candidatePeaks(motion, audio, WINDOW_SECONDS, durationSeconds);
|
||||
double climax = Double.NaN;
|
||||
if (chooser != null && !candidates.isEmpty()) {
|
||||
climax = chooser.choose(List.copyOf(candidates));
|
||||
}
|
||||
if (!Double.isFinite(climax)) {
|
||||
climax = candidates.isEmpty() ? 0.0 : candidates.get(0); // measured fallback: strongest peak
|
||||
}
|
||||
return composeMontageAt(projectId, sourceFileName, motion, audio, climax, WINDOW_SECONDS, durationSeconds);
|
||||
}
|
||||
|
||||
/** Chooses which candidate moment (seconds) is the real highlight; returns NaN to defer to measurement. */
|
||||
@FunctionalInterface
|
||||
public interface MomentChooser {
|
||||
double choose(List<Double> candidateTimeSeconds);
|
||||
}
|
||||
|
||||
/**
|
||||
* Proposes candidate decisive moments as the local maxima of measured intensity (motion + audio),
|
||||
* strongest first, min-separated. GENERIC: no positional band, no per-video threshold — every real
|
||||
* event (a strike, a celebration, a turn-away, a goal, an anticipation) becomes a candidate, and the
|
||||
* vision judge decides which is the highlight. {@code motion[i]}/{@code audio[i]} cover the window at
|
||||
* {@code i * window} seconds.
|
||||
*/
|
||||
List<Double> candidatePeaks(double[] motion, double[] audio, double window, double duration) {
|
||||
int n = Math.min(motion.length, audio.length);
|
||||
if (n < 4 || duration <= 0) {
|
||||
return List.of();
|
||||
}
|
||||
double[] mN = normalize(smooth(motion, n), n);
|
||||
double[] aN = normalize(smooth(audio, n), n);
|
||||
double[] intensity = new double[n];
|
||||
double globalMax = 0;
|
||||
for (int i = 0; i < n; i++) {
|
||||
intensity[i] = aN[i] + mN[i];
|
||||
globalMax = Math.max(globalMax, intensity[i]);
|
||||
}
|
||||
if (globalMax <= 1e-9) {
|
||||
return List.of();
|
||||
}
|
||||
int neigh = Math.max(1, (int) Math.round(1.2 / window)); // local-max over +/-1.2s
|
||||
double threshold = 0.35 * globalMax; // ignore minor bumps
|
||||
List<Integer> peaks = new ArrayList<>();
|
||||
for (int i = 1; i < n - 1; i++) { // skip degenerate first/last window
|
||||
if (intensity[i] < threshold) {
|
||||
continue;
|
||||
}
|
||||
boolean isMax = true;
|
||||
for (int j = Math.max(0, i - neigh); j <= Math.min(n - 1, i + neigh); j++) {
|
||||
if (intensity[j] > intensity[i]) {
|
||||
isMax = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (isMax) {
|
||||
peaks.add(i);
|
||||
}
|
||||
}
|
||||
peaks.sort((x, y) -> Double.compare(intensity[y], intensity[x])); // strongest first
|
||||
List<Double> chosen = new ArrayList<>();
|
||||
for (int idx : peaks) {
|
||||
double t = idx * window;
|
||||
boolean tooClose = chosen.stream().anyMatch(c -> Math.abs(c - t) < MIN_CANDIDATE_SEPARATION);
|
||||
if (!tooClose) {
|
||||
chosen.add(t);
|
||||
}
|
||||
if (chosen.size() >= MAX_CANDIDATES) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return chosen;
|
||||
}
|
||||
|
||||
MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio,
|
||||
double window, double duration) {
|
||||
return composeMontage(projectId, sourceFileName, motion, audio, null, window, duration);
|
||||
// Measured path (no vision judge): build around the strongest intensity peak.
|
||||
List<Double> candidates = candidatePeaks(motion, audio, window, duration);
|
||||
double climax = candidates.isEmpty() ? 0.0 : candidates.get(0);
|
||||
return composeMontageAt(projectId, sourceFileName, motion, audio, climax, window, duration);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure, deterministic shot composition from the measured curves. Package-visible for unit testing without
|
||||
* ffmpeg. {@code motion[i]} and {@code audio[i]} cover the window starting at {@code i * window} seconds.
|
||||
* Builds the ACTION SEGMENT (start -> peak -> outcome) around a chosen climax time. Pure and deterministic;
|
||||
* package-visible for unit testing without ffmpeg. The onset (start) and resolution (end) are MEASURED from
|
||||
* motion around the given peak — the peak itself is chosen by the caller (the vision judge).
|
||||
*/
|
||||
MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio,
|
||||
double[] semantic, double window, double duration) {
|
||||
MontagePlan composeMontageAt(String projectId, String sourceFileName, double[] motion, double[] audio,
|
||||
double climaxTime, double window, double duration) {
|
||||
int n = Math.min(motion.length, audio.length);
|
||||
if (n < 4 || duration <= 0) {
|
||||
return straightCut(projectId, sourceFileName, duration);
|
||||
}
|
||||
double[] m = smooth(motion, n);
|
||||
double[] a = smooth(audio, n);
|
||||
int climaxIdx = Math.max(1, Math.min(n - 1, (int) Math.round(climaxTime / window)));
|
||||
climaxTime = climaxIdx * window;
|
||||
|
||||
// Localize the DECISIVE MOMENT. The old code searched only the central 25-80% band on audio(+semantic),
|
||||
// which structurally EXCLUDED a payoff in the last fifth of a clip (a late kick/goal) and could land on
|
||||
// the anticipation rather than the action. Now: search almost the whole clip and localize primarily by
|
||||
// MOTION — the decisive action is a motion event — with audio an equal partner and any semantic curve a
|
||||
// weak tie-breaker. Motion catches pure-action payoffs (a kick); audio catches reaction payoffs (a
|
||||
// celebration cheer). This is the generic fix for "the highlight cut off before the actual action".
|
||||
double[] mN = normalize(m, n);
|
||||
double[] aN = normalize(a, n);
|
||||
boolean haveSemantic = semantic != null && semantic.length >= n;
|
||||
double[] sN = haveSemantic ? normalize(semantic, n) : null;
|
||||
double[] payoffSignal = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
payoffSignal[i] = aN[i] + 0.6 * mN[i] + (haveSemantic ? 0.9 * sN[i] : 0.0);
|
||||
}
|
||||
// Exclude a late CAMERA WHIP (a brief motion spike far above the clip's own motion level — the hand-held
|
||||
// pan after the action) from both the climax search and the window. The baseline is the 40th-percentile
|
||||
// motion (robust to the action spikes themselves), so widening the search to nearly the whole clip can't
|
||||
// land the payoff on the messy aftermath — while a genuine late action (well below whip level) stays in.
|
||||
// Onset: the biggest motion spike before the peak (the release/approach); the action shot leads in.
|
||||
int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)),
|
||||
Math.max(2, (int) Math.floor((climaxTime - 1.0) / window)));
|
||||
double actionTime = actionIdx * window;
|
||||
|
||||
// Resolution: extend past the peak through the outcome AND the immediate reaction (the pins falling and
|
||||
// the celebration; the ball settling in the net), stopping when motion settles back to baseline for a
|
||||
// sustained stretch, at a camera whip, or at a cap. This is MEASURED, not positional.
|
||||
double whipLevel = Math.max(1e-6, percentile(m, n, 0.4) * 4.0);
|
||||
int whipCutoff = n;
|
||||
for (int i = (int) Math.floor(0.55 * n); i < n; i++) {
|
||||
for (int i = climaxIdx + 2; i < n; i++) {
|
||||
if (m[i] > whipLevel) {
|
||||
whipCutoff = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
int lo = (int) Math.floor(0.08 * n);
|
||||
int searchHi = Math.min((int) Math.ceil(0.95 * n), whipCutoff);
|
||||
int climaxIdx = argMax(payoffSignal, lo, Math.max(lo + 1, searchHi));
|
||||
double climaxTime = climaxIdx * window;
|
||||
|
||||
// Action = the biggest motion spike before the climax; the action shot leads in ~1.5s before it so
|
||||
// the release is included and the release->roll->watch stays continuous.
|
||||
int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)),
|
||||
Math.max(2, (int) Math.floor((climaxTime - 1.0) / window)));
|
||||
double actionTime = actionIdx * window;
|
||||
|
||||
// Resolution: the highlight must show the OUTCOME, not cut off at the action. Extend past the climax
|
||||
// while motion stays elevated (the action and its result, e.g. the ball rolling away), and stop when
|
||||
// motion SETTLES back near baseline for a sustained stretch, at the whip, or at a cap.
|
||||
double baseMotion = mean(m, 0, Math.max(1, climaxIdx));
|
||||
double settleLevel = baseMotion * 1.4;
|
||||
double maxResolution = climaxTime + 6.0;
|
||||
|
|
|
|||
|
|
@ -228,15 +228,18 @@ public class HighlightSourceScheduler {
|
|||
HighlightSourceAnalysis analysis, CinematicHighlightAnalysis cinematic, long scanId) {
|
||||
try {
|
||||
double duration = analysis.source() == null ? 0.0 : analysis.source().durationSeconds();
|
||||
// Localize the decisive moment from MEASURED motion + audio (robust, whole-clip). A sparse whole-clip
|
||||
// VLM pass used to mislocalize the payoff onto the anticipation, so the VLM is now used only to
|
||||
// GROUND the overlay in the decisive window (below) — never to pick the moment.
|
||||
MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration);
|
||||
Path visionWork = store.directorDirectory(projectId).resolve("vision-work");
|
||||
// GENERIC rule: measurement PROPOSES candidate decisive moments (intensity peaks); the vision judge
|
||||
// PICKS which is the real highlight by meaning (a celebration/goal beats a loud turn-away). No
|
||||
// positional band, no per-video threshold. Falls back to the strongest peak if the model declines.
|
||||
HighlightMontageDirector.MomentChooser chooser = properties.isVisionDirectorEnabled()
|
||||
? candidates -> visionDirector.rankDecisiveMoment(sourcePath, candidates, visionWork)
|
||||
: null;
|
||||
MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration, chooser);
|
||||
if (properties.isVisionDirectorEnabled()) {
|
||||
// Dense pass over the decisive window (payoff shot through the outcome) with the honesty rule:
|
||||
// never assert an action the shown window doesn't contain (a "KICK" over a cut before the kick).
|
||||
montage = visionDirector.groundOverlay(montage, sourcePath,
|
||||
store.directorDirectory(projectId).resolve("vision-work"));
|
||||
// Dense pass over the chosen decisive window (payoff shot through the outcome) with the honesty
|
||||
// rule: never assert an action the shown window doesn't contain (a "KICK" over a cut before it).
|
||||
montage = visionDirector.groundOverlay(montage, sourcePath, visionWork);
|
||||
}
|
||||
store.writeJson(projectId, "director/montage.json", montage);
|
||||
// Minimal edit-plan.json so the render scanner selects the project; montage.json takes precedence.
|
||||
|
|
|
|||
|
|
@ -29,6 +29,14 @@ public class HighlightVisionDirector {
|
|||
private static final Logger log = LoggerFactory.getLogger(HighlightVisionDirector.class);
|
||||
private static final String SCRIPT = "tools/vision_caption.py";
|
||||
private static final long TIMEOUT_SECONDS = 600;
|
||||
/**
|
||||
* A descriptive, open-ended question. Small VLMs give far more discriminative answers to this than to a
|
||||
* terse posture question (an example-laden "arms raised / mid-throw / standing" prompt collapses to a
|
||||
* constant "Standing still" on distant footage). The description feeds both the highlight-worthiness score
|
||||
* and the honest-overlay anticipation check.
|
||||
*/
|
||||
private static final String DESCRIPTIVE_QUESTION =
|
||||
"In one sentence, describe what the person is doing with their body right now.";
|
||||
|
||||
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
|
||||
private final String ffmpegBinary;
|
||||
|
|
@ -115,8 +123,7 @@ public class HighlightVisionDirector {
|
|||
Path frame = workDir.resolve("payoff-" + i + ".jpg");
|
||||
if (extractFrame(source, t, frame)) {
|
||||
String img = frame.toAbsolutePath().toString();
|
||||
manifest.add(new Manifest("d" + i, img,
|
||||
"In one sentence, describe what the person is doing with their body right now."));
|
||||
manifest.add(new Manifest("d" + i, img, DESCRIPTIVE_QUESTION));
|
||||
manifest.add(new Manifest("l" + i, img,
|
||||
"In one to three words, what is the exciting action in this moment?"));
|
||||
manifest.add(new Manifest("q" + i, img,
|
||||
|
|
@ -166,6 +173,64 @@ public class HighlightVisionDirector {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The Tier-2 vision JUDGE. Given candidate decisive moments proposed by measurement, it captions each and
|
||||
* returns the time of the most highlight-worthy one — a celebration or a scored goal beats a loud
|
||||
* camera turn-away or an anticipatory build, because "is this THE highlight?" is a question of meaning, not
|
||||
* of motion or loudness. Returns NaN on any failure so the director falls back to the strongest peak.
|
||||
*/
|
||||
public double rankDecisiveMoment(Path source, List<Double> candidateTimes, Path workDir) {
|
||||
if (candidateTimes == null || candidateTimes.isEmpty()) {
|
||||
return Double.NaN;
|
||||
}
|
||||
try {
|
||||
Files.createDirectories(workDir);
|
||||
List<Manifest> manifest = new ArrayList<>();
|
||||
List<Integer> ids = new ArrayList<>();
|
||||
for (int i = 0; i < candidateTimes.size(); i++) {
|
||||
Path frame = workDir.resolve("cand-" + i + ".jpg");
|
||||
if (extractFrame(source, candidateTimes.get(i), frame)) {
|
||||
manifest.add(new Manifest("d" + i, frame.toAbsolutePath().toString(), DESCRIPTIVE_QUESTION));
|
||||
ids.add(i);
|
||||
}
|
||||
}
|
||||
if (ids.isEmpty()) {
|
||||
return Double.NaN;
|
||||
}
|
||||
List<Caption> caps = runCaptioner(workDir, manifest);
|
||||
int best = -1;
|
||||
double bestScore = -1;
|
||||
for (int i : ids) {
|
||||
double score = highlightWorthiness(answerFor(caps, "d" + i));
|
||||
if (score > bestScore) { // strict >: ties keep the stronger-intensity (earlier) candidate
|
||||
bestScore = score;
|
||||
best = i;
|
||||
}
|
||||
}
|
||||
if (best < 0) {
|
||||
return Double.NaN;
|
||||
}
|
||||
log.info("event=highlight_vision_moment_ranked candidates={} chosen_time={} score={}",
|
||||
ids.size(), candidateTimes.get(best), bestScore);
|
||||
return candidateTimes.get(best);
|
||||
} catch (RuntimeException | IOException | InterruptedException ex) {
|
||||
if (ex instanceof InterruptedException) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
log.warn("event=highlight_vision_director_failed stage=rank error_type={} message={}",
|
||||
ex.getClass().getSimpleName(), ex.getMessage());
|
||||
return Double.NaN;
|
||||
}
|
||||
}
|
||||
|
||||
/** Highlight-worthiness of a described moment: anticipation is NOT the payoff, so it scores low. */
|
||||
static double highlightWorthiness(String description) {
|
||||
if (isAnticipatory(description)) {
|
||||
return 0.15; // "about to kick" / "walking up to the ball" is the build-up, never the highlight
|
||||
}
|
||||
return semanticScore(description);
|
||||
}
|
||||
|
||||
/** True when a description reads as anticipation ("about to…") rather than a shown/ongoing action. */
|
||||
static boolean isAnticipatory(String description) {
|
||||
if (description == null || description.isBlank()) {
|
||||
|
|
@ -228,7 +293,8 @@ public class HighlightVisionDirector {
|
|||
String[] payoff = {"celebrat", "cheer", "victor", "triumph", "raising", "arms up", "arms rais",
|
||||
"excit", "happy", "applau", "fist", "champion", "thumbs", "dancing", "jumping"};
|
||||
String[] action = {"throw", "roll", "release", "swing", "kick", "shoot", "spin", "sliding"};
|
||||
String[] idle = {"stand", "wait", "empty", "prepare", "background", "looking", "watching", "sitting"};
|
||||
String[] idle = {"stand", "wait", "empty", "prepare", "background", "looking", "watching", "sitting",
|
||||
"walk", "turning", "turns", "turned", "away", "leaving", "adjust"};
|
||||
if (containsAny(c, payoff)) {
|
||||
return 1.0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@ package org.example.videoclips.editing;
|
|||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class HighlightMontageDirectorTest {
|
||||
|
|
@ -10,81 +12,84 @@ class HighlightMontageDirectorTest {
|
|||
private final HighlightMontageDirector director = new HighlightMontageDirector(new VideoClippingProperties());
|
||||
|
||||
@Test
|
||||
void composesAStoryStructureFromMeasuredMotionAndAudio() {
|
||||
// 16s at 0.5s windows = 32 buckets. Motion: baseline with a release spike at ~4s, a celebration bump
|
||||
// at ~9-11s, and a strong camera-whip tail from ~12.5s. Audio: climax (celebration) peaking at ~8s;
|
||||
// loud tail spikes after ~13.5s that must be ignored (outside the central band).
|
||||
int n = 32;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0; // release spike
|
||||
if (t >= 9.0 && t <= 11.0) motion[i] = 4.0; // celebration motion
|
||||
if (t >= 12.5 && t <= 13.0) motion[i] = 14.0; // brief camera-whip (far above action motion)
|
||||
audio[i] = -30.0;
|
||||
if (t >= 6.5 && t <= 9.5) audio[i] = -12.0; // celebration band
|
||||
if (Math.abs(t - 8.0) < 0.6) audio[i] = -8.0; // climax peak
|
||||
if (t >= 13.5) audio[i] = -10.0; // loud aftermath (must be ignored)
|
||||
}
|
||||
|
||||
MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 16.0);
|
||||
|
||||
// A real multi-beat structure, not one straight shot.
|
||||
assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4);
|
||||
|
||||
// Exactly one strong slow-motion payoff, and it sits on the audio climax (~8s).
|
||||
var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList();
|
||||
assertThat(slowMo).hasSize(1);
|
||||
assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(7.5, 8.6);
|
||||
|
||||
// There is a continuous action/tension shot (long, not chopped).
|
||||
double longestSpan = plan.shots().stream()
|
||||
.mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0);
|
||||
assertThat(longestSpan).isGreaterThan(2.0);
|
||||
|
||||
// The camera-whip tail (>=12.5s) is trimmed: no shot reads source beyond it.
|
||||
double maxSourceEnd = plan.shots().stream()
|
||||
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
|
||||
assertThat(maxSourceEnd).isLessThanOrEqualTo(13.0);
|
||||
|
||||
assertThat(plan.grade()).isEqualTo("hero");
|
||||
assertThat(plan.musicDirection()).isNotBlank();
|
||||
}
|
||||
|
||||
@Test
|
||||
void selectsALateDecisiveMomentAndIncludesItsOutcome() {
|
||||
// 64s source: nothing happens for most of it; the decisive ACTION (a kick) is at ~53s — in the last
|
||||
// fifth, which the old central-band search structurally excluded — and its OUTCOME (the ball rolling
|
||||
// away) runs to ~58s. The director must select the late action AND extend the window to the outcome.
|
||||
void proposesEveryRealMomentAsACandidateWithoutBias() {
|
||||
// 64s source: an early goal (~8s) AND a late, HIGHER-intensity aftermath (~55s: loud + high motion).
|
||||
// Measurement must propose BOTH as candidates — it does not get to decide which is the highlight
|
||||
// (that is the vision judge's job). Note the aftermath ranks first: measurement alone would pick it.
|
||||
int n = 128;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (t >= 52.0 && t <= 55.0) motion[i] = 7.0; // the kick (a motion event)
|
||||
if (t > 55.0 && t <= 58.0) motion[i] = 4.0; // the ball rolling away (the outcome)
|
||||
if (t >= 7.0 && t <= 10.0) motion[i] = 7.0; // the goal
|
||||
if (t >= 54.0 && t <= 56.0) motion[i] = 8.0; // the aftermath (higher motion)
|
||||
audio[i] = -30.0;
|
||||
if (t >= 53.0 && t <= 56.0) audio[i] = -10.0; // the kick / kids reacting
|
||||
if (t >= 8.0 && t <= 11.0) audio[i] = -10.0; // goal reaction
|
||||
if (t >= 54.0 && t <= 56.0) audio[i] = -12.0; // aftermath (loud)
|
||||
}
|
||||
|
||||
List<Double> candidates = director.candidatePeaks(motion, audio, 0.5, 64.0);
|
||||
|
||||
assertThat(candidates).anySatisfy(t -> assertThat(t).isBetween(6.0, 11.0)); // the goal is a candidate
|
||||
assertThat(candidates).anySatisfy(t -> assertThat(t).isBetween(53.0, 57.0)); // the aftermath too
|
||||
// Distinct moments (min-separated), strongest first -> the aftermath, which a naive pick would take.
|
||||
assertThat(candidates.get(0)).isBetween(53.0, 57.0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void buildsTheActionSegmentAroundTheCHOSENPeakAndSweepsInTheOutcome() {
|
||||
// The judge chose the early goal (~8s), NOT the louder late aftermath. Given that choice, the segment
|
||||
// must place the slow-mo payoff on the goal and extend the window through its outcome (ball settling).
|
||||
int n = 128;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (t >= 7.0 && t <= 10.0) motion[i] = 7.0; // the goal (peak)
|
||||
if (t > 10.0 && t <= 13.0) motion[i] = 4.0; // the ball settling in the net (outcome)
|
||||
if (t >= 54.0 && t <= 56.0) motion[i] = 8.0; // aftermath elsewhere (not chosen)
|
||||
audio[i] = -30.0;
|
||||
if (t >= 8.0 && t <= 11.0) audio[i] = -10.0;
|
||||
}
|
||||
|
||||
MontagePlan plan = director.composeMontageAt("p", "src.mp4", motion, audio, 8.5, 0.5, 64.0);
|
||||
|
||||
double payoff = payoffSourceStart(plan);
|
||||
assertThat(payoff).isBetween(7.5, 10.0); // slow-mo on the goal
|
||||
double maxSourceEnd = plan.shots().stream()
|
||||
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
|
||||
assertThat(maxSourceEnd).isGreaterThan(11.0); // outcome included
|
||||
assertThat(maxSourceEnd).isLessThan(20.0); // not the far aftermath
|
||||
assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4);
|
||||
assertThat(plan.grade()).isEqualTo("hero");
|
||||
assertThat(plan.musicDirection()).isNotBlank();
|
||||
}
|
||||
|
||||
@Test
|
||||
void measuredFallbackUsesTheStrongestPeakWhenThereIsNoJudge() {
|
||||
// With no vision judge, composeMontage builds around the strongest intensity peak (~45s here).
|
||||
int n = 128;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0;
|
||||
if (t >= 44.0 && t <= 46.0) motion[i] = 4.0;
|
||||
audio[i] = -30.0;
|
||||
if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // strongest (loud + motion)
|
||||
}
|
||||
|
||||
MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 64.0);
|
||||
|
||||
// The payoff lands on the late kick (~52-55s), not the long anticipation before it.
|
||||
double payoff = payoffSourceStart(plan);
|
||||
assertThat(payoff).isBetween(51.0, 56.0);
|
||||
// The window reaches into the outcome (ball rolling, ~56-58s) instead of cutting off at the kick.
|
||||
double maxSourceEnd = plan.shots().stream()
|
||||
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
|
||||
assertThat(maxSourceEnd).isGreaterThan(56.0);
|
||||
assertThat(payoffSourceStart(plan)).isBetween(44.0, 46.0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void capsThePreClimaxBuildOnLongSourcesSoNoSingleShotRunsAway() {
|
||||
// 64s source: action spike early (~5s), climax far away (~45s). Without the cap the action/tension
|
||||
// shot would stretch ~38s (one giant shot); the cap must keep it bounded and the montage tight.
|
||||
// Peak far from the pre-peak action spike; the build shot must be bounded by montage-max-build-seconds.
|
||||
int n = 128;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
|
|
@ -92,62 +97,19 @@ class HighlightMontageDirectorTest {
|
|||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (Math.abs(t - 5.0) < 0.6) motion[i] = 6.0; // early action spike
|
||||
if (t >= 44.0 && t <= 46.0) motion[i] = 4.0; // celebration motion near the climax
|
||||
if (t >= 44.0 && t <= 46.0) motion[i] = 4.0;
|
||||
audio[i] = -30.0;
|
||||
if (t >= 43.0 && t <= 46.5) audio[i] = -12.0; // celebration band
|
||||
if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // climax peak ~45s
|
||||
if (Math.abs(t - 45.0) < 0.6) audio[i] = -8.0; // strongest peak ~45s
|
||||
}
|
||||
|
||||
MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 64.0);
|
||||
MontagePlan plan = director.composeMontageAt("p", "src.mp4", motion, audio, 45.0, 0.5, 64.0);
|
||||
|
||||
// No single shot's source span may exceed the configured build cap (6s default) plus a small margin.
|
||||
double longestSpan = plan.shots().stream()
|
||||
.mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0);
|
||||
assertThat(longestSpan).isLessThanOrEqualTo(6.5);
|
||||
// The whole highlight stays tight (a handful of short shots around the climax), not tens of seconds.
|
||||
double total = plan.shots().stream().mapToDouble(MontagePlan.Shot::durationSeconds).sum();
|
||||
assertThat(total).isLessThanOrEqualTo(16.0);
|
||||
// The payoff still lands on the real climax (~45s), so the cap didn't move the story.
|
||||
var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList();
|
||||
assertThat(slowMo).hasSize(1);
|
||||
assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(44.0, 46.0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void semanticCurveMovesThePayoffToTheMeaningfulMoment() {
|
||||
// Audio is slightly LOUDER at t=5 (a bang) than at t=8 (the celebration). Motion has a release spike.
|
||||
int n = 32;
|
||||
double[] motion = new double[n];
|
||||
double[] audio = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
motion[i] = 2.0;
|
||||
if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0;
|
||||
audio[i] = -30.0;
|
||||
if (Math.abs(t - 5.0) < 0.7) audio[i] = -8.0; // loudest (a wide bang), but not the highlight
|
||||
if (Math.abs(t - 8.0) < 0.7) audio[i] = -14.0; // celebration, quieter
|
||||
}
|
||||
// Semantics say the celebration (t~8) is the highlight.
|
||||
double[] semantic = new double[n];
|
||||
for (int i = 0; i < n; i++) {
|
||||
double t = i * 0.5;
|
||||
semantic[i] = 0.2;
|
||||
if (t >= 7.5 && t <= 8.5) semantic[i] = 1.0;
|
||||
}
|
||||
|
||||
double payoffWithoutSemantics = payoffSourceStart(
|
||||
director.composeMontage("p", "s.mp4", motion, audio, 0.5, 16.0));
|
||||
double payoffWithSemantics = payoffSourceStart(
|
||||
director.composeMontage("p", "s.mp4", motion, audio, semantic, 0.5, 16.0));
|
||||
|
||||
// Without semantics the payoff sits on the loud bang (~5s); with them it moves onto the celebration (~8s).
|
||||
assertThat(payoffWithoutSemantics).isLessThan(6.5);
|
||||
assertThat(payoffWithSemantics).isGreaterThan(7.5);
|
||||
}
|
||||
|
||||
private double payoffSourceStart(MontagePlan plan) {
|
||||
return plan.shots().stream().filter(s -> s.speed() < 0.8).findFirst()
|
||||
.map(MontagePlan.Shot::sourceStartSeconds).orElse(-1.0);
|
||||
assertThat(payoffSourceStart(plan)).isBetween(44.0, 46.0);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
@ -157,4 +119,9 @@ class HighlightMontageDirectorTest {
|
|||
assertThat(plan.shots()).hasSize(1);
|
||||
assertThat(plan.shots().get(0).sourceStartSeconds()).isEqualTo(0.0);
|
||||
}
|
||||
|
||||
private double payoffSourceStart(MontagePlan plan) {
|
||||
return plan.shots().stream().filter(s -> s.speed() < 0.8).findFirst()
|
||||
.map(MontagePlan.Shot::sourceStartSeconds).orElse(-1.0);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -47,6 +47,24 @@ class HighlightVisionDirectorTest {
|
|||
assertThat(HighlightVisionDirector.semanticScore("Standing and waiting")).isEqualTo(0.2);
|
||||
assertThat(HighlightVisionDirector.semanticScore("A quiet room")).isEqualTo(0.4); // neutral
|
||||
assertThat(HighlightVisionDirector.semanticScore("")).isEqualTo(0.4);
|
||||
// A subject turning/walking away is idle, not a highlight.
|
||||
assertThat(HighlightVisionDirector.semanticScore("The person is turning away from the lane")).isEqualTo(0.2);
|
||||
assertThat(HighlightVisionDirector.semanticScore("walking back to the seats")).isEqualTo(0.2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void theJudgeRanksTheHighlightAboveLoudOrHighMotionNonHighlights() {
|
||||
// This is the generic rule: a celebration or scored goal outranks a loud turn-away or an anticipatory
|
||||
// build, because highlight-worthiness is a question of MEANING, not of motion/loudness.
|
||||
double celebration = HighlightVisionDirector.highlightWorthiness("raising his arms in celebration");
|
||||
double goal = HighlightVisionDirector.highlightWorthiness("kicking the ball into the net");
|
||||
double turnAway = HighlightVisionDirector.highlightWorthiness("turning away and walking back");
|
||||
double anticipation = HighlightVisionDirector.highlightWorthiness("about to kick the ball");
|
||||
|
||||
assertThat(celebration).isGreaterThan(turnAway);
|
||||
assertThat(goal).isGreaterThan(turnAway);
|
||||
assertThat(celebration).isGreaterThan(anticipation); // anticipation is the build-up, not the payoff
|
||||
assertThat(goal).isGreaterThan(anticipation);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
|
|||
Loading…
Reference in New Issue