Choosing WHICH moment is the highlight is a question of meaning, not motion or
loudness, and there is no generic rule in measurement alone: on the real bowling
clip a camera turn-away has the highest motion AND is louder than the celebration.
Positional bands / thresholds only move which video breaks. So responsibilities
are now split, with zero per-video constants:
- HighlightMontageDirector.candidatePeaks: measurement PROPOSES the intensity
(motion+audio) local maxima, strongest-first, min-separated. No opinion on which
is the highlight; no band, no threshold.
- HighlightVisionDirector.rankDecisiveMoment: the vision model JUDGES each
candidate by highlight-worthiness (a celebration/goal outranks a loud turn-away
or an "about to..." build; anticipation is not the payoff). Highest score wins;
falls back to the strongest peak only if the model declines.
- composeMontageAt: builds the action segment (measured onset -> chosen peak ->
measured resolution that sweeps in the outcome+reaction) around the choice.
- MomentChooser interface makes the judge a drop-in: a stronger local VLM plugs in
with no director changes.
Removes the previous positional-band / semantic-weight heuristics.
Honest, verified ceiling (documented in R15): the judge is only as good as its
eyes. moondream2 perceives some actions (soccer: "kicking a soccer ball" -> the
goal is chosen correctly) but not others -- on distant portrait bowling footage it
describes every frame as "standing"/"walking" and never sees the celebration
(a posture prompt collapsed to a constant "Standing still"). When it can't
discriminate, candidates tie and it falls back to the loudest peak. This is a
model-capability limit, not a design flaw; the fix is a stronger VLM (drop-in).
mvn verify: 293 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A slow-motion shot now decelerates smoothly instead of snapping to slow-mo:
speedRampSetpts builds a log-integrated setpts that ramps playback speed from
normal (1.0) down to below the target across the shot. The shot stays a SINGLE
segment (so the R5 per-shot push-in is preserved) and the existing -t pin keeps
the planned output duration. Normal-speed shots keep a plain constant setpts.
Unit-tested; verified in a real render (payoff carries the ramp, output valid).
Completes R6 (crossfades + speed-ramp).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
MusicGen's internal structure is uncontrolled, so the mix applies a deterministic
swell envelope to the score (volume='min(1,0.5+0.5*t/peak)':eval=frame): the music
amplitude rises from 0.5x to full over the run-up to the payoff, then holds. The
peak is the payoff (slow-motion) beat's timeline midpoint on the crossfade-
compressed timeline -- generic, driven only by the plan, no content assumptions.
No-ops when there is no slow-mo beat. Filter string unit-tested; verified in a real
render (swell peaks at 6.83s, output -16.1 LUFS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
The vision director now captions several beat frames (two questions per frame in
one worker call: a discriminative description + a punchy label) and turns the
descriptions into a per-window "highlight-worthiness" curve via generic
emotion/action/idle keyword scoring (semanticScore/semanticCurve). The montage
director blends that curve with audio to place the payoff on the semantically
strongest moment; the payoff label becomes the bold overlay and the description
flavors the music. Everything is content-agnostic and fails soft to the measured
cut. Verified on bowling: the payoff moved onto moondream's detected celebration.
Honest limit: a small VLM on distant subjects is only weakly discriminative;
descriptive questions beat terse ones (which collapse to a constant answer).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
Overlays are now large (fontsize 84) with a thick outline + strong drop shadow so
they read on any background, and animate in: a snappy 0.18s alpha punch plus a
34px rise-up over 0.22s, with a soft ease-out. Placed on the payoff beat so the
entrance lands on the musical/edit accent. This presents the Tier-2 vision
director semantic caption ("STRIKE") boldly. Overlay style/animation asserted in
the existing overlay test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
Beats now blend instead of hard-cutting (incl. the cut into the slow-mo payoff)
via an xfade + acrossfade chain (xfadeTimelineCommand). The timeline compresses
by (n-1)*xf, so shiftOverlayForCrossfade re-times overlays and the reported
duration is reduced to keep overlays, loudness mastering, and QA aligned. Opt-in
via editing.crossfade-seconds (0 = hard cuts default; localpoc 0.25), clamped to
half the shortest beat. Offset math + overlay shift unit-tested; verified on the
bowling cut (visible dissolve, STRIKE overlay stayed on the payoff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
Single-pass loudnorm in the mix is only ~+/-2 LUFS accurate, so the auto-rendered
output could land quiet (e.g. -18.7 LUFS vs the -16 target). After the mix,
probeIntegratedLoudness measures the file, loudnessGainDb computes the corrective
gain, and masterLoudness applies it with a brickwall limiter for true peak. No-ops
when already on target or when the measurement is implausible; handles MusicGen
loudness variance. loudnessGainDb unit-tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
Rendering (whole service, driven by source measurements, not constants):
- Orientation-aware geometry: FfmpegClipInspector reads display rotation and
stores effective dims; HighlightFfmpegRenderer.outputGeometry renders portrait
sources portrait and skips the 2.39 letterbox on portrait (landscape unchanged).
- Dynamic exposure: probeSourceLuma measures the frames; exposureNormalizationFilter
maps the mean toward a target; the grade is now exposure-preserving (no crushed
subjects: bowling final went ~63 -> ~100 mean luma).
- Motion-adaptive in-shot push-in (zoompan), amount from per-shot YDIF.
- Audio mix ducks source audio under the generated score so it leads.
- Highlight duration is no longer capped (validator + config).
Automatic director (plans were hand-authored before):
- Tier 1 HighlightMontageDirector: composes the montage from measured motion (YDIF)
and audio-energy (RMS) curves -- setup, continuous action/tension, slow-mo payoff
on the audio climax, resolution button, camera-whip tail trimmed.
- Tier 2 HighlightVisionDirector + tools/vision_caption.py: a local, offline
vision-language model (moondream2) captions the payoff frame and augments the
montage with a semantic overlay ("STRIKE") and scene-informed music; fails soft.
- Wired into the scheduler behind auto-director-enabled / vision-director-enabled
(on in the localpoc profile).
Docs: cinematic-quality-rules.md (R1-R5, R9 both tiers), poc-plan milestones.
Tests: mvn -o verify -> 262 passing, 0 failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
JSLMPR