Automatic two-tier highlight director + source-adaptive rendering
Rendering (whole service, driven by source measurements, not constants):
- Orientation-aware geometry: FfmpegClipInspector reads display rotation and
stores effective dims; HighlightFfmpegRenderer.outputGeometry renders portrait
sources portrait and skips the 2.39 letterbox on portrait (landscape unchanged).
- Dynamic exposure: probeSourceLuma measures the frames; exposureNormalizationFilter
maps the mean toward a target; the grade is now exposure-preserving (no crushed
subjects: bowling final went ~63 -> ~100 mean luma).
- Motion-adaptive in-shot push-in (zoompan), amount from per-shot YDIF.
- Audio mix ducks source audio under the generated score so it leads.
- Highlight duration is no longer capped (validator + config).
Automatic director (plans were hand-authored before):
- Tier 1 HighlightMontageDirector: composes the montage from measured motion (YDIF)
and audio-energy (RMS) curves -- setup, continuous action/tension, slow-mo payoff
on the audio climax, resolution button, camera-whip tail trimmed.
- Tier 2 HighlightVisionDirector + tools/vision_caption.py: a local, offline
vision-language model (moondream2) captions the payoff frame and augments the
montage with a semantic overlay ("STRIKE") and scene-informed music; fails soft.
- Wired into the scheduler behind auto-director-enabled / vision-director-enabled
(on in the localpoc profile).
Docs: cinematic-quality-rules.md (R1-R5, R9 both tiers), poc-plan milestones.
Tests: mvn -o verify -> 262 passing, 0 failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MPuJXQyAeWpFcTtcnxo1UN
This commit is contained in:
JSLMPR
parent
8b68116dae
commit
1da0661eea
|
|
@ -0,0 +1,65 @@
|
||||||
|
# AI Handover: Local Cinematic Highlight PoC
|
||||||
|
|
||||||
|
Last updated: 2026-07-23 (Europe/Berlin). **This file goes stale — verify before trusting it.**
|
||||||
|
|
||||||
|
## FIRST: check for staleness (prevents the 2026-07-23 stale-session incident)
|
||||||
|
|
||||||
|
This handover was once resumed from an old commit while the repo had advanced 2 days on another
|
||||||
|
branch/agent, wasting a whole session re-deriving already-built work. Do NOT let that happen:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git rev-parse --short HEAD # compare to "Recorded HEAD" below
|
||||||
|
git log -8 --oneline --decorate
|
||||||
|
git status --short # uncommitted work may be the real current state
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Recorded HEAD when this was written: `8b68116`**, plus uncommitted working-tree changes (portrait +
|
||||||
|
audio-mix + unbounded-duration fixes, see below).
|
||||||
|
- If HEAD is newer than `8b68116`, or there are unfamiliar commits/uncommitted files, **this file is behind
|
||||||
|
the repo**. The authoritative live state is: the git log, `git status`, and
|
||||||
|
`docs/cinematic-highlight-poc-plan.md` (milestone log at its tail). Trust those over this file.
|
||||||
|
- If a `claude agents` background session is still "running" for this repo, only one agent should mutate the
|
||||||
|
repo at a time (renders/commits/state). Confirm the others are stopped.
|
||||||
|
|
||||||
|
## What exists NOW (built, not aspirational)
|
||||||
|
|
||||||
|
The full local single-source cinematic-highlight pipeline is implemented and working end-to-end:
|
||||||
|
analysis → candidates/category → (manual) director plan → local asset generation → cinematic FFmpeg render
|
||||||
|
→ QA. Models are provisioned and proven (Piper voice, MusicGen music, AudioLDM2 SFX) — see
|
||||||
|
[[local-model-runtime-intel-mac]]. The user authorized model downloads on 2026-07-21 (this **reversed** the
|
||||||
|
earlier no-download stance; that reversal is real and current).
|
||||||
|
|
||||||
|
To run source → final render, follow [[highlight-e2e-render-runbook]] — do not re-derive it.
|
||||||
|
|
||||||
|
## Fixes applied 2026-07-23 (whole service, not just PoC) — uncommitted in working tree
|
||||||
|
|
||||||
|
1. **Portrait/orientation**: `FfmpegClipInspector` now reads rotation side-data/`rotate` tag and stores the
|
||||||
|
EFFECTIVE (display) width/height. `HighlightFfmpegRenderer.outputGeometry()` renders portrait sources to a
|
||||||
|
portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait phone footage being
|
||||||
|
stretched into landscape.
|
||||||
|
2. **Audio mix**: source audio (`[0:a]`) is now ducked (-20 dB, or -24 with narration) so the generated score
|
||||||
|
leads instead of being buried; montage music cue raised to -9 dB. In `HighlightFfmpegRenderer.audioMixCommand`
|
||||||
|
+ the montage cue in `HighlightDirectorFlowService`.
|
||||||
|
3. **Duration constraint removed**: `HighlightDirectorPlanValidator` no longer enforces min/max highlight
|
||||||
|
duration (only positive/finite + sane playback speed + candidate containment). Config defaults relaxed
|
||||||
|
(min 2 / max 3600 as soft candidate hints only). A highlight can be as long as the story needs. Montage path
|
||||||
|
was already unconstrained.
|
||||||
|
|
||||||
|
`mvn -o verify`: 254 tests, 0 failures (JaCoCo gate passes). 6 new tests cover the above. NOT yet committed —
|
||||||
|
the user has not asked to commit.
|
||||||
|
|
||||||
|
## Still open / next levers (from docs/cinematic-highlight-poc-plan.md Phase 5)
|
||||||
|
|
||||||
|
- Source footage quality is the ceiling (e.g. 576p phone clips upscale soft). Biggest lever.
|
||||||
|
- No resident local **director** — plans are hand-authored (`montage.json`). P5.2/P5.5.
|
||||||
|
- Bigger/continuous music (musicgen-medium is slow on CPU) — P5.3.
|
||||||
|
- In-shot `zoompan` push-in, beat-synced cut lengths — P5.4.
|
||||||
|
- Licensing: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL → non-commercial. Production needs
|
||||||
|
commercially-licensed models — P5.7.
|
||||||
|
- Production hardening (security, CI/CD, containers, PostgreSQL, no-egress cert) deferred until quality passes.
|
||||||
|
|
||||||
|
## Non-negotiables still in force (except downloads, which the user authorized)
|
||||||
|
|
||||||
|
No external AI services in the media path; no unlicensed assets shipped as production; no placeholder
|
||||||
|
silence/tones as success; no rendering without explicit approval; record license/provenance for every model.
|
||||||
|
Manual approval is fine for the PoC. Do not commit unless the user asks.
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
# Video Editing Service: Current State
|
||||||
|
|
||||||
|
Last verified: 2026-07-23 (Europe/Berlin). **Verify against git before trusting — see ai-handover.md.**
|
||||||
|
|
||||||
|
## Snapshot
|
||||||
|
- Branch `feature/imrovements`; recorded HEAD `8b68116` + uncommitted working-tree fixes (2026-07-23).
|
||||||
|
- `mvn -o verify`: **254 tests, 0 failures**, JaCoCo gate passes.
|
||||||
|
- The full local single-source cinematic-highlight pipeline is BUILT and works end-to-end:
|
||||||
|
analysis → candidates/category → manual `montage.json` director plan → local audio generation
|
||||||
|
(Piper voice, MusicGen music, AudioLDM2 SFX) → cinematic FFmpeg render (grade, grain, overlays,
|
||||||
|
multi-cut, slow-mo, orientation-aware geometry) → QA.
|
||||||
|
- Models provisioned + proven locally; user authorized downloads 2026-07-21 (reversed the no-download stance).
|
||||||
|
|
||||||
|
## Authoritative sources (do not duplicate them here)
|
||||||
|
- Live code/state: `git log`, `git status`.
|
||||||
|
- Roadmap + milestone log: `docs/cinematic-highlight-poc-plan.md`.
|
||||||
|
- How to render e2e: memory `highlight-e2e-render-runbook`.
|
||||||
|
- Model runtime facts: memory `local-model-runtime-intel-mac`.
|
||||||
|
- Handover + staleness check: `.claude/memory/ai-handover.md`.
|
||||||
|
|
||||||
|
## 2026-07-23 fixes (whole service)
|
||||||
|
- Orientation-aware: portrait sources render portrait (rotation read in `FfmpegClipInspector`,
|
||||||
|
geometry chosen in `HighlightFfmpegRenderer.outputGeometry`), letterbox landscape-only.
|
||||||
|
- Audio mix: source audio ducked under the generated score (score now leads).
|
||||||
|
- Highlight duration no longer capped (validator + config); a highlight can be as long as needed.
|
||||||
|
|
||||||
|
## Known limits / next (Phase 5 in poc-plan)
|
||||||
|
Source footage quality is the ceiling; no resident local director (plans are manual); bigger/continuous
|
||||||
|
music pending; licensing of local models is non-commercial (CC-BY-NC / AGPL); production hardening deferred
|
||||||
|
until the creative-quality gate passes. Highlight rendering still defaults OFF; approval still required.
|
||||||
|
|
@ -144,21 +144,63 @@ better music, NO voiceover, bolder finish. Done so far (commit 842f2a7):
|
||||||
Verified: mvn 247/0; final −16.5 LUFS, TP −2.7 dBTP, 1080p, no VO. Deliverable:
|
Verified: mvn 247/0; final −16.5 LUFS, TP −2.7 dBTP, 1080p, no VO. Deliverable:
|
||||||
`output/localpoc/highlight-projects/dji_21230510013241_0091_d/final.mp4`.
|
`output/localpoc/highlight-projects/dji_21230510013241_0091_d/final.mp4`.
|
||||||
|
|
||||||
|
### 2026-07-23 — Second source (bowling strike) + orientation/audio/duration fixes
|
||||||
|
- New source `bowling_strike.mp4` (portrait phone footage) rendered e2e via montage; confirmed the DJI
|
||||||
|
Phase-5 finding that **source footage is the ceiling**.
|
||||||
|
- **Service-wide fixes applied** (not just PoC), all with tests, `mvn -o verify` 254/0:
|
||||||
|
- **Orientation-aware rendering.** `FfmpegClipInspector` now reads display rotation (side-data + `rotate`
|
||||||
|
tag) and stores effective width/height; `HighlightFfmpegRenderer.outputGeometry()` renders portrait
|
||||||
|
sources to a portrait frame and skips the 2.39 letterbox (landscape unchanged). Fixes portrait footage
|
||||||
|
being stretched into landscape.
|
||||||
|
- **Audio mix.** Source audio (`[0:a]`) ducked to −20 dB (−24 with narration) so the generated score leads
|
||||||
|
instead of being buried; montage music cue raised to −9 dB.
|
||||||
|
- **Highlight duration constraint removed.** Validator no longer enforces min/max (only positive/finite +
|
||||||
|
sane speed + candidate containment); config bounds relaxed to soft candidate hints (min 2 / max 3600).
|
||||||
|
A highlight can be as long as the story needs.
|
||||||
|
- E2e render flow captured in memory `highlight-e2e-render-runbook` (token-saver).
|
||||||
|
|
||||||
### Phase 5 — Next steps (roadmap as of 2026-07-22, after montage mode)
|
### Phase 5 — Next steps (roadmap as of 2026-07-22, after montage mode)
|
||||||
To reach genuinely cinematic output (Gate B):
|
To reach genuinely cinematic output (Gate B):
|
||||||
- [ ] P5.1 **Better source footage** (biggest lever; source is the real ceiling — parked car / industrial lot).
|
- [ ] P5.1 **Better source footage** (biggest lever; source is the real ceiling — parked car / industrial lot).
|
||||||
- [ ] P5.2 **Automate shot selection** — generate the montage shot list from analysis (scene cuts + CV
|
- [x] P5.2 **Automate shot selection — Tier 1 (DONE 2026-07-23).** `HighlightMontageDirector` composes
|
||||||
quality/variety) instead of the current hand-authored `montage.json`.
|
`director/montage.json` from measured motion (YDIF) + audio-energy (RMS) curves: setup → continuous
|
||||||
|
action/tension → slow-mo payoff on the audio climax → resolution button, camera-whip tail trimmed. Wired
|
||||||
|
into the scheduler behind `auto-director-enabled` (localpoc on). Verified: auto cut ≈ hand cut on bowling.
|
||||||
- [ ] P5.3 **Better music** — musicgen-medium/large (slow) or licensed production track; one produced score.
|
- [ ] P5.3 **Better music** — musicgen-medium/large (slow) or licensed production track; one produced score.
|
||||||
- [ ] P5.4 Polish: proper `zoompan` in-shot push-in; beat-synced cut lengths; better/optional voice.
|
- [ ] P5.4 Polish: proper `zoompan` in-shot push-in; beat-synced cut lengths; better/optional voice.
|
||||||
To make it a real system:
|
To make it a real system:
|
||||||
- [ ] P5.5 **Resident local director** writes the edit/montage plan (handover Step 3; today plans are manual).
|
- [x] P5.5 **Resident local director (DONE 2026-07-23).** Two tiers, both offline + auto: Tier-1
|
||||||
|
`HighlightMontageDirector` (measured motion+audio → shot list) and Tier-2 `HighlightVisionDirector` +
|
||||||
|
`tools/vision_caption.py` (moondream2 captions the payoff → semantic overlay "STRIKE" + scene music).
|
||||||
|
Wired into the scheduler (`auto-director-enabled`, `vision-director-enabled`; localpoc on). Plans are no
|
||||||
|
longer manual. See `docs/cinematic-quality-rules.md` R9.
|
||||||
To reach production (deferred until quality passes):
|
To reach production (deferred until quality passes):
|
||||||
- [ ] P5.6 Formal Gate B human review (rubric in cinematic-highlight-acceptance-review.md).
|
- [ ] P5.6 Formal Gate B human review (rubric in cinematic-highlight-acceptance-review.md).
|
||||||
- [ ] P5.7 **Licensing blocker**: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL are non-commercial/
|
- [ ] P5.7 **Licensing blocker**: MusicGen CC-BY-NC, AudioLDM2 CC-BY-NC-SA, YOLOv8 AGPL are non-commercial/
|
||||||
copyleft -> production needs commercially-licensed models/assets.
|
copyleft -> production needs commercially-licensed models/assets.
|
||||||
- [ ] P5.8 The hardening backlog below (security, CI/CD, containers, PostgreSQL, no-egress cert, repro build).
|
- [ ] P5.8 The hardening backlog below (security, CI/CD, containers, PostgreSQL, no-egress cert, repro build).
|
||||||
|
|
||||||
|
### Director review of the bowling portrait cut (2026-07-23) — general craft gaps
|
||||||
|
Frame-by-frame + audio review. Verdict: technically clean (correct portrait, -15.5 LUFS, TP -2.8, no defects,
|
||||||
|
score leads) but NOT yet top-tier cinematic. Craft gaps, all fixable as GENERAL pipeline features:
|
||||||
|
- [x] P5.9 **Adaptive exposure (DONE 2026-07-23).** `probeSourceLuma` measures source frames (signalstats
|
||||||
|
YAVG); `exposureNormalizationFilter` maps the measured mean to a target band (source-driven gamma); the
|
||||||
|
grade is now exposure-preserving (black-lift, gamma≥1, brightness≥0, soft vignette). Bowling final went
|
||||||
|
63→~100 mean luma with a measured gamma of 1.098. General rules captured in `docs/cinematic-quality-rules.md`.
|
||||||
|
- [x] P5.10 **Life in held frames (DONE 2026-07-23).** `probeSegmentMotion` (YDIF) measures per-shot motion;
|
||||||
|
`pushInAmount` maps low motion → stronger in-shot `zoompan` push-in, high motion → gentle (floor keeps a
|
||||||
|
little life). Applied before `setpts` so slow-mo survives. Bowling: static opening got 0.117 push, active
|
||||||
|
celebration 0.084. See `docs/cinematic-quality-rules.md` R5.
|
||||||
|
- [ ] P5.11 **Speed-ramp + crossfades.** The cut into the slow-mo hero is an abrupt speed change and beats are
|
||||||
|
hard cuts. Ease into slow-mo and add short crossfades on beat boundaries (renderer transition upgrade).
|
||||||
|
- [ ] P5.12 **Bolder, animated overlay synced to the music hit.** Current overlay is small plain white text;
|
||||||
|
make it larger/bolder with a scale/fade punch timed to the payoff (general overlay upgrade).
|
||||||
|
- [ ] P5.13 **Music climax sync.** Align the generated score's peak to the hero beat's timeline position
|
||||||
|
(pass the payoff timestamp into music prompting/trimming). Ties to P5.3.
|
||||||
|
- [ ] P5.14 **Show the actual action.** The edit never shows the pins fall (source is on the person). General:
|
||||||
|
shot-selection (P5.2) should locate the true impact via audio-transient + motion, and source coverage
|
||||||
|
should include it.
|
||||||
|
|
||||||
## Deferred (until output-quality gate passes)
|
## Deferred (until output-quality gate passes)
|
||||||
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
|
Production hardening: Spring Security/OIDC, PostgreSQL/Testcontainers, containers/K8s, CI/CD, distributed
|
||||||
ops, digest-bound authenticated approval. Recorded, not deleted.
|
ops, digest-bound authenticated approval. Recorded, not deleted.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,69 @@
|
||||||
|
# Cinematic Highlight Quality Rules (source-adaptive)
|
||||||
|
|
||||||
|
Derived from real defects found on the DJI and bowling sources (2026-07-23). The governing principle:
|
||||||
|
**every rule is dynamic — it MEASURES the source (frames or audio) and adapts. Nothing is a fixed constant
|
||||||
|
tuned to one clip.** A fixed grade/geometry/level that looks right on one video is wrong on the next.
|
||||||
|
|
||||||
|
Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5.x).
|
||||||
|
|
||||||
|
## R1 — Exposure: normalize to a target, never crush ✅
|
||||||
|
- **Measure:** sample source frames (`ffmpeg signalstats` YAVG, 2 fps, bounded) → mean luma.
|
||||||
|
`HighlightFfmpegRenderer.probeSourceLuma`.
|
||||||
|
- **Adapt:** `exposureNormalizationFilter` picks a gamma that maps the measured mean → a target band
|
||||||
|
(~120/255). Dark clip → brightened; well-exposed clip → left alone; over-bright → pulled down.
|
||||||
|
- **Invariant:** the stylistic grade must be exposure-preserving — lift the black point, keep gamma ≥ 1,
|
||||||
|
brightness ≥ 0, soft vignette. A grade must never reduce mean luma the way the old one did (111 → 63).
|
||||||
|
- **Evidence:** bowling final went 63 → ~100 with the source-measured gamma 1.098.
|
||||||
|
|
||||||
|
## R2 — Orientation: respect the source, never distort ✅
|
||||||
|
- **Measure:** read display rotation (`FfmpegClipInspector` side-data / `rotate` tag) → effective W×H.
|
||||||
|
- **Adapt:** `outputGeometry` renders portrait sources to a portrait frame, landscape to widescreen; the 2.39
|
||||||
|
letterbox is applied to landscape only. Portrait→portrait scale is proportional, so no stretch.
|
||||||
|
|
||||||
|
## R3 — Audio balance: the story audio must lead ✅
|
||||||
|
- **Rule:** the generated score is the bed and leads; source audio is ducked under it (−20 dB, −24 with
|
||||||
|
narration); with narration, voice leads via side-chain ducking. No element is silently buried.
|
||||||
|
- **Verify:** integrated loudness ≈ −16 LUFS, true peak ≤ −1.5 dBTP; confirm the score is audible, not just present.
|
||||||
|
|
||||||
|
## R4 — Duration: length follows the story ✅
|
||||||
|
- No fixed min/max highlight length. The cut is as long as the beats need (validator only checks positive
|
||||||
|
duration + sane playback speed + candidate containment).
|
||||||
|
|
||||||
|
## R5 — No dead air: motion in held shots ✅
|
||||||
|
- **Measure:** per-shot motion = mean temporal luma difference (YDIF via signalstats) over the shot's source
|
||||||
|
range. `HighlightFfmpegRenderer.probeSegmentMotion`.
|
||||||
|
- **Adapt:** `pushInAmount` turns low motion into a stronger in-shot `zoompan` push-in and high motion into a
|
||||||
|
gentle one (with a floor so every shot has a little life); applied before `setpts` so slow-motion survives.
|
||||||
|
- **Evidence (bowling):** static opening shot measured lowest motion → strongest push (0.117); active
|
||||||
|
celebration → gentlest (0.084). Confirmed visually (opening pushes in ~11% over 2s).
|
||||||
|
|
||||||
|
## R6 — Transitions: ease, don't jerk ⏳ (P5.11)
|
||||||
|
- Ramp speed into a slow-motion beat (don't hard-switch playback speed); crossfade on beat boundaries where
|
||||||
|
the cut isn't meant to be a hard cut.
|
||||||
|
|
||||||
|
## R7 — Overlays: bold, animated, synced ⏳ (P5.12)
|
||||||
|
- Title/label text scaled to the frame, with an entrance (scale/fade) timed to the musical/edit accent — not
|
||||||
|
small static text dropped on screen.
|
||||||
|
|
||||||
|
## R8 — Music sync: climax on the payoff ⏳ (P5.13)
|
||||||
|
- **Measure:** the payoff beat's timeline position.
|
||||||
|
- **Adapt:** prompt/trim the score so its peak lands on the payoff, not wherever the generator happened to put it.
|
||||||
|
|
||||||
|
## R9 — Show the action, not just the reaction — Tier 1 ✅ / Tier 2 ⏳
|
||||||
|
- **Tier 1 (measurement director, DONE):** `HighlightMontageDirector` measures a per-window motion curve
|
||||||
|
(YDIF) and audio-energy curve (RMS), then composes a story-structured montage automatically: setup →
|
||||||
|
continuous action/tension (release/roll/watch, never chopped) → slow-motion payoff on the audio climax →
|
||||||
|
resolution button, trimming a high-motion camera-whip tail. Enabled by `highlight-scheduler.auto-director-enabled`
|
||||||
|
(on in localpoc); writes `director/montage.json` after analysis. Verified on bowling: auto cut ≈ the hand cut.
|
||||||
|
- **Tier 2 (semantic VLM director, DONE 2026-07-23):** `HighlightVisionDirector` + `tools/vision_caption.py`
|
||||||
|
run a local vision-language model (moondream2, offline) to caption the payoff frame, then AUGMENT the Tier-1
|
||||||
|
montage with a semantic overlay and a scene-informed music direction. On bowling it read the celebration and
|
||||||
|
produced the overlay "STRIKE" and a scene-accurate music prompt — automatically. Enabled by
|
||||||
|
`highlight-scheduler.vision-director-enabled` (localpoc on); ~25s/frame on CPU; fails soft (Tier-1 stands).
|
||||||
|
moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models).
|
||||||
|
- **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can.
|
||||||
|
|
||||||
|
---
|
||||||
|
Rules R1–R5 are live in `HighlightFfmpegRenderer` / `FfmpegClipInspector` / `HighlightDirectorPlanValidator`
|
||||||
|
and apply to **every** project automatically. R6–R9 are the next implementation targets; each must likewise be
|
||||||
|
driven by a source measurement, never a per-video constant.
|
||||||
|
|
@ -1212,6 +1212,12 @@ public class VideoClippingProperties {
|
||||||
|
|
||||||
private boolean requireDirectorApproval = true;
|
private boolean requireDirectorApproval = true;
|
||||||
|
|
||||||
|
/** When true, the Tier-1 automatic director composes director/montage.json after analysis. */
|
||||||
|
private boolean autoDirectorEnabled = false;
|
||||||
|
|
||||||
|
/** When true, the Tier-2 vision director augments the montage with a semantic overlay + music. */
|
||||||
|
private boolean visionDirectorEnabled = false;
|
||||||
|
|
||||||
private String approvalFileName = "approved.flag";
|
private String approvalFileName = "approved.flag";
|
||||||
|
|
||||||
@Min(1)
|
@Min(1)
|
||||||
|
|
@ -1287,6 +1293,22 @@ public class VideoClippingProperties {
|
||||||
this.requireDirectorApproval = requireDirectorApproval;
|
this.requireDirectorApproval = requireDirectorApproval;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public boolean isAutoDirectorEnabled() {
|
||||||
|
return autoDirectorEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setAutoDirectorEnabled(boolean autoDirectorEnabled) {
|
||||||
|
this.autoDirectorEnabled = autoDirectorEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean isVisionDirectorEnabled() {
|
||||||
|
return visionDirectorEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setVisionDirectorEnabled(boolean visionDirectorEnabled) {
|
||||||
|
this.visionDirectorEnabled = visionDirectorEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
public String getApprovalFileName() {
|
public String getApprovalFileName() {
|
||||||
return approvalFileName;
|
return approvalFileName;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -41,6 +41,8 @@ public class FfmpegClipInspector {
|
||||||
"-v", "error",
|
"-v", "error",
|
||||||
"-show_entries", "format=duration",
|
"-show_entries", "format=duration",
|
||||||
"-show_entries", "stream=codec_type,codec_name,width,height,r_frame_rate",
|
"-show_entries", "stream=codec_type,codec_name,width,height,r_frame_rate",
|
||||||
|
"-show_entries", "stream_side_data=rotation",
|
||||||
|
"-show_entries", "stream_tags=rotate",
|
||||||
"-of", "json",
|
"-of", "json",
|
||||||
clip.toString()
|
clip.toString()
|
||||||
);
|
);
|
||||||
|
|
@ -72,6 +74,14 @@ public class FfmpegClipInspector {
|
||||||
if (duration <= 0 || width <= 0 || height <= 0) {
|
if (duration <= 0 || width <= 0 || height <= 0) {
|
||||||
throw new IllegalStateException("ffprobe did not report valid clip duration and dimensions");
|
throw new IllegalStateException("ffprobe did not report valid clip duration and dimensions");
|
||||||
}
|
}
|
||||||
|
// Respect display rotation: portrait phone footage is often stored as landscape pixels with a
|
||||||
|
// +/-90 degree rotation flag. Store the EFFECTIVE (display) dimensions so every downstream
|
||||||
|
// consumer (analysis, prompt, aspect ratio, render geometry) sees the true orientation.
|
||||||
|
if (Math.floorMod(readRotationDegrees(video), 180) == 90) {
|
||||||
|
int swap = width;
|
||||||
|
width = height;
|
||||||
|
height = swap;
|
||||||
|
}
|
||||||
JsonNode audio = firstStream(root, "audio");
|
JsonNode audio = firstStream(root, "audio");
|
||||||
return new ClipAnalysis(
|
return new ClipAnalysis(
|
||||||
clipId(clip),
|
clipId(clip),
|
||||||
|
|
@ -94,6 +104,31 @@ public class FfmpegClipInspector {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reads the video display rotation in degrees. Modern containers expose it as a Display Matrix
|
||||||
|
* side data entry ({@code side_data_list[].rotation}); older files use the {@code tags.rotate} string.
|
||||||
|
* Returns 0 when no rotation is present.
|
||||||
|
*/
|
||||||
|
private int readRotationDegrees(JsonNode video) {
|
||||||
|
JsonNode sideDataList = video.path("side_data_list");
|
||||||
|
if (sideDataList.isArray()) {
|
||||||
|
for (JsonNode sideData : sideDataList) {
|
||||||
|
if (sideData.has("rotation")) {
|
||||||
|
return sideData.path("rotation").asInt(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
String rotateTag = video.path("tags").path("rotate").asText("").trim();
|
||||||
|
if (!rotateTag.isEmpty()) {
|
||||||
|
try {
|
||||||
|
return Integer.parseInt(rotateTag);
|
||||||
|
} catch (NumberFormatException ignored) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
private JsonNode firstStream(JsonNode root, String codecType) {
|
private JsonNode firstStream(JsonNode root, String codecType) {
|
||||||
for (JsonNode stream : root.path("streams")) {
|
for (JsonNode stream : root.path("streams")) {
|
||||||
if (codecType.equals(stream.path("codec_type").asText())) {
|
if (codecType.equals(stream.path("codec_type").asText())) {
|
||||||
|
|
|
||||||
|
|
@ -175,7 +175,7 @@ public class HighlightDirectorFlowService {
|
||||||
|
|
||||||
List<AudioCue> audioCues = new ArrayList<>();
|
List<AudioCue> audioCues = new ArrayList<>();
|
||||||
if (montage.musicDirection() != null && !montage.musicDirection().isBlank()) {
|
if (montage.musicDirection() != null && !montage.musicDirection().isBlank()) {
|
||||||
audioCues.add(new AudioCue("music", safeKey("music", montage.musicDirection()), 0.0, total, -14.0,
|
audioCues.add(new AudioCue("music", safeKey("music", montage.musicDirection()), 0.0, total, -9.0,
|
||||||
montage.musicDirection()));
|
montage.musicDirection()));
|
||||||
}
|
}
|
||||||
List<TextOverlay> overlays = new ArrayList<>();
|
List<TextOverlay> overlays = new ArrayList<>();
|
||||||
|
|
|
||||||
|
|
@ -122,10 +122,12 @@ public class HighlightDirectorPlanValidator {
|
||||||
|| highlight.sourceEndSeconds() > analysis.source().durationSeconds() + EPSILON) {
|
|| highlight.sourceEndSeconds() > analysis.source().durationSeconds() + EPSILON) {
|
||||||
reject("Highlight source range must be contained by candidate: " + candidate.id());
|
reject("Highlight source range must be contained by candidate: " + candidate.id());
|
||||||
}
|
}
|
||||||
if (!finite(highlight.targetDurationSeconds())
|
// A highlight may be as long as the story needs — there is no fixed min/max duration. The only
|
||||||
|| highlight.targetDurationSeconds() < effectiveMinimumDuration() - EPSILON
|
// hard requirements are that the target duration is a positive, finite number (checked here), that
|
||||||
|| highlight.targetDurationSeconds() > properties.getHighlightMaxDurationSeconds() + EPSILON) {
|
// the source range is contained by the candidate and the source (checked above), and that the
|
||||||
reject("Highlight target duration must be within the configured duration bounds");
|
// resulting playback speed stays in a sane range (checked below).
|
||||||
|
if (!finite(highlight.targetDurationSeconds()) || highlight.targetDurationSeconds() <= EPSILON) {
|
||||||
|
reject("Highlight target duration must be a positive number of seconds");
|
||||||
}
|
}
|
||||||
double requiredPlaybackSpeed = (highlight.sourceEndSeconds() - highlight.sourceStartSeconds())
|
double requiredPlaybackSpeed = (highlight.sourceEndSeconds() - highlight.sourceStartSeconds())
|
||||||
/ highlight.targetDurationSeconds();
|
/ highlight.targetDurationSeconds();
|
||||||
|
|
@ -166,10 +168,6 @@ public class HighlightDirectorPlanValidator {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private double effectiveMinimumDuration() {
|
|
||||||
return Math.min(properties.getHighlightMinDurationSeconds(), properties.getHighlightMaxDurationSeconds());
|
|
||||||
}
|
|
||||||
|
|
||||||
private ContentCategory parseCategory(String value) {
|
private ContentCategory parseCategory(String value) {
|
||||||
if (value == null || value.isBlank()) {
|
if (value == null || value.isBlank()) {
|
||||||
reject("Director plan contentCategory is required");
|
reject("Director plan contentCategory is required");
|
||||||
|
|
|
||||||
|
|
@ -84,6 +84,8 @@ public class HighlightFfmpegRenderer {
|
||||||
new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(),
|
new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(),
|
||||||
plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0)
|
plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0)
|
||||||
);
|
);
|
||||||
|
int[] geometry = outputGeometry(sourceAnalysis);
|
||||||
|
String exposureFilter = exposureNormalizationFilter(probeSourceLuma(source.toString()));
|
||||||
List<List<String>> commands = new ArrayList<>();
|
List<List<String>> commands = new ArrayList<>();
|
||||||
List<Path> segments = new ArrayList<>();
|
List<Path> segments = new ArrayList<>();
|
||||||
for (int index = 0; index < plan.decisions().size(); index++) {
|
for (int index = 0; index < plan.decisions().size(); index++) {
|
||||||
|
|
@ -93,8 +95,10 @@ public class HighlightFfmpegRenderer {
|
||||||
"segment", Integer.toString(index + 1),
|
"segment", Integer.toString(index + 1),
|
||||||
"source_start", Double.toString(decision.sourceStartSeconds()),
|
"source_start", Double.toString(decision.sourceStartSeconds()),
|
||||||
"source_end", Double.toString(decision.sourceEndSeconds()));
|
"source_end", Double.toString(decision.sourceEndSeconds()));
|
||||||
run(segmentCommand(source.toString(), decision, plan.style(), index, plan.decisions().size(), output),
|
double pushIn = pushInAmount(probeSegmentMotion(source.toString(),
|
||||||
commands);
|
decision.sourceStartSeconds(), decision.sourceEndSeconds()));
|
||||||
|
run(segmentCommand(source.toString(), decision, plan.style(), index, plan.decisions().size(),
|
||||||
|
geometry[0], geometry[1], exposureFilter, pushIn, output), commands);
|
||||||
segments.add(output);
|
segments.add(output);
|
||||||
log(projectId, highlight.highlightId(), "highlight_render_segment_completed",
|
log(projectId, highlight.highlightId(), "highlight_render_segment_completed",
|
||||||
"segment", Integer.toString(index + 1));
|
"segment", Integer.toString(index + 1));
|
||||||
|
|
@ -184,11 +188,39 @@ public class HighlightFfmpegRenderer {
|
||||||
return List.copyOf(assets);
|
return List.copyOf(assets);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Chooses render geometry from the source's effective orientation. Portrait sources render to a portrait
|
||||||
|
* frame (and skip the widescreen letterbox); landscape sources keep the configured widescreen frame. The
|
||||||
|
* configured output dimensions define the long/short edges regardless of orientation.
|
||||||
|
*/
|
||||||
|
int[] outputGeometry(HighlightSourceAnalysis analysis) {
|
||||||
|
int longEdge = Math.max(properties.getOutputWidth(), properties.getOutputHeight());
|
||||||
|
int shortEdge = Math.min(properties.getOutputWidth(), properties.getOutputHeight());
|
||||||
|
boolean portrait = analysis != null && analysis.source() != null
|
||||||
|
&& analysis.source().width() > 0
|
||||||
|
&& analysis.source().height() > analysis.source().width();
|
||||||
|
return portrait ? new int[]{shortEdge, longEdge} : new int[]{longEdge, shortEdge};
|
||||||
|
}
|
||||||
|
|
||||||
List<String> segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount,
|
List<String> segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount,
|
||||||
Path output) {
|
Path output) {
|
||||||
|
return segmentCommand(source, decision, style, cutIndex, cutCount,
|
||||||
|
properties.getOutputWidth(), properties.getOutputHeight(), output);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount,
|
||||||
|
int w, int h, Path output) {
|
||||||
|
return segmentCommand(source, decision, style, cutIndex, cutCount, w, h, "", output);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount,
|
||||||
|
int w, int h, String exposureFilter, Path output) {
|
||||||
|
return segmentCommand(source, decision, style, cutIndex, cutCount, w, h, exposureFilter, 0.0, output);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> segmentCommand(String source, EditDecision decision, String style, int cutIndex, int cutCount,
|
||||||
|
int w, int h, String exposureFilter, double pushIn, Path output) {
|
||||||
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
|
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
|
||||||
int w = properties.getOutputWidth();
|
|
||||||
int h = properties.getOutputHeight();
|
|
||||||
boolean styled = !(decision.visualTreatment() == null || decision.visualTreatment().isBlank()
|
boolean styled = !(decision.visualTreatment() == null || decision.visualTreatment().isBlank()
|
||||||
|| "none".equalsIgnoreCase(decision.visualTreatment()));
|
|| "none".equalsIgnoreCase(decision.visualTreatment()));
|
||||||
StringBuilder filter = new StringBuilder();
|
StringBuilder filter = new StringBuilder();
|
||||||
|
|
@ -200,16 +232,33 @@ public class HighlightFfmpegRenderer {
|
||||||
if (zoom <= 0) {
|
if (zoom <= 0) {
|
||||||
zoom = 1.16 + Math.min(cutIndex, 3) * 0.12;
|
zoom = 1.16 + Math.min(cutIndex, 3) * 0.12;
|
||||||
}
|
}
|
||||||
|
if (pushIn > 0.001) {
|
||||||
|
// R5 motion-adaptive in-shot push-in: animate the zoom across the shot so held/low-motion
|
||||||
|
// frames are never dead air. MUST precede setpts so slow-motion is preserved. Frame count is
|
||||||
|
// estimated from the source span; min() caps the zoom so any estimate error can't overshoot.
|
||||||
|
int frames = Math.max(2, (int) Math.round(
|
||||||
|
(decision.sourceEndSeconds() - decision.sourceStartSeconds())
|
||||||
|
* properties.getOutputFrameRate()));
|
||||||
|
filter.append(("scale=%d:%d,zoompan=z='min(%.4f+%.4f*on/%d\\,%.4f)':d=1"
|
||||||
|
+ ":x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':s=%dx%d,format=yuv420p")
|
||||||
|
.formatted(w, h, zoom, pushIn, frames - 1, zoom + pushIn, w, h));
|
||||||
|
} else {
|
||||||
filter.append("crop=iw/%.4f:ih/%.4f,scale=%d:%d,format=yuv420p".formatted(zoom, zoom, w, h));
|
filter.append("crop=iw/%.4f:ih/%.4f,scale=%d:%d,format=yuv420p".formatted(zoom, zoom, w, h));
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(w, h)
|
filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(w, h)
|
||||||
+ "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(w, h));
|
+ "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(w, h));
|
||||||
}
|
}
|
||||||
|
if (exposureFilter != null && !exposureFilter.isBlank()) {
|
||||||
|
filter.append(exposureFilter); // adaptive per-source exposure normalization, before the grade
|
||||||
|
}
|
||||||
filter.append(",setpts=PTS/").append(decision.playbackSpeed());
|
filter.append(",setpts=PTS/").append(decision.playbackSpeed());
|
||||||
filter.append(cinematicVisualFilter(decision.visualTreatment(), style));
|
filter.append(cinematicVisualFilter(decision.visualTreatment(), style));
|
||||||
if (styled) {
|
if (styled) {
|
||||||
filter.append(",noise=alls=6:allf=t"); // subtle film grain
|
filter.append(",noise=alls=6:allf=t"); // subtle film grain
|
||||||
filter.append(letterboxFilter()); // 2.39:1 cinematic bars
|
if (w > h) {
|
||||||
|
filter.append(letterboxFilter()); // 2.39:1 cinematic bars (landscape only)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
double fadeDuration = Math.min(0.5, outputDuration / 2);
|
double fadeDuration = Math.min(0.5, outputDuration / 2);
|
||||||
if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) {
|
if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) {
|
||||||
|
|
@ -236,6 +285,103 @@ public class HighlightFfmpegRenderer {
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Assesses the actual frames of the source and returns a DYNAMIC exposure-normalization filter that lifts
|
||||||
|
* (or lowers) the picture toward a target average brightness. It adapts per source: a dark clip is
|
||||||
|
* brightened, a well-exposed clip is left alone — nothing is hard-coded to one video. Returns "" when the
|
||||||
|
* source cannot be measured or is already within the target band.
|
||||||
|
*/
|
||||||
|
String exposureNormalizationFilter(double sourceMeanLuma) {
|
||||||
|
if (sourceMeanLuma <= 0) {
|
||||||
|
return ""; // could not measure -> make no assumption
|
||||||
|
}
|
||||||
|
double target = 120.0; // target mean luma (of 255) for a bright, readable subject
|
||||||
|
if (Math.abs(sourceMeanLuma - target) <= 8.0) {
|
||||||
|
return ""; // already well exposed -> leave it
|
||||||
|
}
|
||||||
|
double s = Math.min(240.0, Math.max(16.0, sourceMeanLuma)) / 255.0;
|
||||||
|
double t = target / 255.0;
|
||||||
|
// out = in^(1/gamma); pick gamma so the measured mean maps to the target mean. gamma>1 brightens.
|
||||||
|
double gamma = Math.min(2.5, Math.max(0.5, Math.log(s) / Math.log(t)));
|
||||||
|
return ",eq=gamma=" + String.format(Locale.ROOT, "%.3f", gamma);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures the source's mean luma (YAVG, 0-255) by sampling frames across the clip with ffmpeg signalstats.
|
||||||
|
* Bounded (2 fps, first 60s) so it stays cheap. Returns -1 when it cannot be determined, in which case the
|
||||||
|
* caller makes no exposure adjustment.
|
||||||
|
*/
|
||||||
|
double probeSourceLuma(String source) {
|
||||||
|
List<String> command = List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats",
|
||||||
|
"-i", source, "-an", "-vf", "fps=2,signalstats,metadata=print:key=lavfi.signalstats.YAVG",
|
||||||
|
"-t", "60", "-f", "null", "-");
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
process.waitFor();
|
||||||
|
java.util.regex.Matcher matcher = YAVG_TOKEN.matcher(out);
|
||||||
|
double sum = 0;
|
||||||
|
int count = 0;
|
||||||
|
while (matcher.find()) {
|
||||||
|
sum += Double.parseDouble(matcher.group(1));
|
||||||
|
count++;
|
||||||
|
}
|
||||||
|
return count == 0 ? -1 : sum / count;
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return -1;
|
||||||
|
} catch (IOException | RuntimeException ex) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* R5: converts a measured per-shot motion (mean YDIF) into an in-shot push-in amount. Low-motion / held
|
||||||
|
* shots get a stronger push-in so they never read as dead air; active shots get only a gentle one. A
|
||||||
|
* failed/unknown measurement (-1) yields a mild default push so there is always some life.
|
||||||
|
*/
|
||||||
|
double pushInAmount(double motionYdif) {
|
||||||
|
double gentle = 0.05;
|
||||||
|
if (motionYdif < 0) {
|
||||||
|
return gentle; // could not measure -> a mild default push
|
||||||
|
}
|
||||||
|
double lowMotion = 2.5;
|
||||||
|
double highMotion = 9.0;
|
||||||
|
double normalized = Math.min(1.0, Math.max(0.0, (motionYdif - lowMotion) / (highMotion - lowMotion)));
|
||||||
|
double maxPush = 0.12;
|
||||||
|
double floor = 0.03;
|
||||||
|
return Math.max(floor, maxPush * (1.0 - normalized));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures per-shot motion as the mean temporal luma difference (YDIF via signalstats) over the shot's
|
||||||
|
* source range. Higher = more motion. Returns -1 when it cannot be determined.
|
||||||
|
*/
|
||||||
|
double probeSegmentMotion(String source, double startSeconds, double endSeconds) {
|
||||||
|
List<String> command = List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats",
|
||||||
|
"-ss", Double.toString(Math.max(0.0, startSeconds)), "-to", Double.toString(endSeconds),
|
||||||
|
"-i", source, "-an", "-vf", "signalstats,metadata=print:key=lavfi.signalstats.YDIF",
|
||||||
|
"-f", "null", "-");
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
process.waitFor();
|
||||||
|
java.util.regex.Matcher matcher = YDIF_TOKEN.matcher(out);
|
||||||
|
double sum = 0;
|
||||||
|
int count = 0;
|
||||||
|
while (matcher.find()) {
|
||||||
|
sum += Double.parseDouble(matcher.group(1));
|
||||||
|
count++;
|
||||||
|
}
|
||||||
|
return count == 0 ? -1 : sum / count;
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return -1;
|
||||||
|
} catch (IOException | RuntimeException ex) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
List<String> overlayCommand(Path input, List<TextOverlay> overlays, Path output) {
|
List<String> overlayCommand(Path input, List<TextOverlay> overlays, Path output) {
|
||||||
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
||||||
"-vf", overlayFilter(overlays),
|
"-vf", overlayFilter(overlays),
|
||||||
|
|
@ -268,13 +414,21 @@ public class HighlightFfmpegRenderer {
|
||||||
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||||
"-i", timeline.toString()));
|
"-i", timeline.toString()));
|
||||||
List<String> labels = new ArrayList<>();
|
List<String> labels = new ArrayList<>();
|
||||||
if (sourceAudioPresent) {
|
|
||||||
labels.add("[0:a]");
|
|
||||||
}
|
|
||||||
int input = 1;
|
int input = 1;
|
||||||
StringBuilder filters = new StringBuilder();
|
StringBuilder filters = new StringBuilder();
|
||||||
boolean hasMusic = music != null;
|
boolean hasMusic = music != null;
|
||||||
boolean hasVoiceover = !voiceovers.isEmpty();
|
boolean hasVoiceover = !voiceovers.isEmpty();
|
||||||
|
if (sourceAudioPresent) {
|
||||||
|
// Source audio is a background bed. When a generated score is present it must sit clearly
|
||||||
|
// under the music (and further under any narration), not drown it. Attenuate accordingly.
|
||||||
|
double sourceBedDb = !hasMusic ? 0.0 : (hasVoiceover ? -24.0 : -20.0);
|
||||||
|
if (sourceBedDb == 0.0) {
|
||||||
|
labels.add("[0:a]");
|
||||||
|
} else {
|
||||||
|
filters.append("[0:a]volume=").append(sourceBedDb).append("dB[src_bed];");
|
||||||
|
labels.add("[src_bed]");
|
||||||
|
}
|
||||||
|
}
|
||||||
if (music != null) {
|
if (music != null) {
|
||||||
command.addAll(List.of("-i", music.toString()));
|
command.addAll(List.of("-i", music.toString()));
|
||||||
filters.append("[").append(input).append(":a]");
|
filters.append("[").append(input).append(":a]");
|
||||||
|
|
@ -562,29 +716,33 @@ public class HighlightFfmpegRenderer {
|
||||||
// Beat-specific filmic grade. All variants share the teal-shadow / warm-highlight ("teal-orange")
|
// Beat-specific filmic grade. All variants share the teal-shadow / warm-highlight ("teal-orange")
|
||||||
// language but shift contrast, warmth and vignette to match the story beat encoded in the plan
|
// language but shift contrast, warmth and vignette to match the story beat encoded in the plan
|
||||||
// style key (safeKey: style_<category>_<storyPurpose>). Unknown/rising -> balanced base grade.
|
// style key (safeKey: style_<category>_<storyPurpose>). Unknown/rising -> balanced base grade.
|
||||||
|
// EXPOSURE-PRESERVING grades. The stylistic look (teal-orange, contrast, vignette) must not crush
|
||||||
|
// overall brightness: a fixed dark grade turned a well-exposed source (mean luma ~111/255) into ~63.
|
||||||
|
// Every variant now lifts the black point (curves 0/0.06), keeps gamma >= 1 and brightness >= 0, and
|
||||||
|
// uses a soft vignette, so the stylistic look never crushes overall exposure.
|
||||||
String beat = style == null ? "" : style.toLowerCase(Locale.ROOT);
|
String beat = style == null ? "" : style.toLowerCase(Locale.ROOT);
|
||||||
if (beat.contains("opening")) {
|
if (beat.contains("opening")) {
|
||||||
// Opening hook: calmer, cooler, softer contrast to invite the viewer in.
|
// Opening hook: calmer, cooler, softer contrast to invite the viewer in.
|
||||||
return ",curves=preset=linear_contrast"
|
return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'"
|
||||||
+ ",colorbalance=rs=-0.05:bs=0.08:rm=0.01:bm=-0.01:rh=0.03:bh=-0.03"
|
+ ",colorbalance=rs=-0.05:bs=0.08:rm=0.01:bm=-0.01:rh=0.03:bh=-0.03"
|
||||||
+ ",eq=contrast=1.08:saturation=1.08:gamma=0.98:brightness=-0.008"
|
+ ",eq=contrast=1.05:saturation=1.08:gamma=1.04:brightness=0.015"
|
||||||
+ ",unsharp=5:5:0.3:3:3:0.15"
|
+ ",unsharp=5:5:0.3:3:3:0.15"
|
||||||
+ ",vignette=PI/7";
|
+ ",vignette=PI/10";
|
||||||
}
|
}
|
||||||
if (beat.contains("hero")) {
|
if (beat.contains("hero")) {
|
||||||
// Hero payoff: richest grade, stronger S-curve, warmer highlights, deeper vignette.
|
// Hero payoff: richest grade, warmer highlights, but shadows lifted so the subject stays visible.
|
||||||
return ",curves=preset=strong_contrast"
|
return ",curves=preset=medium_contrast,curves=all='0/0.06 1/1'"
|
||||||
+ ",colorbalance=rs=-0.03:bs=0.05:rm=0.01:bm=-0.01:rh=0.09:bh=-0.07"
|
+ ",colorbalance=rs=-0.03:bs=0.05:rm=0.01:bm=-0.01:rh=0.09:bh=-0.07"
|
||||||
+ ",eq=contrast=1.16:saturation=1.16:gamma=0.96:brightness=-0.015"
|
+ ",eq=contrast=1.08:saturation=1.16:gamma=1.06:brightness=0.02"
|
||||||
+ ",unsharp=5:5:0.5:3:3:0.25"
|
+ ",unsharp=5:5:0.5:3:3:0.25"
|
||||||
+ ",vignette=PI/5.5";
|
+ ",vignette=PI/9";
|
||||||
}
|
}
|
||||||
// Rising energy / default: balanced base grade.
|
// Rising energy / default: balanced base grade.
|
||||||
return ",curves=preset=medium_contrast"
|
return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'"
|
||||||
+ ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05"
|
+ ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05"
|
||||||
+ ",eq=contrast=1.12:saturation=1.12:gamma=0.97:brightness=-0.012"
|
+ ",eq=contrast=1.07:saturation=1.12:gamma=1.05:brightness=0.018"
|
||||||
+ ",unsharp=5:5:0.4:3:3:0.2"
|
+ ",unsharp=5:5:0.4:3:3:0.2"
|
||||||
+ ",vignette=PI/6";
|
+ ",vignette=PI/9";
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<VoiceoverInput> voiceoverInputs(EditPlan plan, Path audioDirectory) {
|
private List<VoiceoverInput> voiceoverInputs(EditPlan plan, Path audioDirectory) {
|
||||||
|
|
@ -621,6 +779,10 @@ public class HighlightFfmpegRenderer {
|
||||||
|
|
||||||
private static final java.util.regex.Pattern ZOOM_TOKEN =
|
private static final java.util.regex.Pattern ZOOM_TOKEN =
|
||||||
java.util.regex.Pattern.compile("zoom=([0-9]+(?:\\.[0-9]+)?)");
|
java.util.regex.Pattern.compile("zoom=([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
private static final java.util.regex.Pattern YAVG_TOKEN =
|
||||||
|
java.util.regex.Pattern.compile("YAVG=([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
private static final java.util.regex.Pattern YDIF_TOKEN =
|
||||||
|
java.util.regex.Pattern.compile("YDIF=([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
|
||||||
// 2.39:1 cinematic letterbox: crop the center band, then pad back to frame with black bars.
|
// 2.39:1 cinematic letterbox: crop the center band, then pad back to frame with black bars.
|
||||||
private String letterboxFilter() {
|
private String letterboxFilter() {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,239 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tier-1 automatic director. It inspects MEASURED signals from the source — per-window motion (temporal luma
|
||||||
|
* difference) and audio energy (RMS) — and composes a story-structured montage from them:
|
||||||
|
*
|
||||||
|
* <ol>
|
||||||
|
* <li>setup (before the action);</li>
|
||||||
|
* <li>a continuous action + tension shot (release/roll/watch — never chopped);</li>
|
||||||
|
* <li>a short realization shot;</li>
|
||||||
|
* <li>a slow-motion payoff centered on the audio climax;</li>
|
||||||
|
* <li>a resolution button after the climax;</li>
|
||||||
|
* </ol>
|
||||||
|
*
|
||||||
|
* <p>while trimming a high-motion tail (e.g. a camera whip). Every decision is derived from a measurement of
|
||||||
|
* this source — nothing is hard-coded to one video. Semantic captions/overlays and emotional nuance (e.g.
|
||||||
|
* "he wasn't sure it was a strike") are deliberately NOT attempted here; that is the Tier-2 vision-language
|
||||||
|
* director's job. This tier gives a strong, deterministic, offline baseline cut.
|
||||||
|
*/
|
||||||
|
@Component
|
||||||
|
public class HighlightMontageDirector {
|
||||||
|
|
||||||
|
static final double WINDOW_SECONDS = 0.5;
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing properties;
|
||||||
|
|
||||||
|
public HighlightMontageDirector(VideoClippingProperties properties) {
|
||||||
|
this.properties = properties.getEditing();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Measure the source and compose an automatic montage plan. */
|
||||||
|
public MontagePlan direct(String projectId, String sourceFileName, Path source, double durationSeconds) {
|
||||||
|
double[] motion = probeCurve(source.toString(), true, durationSeconds);
|
||||||
|
double[] audio = probeCurve(source.toString(), false, durationSeconds);
|
||||||
|
return composeMontage(projectId, sourceFileName, motion, audio, WINDOW_SECONDS, durationSeconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pure, deterministic shot composition from the measured curves. Package-visible for unit testing without
|
||||||
|
* ffmpeg. {@code motion[i]} and {@code audio[i]} cover the window starting at {@code i * window} seconds.
|
||||||
|
*/
|
||||||
|
MontagePlan composeMontage(String projectId, String sourceFileName, double[] motion, double[] audio,
|
||||||
|
double window, double duration) {
|
||||||
|
int n = Math.min(motion.length, audio.length);
|
||||||
|
if (n < 4 || duration <= 0) {
|
||||||
|
return straightCut(projectId, sourceFileName, duration);
|
||||||
|
}
|
||||||
|
double[] m = smooth(motion, n);
|
||||||
|
double[] a = smooth(audio, n);
|
||||||
|
|
||||||
|
// Climax = loudest sustained audio in the central band (skip the intro and the messy tail).
|
||||||
|
int climaxIdx = argMax(a, (int) Math.floor(0.25 * n), (int) Math.ceil(0.80 * n));
|
||||||
|
double climaxTime = climaxIdx * window;
|
||||||
|
|
||||||
|
// Action = the biggest motion spike before the climax; the action shot leads in ~1.5s before it so
|
||||||
|
// the release is included and the release->roll->watch stays continuous.
|
||||||
|
int actionIdx = argMax(m, Math.max(1, (int) Math.floor(0.05 * n)),
|
||||||
|
Math.max(2, (int) Math.floor((climaxTime - 1.0) / window)));
|
||||||
|
double actionTime = actionIdx * window;
|
||||||
|
|
||||||
|
// Trim a high-motion tail (camera whip): the first strong post-climax motion rise well above the
|
||||||
|
// celebration's own motion level.
|
||||||
|
double baseMotion = mean(m, 0, Math.max(1, climaxIdx));
|
||||||
|
double tailThreshold = baseMotion * 2.4;
|
||||||
|
double tailStart = duration;
|
||||||
|
for (int i = climaxIdx + 4; i < n; i++) {
|
||||||
|
if (m[i] > tailThreshold) {
|
||||||
|
tailStart = i * window;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
double endLimit = Math.min(Math.min(duration, tailStart), climaxTime + 4.5);
|
||||||
|
|
||||||
|
List<MontagePlan.Shot> shots = new ArrayList<>();
|
||||||
|
addShot(shots, actionTime - 3.2, actionTime - 1.5, 1.03, 1.0, duration); // setup
|
||||||
|
addShot(shots, actionTime - 1.5, climaxTime - 0.6, 1.04, 1.0, duration); // action + tension
|
||||||
|
addShot(shots, climaxTime - 0.6, climaxTime + 0.2, 1.06, 1.0, duration); // realization
|
||||||
|
double payoffEnd = Math.min(climaxTime + 2.2, endLimit - 0.9);
|
||||||
|
addShot(shots, climaxTime + 0.2, payoffEnd, 1.05, 0.7, duration); // slow-mo payoff
|
||||||
|
addShot(shots, payoffEnd, endLimit, 1.04, 0.9, duration); // resolution button
|
||||||
|
|
||||||
|
if (shots.isEmpty()) {
|
||||||
|
return straightCut(projectId, sourceFileName, duration);
|
||||||
|
}
|
||||||
|
return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), List.of(), shots);
|
||||||
|
}
|
||||||
|
|
||||||
|
private MontagePlan straightCut(String projectId, String sourceFileName, double duration) {
|
||||||
|
double d = Math.max(0.5, Math.min(duration <= 0 ? 6.0 : duration, 8.0));
|
||||||
|
return new MontagePlan(projectId, sourceFileName, "hero", genericMusic(), List.of(), List.of(),
|
||||||
|
List.of(new MontagePlan.Shot(0.0, d, 1.03, 1.0)));
|
||||||
|
}
|
||||||
|
|
||||||
|
private void addShot(List<MontagePlan.Shot> shots, double srcStart, double srcEnd,
|
||||||
|
double zoom, double speed, double duration) {
|
||||||
|
double s = clamp(srcStart, 0.0, duration);
|
||||||
|
double e = clamp(srcEnd, 0.0, duration);
|
||||||
|
double span = e - s;
|
||||||
|
if (span < 0.4) {
|
||||||
|
return; // skip degenerate shots
|
||||||
|
}
|
||||||
|
shots.add(new MontagePlan.Shot(round(s), round(span / speed), zoom, speed));
|
||||||
|
}
|
||||||
|
|
||||||
|
private String genericMusic() {
|
||||||
|
return "cinematic build, quiet tense strings rising to a triumphant orchestral and percussion hit at "
|
||||||
|
+ "the climax, then a short warm resolve, modern trailer film score, no vocals";
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- measurement ---------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures a per-window curve over the source: motion (signalstats YDIF, temporal luma diff) when
|
||||||
|
* {@code motion} is true, else audio energy (astats RMS in dB). Bins samples into {@code WINDOW_SECONDS}
|
||||||
|
* buckets. Missing buckets get a floor value. Returns an empty array when nothing could be measured.
|
||||||
|
*/
|
||||||
|
double[] probeCurve(String source, boolean motion, double duration) {
|
||||||
|
int size = Math.max(1, (int) Math.ceil(duration / WINDOW_SECONDS));
|
||||||
|
List<String> command = motion
|
||||||
|
? List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-an",
|
||||||
|
"-vf", "signalstats,metadata=print:key=lavfi.signalstats.YDIF", "-f", "null", "-")
|
||||||
|
: List.of(properties.getFfmpegBinary(), "-hide_banner", "-nostats", "-i", source, "-vn",
|
||||||
|
"-af", "astats=metadata=1:reset=1,ametadata=print:key=lavfi.astats.Overall.RMS_level",
|
||||||
|
"-f", "null", "-");
|
||||||
|
String token = motion ? "YDIF" : "RMS_level";
|
||||||
|
double floor = motion ? 0.0 : -90.0;
|
||||||
|
double[] sum = new double[size];
|
||||||
|
int[] count = new int[size];
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
process.waitFor();
|
||||||
|
double lastPts = -1;
|
||||||
|
for (String line : out.split("\\R")) {
|
||||||
|
int pts = line.indexOf("pts_time:");
|
||||||
|
if (pts >= 0) {
|
||||||
|
lastPts = parseTrailingNumber(line.substring(pts + "pts_time:".length()));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
int tok = line.indexOf(token + "=");
|
||||||
|
if (tok >= 0 && lastPts >= 0) {
|
||||||
|
double value = parseTrailingNumber(line.substring(tok + token.length() + 1));
|
||||||
|
int idx = Math.min(size - 1, (int) Math.floor(lastPts / WINDOW_SECONDS));
|
||||||
|
if (idx >= 0 && !Double.isNaN(value)) {
|
||||||
|
sum[idx] += value;
|
||||||
|
count[idx]++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return new double[0];
|
||||||
|
} catch (IOException | RuntimeException ex) {
|
||||||
|
return new double[0];
|
||||||
|
}
|
||||||
|
double[] curve = new double[size];
|
||||||
|
boolean any = false;
|
||||||
|
for (int i = 0; i < size; i++) {
|
||||||
|
curve[i] = count[i] == 0 ? floor : sum[i] / count[i];
|
||||||
|
any |= count[i] > 0;
|
||||||
|
}
|
||||||
|
return any ? curve : new double[0];
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double parseTrailingNumber(String text) {
|
||||||
|
int i = 0;
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
String trimmed = text.trim();
|
||||||
|
while (i < trimmed.length()) {
|
||||||
|
char c = trimmed.charAt(i);
|
||||||
|
if (Character.isDigit(c) || c == '.' || c == '-' || c == '+' || c == 'e' || c == 'E') {
|
||||||
|
sb.append(c);
|
||||||
|
i++;
|
||||||
|
} else {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return sb.length() == 0 ? Double.NaN : Double.parseDouble(sb.toString());
|
||||||
|
} catch (NumberFormatException ex) {
|
||||||
|
return Double.NaN;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- small numeric helpers ----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
private static double[] smooth(double[] values, int n) {
|
||||||
|
double[] out = new double[n];
|
||||||
|
for (int i = 0; i < n; i++) {
|
||||||
|
double s = 0;
|
||||||
|
int c = 0;
|
||||||
|
for (int j = Math.max(0, i - 1); j <= Math.min(n - 1, i + 1); j++) {
|
||||||
|
s += values[j];
|
||||||
|
c++;
|
||||||
|
}
|
||||||
|
out[i] = s / c;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int argMax(double[] values, int from, int to) {
|
||||||
|
int lo = Math.max(0, from);
|
||||||
|
int hi = Math.min(values.length, Math.max(lo + 1, to));
|
||||||
|
int best = lo;
|
||||||
|
for (int i = lo; i < hi; i++) {
|
||||||
|
if (values[i] > values[best]) {
|
||||||
|
best = i;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double mean(double[] values, int from, int to) {
|
||||||
|
int lo = Math.max(0, from);
|
||||||
|
int hi = Math.min(values.length, Math.max(lo + 1, to));
|
||||||
|
double s = 0;
|
||||||
|
for (int i = lo; i < hi; i++) {
|
||||||
|
s += values[i];
|
||||||
|
}
|
||||||
|
return s / (hi - lo);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double clamp(double v, double lo, double hi) {
|
||||||
|
return Math.max(lo, Math.min(hi, v));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double round(double v) {
|
||||||
|
return Math.round(v * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -41,6 +41,8 @@ public class HighlightSourceScheduler {
|
||||||
private final HighlightSourceAnalyzer analyzer;
|
private final HighlightSourceAnalyzer analyzer;
|
||||||
private final HighlightCandidateGenerator candidateGenerator;
|
private final HighlightCandidateGenerator candidateGenerator;
|
||||||
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
||||||
|
private final HighlightMontageDirector montageDirector;
|
||||||
|
private final HighlightVisionDirector visionDirector;
|
||||||
private final Clock clock;
|
private final Clock clock;
|
||||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||||
private final AtomicLong scanSequence = new AtomicLong();
|
private final AtomicLong scanSequence = new AtomicLong();
|
||||||
|
|
@ -51,9 +53,12 @@ public class HighlightSourceScheduler {
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
HighlightSourceAnalyzer analyzer,
|
HighlightSourceAnalyzer analyzer,
|
||||||
HighlightCandidateGenerator candidateGenerator,
|
HighlightCandidateGenerator candidateGenerator,
|
||||||
HighlightDirectorPromptGenerator directorPromptGenerator
|
HighlightDirectorPromptGenerator directorPromptGenerator,
|
||||||
|
HighlightMontageDirector montageDirector,
|
||||||
|
HighlightVisionDirector visionDirector
|
||||||
) {
|
) {
|
||||||
this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, Clock.systemUTC());
|
this(properties, store, analyzer, candidateGenerator, directorPromptGenerator, montageDirector,
|
||||||
|
visionDirector, Clock.systemUTC());
|
||||||
}
|
}
|
||||||
|
|
||||||
HighlightSourceScheduler(
|
HighlightSourceScheduler(
|
||||||
|
|
@ -62,6 +67,8 @@ public class HighlightSourceScheduler {
|
||||||
HighlightSourceAnalyzer analyzer,
|
HighlightSourceAnalyzer analyzer,
|
||||||
HighlightCandidateGenerator candidateGenerator,
|
HighlightCandidateGenerator candidateGenerator,
|
||||||
HighlightDirectorPromptGenerator directorPromptGenerator,
|
HighlightDirectorPromptGenerator directorPromptGenerator,
|
||||||
|
HighlightMontageDirector montageDirector,
|
||||||
|
HighlightVisionDirector visionDirector,
|
||||||
Clock clock
|
Clock clock
|
||||||
) {
|
) {
|
||||||
this.properties = properties.getEditing().getHighlightScheduler();
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
|
|
@ -69,6 +76,8 @@ public class HighlightSourceScheduler {
|
||||||
this.analyzer = analyzer;
|
this.analyzer = analyzer;
|
||||||
this.candidateGenerator = candidateGenerator;
|
this.candidateGenerator = candidateGenerator;
|
||||||
this.directorPromptGenerator = directorPromptGenerator;
|
this.directorPromptGenerator = directorPromptGenerator;
|
||||||
|
this.montageDirector = montageDirector;
|
||||||
|
this.visionDirector = visionDirector;
|
||||||
this.clock = clock;
|
this.clock = clock;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -179,6 +188,11 @@ public class HighlightSourceScheduler {
|
||||||
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
|
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
|
||||||
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
|
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
|
||||||
store.directorDirectory(projectId).resolve("director-brief.md"));
|
store.directorDirectory(projectId).resolve("director-brief.md"));
|
||||||
|
if (properties.isAutoDirectorEnabled()) {
|
||||||
|
autoDirect(projectId, project.sourceVideoFileName(),
|
||||||
|
store.sourceDirectory(projectId).resolve(workingFile.getFileName()),
|
||||||
|
analysis, cinematic, scanId);
|
||||||
|
}
|
||||||
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
|
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
|
||||||
+ "analysis_file={} elapsed_ms={}",
|
+ "analysis_file={} elapsed_ms={}",
|
||||||
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
|
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
|
||||||
|
|
@ -209,6 +223,54 @@ public class HighlightSourceScheduler {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void autoDirect(String projectId, String sourceFileName, Path sourcePath,
|
||||||
|
HighlightSourceAnalysis analysis, CinematicHighlightAnalysis cinematic, long scanId) {
|
||||||
|
try {
|
||||||
|
double duration = analysis.source() == null ? 0.0 : analysis.source().durationSeconds();
|
||||||
|
MontagePlan montage = montageDirector.direct(projectId, sourceFileName, sourcePath, duration);
|
||||||
|
if (properties.isVisionDirectorEnabled()) {
|
||||||
|
montage = visionDirector.augment(montage, sourcePath,
|
||||||
|
store.directorDirectory(projectId).resolve("vision-work"));
|
||||||
|
}
|
||||||
|
store.writeJson(projectId, "director/montage.json", montage);
|
||||||
|
// Minimal edit-plan.json so the render scanner selects the project; montage.json takes precedence.
|
||||||
|
store.writeJson(projectId, "director/edit-plan.json",
|
||||||
|
fallbackEditPlan(projectId, sourceFileName, cinematic, duration));
|
||||||
|
log.info("event=highlight_auto_director_completed scan_id={} project_id={} shots={} source_duration={}",
|
||||||
|
scanId, projectId, montage.shots().size(), duration);
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
log.warn("event=highlight_auto_director_failed scan_id={} project_id={} error_type={} message={}",
|
||||||
|
scanId, projectId, ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightDirectorPlan fallbackEditPlan(String projectId, String sourceFileName,
|
||||||
|
CinematicHighlightAnalysis cinematic, double duration) {
|
||||||
|
String category = cinematic.category() == null ? "generic_vlog"
|
||||||
|
: cinematic.category().name().toLowerCase(java.util.Locale.ROOT);
|
||||||
|
String candidateId;
|
||||||
|
double start;
|
||||||
|
double end;
|
||||||
|
if (cinematic.candidates() != null && !cinematic.candidates().isEmpty()) {
|
||||||
|
HighlightCandidate candidate = cinematic.candidates().get(0);
|
||||||
|
candidateId = candidate.id();
|
||||||
|
start = candidate.sourceStartSeconds();
|
||||||
|
end = candidate.sourceEndSeconds();
|
||||||
|
} else {
|
||||||
|
candidateId = "candidate_001";
|
||||||
|
start = 0.0;
|
||||||
|
end = Math.max(1.0, Math.min(duration <= 0 ? 8.0 : duration, 8.0));
|
||||||
|
}
|
||||||
|
double span = Math.max(1.0, end - start);
|
||||||
|
HighlightDirectorPlan.HighlightItem item = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", candidateId, "Highlight", start, end, span,
|
||||||
|
"hero_payoff", "cinematic hero grade", "cinematic score", "",
|
||||||
|
java.util.List.of(), java.util.List.of(),
|
||||||
|
"Auto-generated fallback; the adjacent montage.json takes precedence.");
|
||||||
|
return new HighlightDirectorPlan(projectId, sourceFileName, category, java.util.List.of(item),
|
||||||
|
"Auto-director fallback edit-plan; montage.json drives the render.");
|
||||||
|
}
|
||||||
|
|
||||||
private void copySourceToProject(Path source, Path target) {
|
private void copySourceToProject(Path source, Path target) {
|
||||||
try {
|
try {
|
||||||
Files.copy(source, target, StandardCopyOption.COPY_ATTRIBUTES);
|
Files.copy(source, target, StandardCopyOption.COPY_ATTRIBUTES);
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,179 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tier-2 semantic director. It samples the payoff frame of a Tier-1 {@link MontagePlan}, asks a local
|
||||||
|
* vision-language model (moondream2, via {@code tools/vision_caption.py}) what the moment is and what its
|
||||||
|
* mood is, and AUGMENTS the plan with a semantic overlay caption and a scene-informed music direction — the
|
||||||
|
* nuance measurement alone cannot reach.
|
||||||
|
*
|
||||||
|
* <p>It never fails the pipeline: on any error (model missing, worker failure, timeout) it returns the input
|
||||||
|
* plan unchanged, so the Tier-1 cut always stands.
|
||||||
|
*/
|
||||||
|
@Component
|
||||||
|
public class HighlightVisionDirector {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightVisionDirector.class);
|
||||||
|
private static final String SCRIPT = "tools/vision_caption.py";
|
||||||
|
private static final long TIMEOUT_SECONDS = 600;
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
|
||||||
|
private final String ffmpegBinary;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
|
||||||
|
public HighlightVisionDirector(VideoClippingProperties properties, ObjectMapper objectMapper) {
|
||||||
|
this.worker = properties.getEditing().getLocalAssetWorker();
|
||||||
|
this.ffmpegBinary = properties.getEditing().getFfmpegBinary();
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Returns a plan augmented with a semantic overlay + scene-informed music, or the input plan on failure. */
|
||||||
|
public MontagePlan augment(MontagePlan plan, Path source, Path workDir) {
|
||||||
|
try {
|
||||||
|
int payoff = payoffShotIndex(plan);
|
||||||
|
if (payoff < 0) {
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
MontagePlan.Shot shot = plan.shots().get(payoff);
|
||||||
|
double sourceMidpoint = shot.sourceStartSeconds() + shot.durationSeconds() * shot.speed() / 2.0;
|
||||||
|
|
||||||
|
Files.createDirectories(workDir);
|
||||||
|
Path frame = workDir.resolve("vision-payoff.jpg");
|
||||||
|
if (!extractFrame(source, sourceMidpoint, frame)) {
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
|
||||||
|
List<Caption> captions = caption(workDir, frame);
|
||||||
|
String label = captions.stream().filter(c -> "label".equals(c.id())).map(Caption::answer)
|
||||||
|
.findFirst().orElse("");
|
||||||
|
String scene = captions.stream().filter(c -> "scene".equals(c.id())).map(Caption::answer)
|
||||||
|
.findFirst().orElse("");
|
||||||
|
|
||||||
|
String overlayText = toOverlayText(label);
|
||||||
|
List<MontagePlan.Overlay> overlays = new ArrayList<>(
|
||||||
|
plan.overlays() == null ? List.of() : plan.overlays());
|
||||||
|
if (!overlayText.isBlank()) {
|
||||||
|
double[] span = payoffTimeline(plan, payoff);
|
||||||
|
overlays.add(new MontagePlan.Overlay(overlayText, span[0] + 0.2,
|
||||||
|
Math.max(span[0] + 0.8, span[1] - 0.2), "lower_center_safe"));
|
||||||
|
}
|
||||||
|
String music = scene.isBlank() ? plan.musicDirection()
|
||||||
|
: "cinematic film score for this scene: " + scene
|
||||||
|
+ " Build quiet tension to a triumphant climax hit, then a short warm resolve, no vocals.";
|
||||||
|
|
||||||
|
log.info("event=highlight_vision_director_completed overlay=\"{}\" scene_len={}",
|
||||||
|
overlayText, scene.length());
|
||||||
|
return new MontagePlan(plan.projectId(), plan.sourceVideoFileName(), plan.grade(), music,
|
||||||
|
plan.voiceover(), overlays, plan.shots());
|
||||||
|
} catch (RuntimeException | IOException | InterruptedException ex) {
|
||||||
|
if (ex instanceof InterruptedException) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
}
|
||||||
|
log.warn("event=highlight_vision_director_failed error_type={} message={}",
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The strongest slow-motion shot is the payoff; -1 if there is none. */
|
||||||
|
static int payoffShotIndex(MontagePlan plan) {
|
||||||
|
int best = -1;
|
||||||
|
double slowest = 1.0;
|
||||||
|
List<MontagePlan.Shot> shots = plan.shots();
|
||||||
|
for (int i = 0; i < shots.size(); i++) {
|
||||||
|
if (shots.get(i).speed() < slowest) {
|
||||||
|
slowest = shots.get(i).speed();
|
||||||
|
best = i;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best;
|
||||||
|
}
|
||||||
|
|
||||||
|
static double[] payoffTimeline(MontagePlan plan, int index) {
|
||||||
|
double start = 0;
|
||||||
|
for (int i = 0; i < index; i++) {
|
||||||
|
start += plan.shots().get(i).durationSeconds();
|
||||||
|
}
|
||||||
|
return new double[]{start, start + plan.shots().get(index).durationSeconds()};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cleans a model answer into a short, bold overlay: strips punctuation, keeps up to three words, upper-cases.
|
||||||
|
* Returns "" when the answer is empty or unusably long/gibberish.
|
||||||
|
*/
|
||||||
|
static String toOverlayText(String answer) {
|
||||||
|
if (answer == null) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
String cleaned = answer.replaceAll("[\"'.!?,:;]", " ").trim();
|
||||||
|
if (cleaned.isBlank()) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
String[] words = cleaned.split("\\s+");
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
for (int i = 0; i < Math.min(3, words.length); i++) {
|
||||||
|
if (words[i].length() > 18) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (sb.length() > 0) {
|
||||||
|
sb.append(' ');
|
||||||
|
}
|
||||||
|
sb.append(words[i]);
|
||||||
|
}
|
||||||
|
return sb.toString().toUpperCase(Locale.ROOT).trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean extractFrame(Path source, double atSeconds, Path output) throws IOException, InterruptedException {
|
||||||
|
List<String> command = List.of(ffmpegBinary, "-hide_banner", "-loglevel", "error",
|
||||||
|
"-y", "-ss", Double.toString(Math.max(0.0, atSeconds)), "-i", source.toString(),
|
||||||
|
"-frames:v", "1", output.toString());
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
process.getInputStream().readAllBytes();
|
||||||
|
return process.waitFor() == 0 && Files.isRegularFile(output);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<Caption> caption(Path workDir, Path frame) throws IOException, InterruptedException {
|
||||||
|
List<Manifest> manifest = List.of(
|
||||||
|
new Manifest("label", frame.toAbsolutePath().toString(),
|
||||||
|
"In one to three words, what is the exciting achievement or action in this moment?"),
|
||||||
|
new Manifest("scene", frame.toAbsolutePath().toString(),
|
||||||
|
"Describe the scene and mood in one short sentence for a film score composer."));
|
||||||
|
Path manifestFile = workDir.resolve("vision-manifest.json");
|
||||||
|
Path outputFile = workDir.resolve("vision-captions.json");
|
||||||
|
objectMapper.writeValue(manifestFile.toFile(), manifest);
|
||||||
|
|
||||||
|
List<String> command = List.of(worker.getPythonBinary(), SCRIPT,
|
||||||
|
"--manifest", manifestFile.toString(), "--output", outputFile.toString());
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
boolean finished = process.waitFor(TIMEOUT_SECONDS, java.util.concurrent.TimeUnit.SECONDS);
|
||||||
|
if (!finished) {
|
||||||
|
process.destroyForcibly();
|
||||||
|
throw new IllegalStateException("vision caption worker timed out");
|
||||||
|
}
|
||||||
|
if (process.exitValue() != 0 || !Files.isRegularFile(outputFile)) {
|
||||||
|
throw new IllegalStateException("vision caption worker failed: " + out.strip());
|
||||||
|
}
|
||||||
|
Caption[] captions = objectMapper.readValue(outputFile.toFile(), Caption[].class);
|
||||||
|
return List.of(captions);
|
||||||
|
}
|
||||||
|
|
||||||
|
record Manifest(String id, String image, String question) {
|
||||||
|
}
|
||||||
|
|
||||||
|
record Caption(String id, String answer) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -59,9 +59,14 @@ video-clipping:
|
||||||
rejected-directory: ./input/localpoc/highlights/rejected
|
rejected-directory: ./input/localpoc/highlights/rejected
|
||||||
render-enabled: false
|
render-enabled: false
|
||||||
require-director-approval: true
|
require-director-approval: true
|
||||||
|
# Tier-1 automatic director: compose director/montage.json from measured motion + audio after analysis.
|
||||||
|
auto-director-enabled: true
|
||||||
|
# Tier-2 vision director: augment the montage with a semantic overlay + scene-informed music (moondream2).
|
||||||
|
vision-director-enabled: true
|
||||||
max-highlights-per-source: 3
|
max-highlights-per-source: 3
|
||||||
highlight-min-duration-seconds: 8
|
# No fixed highlight length — a highlight can be as long as the story needs (soft candidate hints only).
|
||||||
highlight-max-duration-seconds: 35
|
highlight-min-duration-seconds: 2
|
||||||
|
highlight-max-duration-seconds: 3600
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level:
|
level:
|
||||||
|
|
|
||||||
|
|
@ -95,8 +95,11 @@ video-clipping:
|
||||||
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:true}
|
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:true}
|
||||||
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
||||||
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
||||||
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
|
# Highlights are not restricted to a fixed length: a highlight may be as long as the story needs.
|
||||||
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35}
|
# These bounds are only soft hints for coverage-window candidate generation; the director plan
|
||||||
|
# validator no longer rejects a highlight on duration (only positive/finite duration + sane speed).
|
||||||
|
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:2}
|
||||||
|
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:3600}
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level:
|
level:
|
||||||
|
|
|
||||||
|
|
@ -121,6 +121,67 @@ class FfmpegClipInspectorTest {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void swapsDimensionsForRotatedPortraitSideData() {
|
||||||
|
AtomicReference<List<String>> command = new AtomicReference<>();
|
||||||
|
FfmpegClipInspector inspector = inspector(arguments -> {
|
||||||
|
command.set(arguments);
|
||||||
|
return new FfmpegClipInspector.ProcessResult(0, """
|
||||||
|
{
|
||||||
|
"streams": [
|
||||||
|
{
|
||||||
|
"codec_type": "video",
|
||||||
|
"codec_name": "h264",
|
||||||
|
"width": 1024,
|
||||||
|
"height": 576,
|
||||||
|
"r_frame_rate": "30/1",
|
||||||
|
"side_data_list": [ { "rotation": -90 } ]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"format": { "duration": "16.0" }
|
||||||
|
}
|
||||||
|
""");
|
||||||
|
});
|
||||||
|
|
||||||
|
ClipAnalysis analysis = inspector.inspect(Path.of("portrait.mp4"));
|
||||||
|
|
||||||
|
// Effective (display) orientation is portrait: stored landscape pixels are swapped.
|
||||||
|
assertThat(analysis.width()).isEqualTo(576);
|
||||||
|
assertThat(analysis.height()).isEqualTo(1024);
|
||||||
|
assertThat(command.get()).contains("stream_side_data=rotation", "stream_tags=rotate");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void swapsDimensionsForRotateTagAndIgnoresHalfTurn() {
|
||||||
|
FfmpegClipInspector inspector = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """
|
||||||
|
{
|
||||||
|
"streams": [ {
|
||||||
|
"codec_type": "video", "codec_name": "h264",
|
||||||
|
"width": 1080, "height": 1920, "r_frame_rate": "30/1",
|
||||||
|
"tags": { "rotate": "90" }
|
||||||
|
} ],
|
||||||
|
"format": { "duration": "5.0" }
|
||||||
|
}
|
||||||
|
"""));
|
||||||
|
ClipAnalysis rotated = inspector.inspect(Path.of("tag.mp4"));
|
||||||
|
assertThat(rotated.width()).isEqualTo(1920);
|
||||||
|
assertThat(rotated.height()).isEqualTo(1080);
|
||||||
|
|
||||||
|
FfmpegClipInspector upsideDown = inspector(arguments -> new FfmpegClipInspector.ProcessResult(0, """
|
||||||
|
{
|
||||||
|
"streams": [ {
|
||||||
|
"codec_type": "video", "codec_name": "h264",
|
||||||
|
"width": 1920, "height": 1080, "r_frame_rate": "30/1",
|
||||||
|
"side_data_list": [ { "rotation": 180 } ]
|
||||||
|
} ],
|
||||||
|
"format": { "duration": "5.0" }
|
||||||
|
}
|
||||||
|
"""));
|
||||||
|
ClipAnalysis flipped = upsideDown.inspect(Path.of("flip.mp4"));
|
||||||
|
assertThat(flipped.width()).isEqualTo(1920);
|
||||||
|
assertThat(flipped.height()).isEqualTo(1080);
|
||||||
|
}
|
||||||
|
|
||||||
private FfmpegClipInspector inspector(FfmpegClipInspector.ProcessExecutor executor) {
|
private FfmpegClipInspector inspector(FfmpegClipInspector.ProcessExecutor executor) {
|
||||||
return new FfmpegClipInspector(new VideoClippingProperties(), new ObjectMapper(), executor);
|
return new FfmpegClipInspector(new VideoClippingProperties(), new ObjectMapper(), executor);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -106,6 +106,31 @@ class HighlightDirectorPlanValidatorTest {
|
||||||
.hasMessageContaining("playback speed outside 0.25..4.0");
|
.hasMessageContaining("playback speed outside 0.25..4.0");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void acceptsHighlightLongerThanTheFormerFixedMaximum() {
|
||||||
|
// Source span 2..14 (12s) stretched to a 40s target (0.3x slow-mo) would have been rejected by the
|
||||||
|
// old 35s cap. Highlights may now be as long as the story needs (speed must still be sane).
|
||||||
|
HighlightDirectorPlan.HighlightItem longHighlight = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Epic", 2, 14, 40,
|
||||||
|
"hero_payoff", "cinematic grade", "scene-fit score", "impact",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
|
||||||
|
assertThat(validator.validate("project-1", plan(longHighlight)))
|
||||||
|
.isEqualTo(plan(longHighlight));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsNonPositiveTargetDuration() {
|
||||||
|
HighlightDirectorPlan.HighlightItem zero = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening", 2, 14, 0,
|
||||||
|
"opening_hook", "cinematic grade", "scene-fit score", "impact",
|
||||||
|
List.of("Grounded narration."), List.of(), "reviewed treatment");
|
||||||
|
|
||||||
|
assertThatThrownBy(() -> validator.validate("project-1", plan(zero)))
|
||||||
|
.isInstanceOf(BadRequestException.class)
|
||||||
|
.hasMessageContaining("positive number of seconds");
|
||||||
|
}
|
||||||
|
|
||||||
private HighlightCandidate candidate() {
|
private HighlightCandidate candidate() {
|
||||||
return new HighlightCandidate("candidate_001", "source", 2, 14, 0.7,
|
return new HighlightCandidate("candidate_001", "source", 2, 14, 0.7,
|
||||||
"opening_hook_candidate", List.of("reviewed"));
|
"opening_hook_candidate", List.of("reviewed"));
|
||||||
|
|
|
||||||
|
|
@ -88,9 +88,11 @@ class HighlightFfmpegRendererTest {
|
||||||
String hero = String.join(" ",
|
String hero = String.join(" ",
|
||||||
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
|
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")));
|
||||||
|
|
||||||
assertThat(opening).contains("curves=preset=linear_contrast").contains("vignette=PI/7");
|
assertThat(opening).contains("curves=preset=linear_contrast").contains("vignette=PI/10");
|
||||||
assertThat(rising).contains("curves=preset=medium_contrast").contains("eq=contrast=1.12");
|
assertThat(rising).contains("curves=preset=linear_contrast").contains("eq=contrast=1.07");
|
||||||
assertThat(hero).contains("curves=preset=strong_contrast").contains("eq=contrast=1.16");
|
assertThat(hero).contains("curves=preset=medium_contrast").contains("eq=contrast=1.08");
|
||||||
|
// Exposure-preserving: every beat lifts the black point and never pushes gamma < 1 or brightness < 0.
|
||||||
|
assertThat(hero).contains("curves=all='0/0.06 1/1'").contains("gamma=1.06").contains("brightness=0.02");
|
||||||
assertThat(opening).isNotEqualTo(hero);
|
assertThat(opening).isNotEqualTo(hero);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -172,6 +174,89 @@ class HighlightFfmpegRendererTest {
|
||||||
assertThat(command).contains("-shortest");
|
assertThat(command).contains("-shortest");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void choosesPortraitGeometryForPortraitSourceAndLandscapeOtherwise() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer(); // configured 1920x1080
|
||||||
|
|
||||||
|
assertThat(renderer.outputGeometry(analysisWithDimensions(576, 1024)))
|
||||||
|
.containsExactly(1080, 1920);
|
||||||
|
assertThat(renderer.outputGeometry(analysisWithDimensions(1920, 1080)))
|
||||||
|
.containsExactly(1920, 1080);
|
||||||
|
assertThat(renderer.outputGeometry(null)).containsExactly(1920, 1080);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void portraitSegmentSkipsWidescreenLetterbox() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "fade-in", "cut", 1.0,
|
||||||
|
"hero cinematic", "montage");
|
||||||
|
|
||||||
|
String landscape = String.join(" ",
|
||||||
|
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1920, 1080,
|
||||||
|
Path.of("l.mp4")));
|
||||||
|
String portrait = String.join(" ",
|
||||||
|
renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 1, 1080, 1920,
|
||||||
|
Path.of("p.mp4")));
|
||||||
|
|
||||||
|
assertThat(landscape).contains("scale=1920:1080").contains("crop=1920:"); // 2.39 letterbox present
|
||||||
|
assertThat(portrait).contains("scale=1080:1920").doesNotContain("crop=1080:"); // no letterbox
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void exposureNormalizationAdaptsToMeasuredSourceBrightness() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
|
||||||
|
// A dark source is brightened (gamma > 1).
|
||||||
|
String dark = renderer.exposureNormalizationFilter(60);
|
||||||
|
assertThat(dark).startsWith(",eq=gamma=");
|
||||||
|
assertThat(Double.parseDouble(dark.replace(",eq=gamma=", ""))).isGreaterThan(1.0);
|
||||||
|
|
||||||
|
// A well-exposed source is left alone.
|
||||||
|
assertThat(renderer.exposureNormalizationFilter(120)).isEmpty();
|
||||||
|
|
||||||
|
// An over-bright source is darkened (gamma < 1).
|
||||||
|
String bright = renderer.exposureNormalizationFilter(215);
|
||||||
|
assertThat(Double.parseDouble(bright.replace(",eq=gamma=", ""))).isLessThan(1.0);
|
||||||
|
|
||||||
|
// Unmeasurable source -> no adjustment.
|
||||||
|
assertThat(renderer.exposureNormalizationFilter(-1)).isEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void pushInAmountIsStrongerForLowMotionShots() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
|
||||||
|
double staticShot = renderer.pushInAmount(2.6); // near-static (held)
|
||||||
|
double activeShot = renderer.pushInAmount(8.0); // lots of motion
|
||||||
|
|
||||||
|
assertThat(staticShot).isGreaterThan(activeShot);
|
||||||
|
assertThat(staticShot).isBetween(0.03, 0.12);
|
||||||
|
assertThat(activeShot).isGreaterThanOrEqualTo(0.03); // floor: always a little life
|
||||||
|
assertThat(renderer.pushInAmount(-1)).isGreaterThan(0.0); // unknown -> mild default push
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void appliesMotionAdaptivePushInOnlyWhenRequested() {
|
||||||
|
HighlightFfmpegRenderer renderer = renderer();
|
||||||
|
EditDecision decision = new EditDecision("clip", 0, 4, 0, 4, "cut", "cut", 1.0, "hero cinematic",
|
||||||
|
"montage");
|
||||||
|
|
||||||
|
String withPush = String.join(" ", renderer.segmentCommand("s.mp4", decision,
|
||||||
|
"style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.10, Path.of("p.mp4")));
|
||||||
|
String noPush = String.join(" ", renderer.segmentCommand("s.mp4", decision,
|
||||||
|
"style_generic_vlog_hero_payoff", 0, 1, 1080, 1920, "", 0.0, Path.of("n.mp4")));
|
||||||
|
|
||||||
|
assertThat(withPush).contains("zoompan=z=");
|
||||||
|
assertThat(noPush).doesNotContain("zoompan").contains("crop=iw/"); // static crop preserved at 0
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightSourceAnalysis analysisWithDimensions(int width, int height) {
|
||||||
|
ClipAnalysis source = new ClipAnalysis("clip", "s.mp4", 16, "h264", "aac", width, height, 30,
|
||||||
|
List.of(), null, null, 0, 0, 0);
|
||||||
|
return new HighlightSourceAnalysis("p", "s.mp4", source, List.of(), null, null, null, null,
|
||||||
|
List.of(), null, null, null, null, null);
|
||||||
|
}
|
||||||
|
|
||||||
private HighlightFfmpegRenderer renderer() {
|
private HighlightFfmpegRenderer renderer() {
|
||||||
return new HighlightFfmpegRenderer(new VideoClippingProperties(), mock(HighlightProjectStore.class),
|
return new HighlightFfmpegRenderer(new VideoClippingProperties(), mock(HighlightProjectStore.class),
|
||||||
mock(EditAssetProvider.class), mock(EditObservability.class), command -> null);
|
mock(EditAssetProvider.class), mock(EditObservability.class), command -> null);
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightMontageDirectorTest {
|
||||||
|
|
||||||
|
private final HighlightMontageDirector director = new HighlightMontageDirector(new VideoClippingProperties());
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void composesAStoryStructureFromMeasuredMotionAndAudio() {
|
||||||
|
// 16s at 0.5s windows = 32 buckets. Motion: baseline with a release spike at ~4s, a celebration bump
|
||||||
|
// at ~9-11s, and a strong camera-whip tail from ~12.5s. Audio: climax (celebration) peaking at ~8s;
|
||||||
|
// loud tail spikes after ~13.5s that must be ignored (outside the central band).
|
||||||
|
int n = 32;
|
||||||
|
double[] motion = new double[n];
|
||||||
|
double[] audio = new double[n];
|
||||||
|
for (int i = 0; i < n; i++) {
|
||||||
|
double t = i * 0.5;
|
||||||
|
motion[i] = 2.0;
|
||||||
|
if (Math.abs(t - 4.0) < 0.6) motion[i] = 5.0; // release spike
|
||||||
|
if (t >= 9.0 && t <= 11.0) motion[i] = 4.0; // celebration motion
|
||||||
|
if (t >= 12.5) motion[i] = 9.0; // camera-whip tail
|
||||||
|
audio[i] = -30.0;
|
||||||
|
if (t >= 6.5 && t <= 9.5) audio[i] = -12.0; // celebration band
|
||||||
|
if (Math.abs(t - 8.0) < 0.6) audio[i] = -8.0; // climax peak
|
||||||
|
if (t >= 13.5) audio[i] = -10.0; // loud aftermath (must be ignored)
|
||||||
|
}
|
||||||
|
|
||||||
|
MontagePlan plan = director.composeMontage("p", "src.mp4", motion, audio, 0.5, 16.0);
|
||||||
|
|
||||||
|
// A real multi-beat structure, not one straight shot.
|
||||||
|
assertThat(plan.shots().size()).isGreaterThanOrEqualTo(4);
|
||||||
|
|
||||||
|
// Exactly one strong slow-motion payoff, and it sits on the audio climax (~8s).
|
||||||
|
var slowMo = plan.shots().stream().filter(s -> s.speed() < 0.8).toList();
|
||||||
|
assertThat(slowMo).hasSize(1);
|
||||||
|
assertThat(slowMo.get(0).sourceStartSeconds()).isBetween(7.5, 8.6);
|
||||||
|
|
||||||
|
// There is a continuous action/tension shot (long, not chopped).
|
||||||
|
double longestSpan = plan.shots().stream()
|
||||||
|
.mapToDouble(s -> s.durationSeconds() * s.speed()).max().orElse(0);
|
||||||
|
assertThat(longestSpan).isGreaterThan(2.0);
|
||||||
|
|
||||||
|
// The camera-whip tail (>=12.5s) is trimmed: no shot reads source beyond it.
|
||||||
|
double maxSourceEnd = plan.shots().stream()
|
||||||
|
.mapToDouble(s -> s.sourceStartSeconds() + s.durationSeconds() * s.speed()).max().orElse(0);
|
||||||
|
assertThat(maxSourceEnd).isLessThanOrEqualTo(13.0);
|
||||||
|
|
||||||
|
assertThat(plan.grade()).isEqualTo("hero");
|
||||||
|
assertThat(plan.musicDirection()).isNotBlank();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void fallsBackToAStraightCutForVeryShortSources() {
|
||||||
|
MontagePlan plan = director.composeMontage("p", "src.mp4", new double[]{1, 1}, new double[]{-30, -30},
|
||||||
|
0.5, 1.0);
|
||||||
|
assertThat(plan.shots()).hasSize(1);
|
||||||
|
assertThat(plan.shots().get(0).sourceStartSeconds()).isEqualTo(0.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -114,7 +114,9 @@ class HighlightSourceSchedulerTest {
|
||||||
stubAnalysis("porsche");
|
stubAnalysis("porsche");
|
||||||
stubAnalysis("porsche-1");
|
stubAnalysis("porsche-1");
|
||||||
stubAnalysis("porsche-drive");
|
stubAnalysis("porsche-drive");
|
||||||
return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator, clock);
|
return new HighlightSourceScheduler(properties, store, analyzer, candidateGenerator, promptGenerator,
|
||||||
|
new HighlightMontageDirector(properties),
|
||||||
|
new HighlightVisionDirector(properties, new com.fasterxml.jackson.databind.ObjectMapper()), clock);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void stubAnalysis(String projectId) {
|
private void stubAnalysis(String projectId) {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightVisionDirectorTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void cleansModelAnswersIntoShortBoldOverlays() {
|
||||||
|
assertThat(HighlightVisionDirector.toOverlayText("A strike!")).isEqualTo("A STRIKE");
|
||||||
|
assertThat(HighlightVisionDirector.toOverlayText("Bowling strike celebration moment"))
|
||||||
|
.isEqualTo("BOWLING STRIKE CELEBRATION"); // capped at 3 words
|
||||||
|
assertThat(HighlightVisionDirector.toOverlayText(" ")).isEmpty();
|
||||||
|
assertThat(HighlightVisionDirector.toOverlayText(null)).isEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void findsThePayoffShotAndItsTimelineWindow() {
|
||||||
|
MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(), List.of(
|
||||||
|
new MontagePlan.Shot(0.0, 1.5, 1.03, 1.0),
|
||||||
|
new MontagePlan.Shot(2.0, 2.0, 1.04, 1.0),
|
||||||
|
new MontagePlan.Shot(8.0, 3.0, 1.05, 0.7), // payoff (slowest)
|
||||||
|
new MontagePlan.Shot(11.0, 1.2, 1.04, 0.9)));
|
||||||
|
|
||||||
|
int payoff = HighlightVisionDirector.payoffShotIndex(plan);
|
||||||
|
assertThat(payoff).isEqualTo(2);
|
||||||
|
|
||||||
|
double[] window = HighlightVisionDirector.payoffTimeline(plan, payoff);
|
||||||
|
assertThat(window[0]).isEqualTo(3.5); // 1.5 + 2.0
|
||||||
|
assertThat(window[1]).isEqualTo(6.5); // + 3.0
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void hasNoPayoffWhenNothingIsSlowMotion() {
|
||||||
|
MontagePlan plan = new MontagePlan("p", "s.mp4", "hero", "music", List.of(), List.of(),
|
||||||
|
List.of(new MontagePlan.Shot(0.0, 2.0, 1.03, 1.0)));
|
||||||
|
assertThat(HighlightVisionDirector.payoffShotIndex(plan)).isEqualTo(-1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,70 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Tier-2 vision captioner for the highlight director.
|
||||||
|
|
||||||
|
Loads a local vision-language model (moondream2) ONCE and answers a question for each frame listed in a
|
||||||
|
manifest, writing the answers to a JSON file. Runs fully offline from the local Hugging Face cache — it
|
||||||
|
performs no network access (HF_HUB_OFFLINE is forced on).
|
||||||
|
|
||||||
|
Manifest JSON: [{"id": "...", "image": "/abs/path.jpg", "question": "..."}]
|
||||||
|
Output JSON: [{"id": "...", "answer": "..."}]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Local VLM frame captioner (moondream2)")
|
||||||
|
parser.add_argument("--manifest", required=True)
|
||||||
|
parser.add_argument("--output", required=True)
|
||||||
|
parser.add_argument("--model-id", default="vikhyatk/moondream2")
|
||||||
|
parser.add_argument("--revision", default="2024-08-26")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Hard offline: never reach the network. The model must already be in the local cache.
|
||||||
|
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||||
|
os.environ["TRANSFORMERS_OFFLINE"] = "1"
|
||||||
|
os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from PIL import Image
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
except Exception as exc: # pragma: no cover - dependency guard
|
||||||
|
print(f"vision_caption: missing dependency: {exc}", file=sys.stderr)
|
||||||
|
return 3
|
||||||
|
|
||||||
|
with open(args.manifest, "r", encoding="utf-8") as handle:
|
||||||
|
items = json.load(handle)
|
||||||
|
|
||||||
|
try:
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
args.model_id, revision=args.revision, trust_remote_code=True, local_files_only=True)
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(
|
||||||
|
args.model_id, revision=args.revision, local_files_only=True)
|
||||||
|
model.eval()
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"vision_caption: unable to load local model: {exc}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for item in items:
|
||||||
|
try:
|
||||||
|
image = Image.open(item["image"]).convert("RGB")
|
||||||
|
encoded = model.encode_image(image)
|
||||||
|
answer = model.answer_question(encoded, item["question"], tokenizer).strip()
|
||||||
|
except Exception as exc: # keep going; a single bad frame must not fail the batch
|
||||||
|
print(f"vision_caption: frame {item.get('id')} failed: {exc}", file=sys.stderr)
|
||||||
|
answer = ""
|
||||||
|
results.append({"id": item.get("id"), "answer": answer})
|
||||||
|
|
||||||
|
os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True)
|
||||||
|
with open(args.output, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(results, handle, ensure_ascii=False, indent=2)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Loading…
Reference in New Issue