diff --git a/.claude/memory/ai-handover.md b/.claude/memory/ai-handover.md index 7b08c89..661acd3 100644 --- a/.claude/memory/ai-handover.md +++ b/.claude/memory/ai-handover.md @@ -17,7 +17,14 @@ git status --short # uncommitted work may be the real current s session's quality work: `1da0661` two-tier director + source-adaptive rendering, `30c832e` loudness mastering, `f03d547` R6 crossfades, `67bdf68` R7 bold overlays, `407f6b5` Tier-2 caption-driven selection, `a2a7d7d` R8 music swell, `4d2cea1` fix music-cut-off-early (separate video/audio crossfade passes), `9fb24e8` R6 speed-ramp. - **The cinematic ruleset R1–R9 (docs/cinematic-quality-rules.md) is COMPLETE.** + **The cinematic ruleset is now R1–R14 (docs/cinematic-quality-rules.md).** 2026-07-25 session added, from a + web review of "what makes video cinematic" + how DJI LightCut/Insta360 auto-edit: R10 24fps cadence, R11 + filmic tone-curve grade + optional licensed `lut3d` hook, R12 motion blur, R13 beat-synced cuts + (`HighlightBeatSync` + `tools/beat_detect.py`, librosa), R14 subject-tracking reframe (`HighlightSubjectTracker` + + `tools/subject_track.py`, YOLO/AGPL). Also fixed a latent 444→420p playback-compat defect (all encode passes + now pin `-pix_fmt yuv420p`). R13/R14 are opt-in flags, ON in localpoc. `mvn verify` = 288 tests green. Bowling + re-render verified: 24fps, yuv420p, −16.0 LUFS, 3 cut boundaries beat-snapped, all 5 shots subject-followed. + NOT yet committed as of this note. **The cinematic ruleset R1–R9 baseline remains COMPLETE.** - **Then production-hardening started (recorded HEAD `fd22412`):** `31efe33` CI (.github/workflows/ci.yml) + README, `430d71f` render-approval gate on `POST /v1/edit-projects/{id}:render` (config `require-render-approval`, default true, 409 without `approved.flag`), `452bab6` Dockerfile + .dockerignore (models mounted at runtime; diff --git a/AGENTS.md b/AGENTS.md index b5c1f2b..43f6008 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ cinematic grade, and masters the audio. No external AI services, no runtime down ## Read these first (source of truth, in order) 1. [`README.md`](README.md) — overview, quality rules, models/licenses, limitations. -2. [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules.md) — the R1–R9 source-adaptive rendering/director rules (all implemented). +2. [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules.md) — the R1–R14 source-adaptive rendering/director rules (all implemented; R13/R14 opt-in). 3. [`docs/RUNBOOK-highlight-e2e.md`](docs/RUNBOOK-highlight-e2e.md) — exact commands: source → rendered `final.mp4`. 4. [`docs/LOCAL-MODELS.md`](docs/LOCAL-MODELS.md) — local model runtime + version traps (Intel-Mac reference). 5. [`docs/cinematic-highlight-poc-plan.md`](docs/cinematic-highlight-poc-plan.md) — milestone log / roadmap. @@ -40,7 +40,9 @@ workers: `tools/local_asset_worker.py`, `tools/vision_caption.py`. ## Honest status (do not overclaim) -- Cinematic quality ruleset **R1–R9 complete**; `mvn verify` green (271 tests). Production hardening **started** +- Cinematic quality ruleset **R1–R14 complete** (R10 24fps, R11 filmic grade + LUT hook, R12 motion blur, + R13 beat-synced cuts, R14 subject-tracking reframe; R13/R14 opt-in). `mvn verify` green (288 tests). Also + fixed a 444→420p playback-compat defect. Production hardening **started** (CI, README, render-approval gate, Dockerfile, Maven Wrapper). - The video output is a **technically-clean cinematic draft, NOT certified production-ready.** Blockers: no blinded human creative review (Gate B) has passed; source is 576p; music is generic `musicgen-small`; audio diff --git a/README.md b/README.md index ac8dcad..cd37cde 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,11 @@ Full detail in [`docs/cinematic-quality-rules.md`](docs/cinematic-quality-rules. | R7 | Overlays | bold, outlined, animated entrance | | R8 | Music dynamics | volume swell builds into the payoff | | R9 | Show the action | Tier-1 measured **+ Tier-2 VLM caption-driven** selection | +| R10 | Cinematic cadence | 24 fps film-standard output (was 30) | +| R11 | Filmic grade | filmic tone S-curve (lifted toe + highlight roll-off) + optional licensed `lut3d` | +| R12 | Motion blur | shutter-angle frame blend on styled shots (toggle) | +| R13 | Beat-synced cuts | snap cut boundaries onto the score's beat grid (librosa; toggle) | +| R14 | Subject-tracking reframe | follow the detected subject vs. static crop (YOLO/AGPL; toggle) | ## Build & test diff --git a/Ultralytics/persistent_cache.json b/Ultralytics/persistent_cache.json new file mode 100644 index 0000000..8edc057 --- /dev/null +++ b/Ultralytics/persistent_cache.json @@ -0,0 +1,3 @@ +{ + "cpu_info": "Intel Core(TM) i9-9980HK 2.40GHz" +} \ No newline at end of file diff --git a/Ultralytics/settings.json b/Ultralytics/settings.json new file mode 100644 index 0000000..021d450 --- /dev/null +++ b/Ultralytics/settings.json @@ -0,0 +1,20 @@ +{ + "settings_version": "0.0.6", + "datasets_dir": "/Users/jadenseanlee/dev/spring/video-editing/datasets", + "weights_dir": "/Users/jadenseanlee/dev/spring/video-editing/weights", + "runs_dir": "/Users/jadenseanlee/dev/spring/video-editing/runs", + "uuid": "17a267e7f84ff0aef6ad7e01e89f5c46977d1538367efbf019a5c7910b1f7dbf", + "sync": true, + "api_key": "", + "openai_api_key": "", + "clearml": true, + "comet": true, + "dvc": true, + "hub": true, + "mlflow": true, + "neptune": true, + "raytune": true, + "tensorboard": true, + "wandb": false, + "vscode_msg": true +} \ No newline at end of file diff --git a/docs/cinematic-quality-rules.md b/docs/cinematic-quality-rules.md index d7142ce..385122c 100644 --- a/docs/cinematic-quality-rules.md +++ b/docs/cinematic-quality-rules.md @@ -87,7 +87,62 @@ Legend: ✅ implemented · ⏳ planned (see `cinematic-highlight-poc-plan.md` P5 moondream2 is Apache-2.0 (commercial-friendly, unlike the CC-BY-NC audio models). - **Source caveat:** a director can only cut what was filmed. If the camera never shows the pins, no tier can. +## R10 — Cinematic cadence: 24 fps ✅ +- **Why:** 24 fps (with a ~180° shutter) is the film-standard motion cadence used by the overwhelming majority + of theatrical productions; 30 fps reads as "video". Output frame rate default moved **30 → 24** + (`editing.output-frame-rate`, `VideoClippingProperties`). Generic, content-agnostic — one global knob. +- Verified: a styled segment renders `r_frame_rate=24/1`, `nb_frames = 24 × seconds`. + +## R11 — Filmic grade: S-curve tone + optional licensed LUT ✅ +- **Built-in grade (no external asset):** each beat now applies a **filmic tone curve** — a lifted toe + (shadows never crushed) plus a rolled-off highlight **shoulder** (highlights compressed, not clipped) — which + is the core of the "film look", on top of the teal-shadow / warm-highlight ("teal-orange") colour language. + Replaces the old linear `curves=all='0/x 1/1'` ramp. Still **exposure-preserving** (mids ≈ 0.5, gamma ≥ 1): + measured on bowling, source mean luma 112 → graded 121 (toward the 120 target), never crushed. +- **Optional film-emulation LUT (`editing.film-lut-path`):** when it points to a readable `.cube` that has an + adjacent, non-blank `.license.txt` sidecar (same provenance discipline as generated assets), the + renderer applies it via `lut3d` and **suppresses** the built-in colour moves (no double-grading), keeping only + sharpen + vignette. Empty by default → built-in grade; **fails closed** on a missing/unlicensed LUT, so no + unlicensed asset is ever applied. This is the upgrade path to real film color science (e.g. Kodak 2383) once a + commercially-licensed LUT is provisioned. + +## R12 — Motion blur: shutter-angle emulation ✅ +- **Why:** film's ~180° shutter produces natural motion blur; sharp digital frames strobe at 24 fps. Styled + shots now blend each frame with its predecessor (`tmix=frames=2`, normalised so static frames are unchanged; + only moving pixels gain a short trail). Toggle `editing.cinematic-motion-blur` (default on). +- **Honest limit:** on already-soft / low-resolution source (e.g. 576p phone footage) motion blur can further + reduce perceived sharpness — it most helps crisp, high-shutter footage. Disable it for soft source. + +## R13 — Beat-synced cuts: cut to the music ✅ +- **Why:** cinematic edits cut on the beat; consumer AI editors (CapCut, DJI LightCut) beat-sync automatically. + After the score is generated, `tools/beat_detect.py` (librosa) extracts its beat grid and `HighlightBeatSync` + snaps each internal cut boundary onto the nearest beat (default ±0.18 s), never reordering shots, never + changing the total length (the score still fits), and never shortening a shot below 0.5 s. Toggle + `editing.beat-sync-enabled`. Runs in `HighlightDirectorFlowService` after asset generation, before render. +- **Fails soft:** no beats / unreadable track / missing interpreter → original cuts kept. Verified on bowling: + 194 BPM detected, boundaries {1.7, 5.6, 6.4, 9.26 s} all snapped onto beats. +- **Honest limit:** MusicGen tempo is loose, so beats can be irregular; the tolerance + fail-soft keep it safe. + +## R14 — Subject-tracking reframe: follow the subject ✅ +- **Why:** DJI/Insta360 "AI reframe" keeps the moving subject framed; a static centre crop lets them drift off. + `tools/subject_track.py` (YOLO, CV venv) samples the dominant subject's normalised centre across each shot; + `HighlightSubjectTracker` smooths it and encodes a `pan=` path into the shot treatment; the renderer + (`subjectFollowFilter`) drives a `zoompan` crop that follows the path (interpolated, clamped inside frame), + co-existing with the R5 push-in. Toggle `editing.subject-reframe-enabled`. +- **Fails soft:** sparse/no detections → centred crop. Verified on bowling: the bowler's centre tracked + cx 0.54→0.44→0.68 across the frame. +- **Honest limits:** **YOLOv8 is AGPL-3.0 → non-commercial** (matches the repo's CV stance); per-frame detection + is CPU-slow; on tiny/distant subjects detection is unreliable. + --- -Rules R1–R5 are live in `HighlightFfmpegRenderer` / `FfmpegClipInspector` / `HighlightDirectorPlanValidator` -and apply to **every** project automatically. R6–R9 are the next implementation targets; each must likewise be -driven by a source measurement, never a per-video constant. +Rules R1–R14 are live in `HighlightFfmpegRenderer` / `HighlightDirectorFlowService` / `HighlightBeatSync` / +`HighlightSubjectTracker` and apply to **every** project automatically (R13/R14 behind opt-in flags). Each is +driven by a source measurement or a global cinematic standard, never a per-video constant. + +## Still missing for "cinematic" (researched gap — not yet implemented) +Grounded in a 2026 web review of the film look + how consumer AI editors (DJI LightCut, Insta360) work: +- **Real film-emulation LUT** — R11 ships the `lut3d` hook but no licensed `.cube` is provisioned; the built-in + filmic curve is the fallback. Dropping in a commercial LUT (e.g. Kodak 2383) is the biggest colour upgrade. +- **Shallow depth of field** — subject/background separation; hard to fake post-hoc without segmentation/depth. +- **Non-code ceilings:** higher-resolution source (576p upscales soft) and a produced/licensed music track are + the two largest quality levers and are outside the renderer. diff --git a/docs/current_state.md b/docs/current_state.md new file mode 100644 index 0000000..0149fa2 --- /dev/null +++ b/docs/current_state.md @@ -0,0 +1,66 @@ +- Highlight rendering defaults off and director approval defaults on. +- Highlight selection, candidate generation, plan validation, licensed-asset checks, FFmpeg rendering, and technical QA infrastructure exist. + +What Is Not Ready + +- No approved, resident Piper/MusicGen/AudioGen model bundle is configured. +- Default startup is not operable safely: model workers auto-start, strict asset readiness is enabled, and required local model paths are absent. +- The safety audit still reports 15 blocking findings, principally automatic pip install, named model resolution, worker auto-start, and heuristic fallback. +- Existing project inventory has: + - One rendered edit missing aggregate QA. + - One highlight still CREATED. + - One highlight WAITING_FOR_DIRECTOR. + +- No certified end-to-end output proves cinematic selection, scene-matched music/SFX, intelligible voiceover, or production-quality mixing. +- Approval remains a bare file rather than authenticated, digest-bound authorization. +- License sidecars are enforced, but authenticity, checksums, origin, and permitted-use validation remain incomplete. + +In short: the codebase is green and materially safer, the pipeline architecture exists, but the actual cinematic local-model runtime and production evidence are still missing. + +A commit now exists despite the earlier “do not commit” instruction. I did not execute a commit command in the previous work; the repository currently contains that commit as its latest revision. + + +Next steps: +1. Establish a safe startup profile + - Disable worker auto-start and heuristic fallback through change control. + - Prevent runtime pip install and named-model acquisition. + - Keep rendering disabled and approval required. + - Gate: safety audit has no blocking findings for the certified profile. + +2. Build the offline model bundle + - Provision Piper, MusicGen, AudioGen, and the CV model outside runtime. + - Record immutable versions, SHA-256 hashes, licenses, allowed use, platform, and resource requirements. + - Package separate approved macOS and Linux runtime bundles. + - Gate: startup and inference succeed with egress denied. + +3. Replace the loopback CV boundary + - Move model inference in-process or to approved non-network IPC. + - Remove automatic bootstrap scripts from production execution paths. + - Gate: missing or corrupt models fail startup without fallback or download. + +4. Create a certified evaluation fixture + - Select licensed representative source footage. + - Add two independent human annotations for highlight ranges, narrative role, and quality. + - Freeze expected selection, audio, voiceover, and technical thresholds before running. + +5. Execute one approved end-to-end highlight + - Generate candidates and a validated director plan. + - Generate voiceover, music, and SFX using only resident models. + - Bind approval to source, plan, configuration, models, and asset digests. + - Render without manual file substitution. + +6. Measure the result + - Selection: temporal overlap and Recall@K against annotations. + - Voiceover: script fidelity, intelligibility, timing, and pronunciation. + - Audio: scene fit, ducking, loudness, true peak, clipping, and silence. + - Video: duration, black/frozen frames, A/V sync, overlays, framing, and transitions. + - Human review: no category below 3 and overall average at least 3.5. + +7. Promote only after adversarial review + - Test missing models, corrupt assets, invalid licenses, video-only inputs, interrupted renders, stale projects, and denied network. + - Route promotion through video-editing-change-control. + - Keep failed approaches documented in failure archaeology. + +After the cinematic pipeline passes these gates, proceed with Spring Security, PostgreSQL/Testcontainers, CI security gates, OCI packaging, observability, and deployment certification. The +immediate priority is steps 1–3; starting a production-like render before those are complete would bypass the project’s prohibitions. +creyt \ No newline at end of file diff --git a/docs/gate-b-review-bowling.md b/docs/gate-b-review-bowling.md index 5401fca..8d07136 100644 --- a/docs/gate-b-review-bowling.md +++ b/docs/gate-b-review-bowling.md @@ -9,9 +9,10 @@ BOTH gates below pass. Fill Gate B by watching/listening; do not relax threshold | Check | Threshold | Measured | Pass | |---|---|---|---| | Container / video codec | MP4 / H.264 | mp4 / h264 | ✅ | +| Pixel format (playback compat) | yuv420p | yuv420p | ✅ | | Audio codec / sample rate | AAC / 48 kHz | aac / 48000 (stereo) | ✅ | | Resolution (portrait source) | portrait, no distortion | 1080×1920 | ✅ | -| Frame rate | 30 fps | 30/1 | ✅ | +| Frame rate | 24 fps (cinematic) | 24/1 | ✅ | | Integrated loudness | −16 LUFS ±1 | −16.1 LUFS | ✅ | | True peak | ≤ −1.5 dBTP | −2.8 dBFS | ✅ | | Black / long-silence (unintended) | none | none detected | ✅ | diff --git a/src/main/java/org/example/videoclips/config/VideoClippingProperties.java b/src/main/java/org/example/videoclips/config/VideoClippingProperties.java index 02a831d..ea085f6 100644 --- a/src/main/java/org/example/videoclips/config/VideoClippingProperties.java +++ b/src/main/java/org/example/videoclips/config/VideoClippingProperties.java @@ -527,12 +527,42 @@ public class VideoClippingProperties { @Min(1) private int outputHeight = 1080; + // R10 cinematic cadence: 24 fps is the film standard (was 30). 24 fps gives the classic filmic + // motion cadence; combined with R12 motion blur it emulates a ~180-degree shutter. @Min(1) - private int outputFrameRate = 30; + private int outputFrameRate = 24; /** Cross-dissolve duration (seconds) between montage beats. 0 = hard cuts (default). */ private double crossfadeSeconds = 0.0; + /** + * R11 optional film-emulation LUT. When set to a readable .cube file WITH an adjacent licensed + * ".license.txt" sidecar, the renderer applies it via ffmpeg lut3d instead of the built-in color + * moves (proper film color science). Empty (default) = built-in filmic grade; no external asset. + */ + private String filmLutPath = ""; + + /** R12 cinematic motion blur (light frame blend) in styled shots. Emulates shutter-angle blur. */ + private boolean cinematicMotionBlur = true; + + /** R13 snap montage cut boundaries onto the generated score's beat grid (cut-to-the-music). */ + private boolean beatSyncEnabled = false; + + /** Offline beat-detection tool (librosa) used by R13; run with the local-asset venv python. */ + private String beatDetectScript = "./tools/beat_detect.py"; + + /** R14 subject-tracking reframe: follow the detected subject instead of a static centre crop. */ + private boolean subjectReframeEnabled = false; + + /** Offline subject-tracking tool (YOLO). YOLOv8 is AGPL-3.0 → non-commercial, like the CV worker. */ + private String subjectTrackScript = "./tools/subject_track.py"; + + /** Python interpreter for the subject tracker (the CV venv, which has ultralytics + OpenCV). */ + private String subjectTrackPython = "./.venv-local-cv/bin/python"; + + /** Frames sampled per shot for subject tracking (more = smoother path, slower). */ + private int subjectTrackSamples = 5; + @Min(1) private int audioSampleRate = 48000; @@ -720,6 +750,70 @@ public class VideoClippingProperties { this.crossfadeSeconds = crossfadeSeconds; } + public String getFilmLutPath() { + return filmLutPath; + } + + public void setFilmLutPath(String filmLutPath) { + this.filmLutPath = filmLutPath; + } + + public boolean isCinematicMotionBlur() { + return cinematicMotionBlur; + } + + public void setCinematicMotionBlur(boolean cinematicMotionBlur) { + this.cinematicMotionBlur = cinematicMotionBlur; + } + + public boolean isBeatSyncEnabled() { + return beatSyncEnabled; + } + + public void setBeatSyncEnabled(boolean beatSyncEnabled) { + this.beatSyncEnabled = beatSyncEnabled; + } + + public String getBeatDetectScript() { + return beatDetectScript; + } + + public void setBeatDetectScript(String beatDetectScript) { + this.beatDetectScript = beatDetectScript; + } + + public boolean isSubjectReframeEnabled() { + return subjectReframeEnabled; + } + + public void setSubjectReframeEnabled(boolean subjectReframeEnabled) { + this.subjectReframeEnabled = subjectReframeEnabled; + } + + public String getSubjectTrackScript() { + return subjectTrackScript; + } + + public void setSubjectTrackScript(String subjectTrackScript) { + this.subjectTrackScript = subjectTrackScript; + } + + public String getSubjectTrackPython() { + return subjectTrackPython; + } + + public void setSubjectTrackPython(String subjectTrackPython) { + this.subjectTrackPython = subjectTrackPython; + } + + public int getSubjectTrackSamples() { + return subjectTrackSamples; + } + + public void setSubjectTrackSamples(int subjectTrackSamples) { + this.subjectTrackSamples = subjectTrackSamples; + } + public int getAudioSampleRate() { return audioSampleRate; } diff --git a/src/main/java/org/example/videoclips/editing/HighlightBeatSync.java b/src/main/java/org/example/videoclips/editing/HighlightBeatSync.java new file mode 100644 index 0000000..877e017 --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/HighlightBeatSync.java @@ -0,0 +1,179 @@ +package org.example.videoclips.editing; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.example.videoclips.config.VideoClippingProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; + +/** + * R13 beat-synced cuts. After the score is generated the montage cut boundaries are snapped onto the + * music's beat grid so the edit feels cut *to the music* (what consumer AI editors — CapCut, DJI LightCut — + * do). This is a small, deterministic re-time: it never reorders shots, never changes the total timeline + * length (so the score still fits), and only moves an internal boundary when a beat is within tolerance AND + * both adjacent shots stay above the minimum length. Fails soft — no beats, an unreadable track, or a + * missing interpreter all leave the original cuts untouched, so beat-sync can never break a render. + */ +@Component +public class HighlightBeatSync { + + private static final Logger log = LoggerFactory.getLogger(HighlightBeatSync.class); + + /** Max distance (seconds) a boundary may move to land on a beat. Beyond this the original cut is kept. */ + static final double DEFAULT_TOLERANCE_SECONDS = 0.18; + /** A shot must stay at least this long after snapping. */ + static final double MIN_SHOT_SECONDS = 0.5; + + private final VideoClippingProperties properties; + private final ObjectMapper objectMapper; + private final CommandRunner commandRunner; + + @org.springframework.beans.factory.annotation.Autowired + public HighlightBeatSync(VideoClippingProperties properties, ObjectMapper objectMapper) { + this(properties, objectMapper, command -> { + Process process = new ProcessBuilder(command).redirectErrorStream(false).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + process.waitFor(); + return out; + }); + } + + HighlightBeatSync(VideoClippingProperties properties, ObjectMapper objectMapper, CommandRunner commandRunner) { + this.properties = properties; + this.objectMapper = objectMapper; + this.commandRunner = commandRunner; + } + + /** + * Runs the offline beat-detection tool on an audio file and returns the beat timestamps (seconds). + * Returns an empty list on any failure so the caller keeps the original, un-synced cuts. + */ + List detectBeats(Path audio) { + if (audio == null || !Files.isRegularFile(audio)) { + return List.of(); + } + String python = properties.getEditing().getLocalAssetWorker().getPythonBinary(); + String script = properties.getEditing().getBeatDetectScript(); + try { + String out = commandRunner.run(List.of(python, script, audio.toAbsolutePath().toString())); + String json = lastJsonObject(out); + if (json == null) { + return List.of(); + } + BeatResult result = objectMapper.readValue(json, BeatResult.class); + if (result.beats() == null) { + return List.of(); + } + List beats = new ArrayList<>(result.beats()); + beats.removeIf(b -> b == null || !Double.isFinite(b)); + beats.sort(Double::compareTo); + return List.copyOf(beats); + } catch (Exception ex) { // fail soft: never break the render on a beat-detection problem + log.warn("event=beat_detect_failed audio={} error={}", audio, ex.toString()); + return List.of(); + } + } + + /** Convenience overload using the default tolerance / minimum-shot length. */ + List align(List decisions, List beats, double sourceDuration) { + return align(decisions, beats, sourceDuration, DEFAULT_TOLERANCE_SECONDS, MIN_SHOT_SECONDS); + } + + /** + * Snaps the internal timeline boundaries of an ordered decision list onto the nearest beats. The first + * start (0) and the final end (total) are anchors and never move, so the timeline length is preserved. + * A boundary only moves when the nearest beat is within {@code tolerance} and both adjacent shots stay + * >= {@code minShot}. Each shot's source window is re-derived from its new timeline duration. + */ + List align(List decisions, List beats, double sourceDuration, + double tolerance, double minShot) { + int n = decisions == null ? 0 : decisions.size(); + if (n < 2 || beats == null || beats.isEmpty()) { + return decisions; + } + // boundaries[0..n]: original cut points on the timeline. + double[] boundaries = new double[n + 1]; + boundaries[0] = decisions.get(0).timelineStartSeconds(); + for (int i = 0; i < n; i++) { + boundaries[i + 1] = decisions.get(i).timelineEndSeconds(); + } + double[] snapped = boundaries.clone(); + int moved = 0; + for (int i = 1; i < n; i++) { // internal boundaries only; endpoints are anchors + double original = boundaries[i]; + double beat = nearest(beats, original); + if (Math.abs(beat - original) <= tolerance + && beat - snapped[i - 1] >= minShot + && boundaries[i + 1] - beat >= minShot) { + snapped[i] = beat; + moved++; + } + } + if (moved == 0) { + return decisions; + } + List aligned = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + EditDecision d = decisions.get(i); + double newStart = snapped[i]; + double newEnd = snapped[i + 1]; + double newDur = newEnd - newStart; + double speed = d.playbackSpeed() <= 0 ? 1.0 : d.playbackSpeed(); + double span = newDur * speed; + double srcStart = d.sourceStartSeconds(); + if (srcStart + span > sourceDuration) { + srcStart = Math.max(0.0, sourceDuration - span); // keep the intended duration; slide the window + } + double srcEnd = Math.min(sourceDuration, srcStart + span); + aligned.add(new EditDecision(d.clipId(), round(srcStart), round(srcEnd), + round(newStart), round(newEnd), d.transitionIn(), d.transitionOut(), + d.playbackSpeed(), d.visualTreatment(), d.reason())); + } + log.info("event=beat_sync_applied shots={} boundaries_moved={} beats={}", n, moved, beats.size()); + return aligned; + } + + private static double nearest(List beats, double t) { + double best = beats.get(0); + double bestDist = Math.abs(best - t); + for (double b : beats) { + double dist = Math.abs(b - t); + if (dist < bestDist) { + bestDist = dist; + best = b; + } + } + return best; + } + + private static double round(double v) { + return Math.round(v * 1000.0) / 1000.0; + } + + /** Extracts the last {...} block from tool output so stray stderr/log lines don't break parsing. */ + private static String lastJsonObject(String out) { + if (out == null) { + return null; + } + int end = out.lastIndexOf('}'); + int start = out.lastIndexOf('{', end); + if (start < 0 || end < 0 || end < start) { + return null; + } + return out.substring(start, end + 1); + } + + @FunctionalInterface + interface CommandRunner { + String run(List command) throws Exception; + } + + private record BeatResult(double tempo, List beats, double duration) { + } +} diff --git a/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java b/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java index db70a1d..53af9be 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java +++ b/src/main/java/org/example/videoclips/editing/HighlightDirectorFlowService.java @@ -31,6 +31,9 @@ public class HighlightDirectorFlowService { private final HighlightAssetPreparationService assetPreparationService; private final HighlightLocalAssetWorker assetWorker; private final HighlightFfmpegRenderer renderer; + private final HighlightBeatSync beatSync; + private final HighlightSubjectTracker subjectTracker; + private final VideoClippingProperties.Editing editing; public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper, HighlightProjectStore store, @@ -38,8 +41,11 @@ public class HighlightDirectorFlowService { HighlightVisualEffectsStage visualEffectsStage, HighlightAssetPreparationService assetPreparationService, HighlightLocalAssetWorker assetWorker, - HighlightFfmpegRenderer renderer) { + HighlightFfmpegRenderer renderer, + HighlightBeatSync beatSync, + HighlightSubjectTracker subjectTracker) { this.properties = properties.getEditing().getHighlightScheduler(); + this.editing = properties.getEditing(); this.objectMapper = objectMapper; this.store = store; this.planValidator = planValidator; @@ -47,6 +53,8 @@ public class HighlightDirectorFlowService { this.assetPreparationService = assetPreparationService; this.assetWorker = assetWorker; this.renderer = renderer; + this.beatSync = beatSync; + this.subjectTracker = subjectTracker; } public HighlightFlowResult process(String projectId, long scanId) { @@ -154,6 +162,8 @@ public class HighlightDirectorFlowService { String clipId = analysis.source().clipId(); double sourceDuration = analysis.source().durationSeconds(); + boolean reframe = editing.isSubjectReframeEnabled(); + Path sourceVideo = store.sourceDirectory(projectId).resolve(project.sourceVideoFileName()); List decisions = new ArrayList<>(); double timeline = 0.0; List shots = montage.shots(); @@ -167,6 +177,15 @@ public class HighlightDirectorFlowService { String transitionIn = (i == 0) ? "fade-in" : "cut"; String transitionOut = (i == shots.size() - 1) ? "fade-out" : "cut"; String treatment = "zoom=%.3f cinematic".formatted(shot.zoom() <= 0 ? 1.2 : shot.zoom()); + if (reframe) { + // R14: track the subject over this shot's source range; append the path as a "pan=" token so + // the renderer follows it. Fails soft — an empty path leaves the token off (centred crop). + String panToken = HighlightSubjectTracker.panToken( + subjectTracker.track(sourceVideo, srcStart, srcEnd)); + if (!panToken.isEmpty()) { + treatment = treatment + " " + panToken; + } + } decisions.add(new EditDecision(clipId, srcStart, srcEnd, timeline, timeline + dur, transitionIn, transitionOut, speed, treatment, "montage")); timeline += dur; @@ -216,6 +235,23 @@ public class HighlightDirectorFlowService { markFailed(project, "Montage assets pending: " + assetResult.pendingRequests()); return HighlightFlowResult.skipped(projectId, flowId, "assets_pending"); } + + // R13 beat-synced cuts: the score now exists, so snap the cut boundaries onto its beat grid before + // rendering. Fails soft — align() returns the original decisions when no beat is close enough. + if (editing.isBeatSyncEnabled()) { + Path music = store.highlightsDirectory(projectId) + .resolve("montage").resolve("assets").resolve("music").resolve("music.wav"); + List synced = beatSync.align(decisions, beatSync.detectBeats(music), sourceDuration); + if (synced != decisions) { + decisions = synced; + editPlan = new EditPlan(projectId, style, total, decisions, audioCues, voiceover, overlays, + "mp4-h264-aac-1080p", "cinematic montage"); + store.writeJson(projectId, "highlights/montage/edit-plan.json", editPlan); + log.info("event=highlight_montage_beat_synced flow_id={} project_id={} shots={}", + flowId, projectId, decisions.size()); + } + } + markStatus(project, HighlightProjectStatus.RENDERING, null); HighlightFfmpegRenderer.HighlightRenderResult result; try { diff --git a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java index 8c410cd..9fc6040 100644 --- a/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java +++ b/src/main/java/org/example/videoclips/editing/HighlightFfmpegRenderer.java @@ -263,7 +263,14 @@ public class HighlightFfmpegRenderer { if (zoom <= 0) { zoom = 1.16 + Math.min(cutIndex, 3) * 0.12; } - if (pushIn > 0.001) { + double[][] pan = parsePan(decision.visualTreatment()); + if (pan.length > 0) { + // R14 subject-tracking reframe: crop follows the detected subject path instead of centring. + int frames = Math.max(2, (int) Math.round( + (decision.sourceEndSeconds() - decision.sourceStartSeconds()) + * properties.getOutputFrameRate())); + filter.append(subjectFollowFilter(pan, zoom, pushIn, frames, w, h)); + } else if (pushIn > 0.001) { // R5 motion-adaptive in-shot push-in: animate the zoom across the shot so held/low-motion // frames are never dead air. MUST precede setpts so slow-motion is preserved. Frame count is // estimated from the source span; min() caps the zoom so any estimate error can't overshoot. @@ -286,6 +293,13 @@ public class HighlightFfmpegRenderer { filter.append(speedRampSetpts(decision)); filter.append(cinematicVisualFilter(decision.visualTreatment(), style)); if (styled) { + if (properties.isCinematicMotionBlur()) { + // R12: light shutter-angle motion blur. Blend each frame with its predecessor so 24 fps + // motion reads smooth (film ~180-degree shutter) instead of strobing. tmix normalises by the + // weight sum, so static frames are unchanged; only moving pixels gain a short trail. Applied + // before grain so the grain itself stays crisp. + filter.append(",tmix=frames=2:weights='1 1'"); + } filter.append(",noise=alls=6:allf=t"); // subtle film grain if (w > h) { filter.append(letterboxFilter()); // 2.39:1 cinematic bars (landscape only) @@ -310,7 +324,9 @@ public class HighlightFfmpegRenderer { // cannot accumulate across a many-shot montage (keeps the concat matching the timeline). "-t", Double.toString(outputDuration), "-map", "0:v:0", "-map", "0:a?", - "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", + // pix_fmt yuv420p: filters (curves/vignette/tmix) can negotiate to yuv444p, which libx264 + // would then encode — but 444 H.264 is rejected by browsers/QuickTime/most devices. Pin 420p. + "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", properties.getAudioBitrate(), "-ar", Integer.toString(properties.getAudioSampleRate()), output.toString() )); @@ -477,7 +493,7 @@ public class HighlightFfmpegRenderer { List overlayCommand(Path input, List overlays, Path output) { return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(), "-vf", overlayFilter(overlays), - "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", + "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", "-pix_fmt", "yuv420p", "-c:a", "copy", output.toString()); } @@ -515,7 +531,7 @@ public class HighlightFfmpegRenderer { accum += durations.get(i) - xf; } command.addAll(List.of("-filter_complex", fc.substring(0, fc.length() - 1), "-map", "[vout]", "-an", - "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", output.toString())); + "-c:v", "libx264", "-preset", "veryfast", "-crf", "18", "-pix_fmt", "yuv420p", output.toString())); return List.copyOf(command); } @@ -580,7 +596,7 @@ public class HighlightFfmpegRenderer { List previewCommand(Path input, Path output) { return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(), "-vf", "scale=iw*0.5:ih*0.5", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26", - "-c:a", "copy", output.toString()); + "-pix_fmt", "yuv420p", "-c:a", "copy", output.toString()); } List audioMixCommand(Path timeline, Path music, AudioCue musicCue, List voiceovers, @@ -927,38 +943,154 @@ public class HighlightFfmpegRenderer { if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) { return ""; } - // Beat-specific filmic grade. All variants share the teal-shadow / warm-highlight ("teal-orange") - // language but shift contrast, warmth and vignette to match the story beat encoded in the plan - // style key (safeKey: style__). Unknown/rising -> balanced base grade. - // EXPOSURE-PRESERVING grades. The stylistic look (teal-orange, contrast, vignette) must not crush - // overall brightness: a fixed dark grade turned a well-exposed source (mean luma ~111/255) into ~63. - // Every variant now lifts the black point (curves 0/0.06), keeps gamma >= 1 and brightness >= 0, and - // uses a soft vignette, so the stylistic look never crushes overall exposure. String beat = style == null ? "" : style.toLowerCase(Locale.ROOT); + // R11: when a licensed film-emulation LUT is provisioned, it defines the colour transform. The + // built-in colour moves (curves/colorbalance/eq) are then SUPPRESSED to avoid double-grading; only + // the tone-neutral finishing (sharpen + vignette) is kept. See filmLutFilter(). + String lut = filmLutFilter(); + if (!lut.isEmpty()) { + String vig = beat.contains("opening") ? ",vignette=PI/10" : ",vignette=PI/9"; + String sharp = beat.contains("hero") ? ",unsharp=5:5:0.5:3:3:0.25" + : beat.contains("opening") ? ",unsharp=5:5:0.3:3:3:0.15" : ",unsharp=5:5:0.4:3:3:0.2"; + return lut + sharp + vig; + } + // Built-in FILMIC grade (no external asset). Each variant applies a filmic tone curve — a lifted toe + // (shadows never crushed) plus a rolled-off highlight shoulder (highlights compressed, not clipped) — + // which is the core of the "film look", then the teal-shadow / warm-highlight ("teal-orange") colour + // language shifted per story beat (safeKey: style__). EXPOSURE-PRESERVING: + // mids stay at ~0.5 and gamma>=1, so the stylistic look never darkens a well-exposed source. Contrast + // now comes mostly from the S-curve, so eq contrast is kept light to avoid over-crushing. if (beat.contains("opening")) { - // Opening hook: calmer, cooler, softer contrast to invite the viewer in. - return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'" + // Opening hook: calmer, cooler, gentler S-curve to invite the viewer in. + return ",curves=all='0/0.05 0.5/0.5 0.75/0.78 1/0.95'" + ",colorbalance=rs=-0.05:bs=0.08:rm=0.01:bm=-0.01:rh=0.03:bh=-0.03" - + ",eq=contrast=1.05:saturation=1.08:gamma=1.04:brightness=0.015" + + ",eq=contrast=1.03:saturation=1.08:gamma=1.03:brightness=0.012" + ",unsharp=5:5:0.3:3:3:0.15" + ",vignette=PI/10"; } if (beat.contains("hero")) { - // Hero payoff: richest grade, warmer highlights, but shadows lifted so the subject stays visible. - return ",curves=preset=medium_contrast,curves=all='0/0.06 1/1'" + // Hero payoff: richest grade, deeper filmic contrast, warmer highlights, shadows still lifted. + return ",curves=all='0/0.055 0.25/0.24 0.75/0.82 1/0.97'" + ",colorbalance=rs=-0.03:bs=0.05:rm=0.01:bm=-0.01:rh=0.09:bh=-0.07" - + ",eq=contrast=1.08:saturation=1.16:gamma=1.06:brightness=0.02" + + ",eq=contrast=1.04:saturation=1.16:gamma=1.05:brightness=0.016" + ",unsharp=5:5:0.5:3:3:0.25" + ",vignette=PI/9"; } - // Rising energy / default: balanced base grade. - return ",curves=preset=linear_contrast,curves=all='0/0.05 1/1'" + // Rising energy / default: balanced filmic base grade. + return ",curves=all='0/0.045 0.25/0.23 0.75/0.80 1/0.96'" + ",colorbalance=rs=-0.04:bs=0.06:rm=0.01:bm=-0.01:rh=0.06:bh=-0.05" - + ",eq=contrast=1.07:saturation=1.12:gamma=1.05:brightness=0.018" + + ",eq=contrast=1.035:saturation=1.12:gamma=1.04:brightness=0.014" + ",unsharp=5:5:0.4:3:3:0.2" + ",vignette=PI/9"; } + /** + * R11 film-emulation LUT hook. Returns a {@code ,lut3d=''} filter fragment when a licensed LUT is + * provisioned (config {@code film-lut-path} points to a readable file that has an adjacent, non-blank + * {@code .license.txt} sidecar — the same provenance discipline as generated assets). Returns "" when + * unset, missing, or unlicensed, so the built-in grade is used and no unlicensed asset can ever be applied. + */ + String filmLutFilter() { + String configured = properties.getFilmLutPath(); + if (configured == null || configured.isBlank()) { + return ""; + } + Path lut = Path.of(configured.trim()); + if (!Files.isRegularFile(lut) || !Files.isReadable(lut)) { + return ""; + } + Path license = lut.resolveSibling(lut.getFileName().toString() + ".license.txt"); + try { + if (!Files.isRegularFile(license) || Files.readString(license).isBlank()) { + return ""; // fail closed: a LUT is executable media data; require a licence sidecar + } + } catch (IOException e) { + return ""; + } + // ffmpeg lut3d: escape backslashes/colons/quotes for the filtergraph, wrap in single quotes. + String escaped = lut.toAbsolutePath().toString() + .replace("\\", "\\\\").replace(":", "\\:").replace("'", "\\'"); + return ",lut3d='" + escaped + "'"; + } + + // R14: parse a "pan=cx0:cy0;cx1:cy1;..." token (normalised subject centres) from the treatment. + double[][] parsePan(String visualTreatment) { + if (visualTreatment == null) { + return new double[0][]; + } + java.util.regex.Matcher m = PAN_TOKEN.matcher(visualTreatment); + if (!m.find()) { + return new double[0][]; + } + String[] anchors = m.group(1).split(";"); + List centers = new ArrayList<>(); + for (String anchor : anchors) { + String[] xy = anchor.split(":"); + if (xy.length != 2) { + continue; + } + try { + double cx = Double.parseDouble(xy[0]); + double cy = Double.parseDouble(xy[1]); + if (Double.isFinite(cx) && Double.isFinite(cy)) { + centers.add(new double[]{cx, cy}); + } + } catch (NumberFormatException ignored) { + // skip a malformed anchor rather than fail the render + } + } + return centers.toArray(new double[0][]); + } + + /** + * R14 subject-following crop. Builds a {@code scale,zoompan,format} chain whose crop window tracks the + * (interpolated, clamped) subject centre across the shot. Zoom is animated when {@code pushIn > 0}, else + * held constant. Falls back to the centred formula when only one anchor is present. + */ + String subjectFollowFilter(double[][] pan, double zoom, double pushIn, int frames, int w, int h) { + int lastFrame = Math.max(1, frames - 1); + String z = pushIn > 0.001 + ? "min(%s+%s*on/%d\\,%s)".formatted(fixed(zoom), fixed(pushIn), lastFrame, fixed(zoom + pushIn)) + : fixed(zoom); + double[] cx = new double[pan.length]; + double[] cy = new double[pan.length]; + for (int i = 0; i < pan.length; i++) { + cx[i] = clamp01(pan[i][0]); + cy[i] = clamp01(pan[i][1]); + } + String cxExpr = lerpExpr(cx, lastFrame); + String cyExpr = lerpExpr(cy, lastFrame); + // Place the crop window (size iw/zoom) so its centre sits on the subject, clamped inside the frame. + String x = "max(0\\,min(iw-iw/zoom\\,(%s)*iw-(iw/zoom)/2))".formatted(cxExpr); + String y = "max(0\\,min(ih-ih/zoom\\,(%s)*ih-(ih/zoom)/2))".formatted(cyExpr); + return "scale=%d:%d,zoompan=z='%s':d=1:x='%s':y='%s':s=%dx%d,format=yuv420p" + .formatted(w, h, z, x, y, w, h); + } + + /** Piecewise-linear ffmpeg expression in {@code on} interpolating anchor values across [0, lastFrame]. */ + private static String lerpExpr(double[] vals, int lastFrame) { + if (vals.length == 1) { + return fixed(vals[0]); + } + int k = vals.length; + double[] pos = new double[k]; + for (int i = 0; i < k; i++) { + pos[i] = (double) i / (k - 1) * lastFrame; + } + String expr = fixed(vals[k - 1]); + for (int i = k - 2; i >= 0; i--) { + double denom = Math.max(1e-6, pos[i + 1] - pos[i]); + String seg = "%s+(%s)*(on-%s)/%s".formatted( + fixed(vals[i]), fixed(vals[i + 1] - vals[i]), fixed(pos[i]), fixed(denom)); + expr = "if(lt(on\\,%s)\\,%s\\,%s)".formatted(fixed(pos[i + 1]), seg, expr); + } + return expr; + } + + private static double clamp01(double v) { + return v < 0 ? 0 : (v > 1 ? 1 : v); + } + private List voiceoverInputs(EditPlan plan, Path audioDirectory) { List inputs = new ArrayList<>(); for (int index = 0; index < plan.voiceover().size(); index++) { @@ -993,6 +1125,8 @@ public class HighlightFfmpegRenderer { private static final java.util.regex.Pattern ZOOM_TOKEN = java.util.regex.Pattern.compile("zoom=([0-9]+(?:\\.[0-9]+)?)"); + private static final java.util.regex.Pattern PAN_TOKEN = + java.util.regex.Pattern.compile("pan=([0-9.:;]+)"); private static final java.util.regex.Pattern YAVG_TOKEN = java.util.regex.Pattern.compile("YAVG=([0-9]+(?:\\.[0-9]+)?)"); private static final java.util.regex.Pattern YDIF_TOKEN = diff --git a/src/main/java/org/example/videoclips/editing/HighlightSubjectTracker.java b/src/main/java/org/example/videoclips/editing/HighlightSubjectTracker.java new file mode 100644 index 0000000..76a9f6c --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/HighlightSubjectTracker.java @@ -0,0 +1,145 @@ +package org.example.videoclips.editing; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.example.videoclips.config.VideoClippingProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; + +/** + * R14 subject-tracking reframe. Runs a local YOLO tracker over a shot's source range and returns a smoothed, + * normalised subject-centre path so the renderer can follow the subject instead of using a static centre crop + * (the "AI reframe" consumer editors like DJI/Insta360 do). Fails soft everywhere — a missing interpreter, no + * detections, or a parse error yield an empty path and the renderer keeps its centred crop. + * + *

YOLOv8 is AGPL-3.0, so this path is non-commercial — matching the repo's existing CV stance.

+ */ +@Component +public class HighlightSubjectTracker { + + private static final Logger log = LoggerFactory.getLogger(HighlightSubjectTracker.class); + + private final VideoClippingProperties.Editing editing; + private final ObjectMapper objectMapper; + private final HighlightBeatSync.CommandRunner commandRunner; + + @org.springframework.beans.factory.annotation.Autowired + public HighlightSubjectTracker(VideoClippingProperties properties, ObjectMapper objectMapper) { + this(properties, objectMapper, command -> { + Process process = new ProcessBuilder(command).redirectErrorStream(false).start(); + String out = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8); + process.waitFor(); + return out; + }); + } + + HighlightSubjectTracker(VideoClippingProperties properties, ObjectMapper objectMapper, + HighlightBeatSync.CommandRunner commandRunner) { + this.editing = properties.getEditing(); + this.objectMapper = objectMapper; + this.commandRunner = commandRunner; + } + + /** + * Tracks the dominant subject across {@code [startSeconds, endSeconds)} and returns a smoothed list of + * normalised centres {@code [cx, cy]} (each 0..1), ordered by time. Empty on any failure. + */ + List track(Path source, double startSeconds, double endSeconds) { + if (source == null || endSeconds <= startSeconds) { + return List.of(); + } + int samples = Math.max(1, editing.getSubjectTrackSamples()); + List command = List.of( + editing.getSubjectTrackPython(), editing.getSubjectTrackScript(), + source.toAbsolutePath().toString(), + String.format(Locale.ROOT, "%.3f", Math.max(0.0, startSeconds)), + String.format(Locale.ROOT, "%.3f", endSeconds), + Integer.toString(samples)); + try { + String out = commandRunner.run(command); + String json = lastJsonObject(out); + if (json == null) { + return List.of(); + } + JsonNode path = objectMapper.readTree(json).path("path"); + if (!path.isArray() || path.isEmpty()) { + return List.of(); + } + List centers = new ArrayList<>(); + for (JsonNode point : path) { + double cx = point.path("cx").asDouble(Double.NaN); + double cy = point.path("cy").asDouble(Double.NaN); + if (Double.isFinite(cx) && Double.isFinite(cy)) { + centers.add(new double[]{clamp01(cx), clamp01(cy)}); + } + } + return smooth(centers); + } catch (Exception ex) { // fail soft: keep the centred crop + log.warn("event=subject_track_failed source={} error={}", source, ex.toString()); + return List.of(); + } + } + + /** + * Encodes a centre path as a compact treatment token {@code pan=cx0:cy0;cx1:cy1;...} the renderer parses. + * Returns "" for an empty path so callers can append it unconditionally. + */ + static String panToken(List centers) { + if (centers == null || centers.isEmpty()) { + return ""; + } + StringBuilder token = new StringBuilder("pan="); + for (int i = 0; i < centers.size(); i++) { + if (i > 0) { + token.append(';'); + } + token.append(String.format(Locale.ROOT, "%.4f:%.4f", centers.get(i)[0], centers.get(i)[1])); + } + return token.toString(); + } + + /** 3-tap moving average to take the jitter out of frame-by-frame detections. */ + private static List smooth(List centers) { + int n = centers.size(); + if (n <= 2) { + return List.copyOf(centers); + } + List out = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + int lo = Math.max(0, i - 1); + int hi = Math.min(n - 1, i + 1); + double sx = 0; + double sy = 0; + for (int j = lo; j <= hi; j++) { + sx += centers.get(j)[0]; + sy += centers.get(j)[1]; + } + int count = hi - lo + 1; + out.add(new double[]{sx / count, sy / count}); + } + return out; + } + + private static double clamp01(double v) { + return v < 0 ? 0 : (v > 1 ? 1 : v); + } + + private static String lastJsonObject(String out) { + if (out == null) { + return null; + } + int end = out.lastIndexOf('}'); + int start = out.indexOf('{'); + if (start < 0 || end < 0 || end < start) { + return null; + } + return out.substring(start, end + 1); + } +} diff --git a/src/main/resources/application-localpoc.yml b/src/main/resources/application-localpoc.yml index b79d993..79f64ed 100644 --- a/src/main/resources/application-localpoc.yml +++ b/src/main/resources/application-localpoc.yml @@ -22,6 +22,12 @@ video-clipping: # Soft cross-dissolves between montage beats (0 = hard cuts). Softens the abrupt cut into the slow-mo payoff. crossfade-seconds: 0.25 + # R13 cut-to-the-music: snap cut boundaries onto the generated score's beat grid (needs the asset venv). + beat-sync-enabled: true + + # R14 subject-tracking reframe: follow the detected subject (YOLO/CV venv) instead of a static centre crop. + subject-reframe-enabled: true + assets: # Empty/absent asset folders -> the pipeline generates assets with local models instead of copying. music-folder: ./input/localpoc/assets/music diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index 758a0da..a31da74 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -30,7 +30,7 @@ video-clipping: target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:600} output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920} output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080} - output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30} + output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:24} audio-sample-rate: ${VIDEO_EDITING_AUDIO_SAMPLE_RATE:48000} video-bitrate: ${VIDEO_EDITING_VIDEO_BITRATE:12000k} audio-bitrate: ${VIDEO_EDITING_AUDIO_BITRATE:192k} diff --git a/src/test/java/org/example/videoclips/config/VideoClippingPropertiesTest.java b/src/test/java/org/example/videoclips/config/VideoClippingPropertiesTest.java index 2fafdcb..dae7066 100644 --- a/src/test/java/org/example/videoclips/config/VideoClippingPropertiesTest.java +++ b/src/test/java/org/example/videoclips/config/VideoClippingPropertiesTest.java @@ -34,7 +34,7 @@ class VideoClippingPropertiesTest { assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(600); assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920); assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080); - assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30); + assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(24); assertThat(properties.getEditing().getAudioSampleRate()).isEqualTo(48000); assertThat(properties.getEditing().getVideoBitrate()).isEqualTo("12000k"); assertThat(properties.getEditing().getAudioBitrate()).isEqualTo("192k"); diff --git a/src/test/java/org/example/videoclips/editing/HighlightBeatSyncTest.java b/src/test/java/org/example/videoclips/editing/HighlightBeatSyncTest.java new file mode 100644 index 0000000..b65ca7a --- /dev/null +++ b/src/test/java/org/example/videoclips/editing/HighlightBeatSyncTest.java @@ -0,0 +1,111 @@ +package org.example.videoclips.editing; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.example.videoclips.config.VideoClippingProperties; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.within; + +class HighlightBeatSyncTest { + + @TempDir + Path tempDir; + + private final ObjectMapper mapper = new ObjectMapper(); + + private EditDecision shot(double tlStart, double tlEnd) { + return new EditDecision("clip", tlStart, tlEnd, tlStart, tlEnd, "cut", "cut", 1.0, "grade", "montage"); + } + + @Test + void snapsInternalBoundariesToNearestBeatsAndPreservesTotal() { + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper); + List decisions = List.of( + shot(0.0, 1.7), shot(1.7, 5.6), shot(5.6, 6.4), shot(6.4, 9.26), shot(9.26, 10.7)); + List beats = List.of(1.856, 3.0, 5.7067, 6.3573, 9.2587); + + List aligned = sync.align(decisions, beats, 12.0); + + // internal boundaries moved onto the nearest beat (all within the default 0.18s tolerance) + assertThat(aligned.get(1).timelineStartSeconds()).isCloseTo(1.856, within(1e-3)); + assertThat(aligned.get(2).timelineStartSeconds()).isCloseTo(5.7067, within(1e-3)); + assertThat(aligned.get(3).timelineStartSeconds()).isCloseTo(6.3573, within(1e-3)); + assertThat(aligned.get(4).timelineStartSeconds()).isCloseTo(9.2587, within(1e-3)); + // endpoints are anchors: first start and total length are unchanged + assertThat(aligned.get(0).timelineStartSeconds()).isEqualTo(0.0); + assertThat(aligned.get(4).timelineEndSeconds()).isCloseTo(10.7, within(1e-3)); + // adjacent shots stay contiguous (end == next start) + for (int i = 0; i < aligned.size() - 1; i++) { + assertThat(aligned.get(i).timelineEndSeconds()) + .isCloseTo(aligned.get(i + 1).timelineStartSeconds(), within(1e-6)); + } + } + + @Test + void keepsOriginalCutWhenNoBeatIsWithinTolerance() { + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper); + List decisions = List.of(shot(0.0, 3.0), shot(3.0, 6.0)); + List farBeats = List.of(0.9, 5.4); // nearest to 3.0 is >0.18s away + + List aligned = sync.align(decisions, farBeats, 6.0); + + assertThat(aligned).isSameAs(decisions); // nothing moved -> original list returned + } + + @Test + void doesNotSnapWhenItWouldMakeAShotTooShort() { + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper); + // boundary at 1.0; a beat at 0.9 is within tolerance but would leave shot 0 only 0.9s (< 0.5? no). + // Use a beat that would collapse shot 1 below the 0.5s minimum instead. + List decisions = List.of(shot(0.0, 1.0), shot(1.0, 1.4)); + List beats = List.of(1.05); // shot 1 would become 0.35s (< 0.5) -> reject + + List aligned = sync.align(decisions, beats, 6.0); + + assertThat(aligned).isSameAs(decisions); + } + + @Test + void reDerivesSourceWindowFromNewTimelineDurationForSlowMoShots() { + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper); + // slow-mo shot (speed 0.5): 2.0s timeline consumes 1.0s of source. + EditDecision slow = new EditDecision("clip", 4.0, 5.0, 2.0, 4.0, "cut", "cut", 0.5, "grade", "montage"); + List decisions = List.of(shot(0.0, 2.0), slow); + List beats = List.of(1.9); // move the 2.0 boundary to 1.9 + + List aligned = sync.align(decisions, beats, 20.0); + + EditDecision alignedSlow = aligned.get(1); + double newDur = alignedSlow.timelineEndSeconds() - alignedSlow.timelineStartSeconds(); // 4.0 - 1.9 = 2.1 + double srcSpan = alignedSlow.sourceEndSeconds() - alignedSlow.sourceStartSeconds(); + assertThat(srcSpan).isCloseTo(newDur * 0.5, within(1e-3)); // source span tracks timeline * speed + } + + @Test + void detectBeatsParsesToolJsonEvenWithLeadingLogLines() throws Exception { + Path audio = tempDir.resolve("music.wav"); + Files.writeString(audio, "not really audio"); + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper, + command -> "loading model...\n{\"tempo\": 120.0, \"beats\": [0.5, 1.0, 1.5], \"duration\": 2.0}\n"); + + assertThat(sync.detectBeats(audio)).containsExactly(0.5, 1.0, 1.5); + } + + @Test + void detectBeatsFailsSoftToEmptyOnError() throws Exception { + Path audio = tempDir.resolve("music.wav"); + Files.writeString(audio, "x"); + HighlightBeatSync sync = new HighlightBeatSync(new VideoClippingProperties(), mapper, + command -> { + throw new IllegalStateException("interpreter missing"); + }); + + assertThat(sync.detectBeats(audio)).isEmpty(); + } +} diff --git a/src/test/java/org/example/videoclips/editing/HighlightDirectorFlowServiceTest.java b/src/test/java/org/example/videoclips/editing/HighlightDirectorFlowServiceTest.java index 1576f5b..c2d8beb 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightDirectorFlowServiceTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightDirectorFlowServiceTest.java @@ -59,7 +59,8 @@ class HighlightDirectorFlowServiceTest { when(renderer.render(anyString(), any(), any())).thenReturn(renderResult); HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store, new HighlightDirectorPlanValidator(properties, store), - visualEffectsStage, assetPreparationService, assetWorker, renderer); + visualEffectsStage, assetPreparationService, assetWorker, renderer, + new HighlightBeatSync(properties, mapper), new HighlightSubjectTracker(properties, mapper)); HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1); @@ -104,7 +105,8 @@ class HighlightDirectorFlowServiceTest { HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class); HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store, new HighlightDirectorPlanValidator(properties, store), - visualEffectsStage, assetPreparationService, assetWorker, renderer); + visualEffectsStage, assetPreparationService, assetWorker, renderer, + new HighlightBeatSync(properties, mapper), new HighlightSubjectTracker(properties, mapper)); HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1); @@ -145,7 +147,8 @@ class HighlightDirectorFlowServiceTest { List.of(new RenderQaCheck("audio_clipping", false, "ERROR", "peak=0.0")), Instant.now()))); HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store, new HighlightDirectorPlanValidator(properties, store), mock(HighlightVisualEffectsStage.class), - assetPreparationService, assetWorker, renderer); + assetPreparationService, assetWorker, renderer, new HighlightBeatSync(properties, mapper), + new HighlightSubjectTracker(properties, mapper)); assertThatThrownBy(() -> service.process(project.id(), 1)) .isInstanceOf(IllegalStateException.class) diff --git a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java index ccbaa0b..5f48576 100644 --- a/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java +++ b/src/test/java/org/example/videoclips/editing/HighlightFfmpegRendererTest.java @@ -89,11 +89,14 @@ class HighlightFfmpegRendererTest { String hero = String.join(" ", renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); - assertThat(opening).contains("curves=preset=linear_contrast").contains("vignette=PI/10"); - assertThat(rising).contains("curves=preset=linear_contrast").contains("eq=contrast=1.07"); - assertThat(hero).contains("curves=preset=medium_contrast").contains("eq=contrast=1.08"); - // Exposure-preserving: every beat lifts the black point and never pushes gamma < 1 or brightness < 0. - assertThat(hero).contains("curves=all='0/0.06 1/1'").contains("gamma=1.06").contains("brightness=0.02"); + // R11 filmic tone curve: every beat uses a filmic S-curve (lifted toe + rolled-off highlight + // shoulder) rather than a linear ramp, and the grade differs per story beat. + assertThat(opening).contains("curves=all='0/0.05 0.5/0.5 0.75/0.78 1/0.95'").contains("vignette=PI/10"); + assertThat(rising).contains("curves=all='0/0.045 0.25/0.23 0.75/0.80 1/0.96'").contains("saturation=1.12"); + assertThat(hero).contains("curves=all='0/0.055 0.25/0.24 0.75/0.82 1/0.97'").contains("saturation=1.16"); + // Exposure-preserving: every beat lifts the black point (toe > 0), rolls off highlights (shoulder < 1), + // and never pushes gamma < 1 or brightness < 0. + assertThat(hero).contains("gamma=1.05").contains("brightness=0.016"); assertThat(opening).isNotEqualTo(hero); } @@ -321,8 +324,110 @@ class HighlightFfmpegRendererTest { List.of(), null, null, null, null, null); } + @Test + void rendersAt24fpsCinematicCadence() { + HighlightFfmpegRenderer renderer = renderer(); + EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes"); + + List command = + renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4")); + + assertThat(command).containsSequence("-r", "24"); + // 420p pinned so filter format negotiation can't leave the output as browser-incompatible 444 H.264. + assertThat(command).containsSequence("-pix_fmt", "yuv420p"); + } + + @Test + void pinsYuv420pOnEveryVideoEncodePass() { + HighlightFfmpegRenderer renderer = renderer(); + var segs = List.of(Path.of("s0.mp4"), Path.of("s1.mp4")); + assertThat(renderer.xfadeVideoCommand(segs, List.of(2.0, 2.0), 0.25, Path.of("v.mp4"))) + .containsSequence("-pix_fmt", "yuv420p"); + assertThat(renderer.overlayCommand(Path.of("in.mp4"), List.of(), Path.of("o.mp4"))) + .containsSequence("-pix_fmt", "yuv420p"); + assertThat(renderer.previewCommand(Path.of("in.mp4"), Path.of("p.mp4"))) + .containsSequence("-pix_fmt", "yuv420p"); + } + + @Test + void appliesMotionBlurWhenEnabledAndOmitsWhenDisabled() { + VideoClippingProperties on = new VideoClippingProperties(); + on.getEditing().setCinematicMotionBlur(true); + VideoClippingProperties off = new VideoClippingProperties(); + off.getEditing().setCinematicMotionBlur(false); + EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes"); + + String withBlur = String.join(" ", renderer(on) + .segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); + String noBlur = String.join(" ", renderer(off) + .segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); + + assertThat(withBlur).contains("tmix=frames=2:weights='1 1'"); + assertThat(noBlur).doesNotContain("tmix"); + } + + @Test + void filmLutFilterIsEmptyByDefaultAndAppliedOnlyWhenLicensed() throws Exception { + assertThat(renderer().filmLutFilter()).isEmpty(); + + Path lut = tempDir.resolve("kodak2383.cube"); + Files.writeString(lut, "LUT_3D_SIZE 2\n"); + VideoClippingProperties props = new VideoClippingProperties(); + props.getEditing().setFilmLutPath(lut.toString()); + + // No licence sidecar yet -> fail closed, built-in grade is used. + assertThat(renderer(props).filmLutFilter()).isEmpty(); + + Files.writeString(lut.resolveSibling("kodak2383.cube.license.txt"), "Licensed to Acme, commercial use.\n"); + assertThat(renderer(props).filmLutFilter()) + .startsWith(",lut3d='").contains("kodak2383.cube"); + + // When a LUT is active the built-in colour moves are suppressed (no double-grading). + EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, "grade", "notes"); + String cmd = String.join(" ", renderer(props) + .segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); + assertThat(cmd).contains("lut3d=").doesNotContain("colorbalance").doesNotContain("curves=all"); + } + + @Test + void parsesPanTokenIntoAnchors() { + HighlightFfmpegRenderer renderer = renderer(); + double[][] pan = renderer.parsePan("zoom=1.050 cinematic pan=0.5400:0.6800;0.4444:0.6884"); + assertThat(pan.length).isEqualTo(2); + assertThat(pan[0][0]).isCloseTo(0.54, within(1e-6)); + assertThat(pan[0][1]).isCloseTo(0.68, within(1e-6)); + assertThat(pan[1][0]).isCloseTo(0.4444, within(1e-6)); + } + + @Test + void subjectReframeFollowsPathInsteadOfCentredCrop() { + HighlightFfmpegRenderer renderer = renderer(); + // treatment carries a pan path -> renderer must build a following zoompan, not the centred crop. + EditDecision decision = new EditDecision("clip", 0, 9, 0, 9, "cut", "cut", 1.0, + "zoom=1.050 cinematic pan=0.5400:0.6800;0.4444:0.6884;0.6840:0.6723", "notes"); + + String cmd = String.join(" ", + renderer.segmentCommand("s.mp4", decision, "style_generic_vlog_hero_payoff", 0, 2, Path.of("h.mp4"))); + + assertThat(cmd).contains("zoompan").contains("min(iw-iw/zoom"); // clamped follow crop + assertThat(cmd).doesNotContain("iw/2-(iw/zoom/2)"); // not the centred formula + } + + @Test + void subjectFollowFilterHoldsZoomConstantWithoutPushIn() { + HighlightFfmpegRenderer renderer = renderer(); + String constant = renderer.subjectFollowFilter(new double[][]{{0.5, 0.5}}, 1.2, 0.0, 48, 1080, 1920); + String pushed = renderer.subjectFollowFilter(new double[][]{{0.5, 0.5}}, 1.2, 0.1, 48, 1080, 1920); + assertThat(constant).contains("zoompan=z='1.200'"); + assertThat(pushed).contains("zoompan=z='min(1.200"); // animated zoom when pushing in + } + private HighlightFfmpegRenderer renderer() { - return new HighlightFfmpegRenderer(new VideoClippingProperties(), mock(HighlightProjectStore.class), + return renderer(new VideoClippingProperties()); + } + + private HighlightFfmpegRenderer renderer(VideoClippingProperties properties) { + return new HighlightFfmpegRenderer(properties, mock(HighlightProjectStore.class), mock(EditAssetProvider.class), mock(EditObservability.class), command -> null); } diff --git a/src/test/java/org/example/videoclips/editing/HighlightSubjectTrackerTest.java b/src/test/java/org/example/videoclips/editing/HighlightSubjectTrackerTest.java new file mode 100644 index 0000000..1299ada --- /dev/null +++ b/src/test/java/org/example/videoclips/editing/HighlightSubjectTrackerTest.java @@ -0,0 +1,54 @@ +package org.example.videoclips.editing; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.example.videoclips.config.VideoClippingProperties; +import org.junit.jupiter.api.Test; + +import java.nio.file.Path; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +class HighlightSubjectTrackerTest { + + private final ObjectMapper mapper = new ObjectMapper(); + + @Test + void parsesAndSmoothsTrackPath() { + HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper, + command -> "detecting...\n{\"path\": [{\"t\":0.1,\"cx\":0.2,\"cy\":0.5,\"area\":0.1}," + + "{\"t\":0.2,\"cx\":0.8,\"cy\":0.5,\"area\":0.1}," + + "{\"t\":0.3,\"cx\":0.2,\"cy\":0.5,\"area\":0.1}]}\n"); + + List path = tracker.track(Path.of("s.mp4"), 0.0, 1.0); + + assertThat(path).hasSize(3); + // 3-tap smoothing pulls the spiky middle sample toward its neighbours (0.2,0.8,0.2 -> middle = 0.4). + assertThat(path.get(1)[0]).isCloseTo(0.4, org.assertj.core.api.Assertions.within(1e-9)); + } + + @Test + void trackFailsSoftToEmptyOnError() { + HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper, + command -> { + throw new IllegalStateException("no interpreter"); + }); + + assertThat(tracker.track(Path.of("s.mp4"), 0.0, 1.0)).isEmpty(); + } + + @Test + void trackReturnsEmptyForEmptyPath() { + HighlightSubjectTracker tracker = new HighlightSubjectTracker(new VideoClippingProperties(), mapper, + command -> "{\"path\": []}"); + + assertThat(tracker.track(Path.of("s.mp4"), 0.0, 1.0)).isEmpty(); + } + + @Test + void panTokenEncodesAnchorsAndIsEmptyForNoPath() { + assertThat(HighlightSubjectTracker.panToken(List.of())).isEmpty(); + String token = HighlightSubjectTracker.panToken(List.of(new double[]{0.5, 0.6}, new double[]{0.4, 0.7})); + assertThat(token).isEqualTo("pan=0.5000:0.6000;0.4000:0.7000"); + } +} diff --git a/tools/beat_detect.py b/tools/beat_detect.py new file mode 100644 index 0000000..672beb8 --- /dev/null +++ b/tools/beat_detect.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Offline beat detection for a rendered/generated audio track. + +Usage: + python beat_detect.py + +Prints a single JSON object to stdout: + {"tempo": , "beats": [, ...], "duration": } + +Local/offline only: librosa + numpy (already pinned for the asset venv). No network, no model +downloads. On any failure it prints {"tempo": 0, "beats": [], "duration": 0} and exits 0 so the +caller can fail soft to un-synced cuts rather than failing the render. +""" +import json +import sys + + +def main() -> int: + if len(sys.argv) < 2: + print(json.dumps({"tempo": 0, "beats": [], "duration": 0})) + return 0 + audio_path = sys.argv[1] + try: + import librosa + + # sr=None keeps the file's native rate; mono is enough for beat tracking. + y, sr = librosa.load(audio_path, sr=None, mono=True) + duration = float(librosa.get_duration(y=y, sr=sr)) + tempo, beat_frames = librosa.beat.beat_track(y=y, sr=sr, units="frames") + beat_times = librosa.frames_to_time(beat_frames, sr=sr) + beats = [round(float(t), 4) for t in beat_times.tolist()] + tempo_val = float(tempo[0]) if hasattr(tempo, "__len__") else float(tempo) + print(json.dumps({"tempo": round(tempo_val, 2), "beats": beats, + "duration": round(duration, 4)})) + return 0 + except Exception as exc: # noqa: BLE001 - fail soft, never break the render + sys.stderr.write("beat_detect failed: %s\n" % exc) + print(json.dumps({"tempo": 0, "beats": [], "duration": 0})) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/subject_track.py b/tools/subject_track.py new file mode 100644 index 0000000..eebb813 --- /dev/null +++ b/tools/subject_track.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Offline subject tracking for one source time range (for cinematic auto-reframe). + +Usage: + python subject_track.py + +Samples frames evenly across [start, end], runs a local YOLO detector on each, and reports +the NORMALISED centre (cx, cy in 0..1) of the dominant subject (largest person; else largest box of any +class). Prints one JSON object to stdout: + + {"path": [{"t": , "cx": <0..1>, "cy": <0..1>, "area": <0..1>}, ...]} + +Local/offline only: ultralytics + OpenCV, weights from ./yolov8n.pt (no download). YOLOv8 is AGPL-3.0, so +this reframe path is non-commercial — matching the repo's existing CV stance. On any failure it prints +{"path": []} and exits 0 so the caller can fail soft to a centred crop. +""" +import json +import os +import sys + + +def main() -> int: + if len(sys.argv) < 5: + print(json.dumps({"path": []})) + return 0 + source, start_s, end_s, samples_s = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] + try: + start = max(0.0, float(start_s)) + end = float(end_s) + samples = max(1, int(samples_s)) + if end <= start: + print(json.dumps({"path": []})) + return 0 + + os.environ.setdefault("YOLO_OFFLINE", "1") + import cv2 + from ultralytics import YOLO + + weights = os.environ.get("SUBJECT_TRACK_WEIGHTS", "./yolov8n.pt") + model = YOLO(weights) + cap = cv2.VideoCapture(source) + if not cap.isOpened(): + print(json.dumps({"path": []})) + return 0 + + path = [] + step = (end - start) / samples + for i in range(samples): + t = start + step * (i + 0.5) + cap.set(cv2.CAP_PROP_POS_MSEC, t * 1000.0) + ok, frame = cap.read() + if not ok or frame is None: + continue + h, w = frame.shape[:2] + result = model.predict(frame, verbose=False, device="cpu")[0] + best = _dominant_box(result) + if best is None: + continue + x1, y1, x2, y2 = best + cx = ((x1 + x2) / 2.0) / w + cy = ((y1 + y2) / 2.0) / h + area = ((x2 - x1) * (y2 - y1)) / float(w * h) + path.append({"t": round(t, 3), "cx": round(_clamp01(cx), 4), + "cy": round(_clamp01(cy), 4), "area": round(area, 4)}) + cap.release() + print(json.dumps({"path": path})) + return 0 + except Exception as exc: # noqa: BLE001 - fail soft, never break the render + sys.stderr.write("subject_track failed: %s\n" % exc) + print(json.dumps({"path": []})) + return 0 + + +def _dominant_box(result): + """Largest 'person' box; else the largest box of any class. Returns (x1,y1,x2,y2) or None.""" + boxes = getattr(result, "boxes", None) + if boxes is None or len(boxes) == 0: + return None + names = result.names + best = None + best_area = -1.0 + best_person = None + best_person_area = -1.0 + for b in boxes: + xyxy = b.xyxy[0].tolist() + area = (xyxy[2] - xyxy[0]) * (xyxy[3] - xyxy[1]) + cls_name = names.get(int(b.cls[0]), "") if isinstance(names, dict) else "" + if cls_name == "person" and area > best_person_area: + best_person_area = area + best_person = xyxy + if area > best_area: + best_area = area + best = xyxy + return best_person if best_person is not None else best + + +def _clamp01(v: float) -> float: + return 0.0 if v < 0 else (1.0 if v > 1 else v) + + +if __name__ == "__main__": + raise SystemExit(main())