From 5006dfd50b38c7d8db7e39d92eb094817a5dd84b Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Sat, 11 Jul 2026 00:50:12 +0200 Subject: [PATCH] Add category-aware cinematic highlight planning --- ...uction-cinematic-highlight-editing-plan.md | 636 ++++++++++++++++++ .../editing/AiDirectorPromptGenerator.java | 8 +- .../editing/CinematicHighlightAnalysis.java | 14 + .../editing/CinematicHighlightAnalyzer.java | 157 +++++ .../videoclips/editing/ContentCategory.java | 8 + .../editing/EditProjectAnalyzer.java | 22 +- .../editing/HighlightCandidate.java | 14 + .../editing/LocalDirectorScheduler.java | 2 +- .../editing/StoryboardPromptGenerator.java | 89 ++- .../application-cinematic-editing-local.yml | 2 +- .../AiDirectorPromptGeneratorTest.java | 5 +- .../CinematicEditingIntegrationTest.java | 2 +- .../CinematicHighlightAnalyzerTest.java | 74 ++ .../editing/EditProjectAnalyzerTest.java | 50 ++ .../StoryboardPromptGeneratorTest.java | 27 + 15 files changed, 1096 insertions(+), 14 deletions(-) create mode 100644 docs/production-cinematic-highlight-editing-plan.md create mode 100644 src/main/java/org/example/videoclips/editing/CinematicHighlightAnalysis.java create mode 100644 src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java create mode 100644 src/main/java/org/example/videoclips/editing/ContentCategory.java create mode 100644 src/main/java/org/example/videoclips/editing/HighlightCandidate.java create mode 100644 src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java diff --git a/docs/production-cinematic-highlight-editing-plan.md b/docs/production-cinematic-highlight-editing-plan.md new file mode 100644 index 0000000..eaa32de --- /dev/null +++ b/docs/production-cinematic-highlight-editing-plan.md @@ -0,0 +1,636 @@ +# Production Cinematic Highlight Editing Plan + +## Goal + +Build a local-first workflow where a user drops one source video into a source folder and the service produces multiple production-ready cinematic highlight edits. Each highlight must be adapted to the detected content type, such as family vlog, food vlog, car vlog, or a generic fallback category. + +The target output is not just a technically valid MP4. The output must feel intentionally directed: strong shot selection, a clear mini-story, pacing matched to music, fitting voiceover, sound effects, cinematic visual treatment, tasteful text overlays, and a final audio mix that sounds finished. + +## Research Findings + +- Recent highlight-detection research points toward multimodal narrative understanding, not simple duration slicing. A production system should combine shot segmentation, visual analysis, speech/transcript analysis, audio events, pacing, and narrative role detection. +- Audio matters for highlight detection. Interesting moments are often signaled by speech emphasis, engine sound, laughter, applause, cooking sounds, silence, music changes, or loudness peaks. +- Shot and scene detection should be treated as a core primitive. The service should first split the source video into meaningful temporal units before asking an AI director to choose edits. +- Modern editing tools increasingly use transcript search, visual search, beat detection, montage generation, auto-cropping, and music synchronization. The service should expose similar building blocks internally instead of relying on a prompt alone. +- FFmpeg is suitable as the deterministic render backend, but the current renderer must be expanded. Production-level edits need real filter graphs for speed ramps, transitions, stabilization, text, color, grain, vignettes, audio ducking, loudness normalization, SFX placement, and final mastering. + +Research sources: + +- https://arxiv.org/abs/2507.10495 +- https://arxiv.org/abs/2401.10075 +- https://arxiv.org/abs/2107.09612 +- https://arxiv.org/abs/2306.11918 +- https://en.wikipedia.org/wiki/Shot_transition_detection +- https://en.wikipedia.org/wiki/Video_content_analysis +- https://en.wikipedia.org/wiki/Film_editing +- https://en.wikipedia.org/wiki/FFmpeg +- https://www.digitalcameraworld.com/cameras/video-cameras/these-are-the-new-ai-powered-features-coming-to-final-cut-pro-and-imovie + +## Target Workflow + +1. User places one source video in the configured local source folder. +2. Scheduler detects the video and creates a highlight project. +3. Service probes the source with `ffprobe` and creates stable low-resolution proxy media. +4. Service extracts thumbnails, contact sheets, audio waveform data, audio stems when possible, and transcript data. +5. Service performs shot and scene segmentation. +6. Service classifies the content category. +7. Service generates highlight candidates from visual, audio, transcript, and motion signals. +8. AI director reviews the compact analysis artifacts and creates a strict edit plan. +9. Asset providers create or select voiceover, music, SFX, captions, and overlay text. +10. Renderer executes the edit plan with FFmpeg and produces final highlight MP4s. +11. QA checks validate technical quality and creative completeness. +12. Final clips, manifests, storyboards, and logs are written to the output folder. + +## Proposed Folder Contract + +Input: + +```text +input/highlights/source/ +``` + +Output: + +```text +output/highlight-projects// + project.json + source/ + analysis/ + ffprobe.json + shots.json + scenes.json + transcript.json + audio-events.json + visual-analysis.json + category.json + highlight-candidates.json + director/ + director-brief.md + director-prompt.md + edit-plan.json + storyboard.md + assets/ + voiceover/ + music/ + sfx/ + overlays/ + highlights/ + / + final.mp4 + preview.mp4 + render-manifest.json + qa-report.json +``` + +## Content Categories + +### Family Vlog + +Creative intent: + +- Warm, emotional, human, memory-focused. +- Preserve natural faces, voices, laughter, and real moments. +- Avoid over-editing or aggressive transitions. +- Use gentle music, soft cinematic color, light film grain, and meaningful text overlays. + +Highlight signals: + +- Smiles, hugs, reactions, children, group moments, travel reveal moments, laughter. +- Speech with emotional words or important context. +- Stable shots with faces and clear composition. + +Voiceover style: + +- Personal, warm, reflective, short. +- Should sound like a memory being narrated, not an advertisement. + +### Food Vlog + +Creative intent: + +- Sensory, rhythmic, appetizing, tactile. +- Make ingredients, textures, steam, plating, and reactions feel premium. +- Sync cuts to cooking sounds and music beats. + +Highlight signals: + +- Close-ups of ingredients, cutting, sizzling, steam, pouring, plating, first bite, reaction. +- Strong texture, color, macro framing, hand movement, kitchen action. + +Voiceover style: + +- Descriptive and sensory. +- Mention texture, aroma, craft, freshness, and payoff. + +### Car Vlog + +Creative intent: + +- Heroic, powerful, premium, dynamic. +- Emphasize stance, motion, engine sound, details, acceleration, road presence, reflections, wheels, badges, lights, interior, and reveal shots. +- Use speed ramps, bass hits, whooshes, cinematic grade, controlled text overlays, and strong music. + +Highlight signals: + +- Car fully visible, front/rear three-quarter angles, wheels turning, badge/detail shots, engine/exhaust audio, road motion, camera movement, dramatic lighting. +- Avoid repeated static hood shots unless used as setup. + +Voiceover style: + +- Confident, cinematic, concise. +- Should praise the vehicle specifically based on observed details, not use generic supercar phrases. + +### Generic Vlog Fallback + +Creative intent: + +- Clear story arc, clean pacing, tasteful grade, and context-aware overlays. +- Use the best available moments even when the category is uncertain. + +Highlight signals: + +- High motion, strong composition, faces, speech peaks, laughter, visual novelty, scene changes. + +## AI Director Responsibilities + +The AI director should not render the video. It should produce a validated edit decision plan that the renderer can execute. + +Required AI director inputs: + +- Source metadata: duration, resolution, frame rate, codec, audio streams. +- Shot list with timestamps, representative thumbnails, motion score, blur score, brightness score, face/object labels, and transcript references. +- Audio analysis: speech segments, silence, peaks, loudness, detected audio events, music/noise/speech classification. +- Category classification with confidence and reasons. +- Highlight candidates with ranked evidence. +- Available music, SFX, voice, font, LUT, and transition capabilities. + +Required AI director output: + +- Highlight list with target duration and category. +- Timeline edit decisions with exact source timestamps. +- Music mood, energy curve, and beat-sync points. +- SFX cues with timestamps and purpose. +- Voiceover script split into timed lines. +- Text overlay copy, timing, animation, placement, and safe-area rules. +- Visual treatment: grade, transitions, speed ramps, stabilization, crop, punch-ins, grain, vignette, glow, blur, and overlays. +- Render profile: aspect ratio, resolution, FPS policy, codec, CRF or bitrate mode, loudness target. + +## Renderer Requirements + +The renderer must support these production features before the output can be called cinematic: + +- Precise trims using source timestamps. +- Crossfades, match cuts, hard cuts, whip transitions, fade to black, and dip transitions. +- Speed ramps and short impact pauses. +- Dynamic crop and punch-in keyframes for static shots. +- Stabilization option for shaky footage. +- Category-specific color grade chains or LUT support. +- Film grain, vignette, sharpening, glow, blur, and selective contrast. +- Text overlays with font, position, animation timing, stroke, shadow, and safe area. +- Caption and subtitle rendering. +- Music bed placement with fade in/out. +- Voiceover placement with sidechain ducking. +- SFX placement for whooshes, impacts, risers, camera clicks, engine hits, cooking sounds, and ambient layers. +- Audio loudness normalization to a configured target. +- Final render manifest with all commands, source segments, assets, and durations. + +## Asset Provider Requirements + +The system needs explicit providers instead of pretending a prompt can create finished audio by itself. + +- Voiceover provider: local TTS or remote TTS, configurable voice per category, output WAV. +- Music provider: licensed local library first, optional generated music provider later. +- SFX provider: licensed local SFX library with tags and category mapping. +- Font provider: local fonts with brand/category mapping. +- LUT provider: local LUT files or named FFmpeg grade presets. +- Asset cache: deterministic reuse by prompt, category, duration, and provider settings. + +Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export. + +## Highlight Scoring Model + +Each candidate segment should be scored with a weighted model: + +```text +score = + visual_interest + + motion_energy + + audio_interest + + transcript_importance + + category_relevance + + composition_quality + + novelty + - blur_penalty + - darkness_penalty + - repetition_penalty + - unusable_audio_penalty +``` + +Category-specific examples: + +- Family vlog should boost faces, laughter, dialogue, emotional reactions, and group moments. +- Food vlog should boost close-ups, texture, cooking actions, plating, steam, and bite reactions. +- Car vlog should boost full-car hero shots, motion, wheels, engine audio, badges, roads, and dramatic lighting. + +## Edit Plan Schema + +The plan should be strict JSON so a cheaper model or deterministic renderer can execute it. + +```json +{ + "projectId": "example-project", + "sourceVideo": "input/highlights/source/example.mp4", + "contentCategory": "car_vlog", + "highlights": [ + { + "id": "hero-reveal-001", + "title": "The Reveal", + "targetDurationSeconds": 35, + "aspectRatio": "16:9", + "renderProfile": "cinematic_4k_h264", + "creativeIntent": "Premium Porsche reveal with dynamic pacing and confident narration.", + "timeline": [ + { + "type": "video", + "sourceStart": "00:01:12.400", + "sourceEnd": "00:01:16.900", + "timelineStart": "00:00:00.000", + "effects": ["stabilize", "cinematic_warm_contrast", "slow_push_in"], + "transitionOut": "bass_hit_cut" + } + ], + "voiceover": [ + { + "text": "This is not just a drive. It is precision, presence, and control.", + "timelineStart": "00:00:04.200", + "voice": "confident_cinematic_male", + "delivery": "calm premium" + } + ], + "music": { + "mood": "premium cinematic electronic", + "energyCurve": "slow_build_to_impact", + "assetId": "music/premium-drive-90bpm.wav" + }, + "sfx": [ + { + "assetId": "sfx/bass-hit-01.wav", + "timelineStart": "00:00:07.800", + "gainDb": -6 + } + ], + "overlays": [ + { + "text": "PRECISION IN MOTION", + "timelineStart": "00:00:08.000", + "durationSeconds": 2.5, + "placement": "lower_left_safe", + "animation": "fade_slide_up" + } + ] + } + ] +} +``` + +## Milestones + +- [ ] Milestone 1: Define the cinematic highlight project domain model, folder contract, and persisted manifests. +- [ ] Milestone 2: Add a local scheduler that accepts exactly one source video at a time from the highlight source folder. +- [ ] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation. +- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect. +- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript. +- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails. +- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback. +- [x] Milestone 8: Implement category-specific highlight candidate scoring. +- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans. +- [ ] Milestone 10: Implement edit plan validation so impossible timestamps, missing assets, invalid overlays, and unsupported effects fail before render. +- [ ] Milestone 11: Implement asset provider interfaces for voiceover, music, SFX, fonts, and LUTs. +- [ ] Milestone 12: Add a local licensed asset library with category tags and deterministic asset selection. +- [ ] Milestone 13: Implement renderer v2 with transitions, speed ramps, dynamic crops, overlays, visual effects, audio ducking, loudness normalization, and render manifests. +- [ ] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters. +- [ ] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips. +- [ ] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content. +- [ ] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size. +- [ ] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering. + +## Testing Strategy + +- Unit tests for category scoring, candidate ranking, schema validation, folder discovery, and manifest writing. +- Integration tests with short fixture videos that verify each category can produce at least one valid highlight plan. +- Renderer tests that verify FFmpeg commands include expected filters for transitions, overlays, color treatment, music, SFX, and voiceover. +- Golden-file tests for AI prompt inputs and expected JSON schema shape. +- End-to-end local test that places one video in the source folder and verifies final clips, manifests, storyboards, and QA reports are created. +- Negative tests for corrupt input video, unsupported codec, empty video, missing audio, missing asset, invalid timestamp, and renderer failure. + +## Step-By-Step Local How-To Guide + +This guide describes the intended operator workflow after the milestones are implemented. + +### 1. Prepare The Machine + +Install and verify required binaries: + +```bash +ffmpeg -version +ffprobe -version +``` + +Optional but recommended tools: + +```bash +python3 --version +``` + +Recommended future optional components: + +- PySceneDetect for higher-quality scene detection. +- Whisper or another ASR provider for transcript-based editing. +- A configured TTS provider for voiceover. +- A licensed local music and SFX library. + +### 2. Configure The Application + +Set the local cinematic highlight folders in `application.yml`: + +```yaml +video-clipping: + cinematic-highlights: + enabled: true + source-folder: input/highlights/source + output-folder: output/highlight-projects + scheduler-interval: 10s + max-active-projects: 1 + default-aspect-ratio: "16:9" + default-render-profile: cinematic_1080p_h264 +``` + +Configure the AI director model separately from the renderer: + +```yaml +video-clipping: + cinematic-highlights: + director: + enabled: true + model: gpt-5 + allow-cheaper-model: true + cheaper-model: gpt-5-mini + require-human-approval-before-render: true +``` + +Configure local assets: + +```yaml +video-clipping: + cinematic-highlights: + assets: + music-folder: input/highlights/assets/music + sfx-folder: input/highlights/assets/sfx + fonts-folder: input/highlights/assets/fonts + luts-folder: input/highlights/assets/luts + voiceover-folder: output/highlight-projects/_voiceover-cache +``` + +### 3. Create Folders + +Create the required local folders: + +```bash +mkdir -p input/highlights/source +mkdir -p input/highlights/assets/music +mkdir -p input/highlights/assets/sfx +mkdir -p input/highlights/assets/fonts +mkdir -p input/highlights/assets/luts +mkdir -p output/highlight-projects +``` + +### 4. Add The Source Video + +Place exactly one source video in the source folder: + +```text +input/highlights/source/my-porsche-drive.mp4 +``` + +The scheduler should process one source video at a time. If multiple videos are present, it should process the first valid file, finish it, then move to the next one. + +Invalid files should be ignored and logged with the reason. + +### 5. Start The Application + +Run the Spring Boot service: + +```bash +./mvnw spring-boot:run +``` + +Expected startup logs: + +```text +Cinematic highlight scheduler started +Watching source folder: input/highlights/source +Output folder: output/highlight-projects +Scheduler interval: 10s +Max active projects: 1 +``` + +### 6. Verify Project Creation + +After the scheduler picks up the source file, verify a project folder exists: + +```text +output/highlight-projects// +``` + +Expected early files: + +```text +project.json +analysis/ffprobe.json +analysis/shots.json +analysis/scenes.json +analysis/category.json +director/director-brief.md +director/director-prompt.md +``` + +Expected logs: + +```text +Started cinematic highlight project projectId= source=my-porsche-drive.mp4 +Generated proxy video +Extracted representative thumbnails +Detected shots and scenes +Classified content category category=car_vlog confidence=... +Generated highlight candidates count=... +``` + +### 7. Review The AI Director Prompt + +Open the director prompt: + +```text +output/highlight-projects//director/director-prompt.md +``` + +Check that it includes: + +- Source metadata. +- Category classification. +- Contact sheet or thumbnail references. +- Candidate segments with timestamps. +- Available renderer effects. +- Available music, SFX, voice, font, and LUT assets. +- Strict JSON schema requirements. +- Category-specific creative direction. + +If the prompt is generic, the final edit will probably be generic. The prompt must force concrete choices: exact cuts, specific overlays, voiceover lines, SFX timing, music mood, and visual treatment. + +### 8. Execute The AI Director + +Run the configured AI director against the prompt. The director must output: + +```text +output/highlight-projects//director/edit-plan.json +``` + +If human approval is enabled, inspect the edit plan before rendering. + +Minimum review checklist: + +- The selected shots are not repetitive. +- Each highlight has a clear beginning, middle, and ending. +- The edit uses the correct category style. +- Voiceover text is specific to the observed video. +- Music and SFX cues are not placeholders. +- Text overlays are tasteful and not overused. +- All referenced timestamps are inside the source duration. +- All referenced assets exist. + +### 9. Approve Or Correct The Edit Plan + +If the plan is weak, edit or regenerate it before rendering. + +Common fixes: + +- Replace generic voiceover with concrete lines about the actual footage. +- Remove boring shots or repeated angles. +- Add stronger openers and endings. +- Add music beat cues. +- Add SFX only where they support a cut or reveal. +- Add overlay text only where it improves the scene. +- Adjust target duration if the highlight feels too slow. + +When approved, write: + +```text +output/highlight-projects//director/approved.flag +``` + +### 10. Render The Highlights + +The renderer consumes: + +```text +output/highlight-projects//director/edit-plan.json +``` + +Expected renderer output: + +```text +output/highlight-projects//highlights//final.mp4 +output/highlight-projects//highlights//preview.mp4 +output/highlight-projects//highlights//render-manifest.json +output/highlight-projects//highlights//qa-report.json +``` + +Expected logs: + +```text +Rendering highlight projectId= highlightId= +Applying video trims count=... +Applying transitions count=... +Applying visual effects count=... +Mixing music asset=... +Mixing voiceover lines=... +Mixing SFX count=... +Completed highlight render duration=... output=... +``` + +### 11. Review The Final Clips + +For each `final.mp4`, verify: + +- The first three seconds are interesting. +- The edit has intentional pacing. +- The selected shots fit the content type. +- Music fits the scene and is audible. +- Voiceover is clear and not robotic or generic. +- SFX support the edit without becoming noisy. +- Text overlays are readable and not cheap-looking. +- Color grade improves the footage without crushing detail. +- Audio is not clipping and does not have long silent sections. + +If the final video is boring, inspect in this order: + +1. `analysis/highlight-candidates.json`: weak source moments or wrong scoring. +2. `director/edit-plan.json`: weak AI choices, generic voiceover, no effects. +3. `render-manifest.json`: renderer ignored effects or missing assets. +4. `qa-report.json`: technical failures or missing audio/effects. + +### 12. Find The Final Result + +Final highlight videos are stored here: + +```text +output/highlight-projects//highlights//final.mp4 +``` + +The full audit trail is stored beside the final video: + +```text +render-manifest.json +qa-report.json +``` + +The creative plan is stored here: + +```text +output/highlight-projects//director/storyboard.md +output/highlight-projects//director/edit-plan.json +``` + +### 13. Clean Up After A Run + +After confirming the output is good, move the source video out of the source folder or let the scheduler move it to a processed folder: + +```text +input/highlights/processed/ +``` + +Failed inputs should move to: + +```text +input/highlights/failed/ +``` + +The scheduler must not repeatedly process the same source video. + +## Definition Of Done + +A highlight export is production-ready only when: + +- The selected shots match the detected content category. +- The edit has a clear beginning, middle, and ending. +- Music is present, fitting, and timed to key cuts. +- Voiceover is present when requested and sounds appropriate for the category. +- SFX are present where they add impact and are not randomly overused. +- Text overlays are readable, visually appealing, correctly timed, and safe-area compliant. +- Visual treatment is visible but not destructive. +- Audio is normalized and free of clipping. +- Output includes a render manifest, edit plan, storyboard, and QA report. +- The final MP4 can be played independently without relying on temporary files. + +## Key Risk + +The previous boring output is expected if the system only creates a weak prompt and then renders simple trims. Cinematic output requires three things working together: richer multimodal analysis, a strict category-aware AI director plan, and a renderer that can actually execute music, SFX, voiceover, overlays, transitions, and visual effects. diff --git a/src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java b/src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java index 0e65cd6..3237e8b 100644 --- a/src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java +++ b/src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java @@ -54,21 +54,23 @@ public class AiDirectorPromptGenerator { return """ # Local AI Director Assignment - You are the director for a premium cinematic Porsche promo. + You are the director for a production-ready cinematic highlight edit. Work only with artifacts in this project folder: - Read `analysis.json` for authoritative clip metadata. + - Read `cinematic-highlight-analysis.json`, `category.json`, and `highlight-candidates.json` when present. - Inspect every relevant image in `contact-sheets/` and `thumbnails/` before choosing shots. - Use `proxies/` only when motion needs closer inspection. Complete this task: - 1. Judge visual quality, composition, motion, continuity, and story value from the generated artifacts. - 2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final hero shot. + 1. Judge visual quality, composition, motion, continuity, audio opportunity, and story value from the generated artifacts. + 2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final memorable shot. 3. Follow the strict JSON schema and constraints in the brief below. 4. Write the final JSON object to `inbox/%s`. Do not render video. Do not run FFmpeg. Do not modify source clips or generated analysis artifacts. Do not invent clip IDs, source ranges, durations, or media paths. + Do not write generic voiceover. The narration, text overlays, music mood, SFX, and visual effects must fit the detected content category. Your task is complete only when `inbox/%s` contains valid JSON with no Markdown fences. %s diff --git a/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalysis.java b/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalysis.java new file mode 100644 index 0000000..66cc5f6 --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalysis.java @@ -0,0 +1,14 @@ +package org.example.videoclips.editing; + +import java.time.Instant; +import java.util.List; + +public record CinematicHighlightAnalysis( + String projectId, + ContentCategory category, + double categoryConfidence, + List categoryReasons, + List candidates, + Instant createdAt +) { +} diff --git a/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java b/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java new file mode 100644 index 0000000..67a3e22 --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/CinematicHighlightAnalyzer.java @@ -0,0 +1,157 @@ +package org.example.videoclips.editing; + +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.stereotype.Component; + +import java.time.Clock; +import java.time.Instant; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Locale; + +@Component +@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true) +public class CinematicHighlightAnalyzer { + + private static final double DEFAULT_CANDIDATE_SECONDS = 8.0; + private static final int MAX_CANDIDATES_PER_CLIP = 12; + + private final Clock clock; + + public CinematicHighlightAnalyzer() { + this(Clock.systemUTC()); + } + + CinematicHighlightAnalyzer(Clock clock) { + this.clock = clock; + } + + public CinematicHighlightAnalysis analyze(EditProject project, EditProjectAnalysis analysis) { + CategoryDecision category = classify(project, analysis); + List candidates = analysis.clips().stream() + .flatMap(clip -> candidatesForClip(clip, category.category()).stream()) + .sorted(Comparator.comparingDouble(HighlightCandidate::score).reversed() + .thenComparing(HighlightCandidate::clipId) + .thenComparingDouble(HighlightCandidate::sourceStartSeconds)) + .toList(); + return new CinematicHighlightAnalysis(project.id(), category.category(), category.confidence(), + category.reasons(), candidates, Instant.now(clock)); + } + + private CategoryDecision classify(EditProject project, EditProjectAnalysis analysis) { + String haystack = (project.name() + " " + project.style() + " " + + analysis.clips().stream() + .map(clip -> clip.clipId() + " " + clip.sourcePath()) + .reduce("", (left, right) -> left + " " + right)) + .toLowerCase(Locale.ROOT); + + if (containsAny(haystack, "porsche", "car", "auto", "vehicle", "drive", "driving", "engine", "wheel")) { + return new CategoryDecision(ContentCategory.CAR_VLOG, 0.78, + List.of("Project metadata suggests vehicle or driving content.")); + } + if (containsAny(haystack, "food", "cook", "cooking", "recipe", "kitchen", "restaurant", "meal", "dish")) { + return new CategoryDecision(ContentCategory.FOOD_VLOG, 0.74, + List.of("Project metadata suggests food, cooking, or restaurant content.")); + } + if (containsAny(haystack, "family", "kids", "child", "children", "birthday", "holiday", "vacation", "vlog")) { + return new CategoryDecision(ContentCategory.FAMILY_VLOG, 0.68, + List.of("Project metadata suggests personal or family vlog content.")); + } + return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.45, + List.of("No strong category-specific metadata was available; using generic vlog direction.")); + } + + private boolean containsAny(String haystack, String... needles) { + for (String needle : needles) { + if (haystack.contains(needle)) { + return true; + } + } + return false; + } + + private List candidatesForClip(ClipAnalysis clip, ContentCategory category) { + double duration = clip.durationSeconds(); + if (duration <= 0) { + return List.of(); + } + + double segmentSeconds = Math.min(DEFAULT_CANDIDATE_SECONDS, duration); + int segmentCount = Math.max(1, (int) Math.ceil(duration / segmentSeconds)); + int cappedCount = Math.min(segmentCount, MAX_CANDIDATES_PER_CLIP); + double step = segmentCount <= MAX_CANDIDATES_PER_CLIP + ? segmentSeconds + : Math.max(1.0, (duration - segmentSeconds) / (MAX_CANDIDATES_PER_CLIP - 1)); + + List candidates = new ArrayList<>(); + for (int index = 0; index < cappedCount; index++) { + double start = Math.min(duration - segmentSeconds, index * step); + double end = Math.min(duration, start + segmentSeconds); + candidates.add(new HighlightCandidate( + "%s_candidate_%02d".formatted(clip.clipId(), index + 1), + clip.clipId(), + round(start), + round(end), + round(score(clip, category, index, cappedCount)), + role(index, cappedCount), + reasons(clip, category, index, cappedCount) + )); + } + return candidates; + } + + private double score(ClipAnalysis clip, ContentCategory category, int index, int count) { + double quality = Math.max(0.1, clip.sharpnessScore()) + Math.max(0.1, clip.brightnessScore()) + + Math.max(0.1, clip.motionScore()); + double position = switch (role(index, count)) { + case "opening_hook" -> 0.25; + case "final_hero" -> 0.22; + case "rising_action" -> 0.16; + default -> 0.1; + }; + double categoryBoost = switch (category) { + case CAR_VLOG -> 0.2; + case FOOD_VLOG -> 0.16; + case FAMILY_VLOG -> 0.14; + case GENERIC_VLOG -> 0.08; + }; + return Math.min(1.0, 0.35 + quality * 0.12 + position + categoryBoost); + } + + private String role(int index, int count) { + if (index == 0) { + return "opening_hook"; + } + if (index == count - 1) { + return "final_hero"; + } + if (index <= Math.max(1, count / 3)) { + return "rising_action"; + } + return "detail_or_bridge"; + } + + private List reasons(ClipAnalysis clip, ContentCategory category, int index, int count) { + List reasons = new ArrayList<>(); + reasons.add("Suggested as " + role(index, count).replace('_', ' ') + " based on source position."); + reasons.add("Duration window is short enough for fast cinematic pacing."); + if (clip.motionScore() > 0) { + reasons.add("Motion score is available for director ranking."); + } + reasons.add(switch (category) { + case CAR_VLOG -> "For car content, inspect this window for hero angles, wheels, badges, engine sound, or road motion."; + case FOOD_VLOG -> "For food content, inspect this window for texture, preparation, plating, steam, or bite reactions."; + case FAMILY_VLOG -> "For family content, inspect this window for faces, reactions, laughter, hugs, or emotional dialogue."; + case GENERIC_VLOG -> "For generic vlog content, inspect this window for visual novelty, clear action, or useful story context."; + }); + return List.copyOf(reasons); + } + + private double round(double value) { + return Math.round(value * 1000.0) / 1000.0; + } + + private record CategoryDecision(ContentCategory category, double confidence, List reasons) { + } +} diff --git a/src/main/java/org/example/videoclips/editing/ContentCategory.java b/src/main/java/org/example/videoclips/editing/ContentCategory.java new file mode 100644 index 0000000..0ddeda1 --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/ContentCategory.java @@ -0,0 +1,8 @@ +package org.example.videoclips.editing; + +public enum ContentCategory { + FAMILY_VLOG, + FOOD_VLOG, + CAR_VLOG, + GENERIC_VLOG +} diff --git a/src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java b/src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java index 3010396..3b8cc2c 100644 --- a/src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java +++ b/src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java @@ -21,6 +21,7 @@ public class EditProjectAnalyzer { private final ThumbnailExtractor thumbnailExtractor; private final ContactSheetGenerator contactSheetGenerator; private final ProxyGenerator proxyGenerator; + private final CinematicHighlightAnalyzer cinematicHighlightAnalyzer; private final Clock clock; private final EditObservability observability; @@ -32,10 +33,11 @@ public class EditProjectAnalyzer { ThumbnailExtractor thumbnailExtractor, ContactSheetGenerator contactSheetGenerator, ProxyGenerator proxyGenerator, + CinematicHighlightAnalyzer cinematicHighlightAnalyzer, EditObservability observability ) { this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, - Clock.systemUTC(), observability); + cinematicHighlightAnalyzer, Clock.systemUTC(), observability); } EditProjectAnalyzer( @@ -47,13 +49,14 @@ public class EditProjectAnalyzer { ProxyGenerator proxyGenerator, Clock clock ) { - this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, clock, null); + this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, null, clock, null); } EditProjectAnalyzer( EditProjectStore store, EditClipDiscovery discovery, FfmpegClipInspector inspector, ThumbnailExtractor thumbnailExtractor, ContactSheetGenerator contactSheetGenerator, - ProxyGenerator proxyGenerator, Clock clock, EditObservability observability + ProxyGenerator proxyGenerator, CinematicHighlightAnalyzer cinematicHighlightAnalyzer, Clock clock, + EditObservability observability ) { this.store = store; this.discovery = discovery; @@ -61,6 +64,7 @@ public class EditProjectAnalyzer { this.thumbnailExtractor = thumbnailExtractor; this.contactSheetGenerator = contactSheetGenerator; this.proxyGenerator = proxyGenerator; + this.cinematicHighlightAnalyzer = cinematicHighlightAnalyzer; this.clock = clock; this.observability = observability; } @@ -100,6 +104,7 @@ public class EditProjectAnalyzer { EditProjectAnalysis analysis = new EditProjectAnalysis(projectId, List.copyOf(clips), List.copyOf(errors), Instant.now(clock)); store.writeJson(projectId, "analysis.json", analysis); + writeCinematicHighlightArtifacts(projectId, analysis); if (observability != null) { observability.analysisCompleted(projectId, System.nanoTime() - startedAt, clips.size(), errors.size()); } @@ -133,4 +138,15 @@ public class EditProjectAnalyzer { inspected.sharpnessScore() ); } + + private void writeCinematicHighlightArtifacts(String projectId, EditProjectAnalysis analysis) { + if (cinematicHighlightAnalyzer == null) { + return; + } + EditProject project = store.readJson(projectId, "project.json", EditProject.class); + CinematicHighlightAnalysis cinematic = cinematicHighlightAnalyzer.analyze(project, analysis); + store.writeJson(projectId, "cinematic-highlight-analysis.json", cinematic); + store.writeJson(projectId, "category.json", cinematic); + store.writeJson(projectId, "highlight-candidates.json", cinematic.candidates()); + } } diff --git a/src/main/java/org/example/videoclips/editing/HighlightCandidate.java b/src/main/java/org/example/videoclips/editing/HighlightCandidate.java new file mode 100644 index 0000000..08b05fe --- /dev/null +++ b/src/main/java/org/example/videoclips/editing/HighlightCandidate.java @@ -0,0 +1,14 @@ +package org.example.videoclips.editing; + +import java.util.List; + +public record HighlightCandidate( + String id, + String clipId, + double sourceStartSeconds, + double sourceEndSeconds, + double score, + String suggestedRole, + List reasons +) { +} diff --git a/src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java b/src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java index 3fc4044..bd06517 100644 --- a/src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java +++ b/src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java @@ -160,7 +160,7 @@ public class LocalDirectorScheduler { try { EditProjectResponse project = projectService.createProject(new CreateEditProjectRequest( working.getFileName().toString(), working.toString(), editing.getTargetDurationSeconds(), - "cinematic-porsche-promo", true, true, true)); + "category-aware-cinematic-highlights", true, true, true)); projectId = project.projectId(); projectService.updateProject(projectId, EditProjectStatus.ANALYZING, working, null); EditProjectAnalysis analysis = analyzer.analyze(projectId, working); diff --git a/src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java b/src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java index e5c1e17..cd0446f 100644 --- a/src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java +++ b/src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java @@ -43,6 +43,7 @@ public class StoryboardPromptGenerator { } String prompt(EditProject project, EditProjectAnalysis analysis) { + CinematicHighlightAnalysis cinematic = readCinematicAnalysis(project.id()); StringBuilder clips = new StringBuilder(); for (ClipAnalysis clip : analysis.clips()) { clips.append("- clipId: ").append(clip.clipId()).append('\n') @@ -54,6 +55,10 @@ public class StoryboardPromptGenerator { .append(" contactSheet: ").append(reference("contact-sheets", clip.contactSheet())).append('\n') .append(" proxy: ").append(reference("proxies", clip.proxyPath())).append('\n'); } + String category = cinematic == null ? "unknown" : cinematic.category().name().toLowerCase(); + String candidates = cinematic == null ? "No precomputed highlight candidates. Inspect all clips carefully." + : candidateBrief(cinematic); + String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category()); return """ # Cinematic Storyboard Director Brief @@ -66,18 +71,29 @@ public class StoryboardPromptGenerator { - Voiceover enabled: %s - Music enabled: %s - Sound effects enabled: %s + - Detected content category: `%s` - Use only the clip IDs and source ranges listed below. - Keep every source range within its clip duration. - Timeline ranges must be chronological and must not overlap. - Prefer an intentional opening hook, escalating middle, and memorable final hero shot. + - Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category. + - Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets. + - Do not write generic praise, generic lifestyle language, or placeholder asset names. - Return strict JSON only. Do not use Markdown fences or add commentary. + Category-specific direction: + %s + + Ranked highlight candidates: + %s + Available clips and project-relative visual references: %s Required JSON object shape: { "projectId": "%s", "style": "%s", + "contentCategory": "%s", "targetDurationSeconds": %d, "decisions": [{ "clipId": "clip_00001", @@ -93,25 +109,90 @@ public class StoryboardPromptGenerator { }], "audioCues": [{ "type": "music|sfx", - "assetKey": "descriptive asset name", + "assetKey": "concrete music or sfx asset description", "timelineStartSeconds": 0.0, "timelineEndSeconds": 2.5, "gainDb": -12.0, "notes": "story purpose" }], "voiceover": [{ - "text": "narration", + "text": "specific narration based on visible footage", "timelineStartSeconds": 0.0, "timelineEndSeconds": 2.5, "delivery": "confident cinematic narrator" }], + "overlays": [{ + "text": "short premium overlay text", + "timelineStartSeconds": 0.0, + "timelineEndSeconds": 2.5, + "placement": "lower_left_safe|center_safe|lower_center_safe", + "animation": "fade_slide_up|fade_in|none", + "reason": "why this text improves the scene" + }], "renderProfile": "mp4-h264-aac-1080p", "summary": "short creative rationale" } """.formatted( project.id(), project.style(), project.targetDurationSeconds(), - project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(), - clips.toString().stripTrailing(), project.id(), project.style(), project.targetDurationSeconds()); + project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(), category, + categoryDirection, candidates, clips.toString().stripTrailing(), project.id(), project.style(), + category, project.targetDurationSeconds()); + } + + private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) { + if (!Files.exists(store.projectDirectory(projectId).resolve("cinematic-highlight-analysis.json"))) { + return null; + } + return store.readJson(projectId, "cinematic-highlight-analysis.json", CinematicHighlightAnalysis.class); + } + + private String candidateBrief(CinematicHighlightAnalysis cinematic) { + StringBuilder builder = new StringBuilder(); + builder.append("- categoryConfidence: ").append(cinematic.categoryConfidence()).append('\n'); + builder.append("- categoryReasons: ").append(cinematic.categoryReasons()).append('\n'); + for (HighlightCandidate candidate : cinematic.candidates()) { + builder.append("- candidateId: ").append(candidate.id()).append('\n') + .append(" clipId: ").append(candidate.clipId()).append('\n') + .append(" sourceStartSeconds: ").append(candidate.sourceStartSeconds()).append('\n') + .append(" sourceEndSeconds: ").append(candidate.sourceEndSeconds()).append('\n') + .append(" score: ").append(candidate.score()).append('\n') + .append(" suggestedRole: ").append(candidate.suggestedRole()).append('\n') + .append(" reasons: ").append(candidate.reasons()).append('\n'); + } + return builder.toString().stripTrailing(); + } + + private String categoryDirection(ContentCategory category) { + return switch (category) { + case CAR_VLOG -> """ + - Make the edit feel premium, powerful, and precise. + - Prioritize hero angles, wheels, badges, motion, reflections, roads, lights, interior details, and engine or exhaust moments. + - Use speed ramps, bass-hit cuts, whooshes, cinematic contrast, subtle grain, and confident short overlay text when the footage supports it. + - Voiceover should praise the specific car footage, not make unsupported claims. + """; + case FOOD_VLOG -> """ + - Make the edit sensory, warm, rhythmic, and appetizing. + - Prioritize texture, slicing, pouring, sizzling, steam, plating, table reveals, and first-bite reactions. + - Use music with light groove, tactile SFX, warm grade, macro-feeling punch-ins, and short ingredient or payoff overlays. + - Voiceover should describe texture, craft, freshness, and payoff without sounding like a generic restaurant ad. + """; + case FAMILY_VLOG -> """ + - Make the edit warm, human, emotional, and memory-focused. + - Prioritize faces, reactions, laughter, hugs, children, travel reveals, and meaningful dialogue. + - Use gentle music, soft grade, light grain, natural pacing, and minimal text overlays. + - Voiceover should feel personal and reflective, not promotional. + """; + case GENERIC_VLOG -> genericDirection(); + }; + } + + private String genericDirection() { + return """ + - Build a clear mini-story from the strongest available moments. + - Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes. + - Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story. + - Voiceover must explain what is actually happening in the footage. + """; } private String references(String directory, List paths) { diff --git a/src/main/resources/application-cinematic-editing-local.yml b/src/main/resources/application-cinematic-editing-local.yml index 21e45de..5bf4be4 100644 --- a/src/main/resources/application-cinematic-editing-local.yml +++ b/src/main/resources/application-cinematic-editing-local.yml @@ -5,7 +5,7 @@ video-clipping: enabled: true local-director: enabled: true - auto-render-when-plan-appears: true + auto-render-when-plan-appears: false logging: level: diff --git a/src/test/java/org/example/videoclips/editing/AiDirectorPromptGeneratorTest.java b/src/test/java/org/example/videoclips/editing/AiDirectorPromptGeneratorTest.java index 9864ad8..9747971 100644 --- a/src/test/java/org/example/videoclips/editing/AiDirectorPromptGeneratorTest.java +++ b/src/test/java/org/example/videoclips/editing/AiDirectorPromptGeneratorTest.java @@ -30,12 +30,15 @@ class AiDirectorPromptGeneratorTest { String prompt = Files.readString(Path.of(files.promptPath())); assertThat(prompt) .contains("`analysis.json`") + .contains("`cinematic-highlight-analysis.json`") .contains("`contact-sheets/`") .contains("`thumbnails/`") .contains("`inbox/edit-plan.json`") .contains("Do not render video") + .contains("production-ready cinematic highlight edit") .contains("clipId: clip_00001") - .contains("\"decisions\""); + .contains("\"decisions\"") + .doesNotContain("premium cinematic Porsche promo"); assertThat(Files.readString(Path.of(files.readmePath()))) .contains("Codex, Claude") .contains("`inbox/edit-plan.json`"); diff --git a/src/test/java/org/example/videoclips/editing/CinematicEditingIntegrationTest.java b/src/test/java/org/example/videoclips/editing/CinematicEditingIntegrationTest.java index 8634a46..4b49f34 100644 --- a/src/test/java/org/example/videoclips/editing/CinematicEditingIntegrationTest.java +++ b/src/test/java/org/example/videoclips/editing/CinematicEditingIntegrationTest.java @@ -87,7 +87,7 @@ class CinematicEditingIntegrationTest { index * 2.0, (index + 1) * 2.0, "cut", "cut", 1, "cinematic grade", "integration sequence")) .toList(); - return new EditPlan(projectId, "cinematic-porsche-promo", 60, decisions, List.of(), List.of(), + return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(), "mp4-h264-aac-320p", "Generated integration edit"); } diff --git a/src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java b/src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java new file mode 100644 index 0000000..4766b65 --- /dev/null +++ b/src/test/java/org/example/videoclips/editing/CinematicHighlightAnalyzerTest.java @@ -0,0 +1,74 @@ +package org.example.videoclips.editing; + +import org.junit.jupiter.api.Test; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +class CinematicHighlightAnalyzerTest { + + private final CinematicHighlightAnalyzer analyzer = new CinematicHighlightAnalyzer( + Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC)); + + @Test + void classifiesCarContentAndCreatesRankedCandidates() { + EditProject project = project("Porsche mountain drive", "category-aware-cinematic-highlights"); + EditProjectAnalysis source = analysis(clip("porsche-drive.mp4", 45.0)); + + CinematicHighlightAnalysis result = analyzer.analyze(project, source); + + assertThat(result.projectId()).isEqualTo("project-1"); + assertThat(result.category()).isEqualTo(ContentCategory.CAR_VLOG); + assertThat(result.categoryConfidence()).isGreaterThan(0.7); + assertThat(result.categoryReasons()).isNotEmpty(); + assertThat(result.candidates()).hasSizeGreaterThan(1); + assertThat(result.candidates().get(0).score()).isGreaterThanOrEqualTo(result.candidates().get(1).score()); + assertThat(result.candidates()).allSatisfy(candidate -> { + assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0); + assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(45.0); + assertThat(candidate.reasons()).anyMatch(reason -> reason.contains("car content")); + }); + assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z")); + } + + @Test + void classifiesFoodAndFamilyFromProjectMetadata() { + assertThat(analyzer.analyze(project("Street food recipe", "cinematic"), analysis(clip("clip.mp4", 8))) + .category()).isEqualTo(ContentCategory.FOOD_VLOG); + assertThat(analyzer.analyze(project("Family birthday vlog", "cinematic"), analysis(clip("clip.mp4", 8))) + .category()).isEqualTo(ContentCategory.FAMILY_VLOG); + } + + @Test + void fallsBackToGenericWhenCategoryIsUnknown() { + CinematicHighlightAnalysis result = analyzer.analyze(project("Untitled", "cinematic"), + analysis(clip("scene.mp4", 5.0))); + + assertThat(result.category()).isEqualTo(ContentCategory.GENERIC_VLOG); + assertThat(result.candidates()).singleElement().satisfies(candidate -> { + assertThat(candidate.sourceStartSeconds()).isEqualTo(0.0); + assertThat(candidate.sourceEndSeconds()).isEqualTo(5.0); + assertThat(candidate.suggestedRole()).isEqualTo("opening_hook"); + }); + } + + private EditProject project(String name, String style) { + Instant now = Instant.parse("2026-07-10T10:00:00Z"); + return new EditProject("project-1", name, EditProjectStatus.ANALYZED, "input", "output", + 60, style, true, true, true, now, now, null); + } + + private EditProjectAnalysis analysis(ClipAnalysis clip) { + return new EditProjectAnalysis("project-1", List.of(clip), List.of(), + Instant.parse("2026-07-10T10:00:00Z")); + } + + private ClipAnalysis clip(String sourcePath, double durationSeconds) { + return new ClipAnalysis("clip_00001", sourcePath, durationSeconds, "h264", "aac", + 1920, 1080, 30.0, List.of(), null, null, 0.7, 0.8, 0.9); + } +} diff --git a/src/test/java/org/example/videoclips/editing/EditProjectAnalyzerTest.java b/src/test/java/org/example/videoclips/editing/EditProjectAnalyzerTest.java index 3e56243..2310b1c 100644 --- a/src/test/java/org/example/videoclips/editing/EditProjectAnalyzerTest.java +++ b/src/test/java/org/example/videoclips/editing/EditProjectAnalyzerTest.java @@ -60,6 +60,49 @@ class EditProjectAnalyzerTest { assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis); } + @Test + void writesCinematicHighlightArtifactsWhenEnabled() throws Exception { + Path input = Files.createDirectory(tempDir.resolve("porsche-input")); + Path clip = Files.writeString(input.resolve("porsche-drive.mp4"), "video"); + FileSystemEditProjectStore store = store(); + store.createProject("project-1"); + writeProject(store, input, "Porsche Drive"); + EditClipDiscovery discovery = mock(EditClipDiscovery.class); + FfmpegClipInspector inspector = mock(FfmpegClipInspector.class); + ThumbnailExtractor thumbnailExtractor = mock(ThumbnailExtractor.class); + ContactSheetGenerator contactSheetGenerator = mock(ContactSheetGenerator.class); + ProxyGenerator proxyGenerator = mock(ProxyGenerator.class); + ClipAnalysis inspected = clipAnalysis(clip); + when(discovery.discover(input)).thenReturn(List.of(clip)); + when(inspector.inspect(clip)).thenReturn(inspected); + when(thumbnailExtractor.extract(inspected, tempDir.resolve("projects/project-1/thumbnails"))) + .thenReturn(List.of("thumb-1.jpg")); + when(contactSheetGenerator.generate(inspected, tempDir.resolve("projects/project-1/contact-sheets"))) + .thenReturn("contact-sheet.jpg"); + when(proxyGenerator.generate(inspected, tempDir.resolve("projects/project-1/proxies"))) + .thenReturn(Optional.empty()); + EditProjectAnalyzer analyzer = new EditProjectAnalyzer( + store, + discovery, + inspector, + thumbnailExtractor, + contactSheetGenerator, + proxyGenerator, + new CinematicHighlightAnalyzer(Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC)), + Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC), + null + ); + + analyzer.analyze("project-1", input); + + CinematicHighlightAnalysis cinematic = store.readJson("project-1", "cinematic-highlight-analysis.json", + CinematicHighlightAnalysis.class); + assertThat(cinematic.category()).isEqualTo(ContentCategory.CAR_VLOG); + assertThat(cinematic.candidates()).isNotEmpty(); + assertThat(Files.exists(tempDir.resolve("projects/project-1/category.json"))).isTrue(); + assertThat(Files.exists(tempDir.resolve("projects/project-1/highlight-candidates.json"))).isTrue(); + } + @Test void recordsInvalidClipErrorsAndKeepsUsableClips() throws Exception { Path input = Files.createDirectory(tempDir.resolve("input")); @@ -148,6 +191,13 @@ class EditProjectAnalyzerTest { return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules()); } + private void writeProject(FileSystemEditProjectStore store, Path input, String name) { + Instant now = Instant.parse("2026-07-10T10:00:00Z"); + store.writeJson("project-1", "project.json", new EditProject( + "project-1", name, EditProjectStatus.ANALYZING, input.toString(), "output", + 60, "category-aware-cinematic-highlights", true, true, true, now, now, null)); + } + private ClipAnalysis clipAnalysis(Path clip) { return new ClipAnalysis( clip.getFileName().toString().replace(".mp4", ""), diff --git a/src/test/java/org/example/videoclips/editing/StoryboardPromptGeneratorTest.java b/src/test/java/org/example/videoclips/editing/StoryboardPromptGeneratorTest.java index cde11c0..87831d7 100644 --- a/src/test/java/org/example/videoclips/editing/StoryboardPromptGeneratorTest.java +++ b/src/test/java/org/example/videoclips/editing/StoryboardPromptGeneratorTest.java @@ -46,6 +46,33 @@ class StoryboardPromptGeneratorTest { assertThat(response.promptPath()).endsWith("storyboard-prompt.md"); } + @Test + void includesCategorySpecificHighlightCandidatesWhenAvailable() { + FileSystemEditProjectStore store = store(); + store.createProject("porsche-edit"); + store.writeJson("porsche-edit", "project.json", project()); + store.writeJson("porsche-edit", "analysis.json", analysis()); + store.writeJson("porsche-edit", "cinematic-highlight-analysis.json", new CinematicHighlightAnalysis( + "porsche-edit", + ContentCategory.CAR_VLOG, + 0.82, + List.of("Project metadata suggests vehicle content."), + List.of(new HighlightCandidate("clip_00001_candidate_01", "clip_00001", + 0.0, 8.0, 0.91, "opening_hook", List.of("Inspect for hero car angle."))), + Instant.parse("2026-07-10T10:01:00Z") + )); + + String prompt = new StoryboardPromptGenerator(store).prompt(project(), analysis()); + + assertThat(prompt) + .contains("Detected content category: `car_vlog`") + .contains("Ranked highlight candidates") + .contains("candidateId: clip_00001_candidate_01") + .contains("Make the edit feel premium, powerful, and precise") + .contains("\"contentCategory\": \"car_vlog\"") + .contains("\"overlays\""); + } + private FileSystemEditProjectStore store() { VideoClippingProperties properties = new VideoClippingProperties(); properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());