forked from jsl/video_editing_poc
Add category-aware cinematic highlight planning
This commit is contained in:
JSLMPR
parent
5d889b00dc
commit
5006dfd50b
|
|
@ -0,0 +1,636 @@
|
||||||
|
# Production Cinematic Highlight Editing Plan
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Build a local-first workflow where a user drops one source video into a source folder and the service produces multiple production-ready cinematic highlight edits. Each highlight must be adapted to the detected content type, such as family vlog, food vlog, car vlog, or a generic fallback category.
|
||||||
|
|
||||||
|
The target output is not just a technically valid MP4. The output must feel intentionally directed: strong shot selection, a clear mini-story, pacing matched to music, fitting voiceover, sound effects, cinematic visual treatment, tasteful text overlays, and a final audio mix that sounds finished.
|
||||||
|
|
||||||
|
## Research Findings
|
||||||
|
|
||||||
|
- Recent highlight-detection research points toward multimodal narrative understanding, not simple duration slicing. A production system should combine shot segmentation, visual analysis, speech/transcript analysis, audio events, pacing, and narrative role detection.
|
||||||
|
- Audio matters for highlight detection. Interesting moments are often signaled by speech emphasis, engine sound, laughter, applause, cooking sounds, silence, music changes, or loudness peaks.
|
||||||
|
- Shot and scene detection should be treated as a core primitive. The service should first split the source video into meaningful temporal units before asking an AI director to choose edits.
|
||||||
|
- Modern editing tools increasingly use transcript search, visual search, beat detection, montage generation, auto-cropping, and music synchronization. The service should expose similar building blocks internally instead of relying on a prompt alone.
|
||||||
|
- FFmpeg is suitable as the deterministic render backend, but the current renderer must be expanded. Production-level edits need real filter graphs for speed ramps, transitions, stabilization, text, color, grain, vignettes, audio ducking, loudness normalization, SFX placement, and final mastering.
|
||||||
|
|
||||||
|
Research sources:
|
||||||
|
|
||||||
|
- https://arxiv.org/abs/2507.10495
|
||||||
|
- https://arxiv.org/abs/2401.10075
|
||||||
|
- https://arxiv.org/abs/2107.09612
|
||||||
|
- https://arxiv.org/abs/2306.11918
|
||||||
|
- https://en.wikipedia.org/wiki/Shot_transition_detection
|
||||||
|
- https://en.wikipedia.org/wiki/Video_content_analysis
|
||||||
|
- https://en.wikipedia.org/wiki/Film_editing
|
||||||
|
- https://en.wikipedia.org/wiki/FFmpeg
|
||||||
|
- https://www.digitalcameraworld.com/cameras/video-cameras/these-are-the-new-ai-powered-features-coming-to-final-cut-pro-and-imovie
|
||||||
|
|
||||||
|
## Target Workflow
|
||||||
|
|
||||||
|
1. User places one source video in the configured local source folder.
|
||||||
|
2. Scheduler detects the video and creates a highlight project.
|
||||||
|
3. Service probes the source with `ffprobe` and creates stable low-resolution proxy media.
|
||||||
|
4. Service extracts thumbnails, contact sheets, audio waveform data, audio stems when possible, and transcript data.
|
||||||
|
5. Service performs shot and scene segmentation.
|
||||||
|
6. Service classifies the content category.
|
||||||
|
7. Service generates highlight candidates from visual, audio, transcript, and motion signals.
|
||||||
|
8. AI director reviews the compact analysis artifacts and creates a strict edit plan.
|
||||||
|
9. Asset providers create or select voiceover, music, SFX, captions, and overlay text.
|
||||||
|
10. Renderer executes the edit plan with FFmpeg and produces final highlight MP4s.
|
||||||
|
11. QA checks validate technical quality and creative completeness.
|
||||||
|
12. Final clips, manifests, storyboards, and logs are written to the output folder.
|
||||||
|
|
||||||
|
## Proposed Folder Contract
|
||||||
|
|
||||||
|
Input:
|
||||||
|
|
||||||
|
```text
|
||||||
|
input/highlights/source/<source-video-file>
|
||||||
|
```
|
||||||
|
|
||||||
|
Output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/
|
||||||
|
project.json
|
||||||
|
source/
|
||||||
|
analysis/
|
||||||
|
ffprobe.json
|
||||||
|
shots.json
|
||||||
|
scenes.json
|
||||||
|
transcript.json
|
||||||
|
audio-events.json
|
||||||
|
visual-analysis.json
|
||||||
|
category.json
|
||||||
|
highlight-candidates.json
|
||||||
|
director/
|
||||||
|
director-brief.md
|
||||||
|
director-prompt.md
|
||||||
|
edit-plan.json
|
||||||
|
storyboard.md
|
||||||
|
assets/
|
||||||
|
voiceover/
|
||||||
|
music/
|
||||||
|
sfx/
|
||||||
|
overlays/
|
||||||
|
highlights/
|
||||||
|
<highlight-id>/
|
||||||
|
final.mp4
|
||||||
|
preview.mp4
|
||||||
|
render-manifest.json
|
||||||
|
qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Content Categories
|
||||||
|
|
||||||
|
### Family Vlog
|
||||||
|
|
||||||
|
Creative intent:
|
||||||
|
|
||||||
|
- Warm, emotional, human, memory-focused.
|
||||||
|
- Preserve natural faces, voices, laughter, and real moments.
|
||||||
|
- Avoid over-editing or aggressive transitions.
|
||||||
|
- Use gentle music, soft cinematic color, light film grain, and meaningful text overlays.
|
||||||
|
|
||||||
|
Highlight signals:
|
||||||
|
|
||||||
|
- Smiles, hugs, reactions, children, group moments, travel reveal moments, laughter.
|
||||||
|
- Speech with emotional words or important context.
|
||||||
|
- Stable shots with faces and clear composition.
|
||||||
|
|
||||||
|
Voiceover style:
|
||||||
|
|
||||||
|
- Personal, warm, reflective, short.
|
||||||
|
- Should sound like a memory being narrated, not an advertisement.
|
||||||
|
|
||||||
|
### Food Vlog
|
||||||
|
|
||||||
|
Creative intent:
|
||||||
|
|
||||||
|
- Sensory, rhythmic, appetizing, tactile.
|
||||||
|
- Make ingredients, textures, steam, plating, and reactions feel premium.
|
||||||
|
- Sync cuts to cooking sounds and music beats.
|
||||||
|
|
||||||
|
Highlight signals:
|
||||||
|
|
||||||
|
- Close-ups of ingredients, cutting, sizzling, steam, pouring, plating, first bite, reaction.
|
||||||
|
- Strong texture, color, macro framing, hand movement, kitchen action.
|
||||||
|
|
||||||
|
Voiceover style:
|
||||||
|
|
||||||
|
- Descriptive and sensory.
|
||||||
|
- Mention texture, aroma, craft, freshness, and payoff.
|
||||||
|
|
||||||
|
### Car Vlog
|
||||||
|
|
||||||
|
Creative intent:
|
||||||
|
|
||||||
|
- Heroic, powerful, premium, dynamic.
|
||||||
|
- Emphasize stance, motion, engine sound, details, acceleration, road presence, reflections, wheels, badges, lights, interior, and reveal shots.
|
||||||
|
- Use speed ramps, bass hits, whooshes, cinematic grade, controlled text overlays, and strong music.
|
||||||
|
|
||||||
|
Highlight signals:
|
||||||
|
|
||||||
|
- Car fully visible, front/rear three-quarter angles, wheels turning, badge/detail shots, engine/exhaust audio, road motion, camera movement, dramatic lighting.
|
||||||
|
- Avoid repeated static hood shots unless used as setup.
|
||||||
|
|
||||||
|
Voiceover style:
|
||||||
|
|
||||||
|
- Confident, cinematic, concise.
|
||||||
|
- Should praise the vehicle specifically based on observed details, not use generic supercar phrases.
|
||||||
|
|
||||||
|
### Generic Vlog Fallback
|
||||||
|
|
||||||
|
Creative intent:
|
||||||
|
|
||||||
|
- Clear story arc, clean pacing, tasteful grade, and context-aware overlays.
|
||||||
|
- Use the best available moments even when the category is uncertain.
|
||||||
|
|
||||||
|
Highlight signals:
|
||||||
|
|
||||||
|
- High motion, strong composition, faces, speech peaks, laughter, visual novelty, scene changes.
|
||||||
|
|
||||||
|
## AI Director Responsibilities
|
||||||
|
|
||||||
|
The AI director should not render the video. It should produce a validated edit decision plan that the renderer can execute.
|
||||||
|
|
||||||
|
Required AI director inputs:
|
||||||
|
|
||||||
|
- Source metadata: duration, resolution, frame rate, codec, audio streams.
|
||||||
|
- Shot list with timestamps, representative thumbnails, motion score, blur score, brightness score, face/object labels, and transcript references.
|
||||||
|
- Audio analysis: speech segments, silence, peaks, loudness, detected audio events, music/noise/speech classification.
|
||||||
|
- Category classification with confidence and reasons.
|
||||||
|
- Highlight candidates with ranked evidence.
|
||||||
|
- Available music, SFX, voice, font, LUT, and transition capabilities.
|
||||||
|
|
||||||
|
Required AI director output:
|
||||||
|
|
||||||
|
- Highlight list with target duration and category.
|
||||||
|
- Timeline edit decisions with exact source timestamps.
|
||||||
|
- Music mood, energy curve, and beat-sync points.
|
||||||
|
- SFX cues with timestamps and purpose.
|
||||||
|
- Voiceover script split into timed lines.
|
||||||
|
- Text overlay copy, timing, animation, placement, and safe-area rules.
|
||||||
|
- Visual treatment: grade, transitions, speed ramps, stabilization, crop, punch-ins, grain, vignette, glow, blur, and overlays.
|
||||||
|
- Render profile: aspect ratio, resolution, FPS policy, codec, CRF or bitrate mode, loudness target.
|
||||||
|
|
||||||
|
## Renderer Requirements
|
||||||
|
|
||||||
|
The renderer must support these production features before the output can be called cinematic:
|
||||||
|
|
||||||
|
- Precise trims using source timestamps.
|
||||||
|
- Crossfades, match cuts, hard cuts, whip transitions, fade to black, and dip transitions.
|
||||||
|
- Speed ramps and short impact pauses.
|
||||||
|
- Dynamic crop and punch-in keyframes for static shots.
|
||||||
|
- Stabilization option for shaky footage.
|
||||||
|
- Category-specific color grade chains or LUT support.
|
||||||
|
- Film grain, vignette, sharpening, glow, blur, and selective contrast.
|
||||||
|
- Text overlays with font, position, animation timing, stroke, shadow, and safe area.
|
||||||
|
- Caption and subtitle rendering.
|
||||||
|
- Music bed placement with fade in/out.
|
||||||
|
- Voiceover placement with sidechain ducking.
|
||||||
|
- SFX placement for whooshes, impacts, risers, camera clicks, engine hits, cooking sounds, and ambient layers.
|
||||||
|
- Audio loudness normalization to a configured target.
|
||||||
|
- Final render manifest with all commands, source segments, assets, and durations.
|
||||||
|
|
||||||
|
## Asset Provider Requirements
|
||||||
|
|
||||||
|
The system needs explicit providers instead of pretending a prompt can create finished audio by itself.
|
||||||
|
|
||||||
|
- Voiceover provider: local TTS or remote TTS, configurable voice per category, output WAV.
|
||||||
|
- Music provider: licensed local library first, optional generated music provider later.
|
||||||
|
- SFX provider: licensed local SFX library with tags and category mapping.
|
||||||
|
- Font provider: local fonts with brand/category mapping.
|
||||||
|
- LUT provider: local LUT files or named FFmpeg grade presets.
|
||||||
|
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
|
||||||
|
|
||||||
|
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
|
||||||
|
|
||||||
|
## Highlight Scoring Model
|
||||||
|
|
||||||
|
Each candidate segment should be scored with a weighted model:
|
||||||
|
|
||||||
|
```text
|
||||||
|
score =
|
||||||
|
visual_interest
|
||||||
|
+ motion_energy
|
||||||
|
+ audio_interest
|
||||||
|
+ transcript_importance
|
||||||
|
+ category_relevance
|
||||||
|
+ composition_quality
|
||||||
|
+ novelty
|
||||||
|
- blur_penalty
|
||||||
|
- darkness_penalty
|
||||||
|
- repetition_penalty
|
||||||
|
- unusable_audio_penalty
|
||||||
|
```
|
||||||
|
|
||||||
|
Category-specific examples:
|
||||||
|
|
||||||
|
- Family vlog should boost faces, laughter, dialogue, emotional reactions, and group moments.
|
||||||
|
- Food vlog should boost close-ups, texture, cooking actions, plating, steam, and bite reactions.
|
||||||
|
- Car vlog should boost full-car hero shots, motion, wheels, engine audio, badges, roads, and dramatic lighting.
|
||||||
|
|
||||||
|
## Edit Plan Schema
|
||||||
|
|
||||||
|
The plan should be strict JSON so a cheaper model or deterministic renderer can execute it.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"projectId": "example-project",
|
||||||
|
"sourceVideo": "input/highlights/source/example.mp4",
|
||||||
|
"contentCategory": "car_vlog",
|
||||||
|
"highlights": [
|
||||||
|
{
|
||||||
|
"id": "hero-reveal-001",
|
||||||
|
"title": "The Reveal",
|
||||||
|
"targetDurationSeconds": 35,
|
||||||
|
"aspectRatio": "16:9",
|
||||||
|
"renderProfile": "cinematic_4k_h264",
|
||||||
|
"creativeIntent": "Premium Porsche reveal with dynamic pacing and confident narration.",
|
||||||
|
"timeline": [
|
||||||
|
{
|
||||||
|
"type": "video",
|
||||||
|
"sourceStart": "00:01:12.400",
|
||||||
|
"sourceEnd": "00:01:16.900",
|
||||||
|
"timelineStart": "00:00:00.000",
|
||||||
|
"effects": ["stabilize", "cinematic_warm_contrast", "slow_push_in"],
|
||||||
|
"transitionOut": "bass_hit_cut"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"voiceover": [
|
||||||
|
{
|
||||||
|
"text": "This is not just a drive. It is precision, presence, and control.",
|
||||||
|
"timelineStart": "00:00:04.200",
|
||||||
|
"voice": "confident_cinematic_male",
|
||||||
|
"delivery": "calm premium"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"music": {
|
||||||
|
"mood": "premium cinematic electronic",
|
||||||
|
"energyCurve": "slow_build_to_impact",
|
||||||
|
"assetId": "music/premium-drive-90bpm.wav"
|
||||||
|
},
|
||||||
|
"sfx": [
|
||||||
|
{
|
||||||
|
"assetId": "sfx/bass-hit-01.wav",
|
||||||
|
"timelineStart": "00:00:07.800",
|
||||||
|
"gainDb": -6
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"overlays": [
|
||||||
|
{
|
||||||
|
"text": "PRECISION IN MOTION",
|
||||||
|
"timelineStart": "00:00:08.000",
|
||||||
|
"durationSeconds": 2.5,
|
||||||
|
"placement": "lower_left_safe",
|
||||||
|
"animation": "fade_slide_up"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Milestones
|
||||||
|
|
||||||
|
- [ ] Milestone 1: Define the cinematic highlight project domain model, folder contract, and persisted manifests.
|
||||||
|
- [ ] Milestone 2: Add a local scheduler that accepts exactly one source video at a time from the highlight source folder.
|
||||||
|
- [ ] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
|
||||||
|
- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
||||||
|
- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
||||||
|
- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
||||||
|
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
|
||||||
|
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
|
||||||
|
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
|
||||||
|
- [ ] Milestone 10: Implement edit plan validation so impossible timestamps, missing assets, invalid overlays, and unsupported effects fail before render.
|
||||||
|
- [ ] Milestone 11: Implement asset provider interfaces for voiceover, music, SFX, fonts, and LUTs.
|
||||||
|
- [ ] Milestone 12: Add a local licensed asset library with category tags and deterministic asset selection.
|
||||||
|
- [ ] Milestone 13: Implement renderer v2 with transitions, speed ramps, dynamic crops, overlays, visual effects, audio ducking, loudness normalization, and render manifests.
|
||||||
|
- [ ] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters.
|
||||||
|
- [ ] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips.
|
||||||
|
- [ ] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content.
|
||||||
|
- [ ] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size.
|
||||||
|
- [ ] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering.
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
- Unit tests for category scoring, candidate ranking, schema validation, folder discovery, and manifest writing.
|
||||||
|
- Integration tests with short fixture videos that verify each category can produce at least one valid highlight plan.
|
||||||
|
- Renderer tests that verify FFmpeg commands include expected filters for transitions, overlays, color treatment, music, SFX, and voiceover.
|
||||||
|
- Golden-file tests for AI prompt inputs and expected JSON schema shape.
|
||||||
|
- End-to-end local test that places one video in the source folder and verifies final clips, manifests, storyboards, and QA reports are created.
|
||||||
|
- Negative tests for corrupt input video, unsupported codec, empty video, missing audio, missing asset, invalid timestamp, and renderer failure.
|
||||||
|
|
||||||
|
## Step-By-Step Local How-To Guide
|
||||||
|
|
||||||
|
This guide describes the intended operator workflow after the milestones are implemented.
|
||||||
|
|
||||||
|
### 1. Prepare The Machine
|
||||||
|
|
||||||
|
Install and verify required binaries:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ffmpeg -version
|
||||||
|
ffprobe -version
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional but recommended tools:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 --version
|
||||||
|
```
|
||||||
|
|
||||||
|
Recommended future optional components:
|
||||||
|
|
||||||
|
- PySceneDetect for higher-quality scene detection.
|
||||||
|
- Whisper or another ASR provider for transcript-based editing.
|
||||||
|
- A configured TTS provider for voiceover.
|
||||||
|
- A licensed local music and SFX library.
|
||||||
|
|
||||||
|
### 2. Configure The Application
|
||||||
|
|
||||||
|
Set the local cinematic highlight folders in `application.yml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
cinematic-highlights:
|
||||||
|
enabled: true
|
||||||
|
source-folder: input/highlights/source
|
||||||
|
output-folder: output/highlight-projects
|
||||||
|
scheduler-interval: 10s
|
||||||
|
max-active-projects: 1
|
||||||
|
default-aspect-ratio: "16:9"
|
||||||
|
default-render-profile: cinematic_1080p_h264
|
||||||
|
```
|
||||||
|
|
||||||
|
Configure the AI director model separately from the renderer:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
cinematic-highlights:
|
||||||
|
director:
|
||||||
|
enabled: true
|
||||||
|
model: gpt-5
|
||||||
|
allow-cheaper-model: true
|
||||||
|
cheaper-model: gpt-5-mini
|
||||||
|
require-human-approval-before-render: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Configure local assets:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
cinematic-highlights:
|
||||||
|
assets:
|
||||||
|
music-folder: input/highlights/assets/music
|
||||||
|
sfx-folder: input/highlights/assets/sfx
|
||||||
|
fonts-folder: input/highlights/assets/fonts
|
||||||
|
luts-folder: input/highlights/assets/luts
|
||||||
|
voiceover-folder: output/highlight-projects/_voiceover-cache
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Create Folders
|
||||||
|
|
||||||
|
Create the required local folders:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p input/highlights/source
|
||||||
|
mkdir -p input/highlights/assets/music
|
||||||
|
mkdir -p input/highlights/assets/sfx
|
||||||
|
mkdir -p input/highlights/assets/fonts
|
||||||
|
mkdir -p input/highlights/assets/luts
|
||||||
|
mkdir -p output/highlight-projects
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Add The Source Video
|
||||||
|
|
||||||
|
Place exactly one source video in the source folder:
|
||||||
|
|
||||||
|
```text
|
||||||
|
input/highlights/source/my-porsche-drive.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
The scheduler should process one source video at a time. If multiple videos are present, it should process the first valid file, finish it, then move to the next one.
|
||||||
|
|
||||||
|
Invalid files should be ignored and logged with the reason.
|
||||||
|
|
||||||
|
### 5. Start The Application
|
||||||
|
|
||||||
|
Run the Spring Boot service:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mvnw spring-boot:run
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected startup logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cinematic highlight scheduler started
|
||||||
|
Watching source folder: input/highlights/source
|
||||||
|
Output folder: output/highlight-projects
|
||||||
|
Scheduler interval: 10s
|
||||||
|
Max active projects: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Verify Project Creation
|
||||||
|
|
||||||
|
After the scheduler picks up the source file, verify a project folder exists:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected early files:
|
||||||
|
|
||||||
|
```text
|
||||||
|
project.json
|
||||||
|
analysis/ffprobe.json
|
||||||
|
analysis/shots.json
|
||||||
|
analysis/scenes.json
|
||||||
|
analysis/category.json
|
||||||
|
director/director-brief.md
|
||||||
|
director/director-prompt.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Started cinematic highlight project projectId=<project-id> source=my-porsche-drive.mp4
|
||||||
|
Generated proxy video
|
||||||
|
Extracted representative thumbnails
|
||||||
|
Detected shots and scenes
|
||||||
|
Classified content category category=car_vlog confidence=...
|
||||||
|
Generated highlight candidates count=...
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. Review The AI Director Prompt
|
||||||
|
|
||||||
|
Open the director prompt:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/director/director-prompt.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Check that it includes:
|
||||||
|
|
||||||
|
- Source metadata.
|
||||||
|
- Category classification.
|
||||||
|
- Contact sheet or thumbnail references.
|
||||||
|
- Candidate segments with timestamps.
|
||||||
|
- Available renderer effects.
|
||||||
|
- Available music, SFX, voice, font, and LUT assets.
|
||||||
|
- Strict JSON schema requirements.
|
||||||
|
- Category-specific creative direction.
|
||||||
|
|
||||||
|
If the prompt is generic, the final edit will probably be generic. The prompt must force concrete choices: exact cuts, specific overlays, voiceover lines, SFX timing, music mood, and visual treatment.
|
||||||
|
|
||||||
|
### 8. Execute The AI Director
|
||||||
|
|
||||||
|
Run the configured AI director against the prompt. The director must output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||||
|
```
|
||||||
|
|
||||||
|
If human approval is enabled, inspect the edit plan before rendering.
|
||||||
|
|
||||||
|
Minimum review checklist:
|
||||||
|
|
||||||
|
- The selected shots are not repetitive.
|
||||||
|
- Each highlight has a clear beginning, middle, and ending.
|
||||||
|
- The edit uses the correct category style.
|
||||||
|
- Voiceover text is specific to the observed video.
|
||||||
|
- Music and SFX cues are not placeholders.
|
||||||
|
- Text overlays are tasteful and not overused.
|
||||||
|
- All referenced timestamps are inside the source duration.
|
||||||
|
- All referenced assets exist.
|
||||||
|
|
||||||
|
### 9. Approve Or Correct The Edit Plan
|
||||||
|
|
||||||
|
If the plan is weak, edit or regenerate it before rendering.
|
||||||
|
|
||||||
|
Common fixes:
|
||||||
|
|
||||||
|
- Replace generic voiceover with concrete lines about the actual footage.
|
||||||
|
- Remove boring shots or repeated angles.
|
||||||
|
- Add stronger openers and endings.
|
||||||
|
- Add music beat cues.
|
||||||
|
- Add SFX only where they support a cut or reveal.
|
||||||
|
- Add overlay text only where it improves the scene.
|
||||||
|
- Adjust target duration if the highlight feels too slow.
|
||||||
|
|
||||||
|
When approved, write:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/director/approved.flag
|
||||||
|
```
|
||||||
|
|
||||||
|
### 10. Render The Highlights
|
||||||
|
|
||||||
|
The renderer consumes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected renderer output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Rendering highlight projectId=<project-id> highlightId=<highlight-id>
|
||||||
|
Applying video trims count=...
|
||||||
|
Applying transitions count=...
|
||||||
|
Applying visual effects count=...
|
||||||
|
Mixing music asset=...
|
||||||
|
Mixing voiceover lines=...
|
||||||
|
Mixing SFX count=...
|
||||||
|
Completed highlight render duration=... output=...
|
||||||
|
```
|
||||||
|
|
||||||
|
### 11. Review The Final Clips
|
||||||
|
|
||||||
|
For each `final.mp4`, verify:
|
||||||
|
|
||||||
|
- The first three seconds are interesting.
|
||||||
|
- The edit has intentional pacing.
|
||||||
|
- The selected shots fit the content type.
|
||||||
|
- Music fits the scene and is audible.
|
||||||
|
- Voiceover is clear and not robotic or generic.
|
||||||
|
- SFX support the edit without becoming noisy.
|
||||||
|
- Text overlays are readable and not cheap-looking.
|
||||||
|
- Color grade improves the footage without crushing detail.
|
||||||
|
- Audio is not clipping and does not have long silent sections.
|
||||||
|
|
||||||
|
If the final video is boring, inspect in this order:
|
||||||
|
|
||||||
|
1. `analysis/highlight-candidates.json`: weak source moments or wrong scoring.
|
||||||
|
2. `director/edit-plan.json`: weak AI choices, generic voiceover, no effects.
|
||||||
|
3. `render-manifest.json`: renderer ignored effects or missing assets.
|
||||||
|
4. `qa-report.json`: technical failures or missing audio/effects.
|
||||||
|
|
||||||
|
### 12. Find The Final Result
|
||||||
|
|
||||||
|
Final highlight videos are stored here:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||||
|
```
|
||||||
|
|
||||||
|
The full audit trail is stored beside the final video:
|
||||||
|
|
||||||
|
```text
|
||||||
|
render-manifest.json
|
||||||
|
qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The creative plan is stored here:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/director/storyboard.md
|
||||||
|
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### 13. Clean Up After A Run
|
||||||
|
|
||||||
|
After confirming the output is good, move the source video out of the source folder or let the scheduler move it to a processed folder:
|
||||||
|
|
||||||
|
```text
|
||||||
|
input/highlights/processed/<source-video-file>
|
||||||
|
```
|
||||||
|
|
||||||
|
Failed inputs should move to:
|
||||||
|
|
||||||
|
```text
|
||||||
|
input/highlights/failed/<source-video-file>
|
||||||
|
```
|
||||||
|
|
||||||
|
The scheduler must not repeatedly process the same source video.
|
||||||
|
|
||||||
|
## Definition Of Done
|
||||||
|
|
||||||
|
A highlight export is production-ready only when:
|
||||||
|
|
||||||
|
- The selected shots match the detected content category.
|
||||||
|
- The edit has a clear beginning, middle, and ending.
|
||||||
|
- Music is present, fitting, and timed to key cuts.
|
||||||
|
- Voiceover is present when requested and sounds appropriate for the category.
|
||||||
|
- SFX are present where they add impact and are not randomly overused.
|
||||||
|
- Text overlays are readable, visually appealing, correctly timed, and safe-area compliant.
|
||||||
|
- Visual treatment is visible but not destructive.
|
||||||
|
- Audio is normalized and free of clipping.
|
||||||
|
- Output includes a render manifest, edit plan, storyboard, and QA report.
|
||||||
|
- The final MP4 can be played independently without relying on temporary files.
|
||||||
|
|
||||||
|
## Key Risk
|
||||||
|
|
||||||
|
The previous boring output is expected if the system only creates a weak prompt and then renders simple trims. Cinematic output requires three things working together: richer multimodal analysis, a strict category-aware AI director plan, and a renderer that can actually execute music, SFX, voiceover, overlays, transitions, and visual effects.
|
||||||
|
|
@ -54,21 +54,23 @@ public class AiDirectorPromptGenerator {
|
||||||
return """
|
return """
|
||||||
# Local AI Director Assignment
|
# Local AI Director Assignment
|
||||||
|
|
||||||
You are the director for a premium cinematic Porsche promo.
|
You are the director for a production-ready cinematic highlight edit.
|
||||||
|
|
||||||
Work only with artifacts in this project folder:
|
Work only with artifacts in this project folder:
|
||||||
- Read `analysis.json` for authoritative clip metadata.
|
- Read `analysis.json` for authoritative clip metadata.
|
||||||
|
- Read `cinematic-highlight-analysis.json`, `category.json`, and `highlight-candidates.json` when present.
|
||||||
- Inspect every relevant image in `contact-sheets/` and `thumbnails/` before choosing shots.
|
- Inspect every relevant image in `contact-sheets/` and `thumbnails/` before choosing shots.
|
||||||
- Use `proxies/` only when motion needs closer inspection.
|
- Use `proxies/` only when motion needs closer inspection.
|
||||||
|
|
||||||
Complete this task:
|
Complete this task:
|
||||||
1. Judge visual quality, composition, motion, continuity, and story value from the generated artifacts.
|
1. Judge visual quality, composition, motion, continuity, audio opportunity, and story value from the generated artifacts.
|
||||||
2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final hero shot.
|
2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final memorable shot.
|
||||||
3. Follow the strict JSON schema and constraints in the brief below.
|
3. Follow the strict JSON schema and constraints in the brief below.
|
||||||
4. Write the final JSON object to `inbox/%s`.
|
4. Write the final JSON object to `inbox/%s`.
|
||||||
|
|
||||||
Do not render video. Do not run FFmpeg. Do not modify source clips or generated analysis artifacts.
|
Do not render video. Do not run FFmpeg. Do not modify source clips or generated analysis artifacts.
|
||||||
Do not invent clip IDs, source ranges, durations, or media paths.
|
Do not invent clip IDs, source ranges, durations, or media paths.
|
||||||
|
Do not write generic voiceover. The narration, text overlays, music mood, SFX, and visual effects must fit the detected content category.
|
||||||
Your task is complete only when `inbox/%s` contains valid JSON with no Markdown fences.
|
Your task is complete only when `inbox/%s` contains valid JSON with no Markdown fences.
|
||||||
|
|
||||||
%s
|
%s
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public record CinematicHighlightAnalysis(
|
||||||
|
String projectId,
|
||||||
|
ContentCategory category,
|
||||||
|
double categoryConfidence,
|
||||||
|
List<String> categoryReasons,
|
||||||
|
List<HighlightCandidate> candidates,
|
||||||
|
Instant createdAt
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,157 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.time.Clock;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class CinematicHighlightAnalyzer {
|
||||||
|
|
||||||
|
private static final double DEFAULT_CANDIDATE_SECONDS = 8.0;
|
||||||
|
private static final int MAX_CANDIDATES_PER_CLIP = 12;
|
||||||
|
|
||||||
|
private final Clock clock;
|
||||||
|
|
||||||
|
public CinematicHighlightAnalyzer() {
|
||||||
|
this(Clock.systemUTC());
|
||||||
|
}
|
||||||
|
|
||||||
|
CinematicHighlightAnalyzer(Clock clock) {
|
||||||
|
this.clock = clock;
|
||||||
|
}
|
||||||
|
|
||||||
|
public CinematicHighlightAnalysis analyze(EditProject project, EditProjectAnalysis analysis) {
|
||||||
|
CategoryDecision category = classify(project, analysis);
|
||||||
|
List<HighlightCandidate> candidates = analysis.clips().stream()
|
||||||
|
.flatMap(clip -> candidatesForClip(clip, category.category()).stream())
|
||||||
|
.sorted(Comparator.comparingDouble(HighlightCandidate::score).reversed()
|
||||||
|
.thenComparing(HighlightCandidate::clipId)
|
||||||
|
.thenComparingDouble(HighlightCandidate::sourceStartSeconds))
|
||||||
|
.toList();
|
||||||
|
return new CinematicHighlightAnalysis(project.id(), category.category(), category.confidence(),
|
||||||
|
category.reasons(), candidates, Instant.now(clock));
|
||||||
|
}
|
||||||
|
|
||||||
|
private CategoryDecision classify(EditProject project, EditProjectAnalysis analysis) {
|
||||||
|
String haystack = (project.name() + " " + project.style() + " "
|
||||||
|
+ analysis.clips().stream()
|
||||||
|
.map(clip -> clip.clipId() + " " + clip.sourcePath())
|
||||||
|
.reduce("", (left, right) -> left + " " + right))
|
||||||
|
.toLowerCase(Locale.ROOT);
|
||||||
|
|
||||||
|
if (containsAny(haystack, "porsche", "car", "auto", "vehicle", "drive", "driving", "engine", "wheel")) {
|
||||||
|
return new CategoryDecision(ContentCategory.CAR_VLOG, 0.78,
|
||||||
|
List.of("Project metadata suggests vehicle or driving content."));
|
||||||
|
}
|
||||||
|
if (containsAny(haystack, "food", "cook", "cooking", "recipe", "kitchen", "restaurant", "meal", "dish")) {
|
||||||
|
return new CategoryDecision(ContentCategory.FOOD_VLOG, 0.74,
|
||||||
|
List.of("Project metadata suggests food, cooking, or restaurant content."));
|
||||||
|
}
|
||||||
|
if (containsAny(haystack, "family", "kids", "child", "children", "birthday", "holiday", "vacation", "vlog")) {
|
||||||
|
return new CategoryDecision(ContentCategory.FAMILY_VLOG, 0.68,
|
||||||
|
List.of("Project metadata suggests personal or family vlog content."));
|
||||||
|
}
|
||||||
|
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.45,
|
||||||
|
List.of("No strong category-specific metadata was available; using generic vlog direction."));
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean containsAny(String haystack, String... needles) {
|
||||||
|
for (String needle : needles) {
|
||||||
|
if (haystack.contains(needle)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<HighlightCandidate> candidatesForClip(ClipAnalysis clip, ContentCategory category) {
|
||||||
|
double duration = clip.durationSeconds();
|
||||||
|
if (duration <= 0) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
|
||||||
|
double segmentSeconds = Math.min(DEFAULT_CANDIDATE_SECONDS, duration);
|
||||||
|
int segmentCount = Math.max(1, (int) Math.ceil(duration / segmentSeconds));
|
||||||
|
int cappedCount = Math.min(segmentCount, MAX_CANDIDATES_PER_CLIP);
|
||||||
|
double step = segmentCount <= MAX_CANDIDATES_PER_CLIP
|
||||||
|
? segmentSeconds
|
||||||
|
: Math.max(1.0, (duration - segmentSeconds) / (MAX_CANDIDATES_PER_CLIP - 1));
|
||||||
|
|
||||||
|
List<HighlightCandidate> candidates = new ArrayList<>();
|
||||||
|
for (int index = 0; index < cappedCount; index++) {
|
||||||
|
double start = Math.min(duration - segmentSeconds, index * step);
|
||||||
|
double end = Math.min(duration, start + segmentSeconds);
|
||||||
|
candidates.add(new HighlightCandidate(
|
||||||
|
"%s_candidate_%02d".formatted(clip.clipId(), index + 1),
|
||||||
|
clip.clipId(),
|
||||||
|
round(start),
|
||||||
|
round(end),
|
||||||
|
round(score(clip, category, index, cappedCount)),
|
||||||
|
role(index, cappedCount),
|
||||||
|
reasons(clip, category, index, cappedCount)
|
||||||
|
));
|
||||||
|
}
|
||||||
|
return candidates;
|
||||||
|
}
|
||||||
|
|
||||||
|
private double score(ClipAnalysis clip, ContentCategory category, int index, int count) {
|
||||||
|
double quality = Math.max(0.1, clip.sharpnessScore()) + Math.max(0.1, clip.brightnessScore())
|
||||||
|
+ Math.max(0.1, clip.motionScore());
|
||||||
|
double position = switch (role(index, count)) {
|
||||||
|
case "opening_hook" -> 0.25;
|
||||||
|
case "final_hero" -> 0.22;
|
||||||
|
case "rising_action" -> 0.16;
|
||||||
|
default -> 0.1;
|
||||||
|
};
|
||||||
|
double categoryBoost = switch (category) {
|
||||||
|
case CAR_VLOG -> 0.2;
|
||||||
|
case FOOD_VLOG -> 0.16;
|
||||||
|
case FAMILY_VLOG -> 0.14;
|
||||||
|
case GENERIC_VLOG -> 0.08;
|
||||||
|
};
|
||||||
|
return Math.min(1.0, 0.35 + quality * 0.12 + position + categoryBoost);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String role(int index, int count) {
|
||||||
|
if (index == 0) {
|
||||||
|
return "opening_hook";
|
||||||
|
}
|
||||||
|
if (index == count - 1) {
|
||||||
|
return "final_hero";
|
||||||
|
}
|
||||||
|
if (index <= Math.max(1, count / 3)) {
|
||||||
|
return "rising_action";
|
||||||
|
}
|
||||||
|
return "detail_or_bridge";
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<String> reasons(ClipAnalysis clip, ContentCategory category, int index, int count) {
|
||||||
|
List<String> reasons = new ArrayList<>();
|
||||||
|
reasons.add("Suggested as " + role(index, count).replace('_', ' ') + " based on source position.");
|
||||||
|
reasons.add("Duration window is short enough for fast cinematic pacing.");
|
||||||
|
if (clip.motionScore() > 0) {
|
||||||
|
reasons.add("Motion score is available for director ranking.");
|
||||||
|
}
|
||||||
|
reasons.add(switch (category) {
|
||||||
|
case CAR_VLOG -> "For car content, inspect this window for hero angles, wheels, badges, engine sound, or road motion.";
|
||||||
|
case FOOD_VLOG -> "For food content, inspect this window for texture, preparation, plating, steam, or bite reactions.";
|
||||||
|
case FAMILY_VLOG -> "For family content, inspect this window for faces, reactions, laughter, hugs, or emotional dialogue.";
|
||||||
|
case GENERIC_VLOG -> "For generic vlog content, inspect this window for visual novelty, clear action, or useful story context.";
|
||||||
|
});
|
||||||
|
return List.copyOf(reasons);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double round(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private record CategoryDecision(ContentCategory category, double confidence, List<String> reasons) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
public enum ContentCategory {
|
||||||
|
FAMILY_VLOG,
|
||||||
|
FOOD_VLOG,
|
||||||
|
CAR_VLOG,
|
||||||
|
GENERIC_VLOG
|
||||||
|
}
|
||||||
|
|
@ -21,6 +21,7 @@ public class EditProjectAnalyzer {
|
||||||
private final ThumbnailExtractor thumbnailExtractor;
|
private final ThumbnailExtractor thumbnailExtractor;
|
||||||
private final ContactSheetGenerator contactSheetGenerator;
|
private final ContactSheetGenerator contactSheetGenerator;
|
||||||
private final ProxyGenerator proxyGenerator;
|
private final ProxyGenerator proxyGenerator;
|
||||||
|
private final CinematicHighlightAnalyzer cinematicHighlightAnalyzer;
|
||||||
private final Clock clock;
|
private final Clock clock;
|
||||||
private final EditObservability observability;
|
private final EditObservability observability;
|
||||||
|
|
||||||
|
|
@ -32,10 +33,11 @@ public class EditProjectAnalyzer {
|
||||||
ThumbnailExtractor thumbnailExtractor,
|
ThumbnailExtractor thumbnailExtractor,
|
||||||
ContactSheetGenerator contactSheetGenerator,
|
ContactSheetGenerator contactSheetGenerator,
|
||||||
ProxyGenerator proxyGenerator,
|
ProxyGenerator proxyGenerator,
|
||||||
|
CinematicHighlightAnalyzer cinematicHighlightAnalyzer,
|
||||||
EditObservability observability
|
EditObservability observability
|
||||||
) {
|
) {
|
||||||
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator,
|
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator,
|
||||||
Clock.systemUTC(), observability);
|
cinematicHighlightAnalyzer, Clock.systemUTC(), observability);
|
||||||
}
|
}
|
||||||
|
|
||||||
EditProjectAnalyzer(
|
EditProjectAnalyzer(
|
||||||
|
|
@ -47,13 +49,14 @@ public class EditProjectAnalyzer {
|
||||||
ProxyGenerator proxyGenerator,
|
ProxyGenerator proxyGenerator,
|
||||||
Clock clock
|
Clock clock
|
||||||
) {
|
) {
|
||||||
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, clock, null);
|
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, null, clock, null);
|
||||||
}
|
}
|
||||||
|
|
||||||
EditProjectAnalyzer(
|
EditProjectAnalyzer(
|
||||||
EditProjectStore store, EditClipDiscovery discovery, FfmpegClipInspector inspector,
|
EditProjectStore store, EditClipDiscovery discovery, FfmpegClipInspector inspector,
|
||||||
ThumbnailExtractor thumbnailExtractor, ContactSheetGenerator contactSheetGenerator,
|
ThumbnailExtractor thumbnailExtractor, ContactSheetGenerator contactSheetGenerator,
|
||||||
ProxyGenerator proxyGenerator, Clock clock, EditObservability observability
|
ProxyGenerator proxyGenerator, CinematicHighlightAnalyzer cinematicHighlightAnalyzer, Clock clock,
|
||||||
|
EditObservability observability
|
||||||
) {
|
) {
|
||||||
this.store = store;
|
this.store = store;
|
||||||
this.discovery = discovery;
|
this.discovery = discovery;
|
||||||
|
|
@ -61,6 +64,7 @@ public class EditProjectAnalyzer {
|
||||||
this.thumbnailExtractor = thumbnailExtractor;
|
this.thumbnailExtractor = thumbnailExtractor;
|
||||||
this.contactSheetGenerator = contactSheetGenerator;
|
this.contactSheetGenerator = contactSheetGenerator;
|
||||||
this.proxyGenerator = proxyGenerator;
|
this.proxyGenerator = proxyGenerator;
|
||||||
|
this.cinematicHighlightAnalyzer = cinematicHighlightAnalyzer;
|
||||||
this.clock = clock;
|
this.clock = clock;
|
||||||
this.observability = observability;
|
this.observability = observability;
|
||||||
}
|
}
|
||||||
|
|
@ -100,6 +104,7 @@ public class EditProjectAnalyzer {
|
||||||
EditProjectAnalysis analysis = new EditProjectAnalysis(projectId, List.copyOf(clips), List.copyOf(errors),
|
EditProjectAnalysis analysis = new EditProjectAnalysis(projectId, List.copyOf(clips), List.copyOf(errors),
|
||||||
Instant.now(clock));
|
Instant.now(clock));
|
||||||
store.writeJson(projectId, "analysis.json", analysis);
|
store.writeJson(projectId, "analysis.json", analysis);
|
||||||
|
writeCinematicHighlightArtifacts(projectId, analysis);
|
||||||
if (observability != null) {
|
if (observability != null) {
|
||||||
observability.analysisCompleted(projectId, System.nanoTime() - startedAt, clips.size(), errors.size());
|
observability.analysisCompleted(projectId, System.nanoTime() - startedAt, clips.size(), errors.size());
|
||||||
}
|
}
|
||||||
|
|
@ -133,4 +138,15 @@ public class EditProjectAnalyzer {
|
||||||
inspected.sharpnessScore()
|
inspected.sharpnessScore()
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void writeCinematicHighlightArtifacts(String projectId, EditProjectAnalysis analysis) {
|
||||||
|
if (cinematicHighlightAnalyzer == null) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
EditProject project = store.readJson(projectId, "project.json", EditProject.class);
|
||||||
|
CinematicHighlightAnalysis cinematic = cinematicHighlightAnalyzer.analyze(project, analysis);
|
||||||
|
store.writeJson(projectId, "cinematic-highlight-analysis.json", cinematic);
|
||||||
|
store.writeJson(projectId, "category.json", cinematic);
|
||||||
|
store.writeJson(projectId, "highlight-candidates.json", cinematic.candidates());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public record HighlightCandidate(
|
||||||
|
String id,
|
||||||
|
String clipId,
|
||||||
|
double sourceStartSeconds,
|
||||||
|
double sourceEndSeconds,
|
||||||
|
double score,
|
||||||
|
String suggestedRole,
|
||||||
|
List<String> reasons
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -160,7 +160,7 @@ public class LocalDirectorScheduler {
|
||||||
try {
|
try {
|
||||||
EditProjectResponse project = projectService.createProject(new CreateEditProjectRequest(
|
EditProjectResponse project = projectService.createProject(new CreateEditProjectRequest(
|
||||||
working.getFileName().toString(), working.toString(), editing.getTargetDurationSeconds(),
|
working.getFileName().toString(), working.toString(), editing.getTargetDurationSeconds(),
|
||||||
"cinematic-porsche-promo", true, true, true));
|
"category-aware-cinematic-highlights", true, true, true));
|
||||||
projectId = project.projectId();
|
projectId = project.projectId();
|
||||||
projectService.updateProject(projectId, EditProjectStatus.ANALYZING, working, null);
|
projectService.updateProject(projectId, EditProjectStatus.ANALYZING, working, null);
|
||||||
EditProjectAnalysis analysis = analyzer.analyze(projectId, working);
|
EditProjectAnalysis analysis = analyzer.analyze(projectId, working);
|
||||||
|
|
|
||||||
|
|
@ -43,6 +43,7 @@ public class StoryboardPromptGenerator {
|
||||||
}
|
}
|
||||||
|
|
||||||
String prompt(EditProject project, EditProjectAnalysis analysis) {
|
String prompt(EditProject project, EditProjectAnalysis analysis) {
|
||||||
|
CinematicHighlightAnalysis cinematic = readCinematicAnalysis(project.id());
|
||||||
StringBuilder clips = new StringBuilder();
|
StringBuilder clips = new StringBuilder();
|
||||||
for (ClipAnalysis clip : analysis.clips()) {
|
for (ClipAnalysis clip : analysis.clips()) {
|
||||||
clips.append("- clipId: ").append(clip.clipId()).append('\n')
|
clips.append("- clipId: ").append(clip.clipId()).append('\n')
|
||||||
|
|
@ -54,6 +55,10 @@ public class StoryboardPromptGenerator {
|
||||||
.append(" contactSheet: ").append(reference("contact-sheets", clip.contactSheet())).append('\n')
|
.append(" contactSheet: ").append(reference("contact-sheets", clip.contactSheet())).append('\n')
|
||||||
.append(" proxy: ").append(reference("proxies", clip.proxyPath())).append('\n');
|
.append(" proxy: ").append(reference("proxies", clip.proxyPath())).append('\n');
|
||||||
}
|
}
|
||||||
|
String category = cinematic == null ? "unknown" : cinematic.category().name().toLowerCase();
|
||||||
|
String candidates = cinematic == null ? "No precomputed highlight candidates. Inspect all clips carefully."
|
||||||
|
: candidateBrief(cinematic);
|
||||||
|
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
||||||
|
|
||||||
return """
|
return """
|
||||||
# Cinematic Storyboard Director Brief
|
# Cinematic Storyboard Director Brief
|
||||||
|
|
@ -66,18 +71,29 @@ public class StoryboardPromptGenerator {
|
||||||
- Voiceover enabled: %s
|
- Voiceover enabled: %s
|
||||||
- Music enabled: %s
|
- Music enabled: %s
|
||||||
- Sound effects enabled: %s
|
- Sound effects enabled: %s
|
||||||
|
- Detected content category: `%s`
|
||||||
- Use only the clip IDs and source ranges listed below.
|
- Use only the clip IDs and source ranges listed below.
|
||||||
- Keep every source range within its clip duration.
|
- Keep every source range within its clip duration.
|
||||||
- Timeline ranges must be chronological and must not overlap.
|
- Timeline ranges must be chronological and must not overlap.
|
||||||
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
||||||
|
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
|
||||||
|
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
|
||||||
|
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
|
||||||
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
||||||
|
|
||||||
|
Category-specific direction:
|
||||||
|
%s
|
||||||
|
|
||||||
|
Ranked highlight candidates:
|
||||||
|
%s
|
||||||
|
|
||||||
Available clips and project-relative visual references:
|
Available clips and project-relative visual references:
|
||||||
%s
|
%s
|
||||||
Required JSON object shape:
|
Required JSON object shape:
|
||||||
{
|
{
|
||||||
"projectId": "%s",
|
"projectId": "%s",
|
||||||
"style": "%s",
|
"style": "%s",
|
||||||
|
"contentCategory": "%s",
|
||||||
"targetDurationSeconds": %d,
|
"targetDurationSeconds": %d,
|
||||||
"decisions": [{
|
"decisions": [{
|
||||||
"clipId": "clip_00001",
|
"clipId": "clip_00001",
|
||||||
|
|
@ -93,25 +109,90 @@ public class StoryboardPromptGenerator {
|
||||||
}],
|
}],
|
||||||
"audioCues": [{
|
"audioCues": [{
|
||||||
"type": "music|sfx",
|
"type": "music|sfx",
|
||||||
"assetKey": "descriptive asset name",
|
"assetKey": "concrete music or sfx asset description",
|
||||||
"timelineStartSeconds": 0.0,
|
"timelineStartSeconds": 0.0,
|
||||||
"timelineEndSeconds": 2.5,
|
"timelineEndSeconds": 2.5,
|
||||||
"gainDb": -12.0,
|
"gainDb": -12.0,
|
||||||
"notes": "story purpose"
|
"notes": "story purpose"
|
||||||
}],
|
}],
|
||||||
"voiceover": [{
|
"voiceover": [{
|
||||||
"text": "narration",
|
"text": "specific narration based on visible footage",
|
||||||
"timelineStartSeconds": 0.0,
|
"timelineStartSeconds": 0.0,
|
||||||
"timelineEndSeconds": 2.5,
|
"timelineEndSeconds": 2.5,
|
||||||
"delivery": "confident cinematic narrator"
|
"delivery": "confident cinematic narrator"
|
||||||
}],
|
}],
|
||||||
|
"overlays": [{
|
||||||
|
"text": "short premium overlay text",
|
||||||
|
"timelineStartSeconds": 0.0,
|
||||||
|
"timelineEndSeconds": 2.5,
|
||||||
|
"placement": "lower_left_safe|center_safe|lower_center_safe",
|
||||||
|
"animation": "fade_slide_up|fade_in|none",
|
||||||
|
"reason": "why this text improves the scene"
|
||||||
|
}],
|
||||||
"renderProfile": "mp4-h264-aac-1080p",
|
"renderProfile": "mp4-h264-aac-1080p",
|
||||||
"summary": "short creative rationale"
|
"summary": "short creative rationale"
|
||||||
}
|
}
|
||||||
""".formatted(
|
""".formatted(
|
||||||
project.id(), project.style(), project.targetDurationSeconds(),
|
project.id(), project.style(), project.targetDurationSeconds(),
|
||||||
project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(),
|
project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(), category,
|
||||||
clips.toString().stripTrailing(), project.id(), project.style(), project.targetDurationSeconds());
|
categoryDirection, candidates, clips.toString().stripTrailing(), project.id(), project.style(),
|
||||||
|
category, project.targetDurationSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) {
|
||||||
|
if (!Files.exists(store.projectDirectory(projectId).resolve("cinematic-highlight-analysis.json"))) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return store.readJson(projectId, "cinematic-highlight-analysis.json", CinematicHighlightAnalysis.class);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String candidateBrief(CinematicHighlightAnalysis cinematic) {
|
||||||
|
StringBuilder builder = new StringBuilder();
|
||||||
|
builder.append("- categoryConfidence: ").append(cinematic.categoryConfidence()).append('\n');
|
||||||
|
builder.append("- categoryReasons: ").append(cinematic.categoryReasons()).append('\n');
|
||||||
|
for (HighlightCandidate candidate : cinematic.candidates()) {
|
||||||
|
builder.append("- candidateId: ").append(candidate.id()).append('\n')
|
||||||
|
.append(" clipId: ").append(candidate.clipId()).append('\n')
|
||||||
|
.append(" sourceStartSeconds: ").append(candidate.sourceStartSeconds()).append('\n')
|
||||||
|
.append(" sourceEndSeconds: ").append(candidate.sourceEndSeconds()).append('\n')
|
||||||
|
.append(" score: ").append(candidate.score()).append('\n')
|
||||||
|
.append(" suggestedRole: ").append(candidate.suggestedRole()).append('\n')
|
||||||
|
.append(" reasons: ").append(candidate.reasons()).append('\n');
|
||||||
|
}
|
||||||
|
return builder.toString().stripTrailing();
|
||||||
|
}
|
||||||
|
|
||||||
|
private String categoryDirection(ContentCategory category) {
|
||||||
|
return switch (category) {
|
||||||
|
case CAR_VLOG -> """
|
||||||
|
- Make the edit feel premium, powerful, and precise.
|
||||||
|
- Prioritize hero angles, wheels, badges, motion, reflections, roads, lights, interior details, and engine or exhaust moments.
|
||||||
|
- Use speed ramps, bass-hit cuts, whooshes, cinematic contrast, subtle grain, and confident short overlay text when the footage supports it.
|
||||||
|
- Voiceover should praise the specific car footage, not make unsupported claims.
|
||||||
|
""";
|
||||||
|
case FOOD_VLOG -> """
|
||||||
|
- Make the edit sensory, warm, rhythmic, and appetizing.
|
||||||
|
- Prioritize texture, slicing, pouring, sizzling, steam, plating, table reveals, and first-bite reactions.
|
||||||
|
- Use music with light groove, tactile SFX, warm grade, macro-feeling punch-ins, and short ingredient or payoff overlays.
|
||||||
|
- Voiceover should describe texture, craft, freshness, and payoff without sounding like a generic restaurant ad.
|
||||||
|
""";
|
||||||
|
case FAMILY_VLOG -> """
|
||||||
|
- Make the edit warm, human, emotional, and memory-focused.
|
||||||
|
- Prioritize faces, reactions, laughter, hugs, children, travel reveals, and meaningful dialogue.
|
||||||
|
- Use gentle music, soft grade, light grain, natural pacing, and minimal text overlays.
|
||||||
|
- Voiceover should feel personal and reflective, not promotional.
|
||||||
|
""";
|
||||||
|
case GENERIC_VLOG -> genericDirection();
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private String genericDirection() {
|
||||||
|
return """
|
||||||
|
- Build a clear mini-story from the strongest available moments.
|
||||||
|
- Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes.
|
||||||
|
- Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story.
|
||||||
|
- Voiceover must explain what is actually happening in the footage.
|
||||||
|
""";
|
||||||
}
|
}
|
||||||
|
|
||||||
private String references(String directory, List<String> paths) {
|
private String references(String directory, List<String> paths) {
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@ video-clipping:
|
||||||
enabled: true
|
enabled: true
|
||||||
local-director:
|
local-director:
|
||||||
enabled: true
|
enabled: true
|
||||||
auto-render-when-plan-appears: true
|
auto-render-when-plan-appears: false
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level:
|
level:
|
||||||
|
|
|
||||||
|
|
@ -30,12 +30,15 @@ class AiDirectorPromptGeneratorTest {
|
||||||
String prompt = Files.readString(Path.of(files.promptPath()));
|
String prompt = Files.readString(Path.of(files.promptPath()));
|
||||||
assertThat(prompt)
|
assertThat(prompt)
|
||||||
.contains("`analysis.json`")
|
.contains("`analysis.json`")
|
||||||
|
.contains("`cinematic-highlight-analysis.json`")
|
||||||
.contains("`contact-sheets/`")
|
.contains("`contact-sheets/`")
|
||||||
.contains("`thumbnails/`")
|
.contains("`thumbnails/`")
|
||||||
.contains("`inbox/edit-plan.json`")
|
.contains("`inbox/edit-plan.json`")
|
||||||
.contains("Do not render video")
|
.contains("Do not render video")
|
||||||
|
.contains("production-ready cinematic highlight edit")
|
||||||
.contains("clipId: clip_00001")
|
.contains("clipId: clip_00001")
|
||||||
.contains("\"decisions\"");
|
.contains("\"decisions\"")
|
||||||
|
.doesNotContain("premium cinematic Porsche promo");
|
||||||
assertThat(Files.readString(Path.of(files.readmePath())))
|
assertThat(Files.readString(Path.of(files.readmePath())))
|
||||||
.contains("Codex, Claude")
|
.contains("Codex, Claude")
|
||||||
.contains("`inbox/edit-plan.json`");
|
.contains("`inbox/edit-plan.json`");
|
||||||
|
|
|
||||||
|
|
@ -87,7 +87,7 @@ class CinematicEditingIntegrationTest {
|
||||||
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
||||||
"cinematic grade", "integration sequence"))
|
"cinematic grade", "integration sequence"))
|
||||||
.toList();
|
.toList();
|
||||||
return new EditPlan(projectId, "cinematic-porsche-promo", 60, decisions, List.of(), List.of(),
|
return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(),
|
||||||
"mp4-h264-aac-320p", "Generated integration edit");
|
"mp4-h264-aac-320p", "Generated integration edit");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,74 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.time.Clock;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.time.ZoneOffset;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class CinematicHighlightAnalyzerTest {
|
||||||
|
|
||||||
|
private final CinematicHighlightAnalyzer analyzer = new CinematicHighlightAnalyzer(
|
||||||
|
Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC));
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void classifiesCarContentAndCreatesRankedCandidates() {
|
||||||
|
EditProject project = project("Porsche mountain drive", "category-aware-cinematic-highlights");
|
||||||
|
EditProjectAnalysis source = analysis(clip("porsche-drive.mp4", 45.0));
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis result = analyzer.analyze(project, source);
|
||||||
|
|
||||||
|
assertThat(result.projectId()).isEqualTo("project-1");
|
||||||
|
assertThat(result.category()).isEqualTo(ContentCategory.CAR_VLOG);
|
||||||
|
assertThat(result.categoryConfidence()).isGreaterThan(0.7);
|
||||||
|
assertThat(result.categoryReasons()).isNotEmpty();
|
||||||
|
assertThat(result.candidates()).hasSizeGreaterThan(1);
|
||||||
|
assertThat(result.candidates().get(0).score()).isGreaterThanOrEqualTo(result.candidates().get(1).score());
|
||||||
|
assertThat(result.candidates()).allSatisfy(candidate -> {
|
||||||
|
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
|
||||||
|
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(45.0);
|
||||||
|
assertThat(candidate.reasons()).anyMatch(reason -> reason.contains("car content"));
|
||||||
|
});
|
||||||
|
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void classifiesFoodAndFamilyFromProjectMetadata() {
|
||||||
|
assertThat(analyzer.analyze(project("Street food recipe", "cinematic"), analysis(clip("clip.mp4", 8)))
|
||||||
|
.category()).isEqualTo(ContentCategory.FOOD_VLOG);
|
||||||
|
assertThat(analyzer.analyze(project("Family birthday vlog", "cinematic"), analysis(clip("clip.mp4", 8)))
|
||||||
|
.category()).isEqualTo(ContentCategory.FAMILY_VLOG);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void fallsBackToGenericWhenCategoryIsUnknown() {
|
||||||
|
CinematicHighlightAnalysis result = analyzer.analyze(project("Untitled", "cinematic"),
|
||||||
|
analysis(clip("scene.mp4", 5.0)));
|
||||||
|
|
||||||
|
assertThat(result.category()).isEqualTo(ContentCategory.GENERIC_VLOG);
|
||||||
|
assertThat(result.candidates()).singleElement().satisfies(candidate -> {
|
||||||
|
assertThat(candidate.sourceStartSeconds()).isEqualTo(0.0);
|
||||||
|
assertThat(candidate.sourceEndSeconds()).isEqualTo(5.0);
|
||||||
|
assertThat(candidate.suggestedRole()).isEqualTo("opening_hook");
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private EditProject project(String name, String style) {
|
||||||
|
Instant now = Instant.parse("2026-07-10T10:00:00Z");
|
||||||
|
return new EditProject("project-1", name, EditProjectStatus.ANALYZED, "input", "output",
|
||||||
|
60, style, true, true, true, now, now, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private EditProjectAnalysis analysis(ClipAnalysis clip) {
|
||||||
|
return new EditProjectAnalysis("project-1", List.of(clip), List.of(),
|
||||||
|
Instant.parse("2026-07-10T10:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private ClipAnalysis clip(String sourcePath, double durationSeconds) {
|
||||||
|
return new ClipAnalysis("clip_00001", sourcePath, durationSeconds, "h264", "aac",
|
||||||
|
1920, 1080, 30.0, List.of(), null, null, 0.7, 0.8, 0.9);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -60,6 +60,49 @@ class EditProjectAnalyzerTest {
|
||||||
assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis);
|
assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void writesCinematicHighlightArtifactsWhenEnabled() throws Exception {
|
||||||
|
Path input = Files.createDirectory(tempDir.resolve("porsche-input"));
|
||||||
|
Path clip = Files.writeString(input.resolve("porsche-drive.mp4"), "video");
|
||||||
|
FileSystemEditProjectStore store = store();
|
||||||
|
store.createProject("project-1");
|
||||||
|
writeProject(store, input, "Porsche Drive");
|
||||||
|
EditClipDiscovery discovery = mock(EditClipDiscovery.class);
|
||||||
|
FfmpegClipInspector inspector = mock(FfmpegClipInspector.class);
|
||||||
|
ThumbnailExtractor thumbnailExtractor = mock(ThumbnailExtractor.class);
|
||||||
|
ContactSheetGenerator contactSheetGenerator = mock(ContactSheetGenerator.class);
|
||||||
|
ProxyGenerator proxyGenerator = mock(ProxyGenerator.class);
|
||||||
|
ClipAnalysis inspected = clipAnalysis(clip);
|
||||||
|
when(discovery.discover(input)).thenReturn(List.of(clip));
|
||||||
|
when(inspector.inspect(clip)).thenReturn(inspected);
|
||||||
|
when(thumbnailExtractor.extract(inspected, tempDir.resolve("projects/project-1/thumbnails")))
|
||||||
|
.thenReturn(List.of("thumb-1.jpg"));
|
||||||
|
when(contactSheetGenerator.generate(inspected, tempDir.resolve("projects/project-1/contact-sheets")))
|
||||||
|
.thenReturn("contact-sheet.jpg");
|
||||||
|
when(proxyGenerator.generate(inspected, tempDir.resolve("projects/project-1/proxies")))
|
||||||
|
.thenReturn(Optional.empty());
|
||||||
|
EditProjectAnalyzer analyzer = new EditProjectAnalyzer(
|
||||||
|
store,
|
||||||
|
discovery,
|
||||||
|
inspector,
|
||||||
|
thumbnailExtractor,
|
||||||
|
contactSheetGenerator,
|
||||||
|
proxyGenerator,
|
||||||
|
new CinematicHighlightAnalyzer(Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC)),
|
||||||
|
Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC),
|
||||||
|
null
|
||||||
|
);
|
||||||
|
|
||||||
|
analyzer.analyze("project-1", input);
|
||||||
|
|
||||||
|
CinematicHighlightAnalysis cinematic = store.readJson("project-1", "cinematic-highlight-analysis.json",
|
||||||
|
CinematicHighlightAnalysis.class);
|
||||||
|
assertThat(cinematic.category()).isEqualTo(ContentCategory.CAR_VLOG);
|
||||||
|
assertThat(cinematic.candidates()).isNotEmpty();
|
||||||
|
assertThat(Files.exists(tempDir.resolve("projects/project-1/category.json"))).isTrue();
|
||||||
|
assertThat(Files.exists(tempDir.resolve("projects/project-1/highlight-candidates.json"))).isTrue();
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void recordsInvalidClipErrorsAndKeepsUsableClips() throws Exception {
|
void recordsInvalidClipErrorsAndKeepsUsableClips() throws Exception {
|
||||||
Path input = Files.createDirectory(tempDir.resolve("input"));
|
Path input = Files.createDirectory(tempDir.resolve("input"));
|
||||||
|
|
@ -148,6 +191,13 @@ class EditProjectAnalyzerTest {
|
||||||
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void writeProject(FileSystemEditProjectStore store, Path input, String name) {
|
||||||
|
Instant now = Instant.parse("2026-07-10T10:00:00Z");
|
||||||
|
store.writeJson("project-1", "project.json", new EditProject(
|
||||||
|
"project-1", name, EditProjectStatus.ANALYZING, input.toString(), "output",
|
||||||
|
60, "category-aware-cinematic-highlights", true, true, true, now, now, null));
|
||||||
|
}
|
||||||
|
|
||||||
private ClipAnalysis clipAnalysis(Path clip) {
|
private ClipAnalysis clipAnalysis(Path clip) {
|
||||||
return new ClipAnalysis(
|
return new ClipAnalysis(
|
||||||
clip.getFileName().toString().replace(".mp4", ""),
|
clip.getFileName().toString().replace(".mp4", ""),
|
||||||
|
|
|
||||||
|
|
@ -46,6 +46,33 @@ class StoryboardPromptGeneratorTest {
|
||||||
assertThat(response.promptPath()).endsWith("storyboard-prompt.md");
|
assertThat(response.promptPath()).endsWith("storyboard-prompt.md");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void includesCategorySpecificHighlightCandidatesWhenAvailable() {
|
||||||
|
FileSystemEditProjectStore store = store();
|
||||||
|
store.createProject("porsche-edit");
|
||||||
|
store.writeJson("porsche-edit", "project.json", project());
|
||||||
|
store.writeJson("porsche-edit", "analysis.json", analysis());
|
||||||
|
store.writeJson("porsche-edit", "cinematic-highlight-analysis.json", new CinematicHighlightAnalysis(
|
||||||
|
"porsche-edit",
|
||||||
|
ContentCategory.CAR_VLOG,
|
||||||
|
0.82,
|
||||||
|
List.of("Project metadata suggests vehicle content."),
|
||||||
|
List.of(new HighlightCandidate("clip_00001_candidate_01", "clip_00001",
|
||||||
|
0.0, 8.0, 0.91, "opening_hook", List.of("Inspect for hero car angle."))),
|
||||||
|
Instant.parse("2026-07-10T10:01:00Z")
|
||||||
|
));
|
||||||
|
|
||||||
|
String prompt = new StoryboardPromptGenerator(store).prompt(project(), analysis());
|
||||||
|
|
||||||
|
assertThat(prompt)
|
||||||
|
.contains("Detected content category: `car_vlog`")
|
||||||
|
.contains("Ranked highlight candidates")
|
||||||
|
.contains("candidateId: clip_00001_candidate_01")
|
||||||
|
.contains("Make the edit feel premium, powerful, and precise")
|
||||||
|
.contains("\"contentCategory\": \"car_vlog\"")
|
||||||
|
.contains("\"overlays\"");
|
||||||
|
}
|
||||||
|
|
||||||
private FileSystemEditProjectStore store() {
|
private FileSystemEditProjectStore store() {
|
||||||
VideoClippingProperties properties = new VideoClippingProperties();
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
|
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue