forked from jsl/video_editing_poc
Add category-aware cinematic highlight planning
This commit is contained in:
JSLMPR
parent
5d889b00dc
commit
5006dfd50b
|
|
@ -0,0 +1,636 @@
|
|||
# Production Cinematic Highlight Editing Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Build a local-first workflow where a user drops one source video into a source folder and the service produces multiple production-ready cinematic highlight edits. Each highlight must be adapted to the detected content type, such as family vlog, food vlog, car vlog, or a generic fallback category.
|
||||
|
||||
The target output is not just a technically valid MP4. The output must feel intentionally directed: strong shot selection, a clear mini-story, pacing matched to music, fitting voiceover, sound effects, cinematic visual treatment, tasteful text overlays, and a final audio mix that sounds finished.
|
||||
|
||||
## Research Findings
|
||||
|
||||
- Recent highlight-detection research points toward multimodal narrative understanding, not simple duration slicing. A production system should combine shot segmentation, visual analysis, speech/transcript analysis, audio events, pacing, and narrative role detection.
|
||||
- Audio matters for highlight detection. Interesting moments are often signaled by speech emphasis, engine sound, laughter, applause, cooking sounds, silence, music changes, or loudness peaks.
|
||||
- Shot and scene detection should be treated as a core primitive. The service should first split the source video into meaningful temporal units before asking an AI director to choose edits.
|
||||
- Modern editing tools increasingly use transcript search, visual search, beat detection, montage generation, auto-cropping, and music synchronization. The service should expose similar building blocks internally instead of relying on a prompt alone.
|
||||
- FFmpeg is suitable as the deterministic render backend, but the current renderer must be expanded. Production-level edits need real filter graphs for speed ramps, transitions, stabilization, text, color, grain, vignettes, audio ducking, loudness normalization, SFX placement, and final mastering.
|
||||
|
||||
Research sources:
|
||||
|
||||
- https://arxiv.org/abs/2507.10495
|
||||
- https://arxiv.org/abs/2401.10075
|
||||
- https://arxiv.org/abs/2107.09612
|
||||
- https://arxiv.org/abs/2306.11918
|
||||
- https://en.wikipedia.org/wiki/Shot_transition_detection
|
||||
- https://en.wikipedia.org/wiki/Video_content_analysis
|
||||
- https://en.wikipedia.org/wiki/Film_editing
|
||||
- https://en.wikipedia.org/wiki/FFmpeg
|
||||
- https://www.digitalcameraworld.com/cameras/video-cameras/these-are-the-new-ai-powered-features-coming-to-final-cut-pro-and-imovie
|
||||
|
||||
## Target Workflow
|
||||
|
||||
1. User places one source video in the configured local source folder.
|
||||
2. Scheduler detects the video and creates a highlight project.
|
||||
3. Service probes the source with `ffprobe` and creates stable low-resolution proxy media.
|
||||
4. Service extracts thumbnails, contact sheets, audio waveform data, audio stems when possible, and transcript data.
|
||||
5. Service performs shot and scene segmentation.
|
||||
6. Service classifies the content category.
|
||||
7. Service generates highlight candidates from visual, audio, transcript, and motion signals.
|
||||
8. AI director reviews the compact analysis artifacts and creates a strict edit plan.
|
||||
9. Asset providers create or select voiceover, music, SFX, captions, and overlay text.
|
||||
10. Renderer executes the edit plan with FFmpeg and produces final highlight MP4s.
|
||||
11. QA checks validate technical quality and creative completeness.
|
||||
12. Final clips, manifests, storyboards, and logs are written to the output folder.
|
||||
|
||||
## Proposed Folder Contract
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
input/highlights/source/<source-video-file>
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/
|
||||
project.json
|
||||
source/
|
||||
analysis/
|
||||
ffprobe.json
|
||||
shots.json
|
||||
scenes.json
|
||||
transcript.json
|
||||
audio-events.json
|
||||
visual-analysis.json
|
||||
category.json
|
||||
highlight-candidates.json
|
||||
director/
|
||||
director-brief.md
|
||||
director-prompt.md
|
||||
edit-plan.json
|
||||
storyboard.md
|
||||
assets/
|
||||
voiceover/
|
||||
music/
|
||||
sfx/
|
||||
overlays/
|
||||
highlights/
|
||||
<highlight-id>/
|
||||
final.mp4
|
||||
preview.mp4
|
||||
render-manifest.json
|
||||
qa-report.json
|
||||
```
|
||||
|
||||
## Content Categories
|
||||
|
||||
### Family Vlog
|
||||
|
||||
Creative intent:
|
||||
|
||||
- Warm, emotional, human, memory-focused.
|
||||
- Preserve natural faces, voices, laughter, and real moments.
|
||||
- Avoid over-editing or aggressive transitions.
|
||||
- Use gentle music, soft cinematic color, light film grain, and meaningful text overlays.
|
||||
|
||||
Highlight signals:
|
||||
|
||||
- Smiles, hugs, reactions, children, group moments, travel reveal moments, laughter.
|
||||
- Speech with emotional words or important context.
|
||||
- Stable shots with faces and clear composition.
|
||||
|
||||
Voiceover style:
|
||||
|
||||
- Personal, warm, reflective, short.
|
||||
- Should sound like a memory being narrated, not an advertisement.
|
||||
|
||||
### Food Vlog
|
||||
|
||||
Creative intent:
|
||||
|
||||
- Sensory, rhythmic, appetizing, tactile.
|
||||
- Make ingredients, textures, steam, plating, and reactions feel premium.
|
||||
- Sync cuts to cooking sounds and music beats.
|
||||
|
||||
Highlight signals:
|
||||
|
||||
- Close-ups of ingredients, cutting, sizzling, steam, pouring, plating, first bite, reaction.
|
||||
- Strong texture, color, macro framing, hand movement, kitchen action.
|
||||
|
||||
Voiceover style:
|
||||
|
||||
- Descriptive and sensory.
|
||||
- Mention texture, aroma, craft, freshness, and payoff.
|
||||
|
||||
### Car Vlog
|
||||
|
||||
Creative intent:
|
||||
|
||||
- Heroic, powerful, premium, dynamic.
|
||||
- Emphasize stance, motion, engine sound, details, acceleration, road presence, reflections, wheels, badges, lights, interior, and reveal shots.
|
||||
- Use speed ramps, bass hits, whooshes, cinematic grade, controlled text overlays, and strong music.
|
||||
|
||||
Highlight signals:
|
||||
|
||||
- Car fully visible, front/rear three-quarter angles, wheels turning, badge/detail shots, engine/exhaust audio, road motion, camera movement, dramatic lighting.
|
||||
- Avoid repeated static hood shots unless used as setup.
|
||||
|
||||
Voiceover style:
|
||||
|
||||
- Confident, cinematic, concise.
|
||||
- Should praise the vehicle specifically based on observed details, not use generic supercar phrases.
|
||||
|
||||
### Generic Vlog Fallback
|
||||
|
||||
Creative intent:
|
||||
|
||||
- Clear story arc, clean pacing, tasteful grade, and context-aware overlays.
|
||||
- Use the best available moments even when the category is uncertain.
|
||||
|
||||
Highlight signals:
|
||||
|
||||
- High motion, strong composition, faces, speech peaks, laughter, visual novelty, scene changes.
|
||||
|
||||
## AI Director Responsibilities
|
||||
|
||||
The AI director should not render the video. It should produce a validated edit decision plan that the renderer can execute.
|
||||
|
||||
Required AI director inputs:
|
||||
|
||||
- Source metadata: duration, resolution, frame rate, codec, audio streams.
|
||||
- Shot list with timestamps, representative thumbnails, motion score, blur score, brightness score, face/object labels, and transcript references.
|
||||
- Audio analysis: speech segments, silence, peaks, loudness, detected audio events, music/noise/speech classification.
|
||||
- Category classification with confidence and reasons.
|
||||
- Highlight candidates with ranked evidence.
|
||||
- Available music, SFX, voice, font, LUT, and transition capabilities.
|
||||
|
||||
Required AI director output:
|
||||
|
||||
- Highlight list with target duration and category.
|
||||
- Timeline edit decisions with exact source timestamps.
|
||||
- Music mood, energy curve, and beat-sync points.
|
||||
- SFX cues with timestamps and purpose.
|
||||
- Voiceover script split into timed lines.
|
||||
- Text overlay copy, timing, animation, placement, and safe-area rules.
|
||||
- Visual treatment: grade, transitions, speed ramps, stabilization, crop, punch-ins, grain, vignette, glow, blur, and overlays.
|
||||
- Render profile: aspect ratio, resolution, FPS policy, codec, CRF or bitrate mode, loudness target.
|
||||
|
||||
## Renderer Requirements
|
||||
|
||||
The renderer must support these production features before the output can be called cinematic:
|
||||
|
||||
- Precise trims using source timestamps.
|
||||
- Crossfades, match cuts, hard cuts, whip transitions, fade to black, and dip transitions.
|
||||
- Speed ramps and short impact pauses.
|
||||
- Dynamic crop and punch-in keyframes for static shots.
|
||||
- Stabilization option for shaky footage.
|
||||
- Category-specific color grade chains or LUT support.
|
||||
- Film grain, vignette, sharpening, glow, blur, and selective contrast.
|
||||
- Text overlays with font, position, animation timing, stroke, shadow, and safe area.
|
||||
- Caption and subtitle rendering.
|
||||
- Music bed placement with fade in/out.
|
||||
- Voiceover placement with sidechain ducking.
|
||||
- SFX placement for whooshes, impacts, risers, camera clicks, engine hits, cooking sounds, and ambient layers.
|
||||
- Audio loudness normalization to a configured target.
|
||||
- Final render manifest with all commands, source segments, assets, and durations.
|
||||
|
||||
## Asset Provider Requirements
|
||||
|
||||
The system needs explicit providers instead of pretending a prompt can create finished audio by itself.
|
||||
|
||||
- Voiceover provider: local TTS or remote TTS, configurable voice per category, output WAV.
|
||||
- Music provider: licensed local library first, optional generated music provider later.
|
||||
- SFX provider: licensed local SFX library with tags and category mapping.
|
||||
- Font provider: local fonts with brand/category mapping.
|
||||
- LUT provider: local LUT files or named FFmpeg grade presets.
|
||||
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
|
||||
|
||||
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
|
||||
|
||||
## Highlight Scoring Model
|
||||
|
||||
Each candidate segment should be scored with a weighted model:
|
||||
|
||||
```text
|
||||
score =
|
||||
visual_interest
|
||||
+ motion_energy
|
||||
+ audio_interest
|
||||
+ transcript_importance
|
||||
+ category_relevance
|
||||
+ composition_quality
|
||||
+ novelty
|
||||
- blur_penalty
|
||||
- darkness_penalty
|
||||
- repetition_penalty
|
||||
- unusable_audio_penalty
|
||||
```
|
||||
|
||||
Category-specific examples:
|
||||
|
||||
- Family vlog should boost faces, laughter, dialogue, emotional reactions, and group moments.
|
||||
- Food vlog should boost close-ups, texture, cooking actions, plating, steam, and bite reactions.
|
||||
- Car vlog should boost full-car hero shots, motion, wheels, engine audio, badges, roads, and dramatic lighting.
|
||||
|
||||
## Edit Plan Schema
|
||||
|
||||
The plan should be strict JSON so a cheaper model or deterministic renderer can execute it.
|
||||
|
||||
```json
|
||||
{
|
||||
"projectId": "example-project",
|
||||
"sourceVideo": "input/highlights/source/example.mp4",
|
||||
"contentCategory": "car_vlog",
|
||||
"highlights": [
|
||||
{
|
||||
"id": "hero-reveal-001",
|
||||
"title": "The Reveal",
|
||||
"targetDurationSeconds": 35,
|
||||
"aspectRatio": "16:9",
|
||||
"renderProfile": "cinematic_4k_h264",
|
||||
"creativeIntent": "Premium Porsche reveal with dynamic pacing and confident narration.",
|
||||
"timeline": [
|
||||
{
|
||||
"type": "video",
|
||||
"sourceStart": "00:01:12.400",
|
||||
"sourceEnd": "00:01:16.900",
|
||||
"timelineStart": "00:00:00.000",
|
||||
"effects": ["stabilize", "cinematic_warm_contrast", "slow_push_in"],
|
||||
"transitionOut": "bass_hit_cut"
|
||||
}
|
||||
],
|
||||
"voiceover": [
|
||||
{
|
||||
"text": "This is not just a drive. It is precision, presence, and control.",
|
||||
"timelineStart": "00:00:04.200",
|
||||
"voice": "confident_cinematic_male",
|
||||
"delivery": "calm premium"
|
||||
}
|
||||
],
|
||||
"music": {
|
||||
"mood": "premium cinematic electronic",
|
||||
"energyCurve": "slow_build_to_impact",
|
||||
"assetId": "music/premium-drive-90bpm.wav"
|
||||
},
|
||||
"sfx": [
|
||||
{
|
||||
"assetId": "sfx/bass-hit-01.wav",
|
||||
"timelineStart": "00:00:07.800",
|
||||
"gainDb": -6
|
||||
}
|
||||
],
|
||||
"overlays": [
|
||||
{
|
||||
"text": "PRECISION IN MOTION",
|
||||
"timelineStart": "00:00:08.000",
|
||||
"durationSeconds": 2.5,
|
||||
"placement": "lower_left_safe",
|
||||
"animation": "fade_slide_up"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Milestones
|
||||
|
||||
- [ ] Milestone 1: Define the cinematic highlight project domain model, folder contract, and persisted manifests.
|
||||
- [ ] Milestone 2: Add a local scheduler that accepts exactly one source video at a time from the highlight source folder.
|
||||
- [ ] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
|
||||
- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
||||
- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
||||
- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
||||
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
|
||||
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
|
||||
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
|
||||
- [ ] Milestone 10: Implement edit plan validation so impossible timestamps, missing assets, invalid overlays, and unsupported effects fail before render.
|
||||
- [ ] Milestone 11: Implement asset provider interfaces for voiceover, music, SFX, fonts, and LUTs.
|
||||
- [ ] Milestone 12: Add a local licensed asset library with category tags and deterministic asset selection.
|
||||
- [ ] Milestone 13: Implement renderer v2 with transitions, speed ramps, dynamic crops, overlays, visual effects, audio ducking, loudness normalization, and render manifests.
|
||||
- [ ] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters.
|
||||
- [ ] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips.
|
||||
- [ ] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content.
|
||||
- [ ] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size.
|
||||
- [ ] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for category scoring, candidate ranking, schema validation, folder discovery, and manifest writing.
|
||||
- Integration tests with short fixture videos that verify each category can produce at least one valid highlight plan.
|
||||
- Renderer tests that verify FFmpeg commands include expected filters for transitions, overlays, color treatment, music, SFX, and voiceover.
|
||||
- Golden-file tests for AI prompt inputs and expected JSON schema shape.
|
||||
- End-to-end local test that places one video in the source folder and verifies final clips, manifests, storyboards, and QA reports are created.
|
||||
- Negative tests for corrupt input video, unsupported codec, empty video, missing audio, missing asset, invalid timestamp, and renderer failure.
|
||||
|
||||
## Step-By-Step Local How-To Guide
|
||||
|
||||
This guide describes the intended operator workflow after the milestones are implemented.
|
||||
|
||||
### 1. Prepare The Machine
|
||||
|
||||
Install and verify required binaries:
|
||||
|
||||
```bash
|
||||
ffmpeg -version
|
||||
ffprobe -version
|
||||
```
|
||||
|
||||
Optional but recommended tools:
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
Recommended future optional components:
|
||||
|
||||
- PySceneDetect for higher-quality scene detection.
|
||||
- Whisper or another ASR provider for transcript-based editing.
|
||||
- A configured TTS provider for voiceover.
|
||||
- A licensed local music and SFX library.
|
||||
|
||||
### 2. Configure The Application
|
||||
|
||||
Set the local cinematic highlight folders in `application.yml`:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
cinematic-highlights:
|
||||
enabled: true
|
||||
source-folder: input/highlights/source
|
||||
output-folder: output/highlight-projects
|
||||
scheduler-interval: 10s
|
||||
max-active-projects: 1
|
||||
default-aspect-ratio: "16:9"
|
||||
default-render-profile: cinematic_1080p_h264
|
||||
```
|
||||
|
||||
Configure the AI director model separately from the renderer:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
cinematic-highlights:
|
||||
director:
|
||||
enabled: true
|
||||
model: gpt-5
|
||||
allow-cheaper-model: true
|
||||
cheaper-model: gpt-5-mini
|
||||
require-human-approval-before-render: true
|
||||
```
|
||||
|
||||
Configure local assets:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
cinematic-highlights:
|
||||
assets:
|
||||
music-folder: input/highlights/assets/music
|
||||
sfx-folder: input/highlights/assets/sfx
|
||||
fonts-folder: input/highlights/assets/fonts
|
||||
luts-folder: input/highlights/assets/luts
|
||||
voiceover-folder: output/highlight-projects/_voiceover-cache
|
||||
```
|
||||
|
||||
### 3. Create Folders
|
||||
|
||||
Create the required local folders:
|
||||
|
||||
```bash
|
||||
mkdir -p input/highlights/source
|
||||
mkdir -p input/highlights/assets/music
|
||||
mkdir -p input/highlights/assets/sfx
|
||||
mkdir -p input/highlights/assets/fonts
|
||||
mkdir -p input/highlights/assets/luts
|
||||
mkdir -p output/highlight-projects
|
||||
```
|
||||
|
||||
### 4. Add The Source Video
|
||||
|
||||
Place exactly one source video in the source folder:
|
||||
|
||||
```text
|
||||
input/highlights/source/my-porsche-drive.mp4
|
||||
```
|
||||
|
||||
The scheduler should process one source video at a time. If multiple videos are present, it should process the first valid file, finish it, then move to the next one.
|
||||
|
||||
Invalid files should be ignored and logged with the reason.
|
||||
|
||||
### 5. Start The Application
|
||||
|
||||
Run the Spring Boot service:
|
||||
|
||||
```bash
|
||||
./mvnw spring-boot:run
|
||||
```
|
||||
|
||||
Expected startup logs:
|
||||
|
||||
```text
|
||||
Cinematic highlight scheduler started
|
||||
Watching source folder: input/highlights/source
|
||||
Output folder: output/highlight-projects
|
||||
Scheduler interval: 10s
|
||||
Max active projects: 1
|
||||
```
|
||||
|
||||
### 6. Verify Project Creation
|
||||
|
||||
After the scheduler picks up the source file, verify a project folder exists:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/
|
||||
```
|
||||
|
||||
Expected early files:
|
||||
|
||||
```text
|
||||
project.json
|
||||
analysis/ffprobe.json
|
||||
analysis/shots.json
|
||||
analysis/scenes.json
|
||||
analysis/category.json
|
||||
director/director-brief.md
|
||||
director/director-prompt.md
|
||||
```
|
||||
|
||||
Expected logs:
|
||||
|
||||
```text
|
||||
Started cinematic highlight project projectId=<project-id> source=my-porsche-drive.mp4
|
||||
Generated proxy video
|
||||
Extracted representative thumbnails
|
||||
Detected shots and scenes
|
||||
Classified content category category=car_vlog confidence=...
|
||||
Generated highlight candidates count=...
|
||||
```
|
||||
|
||||
### 7. Review The AI Director Prompt
|
||||
|
||||
Open the director prompt:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/director/director-prompt.md
|
||||
```
|
||||
|
||||
Check that it includes:
|
||||
|
||||
- Source metadata.
|
||||
- Category classification.
|
||||
- Contact sheet or thumbnail references.
|
||||
- Candidate segments with timestamps.
|
||||
- Available renderer effects.
|
||||
- Available music, SFX, voice, font, and LUT assets.
|
||||
- Strict JSON schema requirements.
|
||||
- Category-specific creative direction.
|
||||
|
||||
If the prompt is generic, the final edit will probably be generic. The prompt must force concrete choices: exact cuts, specific overlays, voiceover lines, SFX timing, music mood, and visual treatment.
|
||||
|
||||
### 8. Execute The AI Director
|
||||
|
||||
Run the configured AI director against the prompt. The director must output:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||
```
|
||||
|
||||
If human approval is enabled, inspect the edit plan before rendering.
|
||||
|
||||
Minimum review checklist:
|
||||
|
||||
- The selected shots are not repetitive.
|
||||
- Each highlight has a clear beginning, middle, and ending.
|
||||
- The edit uses the correct category style.
|
||||
- Voiceover text is specific to the observed video.
|
||||
- Music and SFX cues are not placeholders.
|
||||
- Text overlays are tasteful and not overused.
|
||||
- All referenced timestamps are inside the source duration.
|
||||
- All referenced assets exist.
|
||||
|
||||
### 9. Approve Or Correct The Edit Plan
|
||||
|
||||
If the plan is weak, edit or regenerate it before rendering.
|
||||
|
||||
Common fixes:
|
||||
|
||||
- Replace generic voiceover with concrete lines about the actual footage.
|
||||
- Remove boring shots or repeated angles.
|
||||
- Add stronger openers and endings.
|
||||
- Add music beat cues.
|
||||
- Add SFX only where they support a cut or reveal.
|
||||
- Add overlay text only where it improves the scene.
|
||||
- Adjust target duration if the highlight feels too slow.
|
||||
|
||||
When approved, write:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/director/approved.flag
|
||||
```
|
||||
|
||||
### 10. Render The Highlights
|
||||
|
||||
The renderer consumes:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||
```
|
||||
|
||||
Expected renderer output:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||
```
|
||||
|
||||
Expected logs:
|
||||
|
||||
```text
|
||||
Rendering highlight projectId=<project-id> highlightId=<highlight-id>
|
||||
Applying video trims count=...
|
||||
Applying transitions count=...
|
||||
Applying visual effects count=...
|
||||
Mixing music asset=...
|
||||
Mixing voiceover lines=...
|
||||
Mixing SFX count=...
|
||||
Completed highlight render duration=... output=...
|
||||
```
|
||||
|
||||
### 11. Review The Final Clips
|
||||
|
||||
For each `final.mp4`, verify:
|
||||
|
||||
- The first three seconds are interesting.
|
||||
- The edit has intentional pacing.
|
||||
- The selected shots fit the content type.
|
||||
- Music fits the scene and is audible.
|
||||
- Voiceover is clear and not robotic or generic.
|
||||
- SFX support the edit without becoming noisy.
|
||||
- Text overlays are readable and not cheap-looking.
|
||||
- Color grade improves the footage without crushing detail.
|
||||
- Audio is not clipping and does not have long silent sections.
|
||||
|
||||
If the final video is boring, inspect in this order:
|
||||
|
||||
1. `analysis/highlight-candidates.json`: weak source moments or wrong scoring.
|
||||
2. `director/edit-plan.json`: weak AI choices, generic voiceover, no effects.
|
||||
3. `render-manifest.json`: renderer ignored effects or missing assets.
|
||||
4. `qa-report.json`: technical failures or missing audio/effects.
|
||||
|
||||
### 12. Find The Final Result
|
||||
|
||||
Final highlight videos are stored here:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||
```
|
||||
|
||||
The full audit trail is stored beside the final video:
|
||||
|
||||
```text
|
||||
render-manifest.json
|
||||
qa-report.json
|
||||
```
|
||||
|
||||
The creative plan is stored here:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/director/storyboard.md
|
||||
output/highlight-projects/<project-id>/director/edit-plan.json
|
||||
```
|
||||
|
||||
### 13. Clean Up After A Run
|
||||
|
||||
After confirming the output is good, move the source video out of the source folder or let the scheduler move it to a processed folder:
|
||||
|
||||
```text
|
||||
input/highlights/processed/<source-video-file>
|
||||
```
|
||||
|
||||
Failed inputs should move to:
|
||||
|
||||
```text
|
||||
input/highlights/failed/<source-video-file>
|
||||
```
|
||||
|
||||
The scheduler must not repeatedly process the same source video.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
A highlight export is production-ready only when:
|
||||
|
||||
- The selected shots match the detected content category.
|
||||
- The edit has a clear beginning, middle, and ending.
|
||||
- Music is present, fitting, and timed to key cuts.
|
||||
- Voiceover is present when requested and sounds appropriate for the category.
|
||||
- SFX are present where they add impact and are not randomly overused.
|
||||
- Text overlays are readable, visually appealing, correctly timed, and safe-area compliant.
|
||||
- Visual treatment is visible but not destructive.
|
||||
- Audio is normalized and free of clipping.
|
||||
- Output includes a render manifest, edit plan, storyboard, and QA report.
|
||||
- The final MP4 can be played independently without relying on temporary files.
|
||||
|
||||
## Key Risk
|
||||
|
||||
The previous boring output is expected if the system only creates a weak prompt and then renders simple trims. Cinematic output requires three things working together: richer multimodal analysis, a strict category-aware AI director plan, and a renderer that can actually execute music, SFX, voiceover, overlays, transitions, and visual effects.
|
||||
|
|
@ -54,21 +54,23 @@ public class AiDirectorPromptGenerator {
|
|||
return """
|
||||
# Local AI Director Assignment
|
||||
|
||||
You are the director for a premium cinematic Porsche promo.
|
||||
You are the director for a production-ready cinematic highlight edit.
|
||||
|
||||
Work only with artifacts in this project folder:
|
||||
- Read `analysis.json` for authoritative clip metadata.
|
||||
- Read `cinematic-highlight-analysis.json`, `category.json`, and `highlight-candidates.json` when present.
|
||||
- Inspect every relevant image in `contact-sheets/` and `thumbnails/` before choosing shots.
|
||||
- Use `proxies/` only when motion needs closer inspection.
|
||||
|
||||
Complete this task:
|
||||
1. Judge visual quality, composition, motion, continuity, and story value from the generated artifacts.
|
||||
2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final hero shot.
|
||||
1. Judge visual quality, composition, motion, continuity, audio opportunity, and story value from the generated artifacts.
|
||||
2. Select the strongest shots and direct a clear opening hook, rising-energy middle, and final memorable shot.
|
||||
3. Follow the strict JSON schema and constraints in the brief below.
|
||||
4. Write the final JSON object to `inbox/%s`.
|
||||
|
||||
Do not render video. Do not run FFmpeg. Do not modify source clips or generated analysis artifacts.
|
||||
Do not invent clip IDs, source ranges, durations, or media paths.
|
||||
Do not write generic voiceover. The narration, text overlays, music mood, SFX, and visual effects must fit the detected content category.
|
||||
Your task is complete only when `inbox/%s` contains valid JSON with no Markdown fences.
|
||||
|
||||
%s
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
public record CinematicHighlightAnalysis(
|
||||
String projectId,
|
||||
ContentCategory category,
|
||||
double categoryConfidence,
|
||||
List<String> categoryReasons,
|
||||
List<HighlightCandidate> candidates,
|
||||
Instant createdAt
|
||||
) {
|
||||
}
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class CinematicHighlightAnalyzer {
|
||||
|
||||
private static final double DEFAULT_CANDIDATE_SECONDS = 8.0;
|
||||
private static final int MAX_CANDIDATES_PER_CLIP = 12;
|
||||
|
||||
private final Clock clock;
|
||||
|
||||
public CinematicHighlightAnalyzer() {
|
||||
this(Clock.systemUTC());
|
||||
}
|
||||
|
||||
CinematicHighlightAnalyzer(Clock clock) {
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
public CinematicHighlightAnalysis analyze(EditProject project, EditProjectAnalysis analysis) {
|
||||
CategoryDecision category = classify(project, analysis);
|
||||
List<HighlightCandidate> candidates = analysis.clips().stream()
|
||||
.flatMap(clip -> candidatesForClip(clip, category.category()).stream())
|
||||
.sorted(Comparator.comparingDouble(HighlightCandidate::score).reversed()
|
||||
.thenComparing(HighlightCandidate::clipId)
|
||||
.thenComparingDouble(HighlightCandidate::sourceStartSeconds))
|
||||
.toList();
|
||||
return new CinematicHighlightAnalysis(project.id(), category.category(), category.confidence(),
|
||||
category.reasons(), candidates, Instant.now(clock));
|
||||
}
|
||||
|
||||
private CategoryDecision classify(EditProject project, EditProjectAnalysis analysis) {
|
||||
String haystack = (project.name() + " " + project.style() + " "
|
||||
+ analysis.clips().stream()
|
||||
.map(clip -> clip.clipId() + " " + clip.sourcePath())
|
||||
.reduce("", (left, right) -> left + " " + right))
|
||||
.toLowerCase(Locale.ROOT);
|
||||
|
||||
if (containsAny(haystack, "porsche", "car", "auto", "vehicle", "drive", "driving", "engine", "wheel")) {
|
||||
return new CategoryDecision(ContentCategory.CAR_VLOG, 0.78,
|
||||
List.of("Project metadata suggests vehicle or driving content."));
|
||||
}
|
||||
if (containsAny(haystack, "food", "cook", "cooking", "recipe", "kitchen", "restaurant", "meal", "dish")) {
|
||||
return new CategoryDecision(ContentCategory.FOOD_VLOG, 0.74,
|
||||
List.of("Project metadata suggests food, cooking, or restaurant content."));
|
||||
}
|
||||
if (containsAny(haystack, "family", "kids", "child", "children", "birthday", "holiday", "vacation", "vlog")) {
|
||||
return new CategoryDecision(ContentCategory.FAMILY_VLOG, 0.68,
|
||||
List.of("Project metadata suggests personal or family vlog content."));
|
||||
}
|
||||
return new CategoryDecision(ContentCategory.GENERIC_VLOG, 0.45,
|
||||
List.of("No strong category-specific metadata was available; using generic vlog direction."));
|
||||
}
|
||||
|
||||
private boolean containsAny(String haystack, String... needles) {
|
||||
for (String needle : needles) {
|
||||
if (haystack.contains(needle)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private List<HighlightCandidate> candidatesForClip(ClipAnalysis clip, ContentCategory category) {
|
||||
double duration = clip.durationSeconds();
|
||||
if (duration <= 0) {
|
||||
return List.of();
|
||||
}
|
||||
|
||||
double segmentSeconds = Math.min(DEFAULT_CANDIDATE_SECONDS, duration);
|
||||
int segmentCount = Math.max(1, (int) Math.ceil(duration / segmentSeconds));
|
||||
int cappedCount = Math.min(segmentCount, MAX_CANDIDATES_PER_CLIP);
|
||||
double step = segmentCount <= MAX_CANDIDATES_PER_CLIP
|
||||
? segmentSeconds
|
||||
: Math.max(1.0, (duration - segmentSeconds) / (MAX_CANDIDATES_PER_CLIP - 1));
|
||||
|
||||
List<HighlightCandidate> candidates = new ArrayList<>();
|
||||
for (int index = 0; index < cappedCount; index++) {
|
||||
double start = Math.min(duration - segmentSeconds, index * step);
|
||||
double end = Math.min(duration, start + segmentSeconds);
|
||||
candidates.add(new HighlightCandidate(
|
||||
"%s_candidate_%02d".formatted(clip.clipId(), index + 1),
|
||||
clip.clipId(),
|
||||
round(start),
|
||||
round(end),
|
||||
round(score(clip, category, index, cappedCount)),
|
||||
role(index, cappedCount),
|
||||
reasons(clip, category, index, cappedCount)
|
||||
));
|
||||
}
|
||||
return candidates;
|
||||
}
|
||||
|
||||
private double score(ClipAnalysis clip, ContentCategory category, int index, int count) {
|
||||
double quality = Math.max(0.1, clip.sharpnessScore()) + Math.max(0.1, clip.brightnessScore())
|
||||
+ Math.max(0.1, clip.motionScore());
|
||||
double position = switch (role(index, count)) {
|
||||
case "opening_hook" -> 0.25;
|
||||
case "final_hero" -> 0.22;
|
||||
case "rising_action" -> 0.16;
|
||||
default -> 0.1;
|
||||
};
|
||||
double categoryBoost = switch (category) {
|
||||
case CAR_VLOG -> 0.2;
|
||||
case FOOD_VLOG -> 0.16;
|
||||
case FAMILY_VLOG -> 0.14;
|
||||
case GENERIC_VLOG -> 0.08;
|
||||
};
|
||||
return Math.min(1.0, 0.35 + quality * 0.12 + position + categoryBoost);
|
||||
}
|
||||
|
||||
private String role(int index, int count) {
|
||||
if (index == 0) {
|
||||
return "opening_hook";
|
||||
}
|
||||
if (index == count - 1) {
|
||||
return "final_hero";
|
||||
}
|
||||
if (index <= Math.max(1, count / 3)) {
|
||||
return "rising_action";
|
||||
}
|
||||
return "detail_or_bridge";
|
||||
}
|
||||
|
||||
private List<String> reasons(ClipAnalysis clip, ContentCategory category, int index, int count) {
|
||||
List<String> reasons = new ArrayList<>();
|
||||
reasons.add("Suggested as " + role(index, count).replace('_', ' ') + " based on source position.");
|
||||
reasons.add("Duration window is short enough for fast cinematic pacing.");
|
||||
if (clip.motionScore() > 0) {
|
||||
reasons.add("Motion score is available for director ranking.");
|
||||
}
|
||||
reasons.add(switch (category) {
|
||||
case CAR_VLOG -> "For car content, inspect this window for hero angles, wheels, badges, engine sound, or road motion.";
|
||||
case FOOD_VLOG -> "For food content, inspect this window for texture, preparation, plating, steam, or bite reactions.";
|
||||
case FAMILY_VLOG -> "For family content, inspect this window for faces, reactions, laughter, hugs, or emotional dialogue.";
|
||||
case GENERIC_VLOG -> "For generic vlog content, inspect this window for visual novelty, clear action, or useful story context.";
|
||||
});
|
||||
return List.copyOf(reasons);
|
||||
}
|
||||
|
||||
private double round(double value) {
|
||||
return Math.round(value * 1000.0) / 1000.0;
|
||||
}
|
||||
|
||||
private record CategoryDecision(ContentCategory category, double confidence, List<String> reasons) {
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
public enum ContentCategory {
|
||||
FAMILY_VLOG,
|
||||
FOOD_VLOG,
|
||||
CAR_VLOG,
|
||||
GENERIC_VLOG
|
||||
}
|
||||
|
|
@ -21,6 +21,7 @@ public class EditProjectAnalyzer {
|
|||
private final ThumbnailExtractor thumbnailExtractor;
|
||||
private final ContactSheetGenerator contactSheetGenerator;
|
||||
private final ProxyGenerator proxyGenerator;
|
||||
private final CinematicHighlightAnalyzer cinematicHighlightAnalyzer;
|
||||
private final Clock clock;
|
||||
private final EditObservability observability;
|
||||
|
||||
|
|
@ -32,10 +33,11 @@ public class EditProjectAnalyzer {
|
|||
ThumbnailExtractor thumbnailExtractor,
|
||||
ContactSheetGenerator contactSheetGenerator,
|
||||
ProxyGenerator proxyGenerator,
|
||||
CinematicHighlightAnalyzer cinematicHighlightAnalyzer,
|
||||
EditObservability observability
|
||||
) {
|
||||
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator,
|
||||
Clock.systemUTC(), observability);
|
||||
cinematicHighlightAnalyzer, Clock.systemUTC(), observability);
|
||||
}
|
||||
|
||||
EditProjectAnalyzer(
|
||||
|
|
@ -47,13 +49,14 @@ public class EditProjectAnalyzer {
|
|||
ProxyGenerator proxyGenerator,
|
||||
Clock clock
|
||||
) {
|
||||
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, clock, null);
|
||||
this(store, discovery, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, null, clock, null);
|
||||
}
|
||||
|
||||
EditProjectAnalyzer(
|
||||
EditProjectStore store, EditClipDiscovery discovery, FfmpegClipInspector inspector,
|
||||
ThumbnailExtractor thumbnailExtractor, ContactSheetGenerator contactSheetGenerator,
|
||||
ProxyGenerator proxyGenerator, Clock clock, EditObservability observability
|
||||
ProxyGenerator proxyGenerator, CinematicHighlightAnalyzer cinematicHighlightAnalyzer, Clock clock,
|
||||
EditObservability observability
|
||||
) {
|
||||
this.store = store;
|
||||
this.discovery = discovery;
|
||||
|
|
@ -61,6 +64,7 @@ public class EditProjectAnalyzer {
|
|||
this.thumbnailExtractor = thumbnailExtractor;
|
||||
this.contactSheetGenerator = contactSheetGenerator;
|
||||
this.proxyGenerator = proxyGenerator;
|
||||
this.cinematicHighlightAnalyzer = cinematicHighlightAnalyzer;
|
||||
this.clock = clock;
|
||||
this.observability = observability;
|
||||
}
|
||||
|
|
@ -100,6 +104,7 @@ public class EditProjectAnalyzer {
|
|||
EditProjectAnalysis analysis = new EditProjectAnalysis(projectId, List.copyOf(clips), List.copyOf(errors),
|
||||
Instant.now(clock));
|
||||
store.writeJson(projectId, "analysis.json", analysis);
|
||||
writeCinematicHighlightArtifacts(projectId, analysis);
|
||||
if (observability != null) {
|
||||
observability.analysisCompleted(projectId, System.nanoTime() - startedAt, clips.size(), errors.size());
|
||||
}
|
||||
|
|
@ -133,4 +138,15 @@ public class EditProjectAnalyzer {
|
|||
inspected.sharpnessScore()
|
||||
);
|
||||
}
|
||||
|
||||
private void writeCinematicHighlightArtifacts(String projectId, EditProjectAnalysis analysis) {
|
||||
if (cinematicHighlightAnalyzer == null) {
|
||||
return;
|
||||
}
|
||||
EditProject project = store.readJson(projectId, "project.json", EditProject.class);
|
||||
CinematicHighlightAnalysis cinematic = cinematicHighlightAnalyzer.analyze(project, analysis);
|
||||
store.writeJson(projectId, "cinematic-highlight-analysis.json", cinematic);
|
||||
store.writeJson(projectId, "category.json", cinematic);
|
||||
store.writeJson(projectId, "highlight-candidates.json", cinematic.candidates());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record HighlightCandidate(
|
||||
String id,
|
||||
String clipId,
|
||||
double sourceStartSeconds,
|
||||
double sourceEndSeconds,
|
||||
double score,
|
||||
String suggestedRole,
|
||||
List<String> reasons
|
||||
) {
|
||||
}
|
||||
|
|
@ -160,7 +160,7 @@ public class LocalDirectorScheduler {
|
|||
try {
|
||||
EditProjectResponse project = projectService.createProject(new CreateEditProjectRequest(
|
||||
working.getFileName().toString(), working.toString(), editing.getTargetDurationSeconds(),
|
||||
"cinematic-porsche-promo", true, true, true));
|
||||
"category-aware-cinematic-highlights", true, true, true));
|
||||
projectId = project.projectId();
|
||||
projectService.updateProject(projectId, EditProjectStatus.ANALYZING, working, null);
|
||||
EditProjectAnalysis analysis = analyzer.analyze(projectId, working);
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ public class StoryboardPromptGenerator {
|
|||
}
|
||||
|
||||
String prompt(EditProject project, EditProjectAnalysis analysis) {
|
||||
CinematicHighlightAnalysis cinematic = readCinematicAnalysis(project.id());
|
||||
StringBuilder clips = new StringBuilder();
|
||||
for (ClipAnalysis clip : analysis.clips()) {
|
||||
clips.append("- clipId: ").append(clip.clipId()).append('\n')
|
||||
|
|
@ -54,6 +55,10 @@ public class StoryboardPromptGenerator {
|
|||
.append(" contactSheet: ").append(reference("contact-sheets", clip.contactSheet())).append('\n')
|
||||
.append(" proxy: ").append(reference("proxies", clip.proxyPath())).append('\n');
|
||||
}
|
||||
String category = cinematic == null ? "unknown" : cinematic.category().name().toLowerCase();
|
||||
String candidates = cinematic == null ? "No precomputed highlight candidates. Inspect all clips carefully."
|
||||
: candidateBrief(cinematic);
|
||||
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
||||
|
||||
return """
|
||||
# Cinematic Storyboard Director Brief
|
||||
|
|
@ -66,18 +71,29 @@ public class StoryboardPromptGenerator {
|
|||
- Voiceover enabled: %s
|
||||
- Music enabled: %s
|
||||
- Sound effects enabled: %s
|
||||
- Detected content category: `%s`
|
||||
- Use only the clip IDs and source ranges listed below.
|
||||
- Keep every source range within its clip duration.
|
||||
- Timeline ranges must be chronological and must not overlap.
|
||||
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
||||
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
|
||||
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
|
||||
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
|
||||
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
||||
|
||||
Category-specific direction:
|
||||
%s
|
||||
|
||||
Ranked highlight candidates:
|
||||
%s
|
||||
|
||||
Available clips and project-relative visual references:
|
||||
%s
|
||||
Required JSON object shape:
|
||||
{
|
||||
"projectId": "%s",
|
||||
"style": "%s",
|
||||
"contentCategory": "%s",
|
||||
"targetDurationSeconds": %d,
|
||||
"decisions": [{
|
||||
"clipId": "clip_00001",
|
||||
|
|
@ -93,25 +109,90 @@ public class StoryboardPromptGenerator {
|
|||
}],
|
||||
"audioCues": [{
|
||||
"type": "music|sfx",
|
||||
"assetKey": "descriptive asset name",
|
||||
"assetKey": "concrete music or sfx asset description",
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": 2.5,
|
||||
"gainDb": -12.0,
|
||||
"notes": "story purpose"
|
||||
}],
|
||||
"voiceover": [{
|
||||
"text": "narration",
|
||||
"text": "specific narration based on visible footage",
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": 2.5,
|
||||
"delivery": "confident cinematic narrator"
|
||||
}],
|
||||
"overlays": [{
|
||||
"text": "short premium overlay text",
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": 2.5,
|
||||
"placement": "lower_left_safe|center_safe|lower_center_safe",
|
||||
"animation": "fade_slide_up|fade_in|none",
|
||||
"reason": "why this text improves the scene"
|
||||
}],
|
||||
"renderProfile": "mp4-h264-aac-1080p",
|
||||
"summary": "short creative rationale"
|
||||
}
|
||||
""".formatted(
|
||||
project.id(), project.style(), project.targetDurationSeconds(),
|
||||
project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(),
|
||||
clips.toString().stripTrailing(), project.id(), project.style(), project.targetDurationSeconds());
|
||||
project.voiceoverEnabled(), project.musicEnabled(), project.soundEffectsEnabled(), category,
|
||||
categoryDirection, candidates, clips.toString().stripTrailing(), project.id(), project.style(),
|
||||
category, project.targetDurationSeconds());
|
||||
}
|
||||
|
||||
private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) {
|
||||
if (!Files.exists(store.projectDirectory(projectId).resolve("cinematic-highlight-analysis.json"))) {
|
||||
return null;
|
||||
}
|
||||
return store.readJson(projectId, "cinematic-highlight-analysis.json", CinematicHighlightAnalysis.class);
|
||||
}
|
||||
|
||||
private String candidateBrief(CinematicHighlightAnalysis cinematic) {
|
||||
StringBuilder builder = new StringBuilder();
|
||||
builder.append("- categoryConfidence: ").append(cinematic.categoryConfidence()).append('\n');
|
||||
builder.append("- categoryReasons: ").append(cinematic.categoryReasons()).append('\n');
|
||||
for (HighlightCandidate candidate : cinematic.candidates()) {
|
||||
builder.append("- candidateId: ").append(candidate.id()).append('\n')
|
||||
.append(" clipId: ").append(candidate.clipId()).append('\n')
|
||||
.append(" sourceStartSeconds: ").append(candidate.sourceStartSeconds()).append('\n')
|
||||
.append(" sourceEndSeconds: ").append(candidate.sourceEndSeconds()).append('\n')
|
||||
.append(" score: ").append(candidate.score()).append('\n')
|
||||
.append(" suggestedRole: ").append(candidate.suggestedRole()).append('\n')
|
||||
.append(" reasons: ").append(candidate.reasons()).append('\n');
|
||||
}
|
||||
return builder.toString().stripTrailing();
|
||||
}
|
||||
|
||||
private String categoryDirection(ContentCategory category) {
|
||||
return switch (category) {
|
||||
case CAR_VLOG -> """
|
||||
- Make the edit feel premium, powerful, and precise.
|
||||
- Prioritize hero angles, wheels, badges, motion, reflections, roads, lights, interior details, and engine or exhaust moments.
|
||||
- Use speed ramps, bass-hit cuts, whooshes, cinematic contrast, subtle grain, and confident short overlay text when the footage supports it.
|
||||
- Voiceover should praise the specific car footage, not make unsupported claims.
|
||||
""";
|
||||
case FOOD_VLOG -> """
|
||||
- Make the edit sensory, warm, rhythmic, and appetizing.
|
||||
- Prioritize texture, slicing, pouring, sizzling, steam, plating, table reveals, and first-bite reactions.
|
||||
- Use music with light groove, tactile SFX, warm grade, macro-feeling punch-ins, and short ingredient or payoff overlays.
|
||||
- Voiceover should describe texture, craft, freshness, and payoff without sounding like a generic restaurant ad.
|
||||
""";
|
||||
case FAMILY_VLOG -> """
|
||||
- Make the edit warm, human, emotional, and memory-focused.
|
||||
- Prioritize faces, reactions, laughter, hugs, children, travel reveals, and meaningful dialogue.
|
||||
- Use gentle music, soft grade, light grain, natural pacing, and minimal text overlays.
|
||||
- Voiceover should feel personal and reflective, not promotional.
|
||||
""";
|
||||
case GENERIC_VLOG -> genericDirection();
|
||||
};
|
||||
}
|
||||
|
||||
private String genericDirection() {
|
||||
return """
|
||||
- Build a clear mini-story from the strongest available moments.
|
||||
- Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes.
|
||||
- Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story.
|
||||
- Voiceover must explain what is actually happening in the footage.
|
||||
""";
|
||||
}
|
||||
|
||||
private String references(String directory, List<String> paths) {
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ video-clipping:
|
|||
enabled: true
|
||||
local-director:
|
||||
enabled: true
|
||||
auto-render-when-plan-appears: true
|
||||
auto-render-when-plan-appears: false
|
||||
|
||||
logging:
|
||||
level:
|
||||
|
|
|
|||
|
|
@ -30,12 +30,15 @@ class AiDirectorPromptGeneratorTest {
|
|||
String prompt = Files.readString(Path.of(files.promptPath()));
|
||||
assertThat(prompt)
|
||||
.contains("`analysis.json`")
|
||||
.contains("`cinematic-highlight-analysis.json`")
|
||||
.contains("`contact-sheets/`")
|
||||
.contains("`thumbnails/`")
|
||||
.contains("`inbox/edit-plan.json`")
|
||||
.contains("Do not render video")
|
||||
.contains("production-ready cinematic highlight edit")
|
||||
.contains("clipId: clip_00001")
|
||||
.contains("\"decisions\"");
|
||||
.contains("\"decisions\"")
|
||||
.doesNotContain("premium cinematic Porsche promo");
|
||||
assertThat(Files.readString(Path.of(files.readmePath())))
|
||||
.contains("Codex, Claude")
|
||||
.contains("`inbox/edit-plan.json`");
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ class CinematicEditingIntegrationTest {
|
|||
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
||||
"cinematic grade", "integration sequence"))
|
||||
.toList();
|
||||
return new EditPlan(projectId, "cinematic-porsche-promo", 60, decisions, List.of(), List.of(),
|
||||
return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(),
|
||||
"mp4-h264-aac-320p", "Generated integration edit");
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class CinematicHighlightAnalyzerTest {
|
||||
|
||||
private final CinematicHighlightAnalyzer analyzer = new CinematicHighlightAnalyzer(
|
||||
Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC));
|
||||
|
||||
@Test
|
||||
void classifiesCarContentAndCreatesRankedCandidates() {
|
||||
EditProject project = project("Porsche mountain drive", "category-aware-cinematic-highlights");
|
||||
EditProjectAnalysis source = analysis(clip("porsche-drive.mp4", 45.0));
|
||||
|
||||
CinematicHighlightAnalysis result = analyzer.analyze(project, source);
|
||||
|
||||
assertThat(result.projectId()).isEqualTo("project-1");
|
||||
assertThat(result.category()).isEqualTo(ContentCategory.CAR_VLOG);
|
||||
assertThat(result.categoryConfidence()).isGreaterThan(0.7);
|
||||
assertThat(result.categoryReasons()).isNotEmpty();
|
||||
assertThat(result.candidates()).hasSizeGreaterThan(1);
|
||||
assertThat(result.candidates().get(0).score()).isGreaterThanOrEqualTo(result.candidates().get(1).score());
|
||||
assertThat(result.candidates()).allSatisfy(candidate -> {
|
||||
assertThat(candidate.sourceStartSeconds()).isGreaterThanOrEqualTo(0);
|
||||
assertThat(candidate.sourceEndSeconds()).isLessThanOrEqualTo(45.0);
|
||||
assertThat(candidate.reasons()).anyMatch(reason -> reason.contains("car content"));
|
||||
});
|
||||
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void classifiesFoodAndFamilyFromProjectMetadata() {
|
||||
assertThat(analyzer.analyze(project("Street food recipe", "cinematic"), analysis(clip("clip.mp4", 8)))
|
||||
.category()).isEqualTo(ContentCategory.FOOD_VLOG);
|
||||
assertThat(analyzer.analyze(project("Family birthday vlog", "cinematic"), analysis(clip("clip.mp4", 8)))
|
||||
.category()).isEqualTo(ContentCategory.FAMILY_VLOG);
|
||||
}
|
||||
|
||||
@Test
|
||||
void fallsBackToGenericWhenCategoryIsUnknown() {
|
||||
CinematicHighlightAnalysis result = analyzer.analyze(project("Untitled", "cinematic"),
|
||||
analysis(clip("scene.mp4", 5.0)));
|
||||
|
||||
assertThat(result.category()).isEqualTo(ContentCategory.GENERIC_VLOG);
|
||||
assertThat(result.candidates()).singleElement().satisfies(candidate -> {
|
||||
assertThat(candidate.sourceStartSeconds()).isEqualTo(0.0);
|
||||
assertThat(candidate.sourceEndSeconds()).isEqualTo(5.0);
|
||||
assertThat(candidate.suggestedRole()).isEqualTo("opening_hook");
|
||||
});
|
||||
}
|
||||
|
||||
private EditProject project(String name, String style) {
|
||||
Instant now = Instant.parse("2026-07-10T10:00:00Z");
|
||||
return new EditProject("project-1", name, EditProjectStatus.ANALYZED, "input", "output",
|
||||
60, style, true, true, true, now, now, null);
|
||||
}
|
||||
|
||||
private EditProjectAnalysis analysis(ClipAnalysis clip) {
|
||||
return new EditProjectAnalysis("project-1", List.of(clip), List.of(),
|
||||
Instant.parse("2026-07-10T10:00:00Z"));
|
||||
}
|
||||
|
||||
private ClipAnalysis clip(String sourcePath, double durationSeconds) {
|
||||
return new ClipAnalysis("clip_00001", sourcePath, durationSeconds, "h264", "aac",
|
||||
1920, 1080, 30.0, List.of(), null, null, 0.7, 0.8, 0.9);
|
||||
}
|
||||
}
|
||||
|
|
@ -60,6 +60,49 @@ class EditProjectAnalyzerTest {
|
|||
assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis);
|
||||
}
|
||||
|
||||
@Test
|
||||
void writesCinematicHighlightArtifactsWhenEnabled() throws Exception {
|
||||
Path input = Files.createDirectory(tempDir.resolve("porsche-input"));
|
||||
Path clip = Files.writeString(input.resolve("porsche-drive.mp4"), "video");
|
||||
FileSystemEditProjectStore store = store();
|
||||
store.createProject("project-1");
|
||||
writeProject(store, input, "Porsche Drive");
|
||||
EditClipDiscovery discovery = mock(EditClipDiscovery.class);
|
||||
FfmpegClipInspector inspector = mock(FfmpegClipInspector.class);
|
||||
ThumbnailExtractor thumbnailExtractor = mock(ThumbnailExtractor.class);
|
||||
ContactSheetGenerator contactSheetGenerator = mock(ContactSheetGenerator.class);
|
||||
ProxyGenerator proxyGenerator = mock(ProxyGenerator.class);
|
||||
ClipAnalysis inspected = clipAnalysis(clip);
|
||||
when(discovery.discover(input)).thenReturn(List.of(clip));
|
||||
when(inspector.inspect(clip)).thenReturn(inspected);
|
||||
when(thumbnailExtractor.extract(inspected, tempDir.resolve("projects/project-1/thumbnails")))
|
||||
.thenReturn(List.of("thumb-1.jpg"));
|
||||
when(contactSheetGenerator.generate(inspected, tempDir.resolve("projects/project-1/contact-sheets")))
|
||||
.thenReturn("contact-sheet.jpg");
|
||||
when(proxyGenerator.generate(inspected, tempDir.resolve("projects/project-1/proxies")))
|
||||
.thenReturn(Optional.empty());
|
||||
EditProjectAnalyzer analyzer = new EditProjectAnalyzer(
|
||||
store,
|
||||
discovery,
|
||||
inspector,
|
||||
thumbnailExtractor,
|
||||
contactSheetGenerator,
|
||||
proxyGenerator,
|
||||
new CinematicHighlightAnalyzer(Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC)),
|
||||
Clock.fixed(Instant.parse("2026-07-10T10:00:00Z"), ZoneOffset.UTC),
|
||||
null
|
||||
);
|
||||
|
||||
analyzer.analyze("project-1", input);
|
||||
|
||||
CinematicHighlightAnalysis cinematic = store.readJson("project-1", "cinematic-highlight-analysis.json",
|
||||
CinematicHighlightAnalysis.class);
|
||||
assertThat(cinematic.category()).isEqualTo(ContentCategory.CAR_VLOG);
|
||||
assertThat(cinematic.candidates()).isNotEmpty();
|
||||
assertThat(Files.exists(tempDir.resolve("projects/project-1/category.json"))).isTrue();
|
||||
assertThat(Files.exists(tempDir.resolve("projects/project-1/highlight-candidates.json"))).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
void recordsInvalidClipErrorsAndKeepsUsableClips() throws Exception {
|
||||
Path input = Files.createDirectory(tempDir.resolve("input"));
|
||||
|
|
@ -148,6 +191,13 @@ class EditProjectAnalyzerTest {
|
|||
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||
}
|
||||
|
||||
private void writeProject(FileSystemEditProjectStore store, Path input, String name) {
|
||||
Instant now = Instant.parse("2026-07-10T10:00:00Z");
|
||||
store.writeJson("project-1", "project.json", new EditProject(
|
||||
"project-1", name, EditProjectStatus.ANALYZING, input.toString(), "output",
|
||||
60, "category-aware-cinematic-highlights", true, true, true, now, now, null));
|
||||
}
|
||||
|
||||
private ClipAnalysis clipAnalysis(Path clip) {
|
||||
return new ClipAnalysis(
|
||||
clip.getFileName().toString().replace(".mp4", ""),
|
||||
|
|
|
|||
|
|
@ -46,6 +46,33 @@ class StoryboardPromptGeneratorTest {
|
|||
assertThat(response.promptPath()).endsWith("storyboard-prompt.md");
|
||||
}
|
||||
|
||||
@Test
|
||||
void includesCategorySpecificHighlightCandidatesWhenAvailable() {
|
||||
FileSystemEditProjectStore store = store();
|
||||
store.createProject("porsche-edit");
|
||||
store.writeJson("porsche-edit", "project.json", project());
|
||||
store.writeJson("porsche-edit", "analysis.json", analysis());
|
||||
store.writeJson("porsche-edit", "cinematic-highlight-analysis.json", new CinematicHighlightAnalysis(
|
||||
"porsche-edit",
|
||||
ContentCategory.CAR_VLOG,
|
||||
0.82,
|
||||
List.of("Project metadata suggests vehicle content."),
|
||||
List.of(new HighlightCandidate("clip_00001_candidate_01", "clip_00001",
|
||||
0.0, 8.0, 0.91, "opening_hook", List.of("Inspect for hero car angle."))),
|
||||
Instant.parse("2026-07-10T10:01:00Z")
|
||||
));
|
||||
|
||||
String prompt = new StoryboardPromptGenerator(store).prompt(project(), analysis());
|
||||
|
||||
assertThat(prompt)
|
||||
.contains("Detected content category: `car_vlog`")
|
||||
.contains("Ranked highlight candidates")
|
||||
.contains("candidateId: clip_00001_candidate_01")
|
||||
.contains("Make the edit feel premium, powerful, and precise")
|
||||
.contains("\"contentCategory\": \"car_vlog\"")
|
||||
.contains("\"overlays\"");
|
||||
}
|
||||
|
||||
private FileSystemEditProjectStore store() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
|
||||
|
|
|
|||
Loading…
Reference in New Issue