forked from jsl/video_editing_poc
1487 lines
39 KiB
Markdown
1487 lines
39 KiB
Markdown
# Cinematic Video Editing Service Implementation Plan
|
|
|
|
## Goal
|
|
|
|
Extend the video clipping service so it can turn a folder or set of existing clips into a new cinematic promotional video.
|
|
|
|
Primary use case:
|
|
|
|
- input: about 35 Porsche clips
|
|
- output: one cinematic MP4 video
|
|
- style: appealing, exciting, premium automotive promo
|
|
- additions: music, sound effects, voiceover praising the Porsche, transitions, timing, and color treatment
|
|
|
|
The expensive AI model should do high-value creative planning only. The service should do the heavy media work with deterministic tools such as FFmpeg. A lower-cost coding model should be able to implement this plan milestone by milestone.
|
|
|
|
## Core Principle
|
|
|
|
Do not send full-resolution video through an AI model.
|
|
|
|
Use the service to extract compact metadata and preview assets:
|
|
|
|
- durations
|
|
- codecs
|
|
- dimensions
|
|
- frame thumbnails
|
|
- contact sheets
|
|
- low-resolution preview proxies
|
|
- optional audio loudness data
|
|
- optional scene or motion scores
|
|
|
|
Then use AI to generate:
|
|
|
|
- creative direction
|
|
- shot selection
|
|
- edit decision list
|
|
- voiceover script
|
|
- music and sound effect cue list
|
|
- final render instructions
|
|
|
|
The service renders the final video from the plan.
|
|
|
|
## Non-Goals
|
|
|
|
- Browser-based nonlinear editor UI.
|
|
- Manual timeline editing interface.
|
|
- Real-time collaborative editing.
|
|
- Full professional color grading system.
|
|
- Training custom AI models.
|
|
- Passing raw full-resolution video to an LLM.
|
|
|
|
## Proposed Workflow
|
|
|
|
```text
|
|
Input clips
|
|
-> analyze media with ffprobe and ffmpeg
|
|
-> extract thumbnails and low-resolution proxies
|
|
-> build compact clip manifest
|
|
-> AI creates cinematic storyboard and edit decision list
|
|
-> service validates the plan
|
|
-> service generates or imports voiceover, music, and SFX assets
|
|
-> service renders final MP4 with ffmpeg
|
|
-> service stores render output and manifest
|
|
```
|
|
|
|
## Local AI Director Workflow
|
|
|
|
The service should support a local workflow where the video editing service runs on the user's machine and an AI coding/chat agent such as Codex or Claude acts as the director.
|
|
|
|
This mode should not require the AI agent to directly process video files. The service prepares a project folder containing metadata, thumbnails, contact sheets, and a strict prompt. The AI agent reviews those artifacts, writes an `edit-plan.json`, and asks the service to render the result.
|
|
|
|
Recommended local folder flow:
|
|
|
|
```text
|
|
input/editing/source/
|
|
porsche-drive-001.mp4
|
|
porsche-drive-002.mp4
|
|
|
|
service scheduler
|
|
-> detects a new source folder or batch
|
|
-> creates output/edit-projects/<project-id>/
|
|
-> analyzes clips
|
|
-> extracts thumbnails and contact sheets
|
|
-> writes analysis.json
|
|
-> writes ai-director-prompt.md
|
|
-> marks project as WAITING_FOR_DIRECTOR
|
|
|
|
AI director
|
|
-> reads ai-director-prompt.md
|
|
-> reviews contact-sheets/ and thumbnails/
|
|
-> selects the strongest shots
|
|
-> writes edit-plan.json
|
|
-> optionally writes voiceover-script.txt and audio shopping list
|
|
|
|
service renderer
|
|
-> validates edit-plan.json
|
|
-> renders final.mp4
|
|
-> writes render-manifest.json
|
|
```
|
|
|
|
Suggested local command:
|
|
|
|
```bash
|
|
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
|
```
|
|
|
|
Then place a folder of clips here:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/
|
|
```
|
|
|
|
The service should produce:
|
|
|
|
```text
|
|
output/edit-projects/<project-id>/ai-director-prompt.md
|
|
output/edit-projects/<project-id>/analysis.json
|
|
output/edit-projects/<project-id>/contact-sheets/
|
|
output/edit-projects/<project-id>/thumbnails/
|
|
```
|
|
|
|
The AI director should only need those generated artifacts to make editing decisions. It should not need to inspect full-resolution clips unless the user explicitly asks for manual review of a specific moment.
|
|
|
|
### AI Director Responsibilities
|
|
|
|
The AI director should:
|
|
|
|
- judge clips from thumbnails, contact sheets, metadata, and optional proxies
|
|
- identify the best hero shots, detail shots, motion shots, and ending shot
|
|
- choose clip order and trim ranges
|
|
- write a strict `edit-plan.json`
|
|
- write or update voiceover lines
|
|
- suggest music and SFX cues
|
|
- keep the final timeline within the configured target duration
|
|
|
|
The AI director should not:
|
|
|
|
- render video
|
|
- run heavy FFmpeg commands unless troubleshooting
|
|
- modify source clips
|
|
- invent clip IDs or timestamps outside `analysis.json`
|
|
- depend on private absolute paths in viewer-facing text
|
|
|
|
### Service Responsibilities In AI Director Mode
|
|
|
|
The service should:
|
|
|
|
- watch a configurable local source directory for new project folders
|
|
- claim one project folder at a time
|
|
- ignore folders ending in `.tmp`, `.part`, `.download`, or `.processing`
|
|
- create deterministic project IDs from folder names with numeric suffixes on collision
|
|
- extract enough thumbnails for visual judgment
|
|
- create contact sheets that are easy for an AI vision model to inspect
|
|
- write a director prompt with exact instructions and JSON schema
|
|
- wait for `edit-plan.json`
|
|
- validate the plan before rendering
|
|
- render final output deterministically
|
|
|
|
This mode allows a strong AI model to do creative direction once, then a cheaper model or the service itself can execute the mechanical steps.
|
|
|
|
## Output Targets
|
|
|
|
Default final render:
|
|
|
|
- container: MP4
|
|
- video codec: H.264
|
|
- audio codec: AAC
|
|
- resolution: same as source if all clips match, otherwise configurable default `1920x1080`
|
|
- frame rate: same as source if all clips match, otherwise configurable default `30`
|
|
- audio sample rate: `48000`
|
|
- max duration: configurable, default `60` seconds
|
|
|
|
For Porsche promo videos, default creative structure:
|
|
|
|
```text
|
|
0-5s: hook, hero reveal, engine or impact sound
|
|
5-15s: exterior beauty shots and badge/details
|
|
15-30s: driving motion, acceleration, road energy
|
|
30-45s: interior, craftsmanship, handling, lifestyle shots
|
|
45-55s: strongest montage sequence
|
|
55-60s: final hero shot and short tagline
|
|
```
|
|
|
|
## Data Model
|
|
|
|
Add package:
|
|
|
|
```text
|
|
src/main/java/org/example/videoclips/editing
|
|
```
|
|
|
|
Suggested domain records:
|
|
|
|
```java
|
|
record EditProject(
|
|
String id,
|
|
String name,
|
|
EditProjectStatus status,
|
|
Path inputDirectory,
|
|
Path outputDirectory,
|
|
Instant createdAt,
|
|
Instant updatedAt
|
|
) {}
|
|
|
|
enum EditProjectStatus {
|
|
CREATED,
|
|
ANALYZING,
|
|
ANALYZED,
|
|
WAITING_FOR_DIRECTOR,
|
|
PLANNING,
|
|
PLANNED,
|
|
RENDERING,
|
|
RENDERED,
|
|
FAILED
|
|
}
|
|
|
|
record ClipAnalysis(
|
|
String clipId,
|
|
Path sourcePath,
|
|
double durationSeconds,
|
|
String videoCodec,
|
|
String audioCodec,
|
|
int width,
|
|
int height,
|
|
double frameRate,
|
|
List<Path> thumbnails,
|
|
Path contactSheet,
|
|
Path proxyPath,
|
|
double motionScore,
|
|
double brightnessScore,
|
|
double sharpnessScore
|
|
) {}
|
|
|
|
record EditDecision(
|
|
String clipId,
|
|
double sourceStartSeconds,
|
|
double sourceEndSeconds,
|
|
double timelineStartSeconds,
|
|
double timelineEndSeconds,
|
|
String transitionIn,
|
|
String transitionOut,
|
|
double playbackSpeed,
|
|
String visualTreatment,
|
|
String reason
|
|
) {}
|
|
|
|
record AudioCue(
|
|
String type,
|
|
String assetKey,
|
|
double timelineStartSeconds,
|
|
double timelineEndSeconds,
|
|
double gainDb,
|
|
String notes
|
|
) {}
|
|
|
|
record VoiceoverLine(
|
|
String text,
|
|
double timelineStartSeconds,
|
|
double timelineEndSeconds,
|
|
String delivery
|
|
) {}
|
|
|
|
record EditPlan(
|
|
String projectId,
|
|
String style,
|
|
double targetDurationSeconds,
|
|
List<EditDecision> decisions,
|
|
List<AudioCue> audioCues,
|
|
List<VoiceoverLine> voiceover,
|
|
String renderProfile,
|
|
String summary
|
|
) {}
|
|
```
|
|
|
|
For the first implementation, these can be JSON files on disk. Do not start with JPA unless project persistence is required by the user later.
|
|
|
|
## API Contract
|
|
|
|
Add REST endpoints under `/v1/edit-projects`.
|
|
|
|
### Create Edit Project
|
|
|
|
`POST /v1/edit-projects`
|
|
|
|
Request:
|
|
|
|
```json
|
|
{
|
|
"name": "porsche-cinematic",
|
|
"inputDirectory": "./output/clips/20240708_173213",
|
|
"targetDurationSeconds": 60,
|
|
"style": "cinematic-porsche-promo",
|
|
"voiceoverEnabled": true,
|
|
"musicEnabled": true,
|
|
"soundEffectsEnabled": true
|
|
}
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"projectId": "edit_01JZABCDE",
|
|
"status": "CREATED"
|
|
}
|
|
```
|
|
|
|
Validation:
|
|
|
|
- `name`: required, filesystem-safe after normalization
|
|
- `inputDirectory`: required and must exist
|
|
- `targetDurationSeconds`: `15..600`
|
|
- `style`: required, initially allow `cinematic-porsche-promo`
|
|
|
|
### Analyze Clips
|
|
|
|
`POST /v1/edit-projects/{projectId}:analyze`
|
|
|
|
Behavior:
|
|
|
|
- scans project input directory
|
|
- accepts valid video files only
|
|
- runs `ffprobe`
|
|
- extracts thumbnails
|
|
- creates contact sheets
|
|
- optionally creates low-resolution proxy files
|
|
- writes `analysis.json`
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"projectId": "edit_01JZABCDE",
|
|
"status": "ANALYZED",
|
|
"clipCount": 35,
|
|
"analysisPath": "./output/edit-projects/edit_01JZABCDE/analysis.json"
|
|
}
|
|
```
|
|
|
|
### Generate Storyboard Prompt
|
|
|
|
`POST /v1/edit-projects/{projectId}:storyboard-prompt`
|
|
|
|
Behavior:
|
|
|
|
- reads `analysis.json`
|
|
- produces a compact AI prompt
|
|
- includes clip summaries, thumbnail references, timing constraints, and output schema
|
|
- writes `storyboard-prompt.md`
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"projectId": "edit_01JZABCDE",
|
|
"promptPath": "./output/edit-projects/edit_01JZABCDE/storyboard-prompt.md"
|
|
}
|
|
```
|
|
|
|
### Save Edit Plan
|
|
|
|
`PUT /v1/edit-projects/{projectId}/plan`
|
|
|
|
Request:
|
|
|
|
```json
|
|
{
|
|
"style": "cinematic-porsche-promo",
|
|
"targetDurationSeconds": 60,
|
|
"decisions": [],
|
|
"audioCues": [],
|
|
"voiceover": [],
|
|
"renderProfile": "mp4-h264-aac-1080p",
|
|
"summary": "Cinematic Porsche promo edit."
|
|
}
|
|
```
|
|
|
|
Behavior:
|
|
|
|
- validates the AI-generated edit plan
|
|
- ensures all clip IDs exist
|
|
- ensures source time ranges are inside clip durations
|
|
- ensures timeline ranges do not overlap incorrectly
|
|
- writes `edit-plan.json`
|
|
|
|
### Render Edit
|
|
|
|
`POST /v1/edit-projects/{projectId}:render`
|
|
|
|
Behavior:
|
|
|
|
- reads `edit-plan.json`
|
|
- renders the final video with FFmpeg
|
|
- mixes music, SFX, and voiceover if available
|
|
- writes final MP4 and `render-manifest.json`
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"projectId": "edit_01JZABCDE",
|
|
"status": "RENDERED",
|
|
"outputPath": "./output/edit-projects/edit_01JZABCDE/final.mp4",
|
|
"durationSeconds": 60
|
|
}
|
|
```
|
|
|
|
### Read Project
|
|
|
|
`GET /v1/edit-projects/{projectId}`
|
|
|
|
Return current status, paths, timestamps, and error details if failed.
|
|
|
|
## File Layout
|
|
|
|
For local filesystem implementation:
|
|
|
|
```text
|
|
output/edit-projects/<project-id>/
|
|
project.json
|
|
analysis.json
|
|
storyboard-prompt.md
|
|
edit-plan.json
|
|
render-manifest.json
|
|
final.mp4
|
|
thumbnails/
|
|
<clip-id>_0001.jpg
|
|
<clip-id>_0002.jpg
|
|
contact-sheets/
|
|
<clip-id>.jpg
|
|
proxies/
|
|
<clip-id>.mp4
|
|
audio/
|
|
voiceover.wav
|
|
music.wav
|
|
sfx/
|
|
inbox/
|
|
edit-plan.json
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Add YAML config under `video-clipping.editing`.
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
enabled: true
|
|
project-directory: ./output/edit-projects
|
|
ffmpeg-binary: ffmpeg
|
|
ffprobe-binary: ffprobe
|
|
thumbnail-count-per-clip: 5
|
|
contact-sheet-columns: 5
|
|
proxy-enabled: true
|
|
proxy-width: 640
|
|
target-duration-seconds: 60
|
|
output-width: 1920
|
|
output-height: 1080
|
|
output-frame-rate: 30
|
|
audio-sample-rate: 48000
|
|
video-bitrate: 12000k
|
|
audio-bitrate: 192k
|
|
local-director:
|
|
enabled: true
|
|
source-directory: ./input/editing/source
|
|
working-directory: ./input/editing/working
|
|
processed-directory: ./input/editing/processed
|
|
rejected-directory: ./input/editing/rejected
|
|
poll-interval-ms: 5000
|
|
director-prompt-file-name: ai-director-prompt.md
|
|
expected-plan-file-name: edit-plan.json
|
|
auto-render-when-plan-appears: false
|
|
```
|
|
|
|
Local director defaults:
|
|
|
|
- `auto-render-when-plan-appears=false` is safer for early development because the user can inspect `edit-plan.json` before rendering.
|
|
- When set to `true`, the service should render automatically after a valid plan appears in the project `inbox/` folder.
|
|
- The scheduler should process one project folder at a time, using the same repeat-prevention pattern as the input folder scheduler.
|
|
|
|
## FFmpeg Commands
|
|
|
|
### Probe Clip
|
|
|
|
```bash
|
|
ffprobe -v error \
|
|
-show_entries format=duration \
|
|
-show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
|
|
-of json \
|
|
clip_00001.mp4
|
|
```
|
|
|
|
### Extract Thumbnail
|
|
|
|
Use timestamps spread across clip duration.
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-ss 3.2 \
|
|
-i clip_00001.mp4 \
|
|
-frames:v 1 \
|
|
-q:v 2 \
|
|
thumbnails/clip_00001_0001.jpg
|
|
```
|
|
|
|
### Create Contact Sheet
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-i clip_00001.mp4 \
|
|
-vf "fps=1/2,scale=320:-1,tile=5x5" \
|
|
-frames:v 1 \
|
|
contact-sheets/clip_00001.jpg
|
|
```
|
|
|
|
### Create Proxy
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-i clip_00001.mp4 \
|
|
-vf "scale=640:-2" \
|
|
-c:v libx264 -preset veryfast -crf 28 \
|
|
-c:a aac -b:a 96k \
|
|
proxies/clip_00001.mp4
|
|
```
|
|
|
|
### Render Final Video
|
|
|
|
First implementation should use a concat-based render:
|
|
|
|
1. Trim each selected segment into an intermediate file.
|
|
2. Normalize resolution, frame rate, pixel format, and audio.
|
|
3. Concatenate intermediates.
|
|
4. Mix music, SFX, and voiceover.
|
|
5. Write final MP4.
|
|
|
|
This is simpler and easier for a lower-cost coding model than building one huge `filter_complex` command.
|
|
|
|
Intermediate segment command:
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-ss <sourceStart> \
|
|
-to <sourceEnd> \
|
|
-i <sourceClip> \
|
|
-vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,format=yuv420p" \
|
|
-r 30 \
|
|
-c:v libx264 -preset veryfast -crf 18 \
|
|
-c:a aac -b:a 192k -ar 48000 \
|
|
<work>/segment_0001.mp4
|
|
```
|
|
|
|
Concat command:
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-f concat \
|
|
-safe 0 \
|
|
-i concat.txt \
|
|
-c copy \
|
|
<work>/timeline-video.mp4
|
|
```
|
|
|
|
Final mix command:
|
|
|
|
```bash
|
|
ffmpeg -hide_banner -y \
|
|
-i <work>/timeline-video.mp4 \
|
|
-i audio/music.wav \
|
|
-i audio/voiceover.wav \
|
|
-filter_complex "[1:a]volume=0.25[music];[2:a]volume=1.0[voice];[0:a][music][voice]amix=inputs=3:duration=first:dropout_transition=2[a]" \
|
|
-map 0:v -map "[a]" \
|
|
-c:v copy \
|
|
-c:a aac -b:a 192k \
|
|
final.mp4
|
|
```
|
|
|
|
Add SFX later after the basic music and voiceover mix works.
|
|
|
|
## AI Storyboard Prompt Requirements
|
|
|
|
The generated prompt should instruct the AI to output strict JSON only.
|
|
|
|
Required input context:
|
|
|
|
- project style
|
|
- target duration
|
|
- clip IDs
|
|
- duration per clip
|
|
- dimensions and frame rate
|
|
- extracted thumbnail paths or descriptions
|
|
- any motion, brightness, and sharpness scores
|
|
- known constraints
|
|
|
|
Required output schema:
|
|
|
|
```json
|
|
{
|
|
"style": "cinematic-porsche-promo",
|
|
"targetDurationSeconds": 60,
|
|
"summary": "string",
|
|
"decisions": [
|
|
{
|
|
"clipId": "clip_00001",
|
|
"sourceStartSeconds": 0.0,
|
|
"sourceEndSeconds": 2.5,
|
|
"timelineStartSeconds": 0.0,
|
|
"timelineEndSeconds": 2.5,
|
|
"transitionIn": "cut",
|
|
"transitionOut": "crossfade",
|
|
"playbackSpeed": 1.0,
|
|
"visualTreatment": "high contrast warm cinematic grade",
|
|
"reason": "hero front angle"
|
|
}
|
|
],
|
|
"voiceover": [
|
|
{
|
|
"text": "This Porsche turns precision into emotion.",
|
|
"timelineStartSeconds": 3.0,
|
|
"timelineEndSeconds": 6.0,
|
|
"delivery": "confident cinematic narrator"
|
|
}
|
|
],
|
|
"audioCues": [
|
|
{
|
|
"type": "music",
|
|
"assetKey": "cinematic-driving-bed",
|
|
"timelineStartSeconds": 0.0,
|
|
"timelineEndSeconds": 60.0,
|
|
"gainDb": -12.0,
|
|
"notes": "build energy toward final montage"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Prompt rules:
|
|
|
|
- Never reference clip paths directly in final viewer-facing text.
|
|
- Keep total timeline duration within the requested target duration.
|
|
- Use at least 12 clips if enough usable clips exist.
|
|
- Avoid using any single source clip for more than 20% of the final runtime.
|
|
- Prefer a strong first 5 seconds.
|
|
- End with the strongest exterior or motion shot.
|
|
- Voiceover should praise the car without sounding exaggerated or fake.
|
|
|
|
## AI Director Prompt File
|
|
|
|
In local director mode, generate `ai-director-prompt.md` for Codex, Claude, or another AI instance.
|
|
|
|
The prompt must include:
|
|
|
|
- project ID and style
|
|
- target duration
|
|
- path to `analysis.json`
|
|
- paths to contact sheets and thumbnails
|
|
- exact output location for the plan
|
|
- strict JSON schema for `edit-plan.json`
|
|
- validation rules
|
|
- instruction to judge shots from thumbnails/contact sheets
|
|
- instruction to produce a cinematic Porsche promo edit
|
|
|
|
Suggested prompt content:
|
|
|
|
```md
|
|
You are the director for a cinematic Porsche promo edit.
|
|
|
|
Use only the project artifacts in this folder:
|
|
|
|
- `analysis.json`
|
|
- `contact-sheets/`
|
|
- `thumbnails/`
|
|
- optional `proxies/`
|
|
|
|
Your task:
|
|
|
|
1. Review the contact sheets and thumbnails.
|
|
2. Pick the strongest shots.
|
|
3. Create a cinematic edit plan with a strong opening, rising energy, and final hero shot.
|
|
4. Write voiceover lines that praise the Porsche in a premium but believable tone.
|
|
5. Add music and SFX cues.
|
|
6. Write strict JSON to `inbox/edit-plan.json`.
|
|
|
|
Do not render video. Do not modify source clips. Do not invent clip IDs.
|
|
```
|
|
|
|
The service should write a matching `director-readme.md` with short human instructions:
|
|
|
|
```text
|
|
Open ai-director-prompt.md in Codex or Claude. Let the AI inspect the generated thumbnails and contact sheets. When it writes inbox/edit-plan.json, call the render endpoint or enable auto-render.
|
|
```
|
|
|
|
## Local Director Scheduler
|
|
|
|
Add a scheduler similar to the input folder scheduler, but for edit project folders.
|
|
|
|
States:
|
|
|
|
```text
|
|
source -> working -> processed
|
|
source -> working -> rejected
|
|
```
|
|
|
|
Folder example:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/
|
|
clip_00000.mp4
|
|
clip_00001.mp4
|
|
```
|
|
|
|
Claimed folder:
|
|
|
|
```text
|
|
input/editing/working/porsche-session-001/
|
|
```
|
|
|
|
Project output:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/
|
|
```
|
|
|
|
Scheduler behavior:
|
|
|
|
1. scan `local-director.source-directory`
|
|
2. pick the first valid project folder in deterministic order
|
|
3. move it to `working-directory`
|
|
4. create an edit project
|
|
5. analyze clips
|
|
6. generate `ai-director-prompt.md`
|
|
7. write status `WAITING_FOR_DIRECTOR`
|
|
8. move the source folder to `processed-directory` after project creation succeeds
|
|
9. leave the project output folder ready for the AI director
|
|
|
|
Plan pickup behavior:
|
|
|
|
1. scan project `inbox/` folders for `edit-plan.json`
|
|
2. validate the plan
|
|
3. if `auto-render-when-plan-appears=true`, render automatically
|
|
4. otherwise expose the project as `PLANNED` and wait for `POST /v1/edit-projects/{projectId}:render`
|
|
|
|
Tests:
|
|
|
|
- ignores temporary project folders
|
|
- processes one project folder at a time
|
|
- creates project artifacts and director prompt
|
|
- does not reprocess a processed folder
|
|
- rejects empty folders
|
|
- validates plan pickup
|
|
- respects `auto-render-when-plan-appears`
|
|
|
|
## Voiceover And Audio
|
|
|
|
First implementation can support imported audio assets only:
|
|
|
|
- user places `music.wav` and `voiceover.wav` under project `audio/`
|
|
- service mixes those files into the final render
|
|
|
|
Second implementation can add generated voiceover:
|
|
|
|
- create `voiceover-script.txt`
|
|
- call an external TTS provider or OpenAI audio model outside the core renderer
|
|
- save `voiceover.wav`
|
|
- keep generation behind an interface so it can be mocked in tests
|
|
|
|
Suggested interface:
|
|
|
|
```java
|
|
interface VoiceoverGenerator {
|
|
Path generateVoiceover(String projectId, List<VoiceoverLine> lines);
|
|
}
|
|
```
|
|
|
|
Suggested first adapter:
|
|
|
|
```text
|
|
NoopVoiceoverGenerator
|
|
```
|
|
|
|
It writes the script only and requires an imported audio file.
|
|
|
|
## Milestones
|
|
|
|
1. [x] Add editing configuration under `video-clipping.editing`.
|
|
2. [ ] Add filesystem project store for `project.json`, `analysis.json`, `edit-plan.json`, and render output.
|
|
3. [ ] Add edit project domain records and status enum.
|
|
4. [ ] Add `POST /v1/edit-projects` and `GET /v1/edit-projects/{projectId}`.
|
|
5. [ ] Add clip discovery for project input directories.
|
|
6. [ ] Add `ffprobe` analysis and validation for all input clips.
|
|
7. [ ] Add thumbnail extraction.
|
|
8. [ ] Add contact sheet generation.
|
|
9. [ ] Add optional proxy generation.
|
|
10. [ ] Add `analysis.json` writing and reading.
|
|
11. [ ] Add storyboard prompt generation from `analysis.json`.
|
|
12. [ ] Add local AI director prompt generation with thumbnail/contact-sheet instructions.
|
|
13. [ ] Add local director source-folder scheduler.
|
|
14. [ ] Add project `inbox/` plan pickup.
|
|
15. [ ] Add strict JSON edit plan schema and validation.
|
|
16. [ ] Add `PUT /v1/edit-projects/{projectId}/plan`.
|
|
17. [ ] Add simple segment rendering without transitions.
|
|
18. [ ] Add concat-based final timeline rendering.
|
|
19. [ ] Add music and voiceover audio mixing.
|
|
20. [ ] Add render manifest with command metadata and output details.
|
|
21. [ ] Add basic transition support: cut, crossfade, fade in, fade out.
|
|
22. [ ] Add SFX cue support.
|
|
23. [ ] Add optional generated voiceover adapter interface.
|
|
24. [ ] Add structured logs and metrics for analysis, planning, and rendering durations.
|
|
25. [ ] Add integration test with generated fixture clips and a short edit plan.
|
|
26. [ ] Add documentation for running a local cinematic edit with Codex or Claude as director.
|
|
27. [ ] Run `mvn verify` and update this checklist.
|
|
|
|
## Milestone Details
|
|
|
|
### Milestone 1: Editing Configuration
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/config/VideoClippingProperties.java`
|
|
- `src/main/resources/application.yml`
|
|
|
|
Add nested properties class:
|
|
|
|
```java
|
|
private final Editing editing = new Editing();
|
|
|
|
public Editing getEditing() {
|
|
return editing;
|
|
}
|
|
```
|
|
|
|
Add fields matching the YAML in this plan.
|
|
|
|
Tests:
|
|
|
|
- add a focused config binding test or extend an existing Spring context test
|
|
- verify defaults bind correctly
|
|
|
|
Acceptance criteria:
|
|
|
|
- application starts with editing config present
|
|
- editing can be disabled with `video-clipping.editing.enabled=false`
|
|
|
|
### Milestone 2: Filesystem Project Store
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditProjectStore.java`
|
|
- `src/main/java/org/example/videoclips/editing/FileSystemEditProjectStore.java`
|
|
|
|
Responsibilities:
|
|
|
|
- create project directories
|
|
- read/write JSON files with Jackson
|
|
- normalize project IDs and reject path traversal
|
|
- expose methods for project, analysis, plan, and manifest paths
|
|
|
|
Tests:
|
|
|
|
- creates expected folder structure
|
|
- rejects invalid project IDs
|
|
- round-trips `project.json`
|
|
|
|
Acceptance criteria:
|
|
|
|
- no direct filesystem writes from controllers
|
|
- all project path creation goes through the store
|
|
|
|
### Milestone 3: Domain Records
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditProject.java`
|
|
- `src/main/java/org/example/videoclips/editing/EditProjectStatus.java`
|
|
- `src/main/java/org/example/videoclips/editing/ClipAnalysis.java`
|
|
- `src/main/java/org/example/videoclips/editing/EditPlan.java`
|
|
- `src/main/java/org/example/videoclips/editing/EditDecision.java`
|
|
- `src/main/java/org/example/videoclips/editing/AudioCue.java`
|
|
- `src/main/java/org/example/videoclips/editing/VoiceoverLine.java`
|
|
|
|
Tests:
|
|
|
|
- JSON serialization for representative records
|
|
|
|
Acceptance criteria:
|
|
|
|
- records serialize with stable camelCase JSON
|
|
- no business logic in records beyond validation helpers if needed
|
|
|
|
### Milestone 4: Project API
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/api/EditProjectController.java`
|
|
- `src/main/java/org/example/videoclips/api/dto/CreateEditProjectRequest.java`
|
|
- `src/main/java/org/example/videoclips/api/dto/EditProjectResponse.java`
|
|
|
|
Tests:
|
|
|
|
- create project success
|
|
- reject missing input directory
|
|
- reject invalid target duration
|
|
- get project success
|
|
- get missing project returns `404`
|
|
|
|
Acceptance criteria:
|
|
|
|
- API returns project ID and status
|
|
- project state is persisted to `project.json`
|
|
|
|
### Milestone 5-10: Analysis Pipeline
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java`
|
|
- `src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java`
|
|
- `src/main/java/org/example/videoclips/editing/ThumbnailExtractor.java`
|
|
- `src/main/java/org/example/videoclips/editing/ContactSheetGenerator.java`
|
|
- `src/main/java/org/example/videoclips/editing/ProxyGenerator.java`
|
|
|
|
Implementation notes:
|
|
|
|
- reuse patterns from `FolderVideoValidator` and `FolderFfmpegClipper`
|
|
- keep process execution behind package-private interfaces for tests
|
|
- sanitize process output before logging
|
|
- process clips in deterministic filename order
|
|
|
|
Tests:
|
|
|
|
- analysis ignores non-video files
|
|
- invalid videos are reported in analysis errors
|
|
- command construction is correct
|
|
- output JSON includes all analyzed clips
|
|
- proxy generation can be disabled
|
|
|
|
Acceptance criteria:
|
|
|
|
- `analysis.json` is enough for an AI prompt without reading raw video
|
|
- failure of one invalid clip does not fail the entire project if at least one valid clip exists
|
|
|
|
### Milestone 11: Storyboard Prompt
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java`
|
|
|
|
Tests:
|
|
|
|
- prompt includes clip IDs, durations, target duration, style, and JSON schema
|
|
- prompt excludes raw absolute system paths unless explicitly configured
|
|
|
|
Acceptance criteria:
|
|
|
|
- generated prompt can be pasted into an AI model
|
|
- model output requirements are strict JSON
|
|
|
|
### Milestone 12: Local AI Director Prompt
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java`
|
|
|
|
Behavior:
|
|
|
|
- generate `ai-director-prompt.md`
|
|
- include project style, target duration, clip metadata, contact sheet paths, thumbnail paths, and strict output schema
|
|
- write human-readable `director-readme.md`
|
|
- instruct the AI to write `inbox/edit-plan.json`
|
|
|
|
Tests:
|
|
|
|
- prompt contains `analysis.json`
|
|
- prompt contains `contact-sheets/` and `thumbnails/`
|
|
- prompt contains exact output path `inbox/edit-plan.json`
|
|
- prompt instructs the AI not to render video
|
|
|
|
Acceptance criteria:
|
|
|
|
- a user can open the prompt in Codex or Claude and get a valid edit plan without additional explanation
|
|
|
|
### Milestone 13: Local Director Source-Folder Scheduler
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java`
|
|
- `src/main/java/org/example/videoclips/editing/EditProjectDirectoryInitializer.java`
|
|
|
|
Behavior:
|
|
|
|
- create configured source, working, processed, rejected, and project output directories on startup
|
|
- scan `video-clipping.editing.local-director.source-directory`
|
|
- claim one project folder by moving it to working
|
|
- ignore hidden and temporary folders
|
|
- create an edit project
|
|
- analyze the clips
|
|
- generate `ai-director-prompt.md`
|
|
- set project status to `WAITING_FOR_DIRECTOR`
|
|
- move the claimed source folder to processed after successful analysis
|
|
|
|
Tests:
|
|
|
|
- creates all local director directories
|
|
- selects project folders in deterministic order
|
|
- ignores `.tmp`, `.part`, `.download`, and `.processing` folders
|
|
- processes one project folder per scan
|
|
- writes project prompt and analysis files
|
|
- rejects empty project folders
|
|
|
|
Acceptance criteria:
|
|
|
|
- dropping a folder into `input/editing/source` produces a ready-to-direct project under `output/edit-projects`
|
|
|
|
### Milestone 14: Plan Inbox Pickup
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditPlanInboxScanner.java`
|
|
|
|
Behavior:
|
|
|
|
- scan project `inbox/` folders for `edit-plan.json`
|
|
- ignore partially written files with `.tmp`, `.part`, or `.download` suffix
|
|
- validate the plan
|
|
- save normalized plan to project root
|
|
- mark status `PLANNED`
|
|
- render automatically only when `auto-render-when-plan-appears=true`
|
|
|
|
Tests:
|
|
|
|
- ignores temporary plan files
|
|
- rejects invalid plan files
|
|
- accepts valid plan files
|
|
- does not render when auto-render is disabled
|
|
- invokes renderer when auto-render is enabled
|
|
|
|
Acceptance criteria:
|
|
|
|
- Codex or Claude can write `inbox/edit-plan.json` and the service can pick it up without a manual API call
|
|
|
|
### Milestone 15-16: Edit Plan Validation
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditPlanValidator.java`
|
|
- `src/main/java/org/example/videoclips/api/dto/SaveEditPlanRequest.java`
|
|
|
|
Validation rules:
|
|
|
|
- all clip IDs exist in `analysis.json`
|
|
- `sourceStartSeconds >= 0`
|
|
- `sourceEndSeconds > sourceStartSeconds`
|
|
- source range is inside clip duration
|
|
- timeline ranges are non-negative
|
|
- timeline ranges do not overlap unless the transition requires overlap
|
|
- final duration is within target duration tolerance
|
|
- playback speed is between `0.25` and `4.0`
|
|
- only supported transitions are accepted
|
|
|
|
Tests:
|
|
|
|
- accepts valid plan
|
|
- rejects unknown clip ID
|
|
- rejects out-of-range source timestamps
|
|
- rejects overlapping timeline decisions
|
|
- rejects unsupported transition
|
|
|
|
Acceptance criteria:
|
|
|
|
- renderer never receives an invalid plan
|
|
|
|
### Milestone 17-20: Render Pipeline
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/EditRenderer.java`
|
|
- `src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java`
|
|
- `src/main/java/org/example/videoclips/editing/RenderManifest.java`
|
|
|
|
Implementation sequence:
|
|
|
|
1. create working directory
|
|
2. render each selected decision into normalized segment MP4
|
|
3. write `concat.txt`
|
|
4. concatenate segments into `timeline-video.mp4`
|
|
5. mix audio when audio files exist
|
|
6. write `final.mp4`
|
|
7. write `render-manifest.json`
|
|
|
|
Tests:
|
|
|
|
- command construction for segment render
|
|
- command construction for concat
|
|
- command construction for audio mix
|
|
- render fails if no decisions exist
|
|
- manifest is written on success
|
|
|
|
Acceptance criteria:
|
|
|
|
- final MP4 exists after render
|
|
- render manifest includes input clip IDs, output path, duration, and command summaries
|
|
|
|
### Milestone 21-22: Transitions And SFX
|
|
|
|
Start simple.
|
|
|
|
Supported transitions:
|
|
|
|
- `cut`
|
|
- `fade-in`
|
|
- `fade-out`
|
|
- `crossfade`
|
|
|
|
Supported SFX cues:
|
|
|
|
- imported WAV files only
|
|
- cue has asset key, start time, gain, and optional fade
|
|
|
|
Tests:
|
|
|
|
- unsupported transition rejected
|
|
- SFX file missing is a validation error
|
|
- final mix includes SFX input when cue exists
|
|
|
|
Acceptance criteria:
|
|
|
|
- basic cinematic polish is possible without needing a full timeline engine
|
|
|
|
### Milestone 23: Voiceover Generation Hook
|
|
|
|
Files:
|
|
|
|
- `src/main/java/org/example/videoclips/editing/VoiceoverGenerator.java`
|
|
- `src/main/java/org/example/videoclips/editing/NoopVoiceoverGenerator.java`
|
|
|
|
Behavior:
|
|
|
|
- first version writes `voiceover-script.txt`
|
|
- does not call external APIs
|
|
- later adapters can call TTS providers
|
|
|
|
Tests:
|
|
|
|
- script file generated from voiceover lines
|
|
- noop adapter does not fail when TTS is disabled
|
|
|
|
Acceptance criteria:
|
|
|
|
- service supports the workflow without requiring paid audio generation
|
|
|
|
### Milestone 24: Logs And Metrics
|
|
|
|
Logs:
|
|
|
|
- `event=edit_project_created`
|
|
- `event=local_director_project_claimed`
|
|
- `event=edit_analysis_started`
|
|
- `event=edit_analysis_completed`
|
|
- `event=ai_director_prompt_generated`
|
|
- `event=storyboard_prompt_generated`
|
|
- `event=edit_plan_inbox_detected`
|
|
- `event=edit_plan_saved`
|
|
- `event=edit_render_started`
|
|
- `event=edit_render_completed`
|
|
- `event=edit_render_failed`
|
|
|
|
Metrics:
|
|
|
|
- `video.clipping.edit.projects.created`
|
|
- `video.clipping.edit.analysis.duration`
|
|
- `video.clipping.edit.render.duration`
|
|
- `video.clipping.edit.render.failures`
|
|
|
|
Tests:
|
|
|
|
- metrics binder test if custom metrics are added
|
|
- at minimum, unit tests should cover event-producing paths
|
|
|
|
Acceptance criteria:
|
|
|
|
- an operator can see which stage failed and how long rendering took
|
|
|
|
### Milestone 25: Integration Test
|
|
|
|
Create generated fixture clips with FFmpeg:
|
|
|
|
- three 3-second clips
|
|
- different colors or test patterns
|
|
- simple audio tones
|
|
|
|
Test flow:
|
|
|
|
1. place a folder of generated clips in local director source
|
|
2. run the local director scheduler
|
|
3. assert `ai-director-prompt.md` and `analysis.json` exist
|
|
4. write a short valid `inbox/edit-plan.json`
|
|
5. run the plan inbox scanner
|
|
6. render final video
|
|
7. probe final video
|
|
8. assert duration is greater than zero
|
|
|
|
Skip test when `ffmpeg` or `ffprobe` is unavailable.
|
|
|
|
Acceptance criteria:
|
|
|
|
- proves the end-to-end renderer works on a machine with FFmpeg installed
|
|
|
|
## Lower-Cost Model Implementation Instructions
|
|
|
|
Follow these rules when assigning this plan to a cheaper coding model:
|
|
|
|
- Implement exactly one milestone at a time.
|
|
- Run the relevant tests after each milestone.
|
|
- Update the checklist in this file after each completed milestone.
|
|
- Commit after each milestone if the repository is in a clean, test-passing state.
|
|
- Do not refactor unrelated packages.
|
|
- Reuse existing patterns from the folder scheduler for process execution, logging, and tests.
|
|
- Keep the first renderer deterministic and simple.
|
|
- Do not add external AI, TTS, or music-provider calls until the filesystem workflow works.
|
|
- For local director mode, implement file handoff first. Do not embed Codex, Claude, or another model inside the Spring service.
|
|
|
|
Suggested prompt for the lower-cost coding model:
|
|
|
|
```md
|
|
You are implementing `docs/cinematic-video-editing-service-plan.md`.
|
|
|
|
Implement the next unchecked milestone only. Keep the change scoped. Use existing patterns from the folder scheduler package where possible. Add tests for the milestone. Run the relevant Maven test command. Update the milestone checkbox only when tests pass. Do not implement future milestones early.
|
|
```
|
|
|
|
Suggested prompt for Codex or Claude acting as the local AI director:
|
|
|
|
```md
|
|
You are the AI director for this local edit project.
|
|
|
|
Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.
|
|
```
|
|
|
|
## Local Runbook
|
|
|
|
Use this runbook after the local AI director milestones are implemented.
|
|
|
|
### Prerequisites
|
|
|
|
Install:
|
|
|
|
- Java 21
|
|
- Maven
|
|
- FFmpeg
|
|
- ffprobe
|
|
- Codex, Claude, or another AI instance that can read local project files and images
|
|
|
|
Verify tools:
|
|
|
|
```bash
|
|
java -version
|
|
mvn -version
|
|
ffmpeg -version
|
|
ffprobe -version
|
|
```
|
|
|
|
### Start The Service
|
|
|
|
Run the service with the local cinematic editing profile:
|
|
|
|
```bash
|
|
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
|
```
|
|
|
|
Expected startup behavior:
|
|
|
|
- creates `input/editing/source`
|
|
- creates `input/editing/working`
|
|
- creates `input/editing/processed`
|
|
- creates `input/editing/rejected`
|
|
- creates `output/edit-projects`
|
|
- logs that local director mode is enabled
|
|
|
|
### Add Source Clips
|
|
|
|
Create one project folder under the local director source directory:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/
|
|
```
|
|
|
|
Place the clips inside that folder:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/clip_00000.mp4
|
|
input/editing/source/porsche-session-001/clip_00001.mp4
|
|
input/editing/source/porsche-session-001/clip_00002.mp4
|
|
```
|
|
|
|
Recommended copy flow for large files:
|
|
|
|
1. copy the folder as `porsche-session-001.tmp`
|
|
2. wait until all files are fully copied
|
|
3. rename it to `porsche-session-001`
|
|
|
|
The scheduler should ignore temporary folders ending in `.tmp`, `.part`, `.download`, or `.processing`.
|
|
|
|
### Wait For Analysis
|
|
|
|
The service should claim the folder and move it through:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001
|
|
input/editing/working/porsche-session-001
|
|
input/editing/processed/porsche-session-001
|
|
```
|
|
|
|
After analysis, find the project output:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/
|
|
```
|
|
|
|
Expected files:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/project.json
|
|
output/edit-projects/porsche-session-001/analysis.json
|
|
output/edit-projects/porsche-session-001/ai-director-prompt.md
|
|
output/edit-projects/porsche-session-001/director-readme.md
|
|
output/edit-projects/porsche-session-001/contact-sheets/
|
|
output/edit-projects/porsche-session-001/thumbnails/
|
|
output/edit-projects/porsche-session-001/inbox/
|
|
```
|
|
|
|
Expected status:
|
|
|
|
```text
|
|
WAITING_FOR_DIRECTOR
|
|
```
|
|
|
|
### Ask The AI Director To Edit
|
|
|
|
Open Codex, Claude, or the selected AI instance in the project output folder.
|
|
|
|
Use this prompt:
|
|
|
|
```md
|
|
You are the AI director for this local edit project.
|
|
|
|
Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.
|
|
```
|
|
|
|
The AI should inspect:
|
|
|
|
- `analysis.json`
|
|
- `contact-sheets/*.jpg`
|
|
- `thumbnails/*.jpg`
|
|
- optional `proxies/*.mp4`
|
|
|
|
The AI should write:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
|
```
|
|
|
|
### Validate And Render
|
|
|
|
If `auto-render-when-plan-appears=false`, trigger rendering manually:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render
|
|
```
|
|
|
|
If `auto-render-when-plan-appears=true`, the service should render automatically after it detects and validates:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
|
```
|
|
|
|
Expected final output:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/final.mp4
|
|
output/edit-projects/porsche-session-001/render-manifest.json
|
|
```
|
|
|
|
### Review Output
|
|
|
|
Check:
|
|
|
|
- final video exists
|
|
- duration is close to target duration
|
|
- music and voiceover are present if configured
|
|
- no black frames at the beginning or end
|
|
- no invalid clip references in `render-manifest.json`
|
|
|
|
Optional probe:
|
|
|
|
```bash
|
|
ffprobe -v error \
|
|
-show_entries format=duration \
|
|
-show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
|
|
-of json \
|
|
output/edit-projects/porsche-session-001/final.mp4
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
No project appears in `output/edit-projects`:
|
|
|
|
- confirm the service is running with `cinematic-editing-local`
|
|
- confirm the source folder is not still named `.tmp`, `.part`, `.download`, or `.processing`
|
|
- confirm the folder contains at least one valid video file
|
|
- check logs for `event=local_director_project_claimed`
|
|
|
|
Analysis fails:
|
|
|
|
- run `ffprobe` manually against one source clip
|
|
- check whether the input file is still being copied
|
|
- move bad files out of the project folder and try again
|
|
|
|
AI cannot decide from thumbnails:
|
|
|
|
- increase `thumbnail-count-per-clip`
|
|
- enable proxies with `proxy-enabled=true`
|
|
- regenerate analysis
|
|
|
|
`edit-plan.json` is rejected:
|
|
|
|
- confirm all `clipId` values exist in `analysis.json`
|
|
- confirm source timestamps are inside each clip duration
|
|
- confirm timeline timestamps do not overlap except for supported transitions
|
|
- confirm the JSON is strict JSON with no markdown wrapper
|
|
|
|
Render fails:
|
|
|
|
- check `render-manifest.json` if it exists
|
|
- check FFmpeg logs
|
|
- confirm all referenced source clips are still available under the project
|
|
- temporarily disable music, SFX, and voiceover to isolate video rendering
|
|
|
|
Final video has no audio:
|
|
|
|
- confirm the source clips have audio or imported audio files exist
|
|
- confirm `music.wav` and `voiceover.wav` are valid WAV files if used
|
|
- check the audio mix command in the render manifest
|
|
|
|
Final video is too slow to render:
|
|
|
|
- lower output resolution
|
|
- disable proxies only after analysis is complete
|
|
- keep the first renderer concat-based before adding expensive transitions
|
|
|
|
## Cost Strategy
|
|
|
|
The service should reduce expensive AI usage by producing compact artifacts.
|
|
|
|
Recommended AI input:
|
|
|
|
- `analysis.json`
|
|
- contact sheet images
|
|
- optional short low-resolution proxy snippets
|
|
- target duration and style
|
|
|
|
Avoid AI input:
|
|
|
|
- full-resolution video files
|
|
- every frame from every clip
|
|
- raw uncompressed audio
|
|
|
|
Expected token ranges:
|
|
|
|
- metadata-only storyboard: `5k-15k`
|
|
- metadata plus contact sheets: `30k-100k`
|
|
- dense frame analysis: `150k-400k+`
|
|
|
|
The target architecture should keep normal cinematic edit planning in the `30k-100k` range before rendering.
|
|
|
|
## Completion Criteria
|
|
|
|
The feature is complete when:
|
|
|
|
- a user can create an edit project from an existing folder of clips
|
|
- a user can drop a folder into `input/editing/source` and get a ready-to-direct project locally
|
|
- `ai-director-prompt.md` tells Codex or Claude exactly how to judge thumbnails and write `inbox/edit-plan.json`
|
|
- the service can pick up an AI-generated `edit-plan.json` from the project inbox
|
|
- the service can analyze clips and generate `analysis.json`
|
|
- the service can generate a storyboard prompt
|
|
- a valid edit plan can be saved
|
|
- the service can render one final MP4
|
|
- music and voiceover can be mixed into the output
|
|
- a realistic FFmpeg integration test passes
|
|
- the plan checklist is fully checked
|