40 KiB

Raw Blame History

Cinematic Video Editing Service Implementation Plan

Goal

Extend the video clipping service so it can turn a folder or set of existing clips into a new cinematic promotional video.

Primary use case:

input: about 35 Porsche clips
output: one cinematic MP4 video
style: appealing, exciting, premium automotive promo
additions: music, sound effects, voiceover praising the Porsche, transitions, timing, and color treatment

The expensive AI model should do high-value creative planning only. The service should do the heavy media work with deterministic tools such as FFmpeg. A lower-cost coding model should be able to implement this plan milestone by milestone.

Core Principle

Do not send full-resolution video through an AI model.

Use the service to extract compact metadata and preview assets:

durations
codecs
dimensions
frame thumbnails
contact sheets
low-resolution preview proxies
optional audio loudness data
optional scene or motion scores

Then use AI to generate:

creative direction
shot selection
edit decision list
voiceover script
music and sound effect cue list
final render instructions

The service renders the final video from the plan.

Non-Goals

Browser-based nonlinear editor UI.
Manual timeline editing interface.
Real-time collaborative editing.
Full professional color grading system.
Training custom AI models.
Passing raw full-resolution video to an LLM.

Proposed Workflow

Input clips
  -> analyze media with ffprobe and ffmpeg
  -> extract thumbnails and low-resolution proxies
  -> build compact clip manifest
  -> AI creates cinematic storyboard and edit decision list
  -> service validates the plan
  -> service generates or imports voiceover, music, and SFX assets
  -> service renders final MP4 with ffmpeg
  -> service stores render output and manifest

Local AI Director Workflow

The service should support a local workflow where the video editing service runs on the user's machine and an AI coding/chat agent such as Codex or Claude acts as the director.

This mode should not require the AI agent to directly process video files. The service prepares a project folder containing metadata, thumbnails, contact sheets, and a strict prompt. The AI agent reviews those artifacts, writes an edit-plan.json, and asks the service to render the result.

Recommended local folder flow:

input/editing/source/
  porsche-drive-001.mp4
  porsche-drive-002.mp4

service scheduler
  -> detects a new source folder or batch
  -> creates output/edit-projects/<project-id>/
  -> analyzes clips
  -> extracts thumbnails and contact sheets
  -> writes analysis.json
  -> writes ai-director-prompt.md
  -> marks project as WAITING_FOR_DIRECTOR

AI director
  -> reads ai-director-prompt.md
  -> reviews contact-sheets/ and thumbnails/
  -> selects the strongest shots
  -> writes edit-plan.json
  -> optionally writes voiceover-script.txt and audio shopping list

service renderer
  -> validates edit-plan.json
  -> renders final.mp4
  -> writes render-manifest.json

Suggested local command:

mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local

Then place a folder of clips here:

input/editing/source/porsche-session-001/

The service should produce:

output/edit-projects/<project-id>/ai-director-prompt.md
output/edit-projects/<project-id>/analysis.json
output/edit-projects/<project-id>/contact-sheets/
output/edit-projects/<project-id>/thumbnails/

The AI director should only need those generated artifacts to make editing decisions. It should not need to inspect full-resolution clips unless the user explicitly asks for manual review of a specific moment.

AI Director Responsibilities

The AI director should:

judge clips from thumbnails, contact sheets, metadata, and optional proxies
identify the best hero shots, detail shots, motion shots, and ending shot
choose clip order and trim ranges
write a strict edit-plan.json
write or update voiceover lines
suggest music and SFX cues
keep the final timeline within the configured target duration

The AI director should not:

render video
run heavy FFmpeg commands unless troubleshooting
modify source clips
invent clip IDs or timestamps outside analysis.json
depend on private absolute paths in viewer-facing text

Service Responsibilities In AI Director Mode

The service should:

watch a configurable local source directory for new project folders
claim one project folder at a time
ignore folders ending in .tmp, .part, .download, or .processing
create deterministic project IDs from folder names with numeric suffixes on collision
extract enough thumbnails for visual judgment
create contact sheets that are easy for an AI vision model to inspect
write a director prompt with exact instructions and JSON schema
wait for edit-plan.json
validate the plan before rendering
render final output deterministically

This mode allows a strong AI model to do creative direction once, then a cheaper model or the service itself can execute the mechanical steps.

Output Targets

Default final render:

container: MP4
video codec: H.264
audio codec: AAC
resolution: same as source if all clips match, otherwise configurable default 1920x1080
frame rate: same as source if all clips match, otherwise configurable default 30
audio sample rate: 48000
max duration: configurable, default 60 seconds

For Porsche promo videos, default creative structure:

0-5s: hook, hero reveal, engine or impact sound
5-15s: exterior beauty shots and badge/details
15-30s: driving motion, acceleration, road energy
30-45s: interior, craftsmanship, handling, lifestyle shots
45-55s: strongest montage sequence
55-60s: final hero shot and short tagline

Data Model

Add package:

src/main/java/org/example/videoclips/editing

Suggested domain records:

record EditProject(
        String id,
        String name,
        EditProjectStatus status,
        Path inputDirectory,
        Path outputDirectory,
        Instant createdAt,
        Instant updatedAt
) {}

enum EditProjectStatus {
    CREATED,
    ANALYZING,
    ANALYZED,
    WAITING_FOR_DIRECTOR,
    PLANNING,
    PLANNED,
    RENDERING,
    RENDERED,
    FAILED
}

record ClipAnalysis(
        String clipId,
        Path sourcePath,
        double durationSeconds,
        String videoCodec,
        String audioCodec,
        int width,
        int height,
        double frameRate,
        List<Path> thumbnails,
        Path contactSheet,
        Path proxyPath,
        double motionScore,
        double brightnessScore,
        double sharpnessScore
) {}

record EditDecision(
        String clipId,
        double sourceStartSeconds,
        double sourceEndSeconds,
        double timelineStartSeconds,
        double timelineEndSeconds,
        String transitionIn,
        String transitionOut,
        double playbackSpeed,
        String visualTreatment,
        String reason
) {}

record AudioCue(
        String type,
        String assetKey,
        double timelineStartSeconds,
        double timelineEndSeconds,
        double gainDb,
        String notes
) {}

record VoiceoverLine(
        String text,
        double timelineStartSeconds,
        double timelineEndSeconds,
        String delivery
) {}

record EditPlan(
        String projectId,
        String style,
        double targetDurationSeconds,
        List<EditDecision> decisions,
        List<AudioCue> audioCues,
        List<VoiceoverLine> voiceover,
        String renderProfile,
        String summary
) {}

For the first implementation, these can be JSON files on disk. Do not start with JPA unless project persistence is required by the user later.

API Contract

Add REST endpoints under /v1/edit-projects.

Create Edit Project

POST /v1/edit-projects

Request:

{
  "name": "porsche-cinematic",
  "inputDirectory": "./output/clips/20240708_173213",
  "targetDurationSeconds": 60,
  "style": "cinematic-porsche-promo",
  "voiceoverEnabled": true,
  "musicEnabled": true,
  "soundEffectsEnabled": true
}

Response:

{
  "projectId": "edit_01JZABCDE",
  "status": "CREATED"
}

Validation:

name: required, filesystem-safe after normalization
inputDirectory: required and must exist
targetDurationSeconds: 15..600
style: required, initially allow cinematic-porsche-promo

Analyze Clips

POST /v1/edit-projects/{projectId}:analyze

Behavior:

scans project input directory
accepts valid video files only
runs ffprobe
extracts thumbnails
creates contact sheets
optionally creates low-resolution proxy files
writes analysis.json

Response:

{
  "projectId": "edit_01JZABCDE",
  "status": "ANALYZED",
  "clipCount": 35,
  "analysisPath": "./output/edit-projects/edit_01JZABCDE/analysis.json"
}

Generate Storyboard Prompt

POST /v1/edit-projects/{projectId}:storyboard-prompt

Behavior:

reads analysis.json
produces a compact AI prompt
includes clip summaries, thumbnail references, timing constraints, and output schema
writes storyboard-prompt.md

Response:

{
  "projectId": "edit_01JZABCDE",
  "promptPath": "./output/edit-projects/edit_01JZABCDE/storyboard-prompt.md"
}

Save Edit Plan

PUT /v1/edit-projects/{projectId}/plan

Request:

{
  "style": "cinematic-porsche-promo",
  "targetDurationSeconds": 60,
  "decisions": [],
  "audioCues": [],
  "voiceover": [],
  "renderProfile": "mp4-h264-aac-1080p",
  "summary": "Cinematic Porsche promo edit."
}

Behavior:

validates the AI-generated edit plan
ensures all clip IDs exist
ensures source time ranges are inside clip durations
ensures timeline ranges do not overlap incorrectly
writes edit-plan.json

Render Edit

POST /v1/edit-projects/{projectId}:render

Behavior:

reads edit-plan.json
renders the final video with FFmpeg
mixes music, SFX, and voiceover if available
writes final MP4 and render-manifest.json

Response:

{
  "projectId": "edit_01JZABCDE",
  "status": "RENDERED",
  "outputPath": "./output/edit-projects/edit_01JZABCDE/final.mp4",
  "durationSeconds": 60
}

Read Project

GET /v1/edit-projects/{projectId}

Return current status, paths, timestamps, and error details if failed.

File Layout

For local filesystem implementation:

output/edit-projects/<project-id>/
  project.json
  analysis.json
  storyboard-prompt.md
  edit-plan.json
  render-manifest.json
  final.mp4
  thumbnails/
    <clip-id>_0001.jpg
    <clip-id>_0002.jpg
  contact-sheets/
    <clip-id>.jpg
  proxies/
    <clip-id>.mp4
  audio/
    voiceover.wav
    music.wav
    sfx/
  inbox/
    edit-plan.json

Configuration

Add YAML config under video-clipping.editing.

video-clipping:
  editing:
    enabled: true
    project-directory: ./output/edit-projects
    ffmpeg-binary: ffmpeg
    ffprobe-binary: ffprobe
    thumbnail-count-per-clip: 5
    contact-sheet-columns: 5
    proxy-enabled: true
    proxy-width: 640
    target-duration-seconds: 60
    output-width: 1920
    output-height: 1080
    output-frame-rate: 30
    audio-sample-rate: 48000
    video-bitrate: 12000k
    audio-bitrate: 192k
    local-director:
      enabled: true
      source-directory: ./input/editing/source
      working-directory: ./input/editing/working
      processed-directory: ./input/editing/processed
      rejected-directory: ./input/editing/rejected
      poll-interval-ms: 5000
      director-prompt-file-name: ai-director-prompt.md
      expected-plan-file-name: edit-plan.json
      auto-render-when-plan-appears: false

Local director defaults:

auto-render-when-plan-appears=false is safer for early development because the user can inspect edit-plan.json before rendering.
When set to true, the service should render automatically after a valid plan appears in the project inbox/ folder.
The scheduler should process one project folder at a time, using the same repeat-prevention pattern as the input folder scheduler.

FFmpeg Commands

Probe Clip

ffprobe -v error \
  -show_entries format=duration \
  -show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
  -of json \
  clip_00001.mp4

Extract Thumbnail

Use timestamps spread across clip duration.

ffmpeg -hide_banner -y \
  -ss 3.2 \
  -i clip_00001.mp4 \
  -frames:v 1 \
  -q:v 2 \
  thumbnails/clip_00001_0001.jpg

Create Contact Sheet

ffmpeg -hide_banner -y \
  -i clip_00001.mp4 \
  -vf "fps=1/2,scale=320:-1,tile=5x5" \
  -frames:v 1 \
  contact-sheets/clip_00001.jpg

Create Proxy

ffmpeg -hide_banner -y \
  -i clip_00001.mp4 \
  -vf "scale=640:-2" \
  -c:v libx264 -preset veryfast -crf 28 \
  -c:a aac -b:a 96k \
  proxies/clip_00001.mp4

Render Final Video

First implementation should use a concat-based render:

Trim each selected segment into an intermediate file.
Normalize resolution, frame rate, pixel format, and audio.
Concatenate intermediates.
Mix music, SFX, and voiceover.
Write final MP4.

This is simpler and easier for a lower-cost coding model than building one huge filter_complex command.

Intermediate segment command:

ffmpeg -hide_banner -y \
  -ss <sourceStart> \
  -to <sourceEnd> \
  -i <sourceClip> \
  -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,format=yuv420p" \
  -r 30 \
  -c:v libx264 -preset veryfast -crf 18 \
  -c:a aac -b:a 192k -ar 48000 \
  <work>/segment_0001.mp4

Concat command:

ffmpeg -hide_banner -y \
  -f concat \
  -safe 0 \
  -i concat.txt \
  -c copy \
  <work>/timeline-video.mp4

Final mix command:

ffmpeg -hide_banner -y \
  -i <work>/timeline-video.mp4 \
  -i audio/music.wav \
  -i audio/voiceover.wav \
  -filter_complex "[1:a]volume=0.25[music];[2:a]volume=1.0[voice];[0:a][music][voice]amix=inputs=3:duration=first:dropout_transition=2[a]" \
  -map 0:v -map "[a]" \
  -c:v copy \
  -c:a aac -b:a 192k \
  final.mp4

Add SFX later after the basic music and voiceover mix works.

AI Storyboard Prompt Requirements

The generated prompt should instruct the AI to output strict JSON only.

Required input context:

project style
target duration
clip IDs
duration per clip
dimensions and frame rate
extracted thumbnail paths or descriptions
any motion, brightness, and sharpness scores
known constraints

Required output schema:

{
  "style": "cinematic-porsche-promo",
  "targetDurationSeconds": 60,
  "summary": "string",
  "decisions": [
    {
      "clipId": "clip_00001",
      "sourceStartSeconds": 0.0,
      "sourceEndSeconds": 2.5,
      "timelineStartSeconds": 0.0,
      "timelineEndSeconds": 2.5,
      "transitionIn": "cut",
      "transitionOut": "crossfade",
      "playbackSpeed": 1.0,
      "visualTreatment": "high contrast warm cinematic grade",
      "reason": "hero front angle"
    }
  ],
  "voiceover": [
    {
      "text": "This Porsche turns precision into emotion.",
      "timelineStartSeconds": 3.0,
      "timelineEndSeconds": 6.0,
      "delivery": "confident cinematic narrator"
    }
  ],
  "audioCues": [
    {
      "type": "music",
      "assetKey": "cinematic-driving-bed",
      "timelineStartSeconds": 0.0,
      "timelineEndSeconds": 60.0,
      "gainDb": -12.0,
      "notes": "build energy toward final montage"
    }
  ]
}

Prompt rules:

Never reference clip paths directly in final viewer-facing text.
Keep total timeline duration within the requested target duration.
Use at least 12 clips if enough usable clips exist.
Avoid using any single source clip for more than 20% of the final runtime.
Prefer a strong first 5 seconds.
End with the strongest exterior or motion shot.
Voiceover should praise the car without sounding exaggerated or fake.

AI Director Prompt File

In local director mode, generate ai-director-prompt.md for Codex, Claude, or another AI instance.

The prompt must include:

project ID and style
target duration
path to analysis.json
paths to contact sheets and thumbnails
exact output location for the plan
strict JSON schema for edit-plan.json
validation rules
instruction to judge shots from thumbnails/contact sheets
instruction to produce a cinematic Porsche promo edit

Local Director Scheduler

Add a scheduler similar to the input folder scheduler, but for edit project folders.

States:

source -> working -> processed
source -> working -> rejected

Folder example:

input/editing/source/porsche-session-001/
  clip_00000.mp4
  clip_00001.mp4

Claimed folder:

input/editing/working/porsche-session-001/

Project output:

output/edit-projects/porsche-session-001/

Scheduler behavior:

scan local-director.source-directory
pick the first valid project folder in deterministic order
move it to working-directory
create an edit project
analyze clips
generate ai-director-prompt.md
write status WAITING_FOR_DIRECTOR
move the source folder to processed-directory after project creation succeeds
leave the project output folder ready for the AI director

Plan pickup behavior:

scan project inbox/ folders for edit-plan.json
validate the plan
if auto-render-when-plan-appears=true, render automatically
otherwise expose the project as PLANNED and wait for POST /v1/edit-projects/{projectId}:render

Tests:

ignores temporary project folders
processes one project folder at a time
creates project artifacts and director prompt
does not reprocess a processed folder
rejects empty folders
validates plan pickup
respects auto-render-when-plan-appears

Voiceover And Audio

First implementation can support imported audio assets only:

user places music.wav and voiceover.wav under project audio/
service mixes those files into the final render

Second implementation can add generated voiceover:

create voiceover-script.txt
call an external TTS provider or OpenAI audio model outside the core renderer
save voiceover.wav
keep generation behind an interface so it can be mocked in tests

Suggested interface:

interface VoiceoverGenerator {
    Path generateVoiceover(String projectId, List<VoiceoverLine> lines);
}

Suggested first adapter:

NoopVoiceoverGenerator

It writes the script only and requires an imported audio file.

Milestones

Add editing configuration under video-clipping.editing.
Add filesystem project store for project.json, analysis.json, edit-plan.json, and render output.
Add edit project domain records and status enum.
Add POST /v1/edit-projects and GET /v1/edit-projects/{projectId}.
Add clip discovery for project input directories.
Add ffprobe analysis and validation for all input clips.
Add thumbnail extraction.
Add contact sheet generation.
Add optional proxy generation.
Add analysis.json writing and reading.
Add storyboard prompt generation from analysis.json.
Add local AI director prompt generation with thumbnail/contact-sheet instructions.
Add local director source-folder scheduler.
Add project inbox/ plan pickup.
Add strict JSON edit plan schema and validation.
Add PUT /v1/edit-projects/{projectId}/plan.
Add simple segment rendering without transitions.
Add concat-based final timeline rendering.
Add music and voiceover audio mixing.
Add render manifest with command metadata and output details.
Add basic transition support: cut, crossfade, fade in, fade out.
Add SFX cue support.
Add optional generated voiceover adapter interface.
Add structured logs and metrics for analysis, planning, and rendering durations.
Add integration test with generated fixture clips and a short edit plan.
Add documentation for running a local cinematic edit with Codex or Claude as director.
Run mvn verify and update this checklist.

Milestone Details

Milestone 1: Editing Configuration

Files:

src/main/java/org/example/videoclips/config/VideoClippingProperties.java
src/main/resources/application.yml

Add nested properties class:

private final Editing editing = new Editing();

public Editing getEditing() {
    return editing;
}

Add fields matching the YAML in this plan.

Tests:

add a focused config binding test or extend an existing Spring context test
verify defaults bind correctly

Acceptance criteria:

application starts with editing config present
editing can be disabled with video-clipping.editing.enabled=false

Milestone 2: Filesystem Project Store

Files:

src/main/java/org/example/videoclips/editing/EditProjectStore.java
src/main/java/org/example/videoclips/editing/FileSystemEditProjectStore.java

Responsibilities:

create project directories
read/write JSON files with Jackson
normalize project IDs and reject path traversal
expose methods for project, analysis, plan, and manifest paths

Tests:

creates expected folder structure
rejects invalid project IDs
round-trips project.json

Acceptance criteria:

no direct filesystem writes from controllers
all project path creation goes through the store

Milestone 3: Domain Records

Files:

src/main/java/org/example/videoclips/editing/EditProject.java
src/main/java/org/example/videoclips/editing/EditProjectStatus.java
src/main/java/org/example/videoclips/editing/ClipAnalysis.java
src/main/java/org/example/videoclips/editing/EditPlan.java
src/main/java/org/example/videoclips/editing/EditDecision.java
src/main/java/org/example/videoclips/editing/AudioCue.java
src/main/java/org/example/videoclips/editing/VoiceoverLine.java

Tests:

JSON serialization for representative records

Acceptance criteria:

records serialize with stable camelCase JSON
no business logic in records beyond validation helpers if needed

Milestone 4: Project API

Files:

src/main/java/org/example/videoclips/api/EditProjectController.java
src/main/java/org/example/videoclips/api/dto/CreateEditProjectRequest.java
src/main/java/org/example/videoclips/api/dto/EditProjectResponse.java

Tests:

create project success
reject missing input directory
reject invalid target duration
get project success
get missing project returns 404

Acceptance criteria:

API returns project ID and status
project state is persisted to project.json

Milestone 5-10: Analysis Pipeline

Files:

src/main/java/org/example/videoclips/editing/EditProjectAnalyzer.java
src/main/java/org/example/videoclips/editing/FfmpegClipInspector.java
src/main/java/org/example/videoclips/editing/ThumbnailExtractor.java
src/main/java/org/example/videoclips/editing/ContactSheetGenerator.java
src/main/java/org/example/videoclips/editing/ProxyGenerator.java

Implementation notes:

reuse patterns from FolderVideoValidator and FolderFfmpegClipper
keep process execution behind package-private interfaces for tests
sanitize process output before logging
process clips in deterministic filename order

Tests:

analysis ignores non-video files
invalid videos are reported in analysis errors
command construction is correct
output JSON includes all analyzed clips
proxy generation can be disabled

Acceptance criteria:

analysis.json is enough for an AI prompt without reading raw video
failure of one invalid clip does not fail the entire project if at least one valid clip exists

Milestone 11: Storyboard Prompt

Files:

src/main/java/org/example/videoclips/editing/StoryboardPromptGenerator.java

Tests:

prompt includes clip IDs, durations, target duration, style, and JSON schema
prompt excludes raw absolute system paths unless explicitly configured

Acceptance criteria:

generated prompt can be pasted into an AI model
model output requirements are strict JSON

Milestone 12: Local AI Director Prompt

Files:

src/main/java/org/example/videoclips/editing/AiDirectorPromptGenerator.java

Behavior:

generate ai-director-prompt.md
include project style, target duration, clip metadata, contact sheet paths, thumbnail paths, and strict output schema
write human-readable director-readme.md
instruct the AI to write inbox/edit-plan.json

Tests:

prompt contains analysis.json
prompt contains contact-sheets/ and thumbnails/
prompt contains exact output path inbox/edit-plan.json
prompt instructs the AI not to render video

Acceptance criteria:

a user can open the prompt in Codex or Claude and get a valid edit plan without additional explanation

Milestone 13: Local Director Source-Folder Scheduler

Files:

src/main/java/org/example/videoclips/editing/LocalDirectorScheduler.java
src/main/java/org/example/videoclips/editing/EditProjectDirectoryInitializer.java

Behavior:

create configured source, working, processed, rejected, and project output directories on startup
scan video-clipping.editing.local-director.source-directory
claim one project folder by moving it to working
ignore hidden and temporary folders
create an edit project
analyze the clips
generate ai-director-prompt.md
set project status to WAITING_FOR_DIRECTOR
move the claimed source folder to processed after successful analysis

Tests:

creates all local director directories
selects project folders in deterministic order
ignores .tmp, .part, .download, and .processing folders
processes one project folder per scan
writes project prompt and analysis files
rejects empty project folders

Acceptance criteria:

dropping a folder into input/editing/source produces a ready-to-direct project under output/edit-projects

Milestone 14: Plan Inbox Pickup

Files:

src/main/java/org/example/videoclips/editing/EditPlanInboxScanner.java

Behavior:

scan project inbox/ folders for edit-plan.json
ignore partially written files with .tmp, .part, or .download suffix
validate the plan
save normalized plan to project root
mark status PLANNED
render automatically only when auto-render-when-plan-appears=true

Tests:

ignores temporary plan files
rejects invalid plan files
accepts valid plan files
does not render when auto-render is disabled
invokes renderer when auto-render is enabled

Acceptance criteria:

Codex or Claude can write inbox/edit-plan.json and the service can pick it up without a manual API call

Milestone 15-16: Edit Plan Validation

Files:

src/main/java/org/example/videoclips/editing/EditPlanValidator.java
src/main/java/org/example/videoclips/api/dto/SaveEditPlanRequest.java

Validation rules:

all clip IDs exist in analysis.json
sourceStartSeconds >= 0
sourceEndSeconds > sourceStartSeconds
source range is inside clip duration
timeline ranges are non-negative
timeline ranges do not overlap unless the transition requires overlap
final duration is within target duration tolerance
playback speed is between 0.25 and 4.0
only supported transitions are accepted

Tests:

accepts valid plan
rejects unknown clip ID
rejects out-of-range source timestamps
rejects overlapping timeline decisions
rejects unsupported transition

Acceptance criteria:

renderer never receives an invalid plan

Milestone 17-20: Render Pipeline

Files:

src/main/java/org/example/videoclips/editing/EditRenderer.java
src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
src/main/java/org/example/videoclips/editing/RenderManifest.java

Implementation sequence:

create working directory
render each selected decision into normalized segment MP4
write concat.txt
concatenate segments into timeline-video.mp4
mix audio when audio files exist
write final.mp4
write render-manifest.json

Tests:

command construction for segment render
command construction for concat
command construction for audio mix
render fails if no decisions exist
manifest is written on success

Acceptance criteria:

final MP4 exists after render
render manifest includes input clip IDs, output path, duration, and command summaries

Milestone 21-22: Transitions And SFX

Start simple.

Supported transitions:

cut
fade-in
fade-out
crossfade

Supported SFX cues:

imported WAV files only
cue has asset key, start time, gain, and optional fade

Tests:

unsupported transition rejected
SFX file missing is a validation error
final mix includes SFX input when cue exists

Acceptance criteria:

basic cinematic polish is possible without needing a full timeline engine

Milestone 23: Voiceover Generation Hook

Files:

src/main/java/org/example/videoclips/editing/VoiceoverGenerator.java
src/main/java/org/example/videoclips/editing/NoopVoiceoverGenerator.java

Behavior:

first version writes voiceover-script.txt
does not call external APIs
later adapters can call TTS providers

Tests:

script file generated from voiceover lines
noop adapter does not fail when TTS is disabled

Acceptance criteria:

service supports the workflow without requiring paid audio generation

Milestone 24: Logs And Metrics

Logs:

event=edit_project_created
event=local_director_project_claimed
event=edit_analysis_started
event=edit_analysis_completed
event=ai_director_prompt_generated
event=storyboard_prompt_generated
event=edit_plan_inbox_detected
event=edit_plan_saved
event=edit_render_started
event=edit_render_completed
event=edit_render_failed

Metrics:

video.clipping.edit.projects.created
video.clipping.edit.analysis.duration
video.clipping.edit.render.duration
video.clipping.edit.render.failures

Tests:

metrics binder test if custom metrics are added
at minimum, unit tests should cover event-producing paths

Acceptance criteria:

an operator can see which stage failed and how long rendering took

Milestone 25: Integration Test

Create generated fixture clips with FFmpeg:

three 3-second clips
different colors or test patterns
simple audio tones

Test flow:

place a folder of generated clips in local director source
run the local director scheduler
assert ai-director-prompt.md and analysis.json exist
write a short valid inbox/edit-plan.json
run the plan inbox scanner
render final video
probe final video
assert duration is greater than zero

Skip test when ffmpeg or ffprobe is unavailable.

Acceptance criteria:

proves the end-to-end renderer works on a machine with FFmpeg installed

Lower-Cost Model Implementation Instructions

Follow these rules when assigning this plan to a cheaper coding model:

Implement exactly one milestone at a time.
Run the relevant tests after each milestone.
Update the checklist in this file after each completed milestone.
Commit after each milestone if the repository is in a clean, test-passing state.
Do not refactor unrelated packages.
Reuse existing patterns from the folder scheduler for process execution, logging, and tests.
Keep the first renderer deterministic and simple.
Do not add external AI, TTS, or music-provider calls until the filesystem workflow works.
For local director mode, implement file handoff first. Do not embed Codex, Claude, or another model inside the Spring service.

Suggested prompt for the lower-cost coding model:

You are implementing `docs/cinematic-video-editing-service-plan.md`.

Implement the next unchecked milestone only. Keep the change scoped. Use existing patterns from the folder scheduler package where possible. Add tests for the milestone. Run the relevant Maven test command. Update the milestone checkbox only when tests pass. Do not implement future milestones early.

Suggested prompt for Codex or Claude acting as the local AI director:

You are the AI director for this local edit project.

Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.

Local Runbook

Use this runbook after the local AI director milestones are implemented.

Prerequisites

Install:

Java 21
Maven
FFmpeg
ffprobe
Codex, Claude, or another AI instance that can read local project files and images

Verify tools:

java -version
mvn -version
ffmpeg -version
ffprobe -version

Start The Service

Run the service with the local cinematic editing profile:

mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local

Expected startup behavior:

creates input/editing/source
creates input/editing/working
creates input/editing/processed
creates input/editing/rejected
creates output/edit-projects
logs that local director mode is enabled

Add Source Clips

Create one project folder under the local director source directory:

input/editing/source/porsche-session-001/

Place the clips inside that folder:

input/editing/source/porsche-session-001/clip_00000.mp4
input/editing/source/porsche-session-001/clip_00001.mp4
input/editing/source/porsche-session-001/clip_00002.mp4

Recommended copy flow for large files:

copy the folder as porsche-session-001.tmp
wait until all files are fully copied
rename it to porsche-session-001

The scheduler should ignore temporary folders ending in .tmp, .part, .download, or .processing.

Wait For Analysis

The service should claim the folder and move it through:

input/editing/source/porsche-session-001
input/editing/working/porsche-session-001
input/editing/processed/porsche-session-001

After analysis, find the project output:

output/edit-projects/porsche-session-001/

Expected files:

output/edit-projects/porsche-session-001/project.json
output/edit-projects/porsche-session-001/analysis.json
output/edit-projects/porsche-session-001/ai-director-prompt.md
output/edit-projects/porsche-session-001/director-readme.md
output/edit-projects/porsche-session-001/contact-sheets/
output/edit-projects/porsche-session-001/thumbnails/
output/edit-projects/porsche-session-001/inbox/

Expected status:

WAITING_FOR_DIRECTOR

Ask The AI Director To Edit

Open Codex, Claude, or the selected AI instance in the project output folder.

Use this prompt:

You are the AI director for this local edit project.

Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.

The AI should inspect:

analysis.json
contact-sheets/*.jpg
thumbnails/*.jpg
optional proxies/*.mp4

The AI should write:

output/edit-projects/porsche-session-001/inbox/edit-plan.json

Validate And Render

Optional imported audio assets must be placed before rendering:

output/edit-projects/porsche-session-001/audio/music.wav
output/edit-projects/porsche-session-001/audio/voiceover.wav
output/edit-projects/porsche-session-001/audio/sfx/engine-rev.wav

SFX plan entries use the filename without .wav as assetKey. When an accepted plan contains voiceover lines, the default noop provider writes voiceover-script.txt; record or generate that script as audio/voiceover.wav before rendering if narration should be included in the final mix.

If auto-render-when-plan-appears=false, trigger rendering manually:

curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render

If auto-render-when-plan-appears=true, the service should render automatically after it detects and validates:

output/edit-projects/porsche-session-001/inbox/edit-plan.json

Expected final output:

output/edit-projects/porsche-session-001/final.mp4
output/edit-projects/porsche-session-001/render-manifest.json

Review Output

Check:

final video exists
duration is close to target duration
music and voiceover are present if configured
no black frames at the beginning or end
no invalid clip references in render-manifest.json

Optional probe:

ffprobe -v error \
  -show_entries format=duration \
  -show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
  -of json \
  output/edit-projects/porsche-session-001/final.mp4

Troubleshooting

No project appears in output/edit-projects:

confirm the service is running with cinematic-editing-local
confirm the source folder is not still named .tmp, .part, .download, or .processing
confirm the folder contains at least one valid video file
check logs for event=local_director_project_claimed

Analysis fails:

run ffprobe manually against one source clip
check whether the input file is still being copied
move bad files out of the project folder and try again

AI cannot decide from thumbnails:

increase thumbnail-count-per-clip
enable proxies with proxy-enabled=true
regenerate analysis

edit-plan.json is rejected:

confirm all clipId values exist in analysis.json
confirm source timestamps are inside each clip duration
confirm timeline timestamps do not overlap except for supported transitions
confirm the JSON is strict JSON with no markdown wrapper

Render fails:

check render-manifest.json if it exists
check FFmpeg logs
confirm all referenced source clips are still available under the project
temporarily disable music, SFX, and voiceover to isolate video rendering

Final video has no audio:

confirm the source clips have audio or imported audio files exist
confirm music.wav and voiceover.wav are valid WAV files if used
check the audio mix command in the render manifest

Final video is too slow to render:

lower output resolution
disable proxies only after analysis is complete
keep the first renderer concat-based before adding expensive transitions

Cost Strategy

The service should reduce expensive AI usage by producing compact artifacts.

Recommended AI input:

analysis.json
contact sheet images
optional short low-resolution proxy snippets
target duration and style

Avoid AI input:

full-resolution video files
every frame from every clip
raw uncompressed audio

Expected token ranges:

metadata-only storyboard: 5k-15k
metadata plus contact sheets: 30k-100k
dense frame analysis: 150k-400k+

The target architecture should keep normal cinematic edit planning in the 30k-100k range before rendering.

Completion Criteria

The feature is complete when:

a user can create an edit project from an existing folder of clips
a user can drop a folder into input/editing/source and get a ready-to-direct project locally
ai-director-prompt.md tells Codex or Claude exactly how to judge thumbnails and write inbox/edit-plan.json
the service can pick up an AI-generated edit-plan.json from the project inbox
the service can analyze clips and generate analysis.json
the service can generate a storyboard prompt
a valid edit plan can be saved
the service can render one final MP4
music and voiceover can be mixed into the output
a realistic FFmpeg integration test passes
the plan checklist is fully checked

40 KiB Raw Blame History

Cinematic Video Editing Service Implementation Plan

Goal

Core Principle

Non-Goals

Proposed Workflow

Local AI Director Workflow

AI Director Responsibilities

Service Responsibilities In AI Director Mode

Output Targets

Data Model

API Contract

Create Edit Project

Analyze Clips

Generate Storyboard Prompt

Save Edit Plan

Render Edit

Read Project

File Layout

Configuration

FFmpeg Commands

Probe Clip

Extract Thumbnail

Create Contact Sheet

Create Proxy

Render Final Video

AI Storyboard Prompt Requirements

AI Director Prompt File

Local Director Scheduler

Voiceover And Audio

Milestones

Milestone Details

Milestone 1: Editing Configuration

Milestone 2: Filesystem Project Store

Milestone 3: Domain Records

Milestone 4: Project API

Milestone 5-10: Analysis Pipeline

Milestone 11: Storyboard Prompt

Milestone 12: Local AI Director Prompt

Milestone 13: Local Director Source-Folder Scheduler

Milestone 14: Plan Inbox Pickup

Milestone 15-16: Edit Plan Validation

Milestone 17-20: Render Pipeline

Milestone 21-22: Transitions And SFX

Milestone 23: Voiceover Generation Hook

Milestone 24: Logs And Metrics

Milestone 25: Integration Test

Lower-Cost Model Implementation Instructions

Local Runbook

Prerequisites

Start The Service

Add Source Clips

Wait For Analysis

Ask The AI Director To Edit

Validate And Render

Review Output

Troubleshooting

Cost Strategy

Completion Criteria

40 KiB

Raw Blame History