16 KiB

Raw Blame History

Cinematic Editing Runbook

This guide explains how to turn a folder of video clips into one edited cinematic MP4 using the local video editing service and an AI director such as Codex or Claude.

The service does the heavy video work with FFmpeg. The AI director reviews generated thumbnails, contact sheets, metadata, and optional proxies, then writes an edit-plan.json. The service validates that plan and renders the final video.

1. Prerequisites

Install these tools on the machine running the service:

Java 21
Maven
FFmpeg
ffprobe
Codex, Claude, or another AI instance that can read local files and images

Verify the tools:

java -version
mvn -version
ffmpeg -version
ffprobe -version

2. Start The Service

From the repository root, start the service with the local cinematic editing profile:

mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local

This profile disables the normal 8-second folder scheduler and enables the cinematic editing workflow.

Expected startup behavior:

The service creates input/editing/source.
The service creates input/editing/working.
The service creates input/editing/processed.
The service creates input/editing/rejected.
The service creates output/edit-projects.
The service logs that local director mode is enabled.

Useful config defaults:

video-clipping:
  editing:
    project-directory: ./output/edit-projects
    thumbnail-count-per-clip: 5
    proxy-enabled: true
    target-duration-seconds: 600
    output-width: 1920
    output-height: 1080
    output-frame-rate: 30
    video-bitrate: 12000k
    audio-bitrate: 192k
    local-director:
      source-directory: ./input/editing/source
      poll-interval-ms: 5000
      auto-render-when-plan-appears: false
      require-approval-before-render: true
      approval-file-name: approved.flag

3. Prepare A Source Project Folder

You can use either a project folder or loose clips directly under input/editing/source.

The clearest layout is one folder per edit project under input/editing/source.

Example:

input/editing/source/porsche-session-001/

Put your source clips in that folder:

input/editing/source/porsche-session-001/clip_00000.mp4
input/editing/source/porsche-session-001/clip_00001.mp4
input/editing/source/porsche-session-001/clip_00002.mp4

For the Porsche example with 35 existing 8-second clips, put all 35 files in the same project folder:

input/editing/source/porsche-session-001/clip_00000.mp4
input/editing/source/porsche-session-001/clip_00001.mp4
...
input/editing/source/porsche-session-001/clip_00034.mp4

The scheduler also accepts loose video clips directly in input/editing/source:

input/editing/source/clip_00000.mp4
input/editing/source/clip_00001.mp4
input/editing/source/clip_00002.mp4

When loose clips are found and no project folder is waiting, the service claims them as one generated project named source-clips.

Recommended flow for large files or many clips:

Copy the folder as porsche-session-001.tmp.
Wait until all clips are fully copied.
Rename the folder to porsche-session-001.

The scheduler ignores temporary project folders ending in .tmp, .part, .download, or .processing.

4. Wait For Service Analysis

The local director scheduler scans every 5 seconds by default.

When it finds the project folder, it moves the folder through this lifecycle:

input/editing/source/porsche-session-001
input/editing/working/porsche-session-001
input/editing/processed/porsche-session-001

After analysis succeeds, the generated project appears here:

output/edit-projects/porsche-session-001/

Expected generated files and folders:

output/edit-projects/porsche-session-001/project.json
output/edit-projects/porsche-session-001/analysis.json
output/edit-projects/porsche-session-001/ai-director-prompt.md
output/edit-projects/porsche-session-001/director-readme.md
output/edit-projects/porsche-session-001/contact-sheets/
output/edit-projects/porsche-session-001/thumbnails/
output/edit-projects/porsche-session-001/proxies/
output/edit-projects/porsche-session-001/inbox/

The expected project status at this point is:

WAITING_FOR_DIRECTOR

Check project state through the API:

curl http://localhost:8080/v1/edit-projects/porsche-session-001

5. Ask Codex Or Claude To Act As Director

Open Codex, Claude, or the selected AI instance in this folder:

output/edit-projects/porsche-session-001/

Model choice matters mostly for creative judgment, not rendering. Use a stronger model when the footage is ambiguous, repetitive, emotionally important, or needs a premium commercial feel. Use a cheaper model when the generated thumbnails/contact sheets are clear and the edit style is straightforward.

Recommended split:

Use a stronger model for the first director pass on high-value edits, because it chooses shots, pacing, voiceover, text overlays, and audio moments.
Use a cheaper model for JSON cleanup, schema repair, or small plan revisions after the creative direction is already clear.
Do not ask any model to process full-resolution video directly unless necessary. Let the service provide thumbnails, contact sheets, metadata, and proxies.

Give the AI this prompt:

You are the AI director for this local edit project.

Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.

The AI director should inspect:

ai-director-prompt.md
analysis.json
contact-sheets/*.jpg
thumbnails/*.jpg
proxies/*.mp4 if more visual context is needed

After the plan is imported, the service also runs an asset materialization stage:

It reuses matching generated music, SFX, and voiceover assets from the shared cache folders.
It copies reusable assets into the project-local audio/ folder so the renderer can use them.
It writes reusable asset requests under assets/requests/ when a needed asset is missing.
It waits to render if required SFX assets are still missing.

The AI director must write this file:

output/edit-projects/porsche-session-001/inbox/edit-plan.json

Do not put markdown fences around the JSON. The file must contain strict JSON only.

6. What The Edit Plan Must Contain

The generated edit-plan.json must match the service schema.

Minimal shape:

{
  "style": "cinematic-porsche-promo",
  "targetDurationSeconds": 60,
  "renderProfile": "mp4-h264-aac-1080p",
  "summary": "A cinematic Porsche promo with a strong opening, detail shots, driving energy, and a final hero shot.",
  "decisions": [
    {
      "clipId": "clip_00000",
      "sourceStartSeconds": 0.0,
      "sourceEndSeconds": 3.5,
      "timelineStartSeconds": 0.0,
      "timelineEndSeconds": 3.5,
      "transitionIn": "fade",
      "transitionOut": "cut",
      "playbackSpeed": 1.0,
      "visualTreatment": "warm high-contrast cinematic grade",
      "reason": "Opening hero angle establishes the car immediately."
    }
  ],
  "audioCues": [],
  "voiceover": []
}

Rules the AI director must follow:

Every clipId must exist in analysis.json.
sourceStartSeconds and sourceEndSeconds must be inside the source clip duration.
sourceEndSeconds must be greater than sourceStartSeconds.
timelineEndSeconds must be greater than timelineStartSeconds.
Timeline decisions should be ordered and non-overlapping.
playbackSpeed should usually stay near 1.0 unless the plan intentionally uses slow motion or speed ramp style.
audioCues and voiceover must be present, even if they are empty arrays.

7. Add Optional Music, Voiceover, And SFX

The renderer can mix optional WAV files into the final output.

Shared cache locations:

Music: input/highlights/assets/music/
SFX: input/highlights/assets/sfx/
Voiceover cache: output/highlight-projects/_voiceover-cache/

If the service or an AI worker generates an asset there, later projects can reuse it automatically.

Place optional assets here before rendering:

output/edit-projects/porsche-session-001/audio/music.wav
output/edit-projects/porsche-session-001/audio/voiceover.wav
output/edit-projects/porsche-session-001/audio/sfx/engine-rev.wav

For SFX cues, the assetKey is the filename without .wav.

Example:

{
  "type": "sfx",
  "assetKey": "engine-rev",
  "timelineStartSeconds": 5.2,
  "timelineEndSeconds": 6.4,
  "gainDb": -3.0,
  "notes": "Short engine accent on acceleration shot."
}

For narration, the default noop voiceover provider writes a script file but does not synthesize audio. If the plan includes voiceover lines, record or generate the narration and place it here before rendering:

output/edit-projects/porsche-session-001/audio/voiceover.wav

8. Render The Final Video

By default, the local profile has:

auto-render-when-plan-appears: false

That means you manually trigger rendering after the AI writes inbox/edit-plan.json:

curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render

If you set VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=true, the service renders automatically after it detects and validates:

output/edit-projects/porsche-session-001/inbox/edit-plan.json

By default, automatic rendering is still gated by human review. To approve a validated plan for automatic rendering, create:

output/edit-projects/porsche-session-001/inbox/approved.flag

On the next inbox scan, the service renders the already imported edit-plan.json. To disable the approval gate for local experiments, set:

VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=false

9. Find The Edited Video

After rendering completes, the final output is:

output/edit-projects/porsche-session-001/final.mp4

The renderer also publishes each final rendered segment clip that was concatenated into the full video:

output/edit-projects/porsche-session-001/rendered-clips/clip_0001.mp4
output/edit-projects/porsche-session-001/rendered-clips/clip_0002.mp4
output/edit-projects/porsche-session-001/rendered-clips/clip_0003.mp4

The render manifest is:

output/edit-projects/porsche-session-001/render-manifest.json

The QA report is:

output/edit-projects/porsche-session-001/qa-report.json

The manifest is useful for debugging because it records the selected clips, rendered segment clip paths, output path, duration, and FFmpeg command summaries.

The QA report is useful for acceptance because it records:

final output existence
rendered segment clip existence
duration consistency with the edit timeline
required asset resolution
text overlay timeline and placement safety
audio mastering presence
FFmpeg command completion
black frame probe result
long silence probe result
audio clipping probe result

Check the final file with ffprobe:

ffprobe -v error \
  -show_entries format=duration \
  -show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
  -of json \
  output/edit-projects/porsche-session-001/final.mp4

10. Review The Result

Review these points before accepting the edited video:

final.mp4 exists.
Duration is close to the requested target duration.
The opening shot is strong and starts cleanly.
There are no black frames at the beginning or end.
Cut timing feels intentional.
Music, voiceover, and SFX are present if configured.
render-manifest.json does not contain invalid clip references.
qa-report.json has no failed ERROR checks.

Current renderer limitation:

Crossfade-style transitions are represented as paired boundary fades in the concat renderer. They are not true overlapping FFmpeg xfade transitions yet.

11. Troubleshooting

No project appears in output/edit-projects:

Confirm the service is running with the cinematic-editing-local profile.
Confirm the source folder is under input/editing/source.
Confirm the source folder is not still named .tmp, .part, .download, or .processing.
Confirm the folder contains at least one valid video file.
Check logs for event=local_director_project_claimed.

Analysis fails:

Run ffprobe manually against one source clip.
Confirm the input file is not still being copied.
Move bad files out of the project folder and try again.
Check input/editing/rejected for rejected project folders.

The AI cannot decide from thumbnails:

Increase VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP.
Keep VIDEO_EDITING_PROXY_ENABLED=true.
Ask the AI to inspect the generated proxy clips.
Delete the failed project output and rerun analysis if you intentionally changed analysis settings.

edit-plan.json is rejected:

Confirm every clipId exists in analysis.json.
Confirm source timestamps are inside each clip duration.
Confirm timeline timestamps are ordered and non-overlapping.
Confirm the JSON has no markdown wrapper.
Confirm audioCues and voiceover are arrays, even when empty.

Render fails:

Check application logs for event=edit_render_failed.
Check render-manifest.json if it exists.
Check qa-report.json if it exists.
Confirm all referenced source clips are still available under the project.
Temporarily remove music, SFX, and voiceover assets to isolate video rendering.

QA report has failed checks:

output_exists: render did not publish final.mp4; check application logs.
duration_matches_timeline: inspect edit-plan.json for timeline gaps or invalid final timestamp.
required_assets_resolved: add the missing SFX or voiceover file, or remove the cue from the edit plan.
text_overlays_safe: adjust overlay placement or timeline.
black_frames: inspect the affected render and replace black source sections or trim the decision.
long_silence: add music, reduce silent sections, or confirm intentional silence.
audio_clipping: lower music, SFX, or voiceover gain and render again.

Final video has no audio:

Confirm the source clips have audio.
Confirm audio/music.wav and audio/voiceover.wav are valid WAV files if used.
Confirm SFX cue assetKey values match files under audio/sfx.
Check the audio mix command in render-manifest.json.

Rendering is too slow:

Lower VIDEO_EDITING_OUTPUT_WIDTH and VIDEO_EDITING_OUTPUT_HEIGHT.
Lower VIDEO_EDITING_VIDEO_BITRATE.
Keep the first edit shorter while testing.
Add music and voiceover only after the video-only render works.

12. End-To-End Quick Path

Use this condensed checklist when the service is already working:

Start the service:

mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local

Create the source project:

input/editing/source/porsche-session-001/

Put all source clips in that folder.

Alternative: put the clips directly in input/editing/source; the service will create a source-clips edit project.

Wait for:

output/edit-projects/porsche-session-001/ai-director-prompt.md
output/edit-projects/porsche-session-001/analysis.json

Run Codex or Claude in:

output/edit-projects/porsche-session-001/

Tell the AI:

Read `ai-director-prompt.md`, `analysis.json`, contact sheets, thumbnails, and proxies. Write the cinematic edit plan to `inbox/edit-plan.json`. Do not render video.

Add optional audio assets under:

output/edit-projects/porsche-session-001/audio/

Render:

curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render

Open the edited video:

output/edit-projects/porsche-session-001/final.mp4

16 KiB Raw Blame History