forked from jsl/video_editing_poc
508 lines
16 KiB
Markdown
508 lines
16 KiB
Markdown
# Cinematic Editing Runbook
|
|
|
|
This guide explains how to turn a folder of video clips into one edited cinematic MP4 using the local video editing service and an AI director such as Codex or Claude.
|
|
|
|
The service does the heavy video work with FFmpeg. The AI director reviews generated thumbnails, contact sheets, metadata, and optional proxies, then writes an `edit-plan.json`. The service validates that plan and renders the final video.
|
|
|
|
## 1. Prerequisites
|
|
|
|
Install these tools on the machine running the service:
|
|
|
|
- Java 21
|
|
- Maven
|
|
- FFmpeg
|
|
- ffprobe
|
|
- Codex, Claude, or another AI instance that can read local files and images
|
|
|
|
Verify the tools:
|
|
|
|
```bash
|
|
java -version
|
|
mvn -version
|
|
ffmpeg -version
|
|
ffprobe -version
|
|
```
|
|
|
|
## 2. Start The Service
|
|
|
|
From the repository root, start the service with the local cinematic editing profile:
|
|
|
|
```bash
|
|
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
|
```
|
|
|
|
This profile disables the normal 8-second folder scheduler and enables the cinematic editing workflow.
|
|
|
|
Expected startup behavior:
|
|
|
|
- The service creates `input/editing/source`.
|
|
- The service creates `input/editing/working`.
|
|
- The service creates `input/editing/processed`.
|
|
- The service creates `input/editing/rejected`.
|
|
- The service creates `output/edit-projects`.
|
|
- The service logs that local director mode is enabled.
|
|
|
|
Useful config defaults:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
project-directory: ./output/edit-projects
|
|
thumbnail-count-per-clip: 5
|
|
proxy-enabled: true
|
|
target-duration-seconds: 600
|
|
output-width: 1920
|
|
output-height: 1080
|
|
output-frame-rate: 30
|
|
video-bitrate: 12000k
|
|
audio-bitrate: 192k
|
|
local-director:
|
|
source-directory: ./input/editing/source
|
|
poll-interval-ms: 5000
|
|
auto-render-when-plan-appears: false
|
|
require-approval-before-render: true
|
|
approval-file-name: approved.flag
|
|
```
|
|
|
|
## 3. Prepare A Source Project Folder
|
|
|
|
You can use either a project folder or loose clips directly under `input/editing/source`.
|
|
|
|
The clearest layout is one folder per edit project under `input/editing/source`.
|
|
|
|
Example:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/
|
|
```
|
|
|
|
Put your source clips in that folder:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/clip_00000.mp4
|
|
input/editing/source/porsche-session-001/clip_00001.mp4
|
|
input/editing/source/porsche-session-001/clip_00002.mp4
|
|
```
|
|
|
|
For the Porsche example with 35 existing 8-second clips, put all 35 files in the same project folder:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/clip_00000.mp4
|
|
input/editing/source/porsche-session-001/clip_00001.mp4
|
|
...
|
|
input/editing/source/porsche-session-001/clip_00034.mp4
|
|
```
|
|
|
|
The scheduler also accepts loose video clips directly in `input/editing/source`:
|
|
|
|
```text
|
|
input/editing/source/clip_00000.mp4
|
|
input/editing/source/clip_00001.mp4
|
|
input/editing/source/clip_00002.mp4
|
|
```
|
|
|
|
When loose clips are found and no project folder is waiting, the service claims them as one generated project named `source-clips`.
|
|
|
|
Recommended flow for large files or many clips:
|
|
|
|
1. Copy the folder as `porsche-session-001.tmp`.
|
|
2. Wait until all clips are fully copied.
|
|
3. Rename the folder to `porsche-session-001`.
|
|
|
|
The scheduler ignores temporary project folders ending in `.tmp`, `.part`, `.download`, or `.processing`.
|
|
|
|
## 4. Wait For Service Analysis
|
|
|
|
The local director scheduler scans every 5 seconds by default.
|
|
|
|
When it finds the project folder, it moves the folder through this lifecycle:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001
|
|
input/editing/working/porsche-session-001
|
|
input/editing/processed/porsche-session-001
|
|
```
|
|
|
|
After analysis succeeds, the generated project appears here:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/
|
|
```
|
|
|
|
Expected generated files and folders:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/project.json
|
|
output/edit-projects/porsche-session-001/analysis.json
|
|
output/edit-projects/porsche-session-001/ai-director-prompt.md
|
|
output/edit-projects/porsche-session-001/director-readme.md
|
|
output/edit-projects/porsche-session-001/contact-sheets/
|
|
output/edit-projects/porsche-session-001/thumbnails/
|
|
output/edit-projects/porsche-session-001/proxies/
|
|
output/edit-projects/porsche-session-001/inbox/
|
|
```
|
|
|
|
The expected project status at this point is:
|
|
|
|
```text
|
|
WAITING_FOR_DIRECTOR
|
|
```
|
|
|
|
Check project state through the API:
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/edit-projects/porsche-session-001
|
|
```
|
|
|
|
## 5. Ask Codex Or Claude To Act As Director
|
|
|
|
Open Codex, Claude, or the selected AI instance in this folder:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/
|
|
```
|
|
|
|
Model choice matters mostly for creative judgment, not rendering. Use a stronger model when the footage is ambiguous, repetitive, emotionally important, or needs a premium commercial feel. Use a cheaper model when the generated thumbnails/contact sheets are clear and the edit style is straightforward.
|
|
|
|
Recommended split:
|
|
|
|
- Use a stronger model for the first director pass on high-value edits, because it chooses shots, pacing, voiceover, text overlays, and audio moments.
|
|
- Use a cheaper model for JSON cleanup, schema repair, or small plan revisions after the creative direction is already clear.
|
|
- Do not ask any model to process full-resolution video directly unless necessary. Let the service provide thumbnails, contact sheets, metadata, and proxies.
|
|
|
|
Give the AI this prompt:
|
|
|
|
```md
|
|
You are the AI director for this local edit project.
|
|
|
|
Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.
|
|
```
|
|
|
|
The AI director should inspect:
|
|
|
|
- `ai-director-prompt.md`
|
|
- `analysis.json`
|
|
- `contact-sheets/*.jpg`
|
|
- `thumbnails/*.jpg`
|
|
- `proxies/*.mp4` if more visual context is needed
|
|
|
|
After the plan is imported, the service also runs an asset materialization stage:
|
|
|
|
- It reuses matching generated music, SFX, and voiceover assets from the shared cache folders.
|
|
- It copies reusable assets into the project-local `audio/` folder so the renderer can use them.
|
|
- It writes reusable asset requests under `assets/requests/` when a needed asset is missing.
|
|
- It waits to render if required SFX assets are still missing.
|
|
|
|
The AI director must write this file:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
|
```
|
|
|
|
Do not put markdown fences around the JSON. The file must contain strict JSON only.
|
|
|
|
## 6. What The Edit Plan Must Contain
|
|
|
|
The generated `edit-plan.json` must match the service schema.
|
|
|
|
Minimal shape:
|
|
|
|
```json
|
|
{
|
|
"style": "cinematic-porsche-promo",
|
|
"targetDurationSeconds": 60,
|
|
"renderProfile": "mp4-h264-aac-1080p",
|
|
"summary": "A cinematic Porsche promo with a strong opening, detail shots, driving energy, and a final hero shot.",
|
|
"decisions": [
|
|
{
|
|
"clipId": "clip_00000",
|
|
"sourceStartSeconds": 0.0,
|
|
"sourceEndSeconds": 3.5,
|
|
"timelineStartSeconds": 0.0,
|
|
"timelineEndSeconds": 3.5,
|
|
"transitionIn": "fade",
|
|
"transitionOut": "cut",
|
|
"playbackSpeed": 1.0,
|
|
"visualTreatment": "warm high-contrast cinematic grade",
|
|
"reason": "Opening hero angle establishes the car immediately."
|
|
}
|
|
],
|
|
"audioCues": [],
|
|
"voiceover": []
|
|
}
|
|
```
|
|
|
|
Rules the AI director must follow:
|
|
|
|
- Every `clipId` must exist in `analysis.json`.
|
|
- `sourceStartSeconds` and `sourceEndSeconds` must be inside the source clip duration.
|
|
- `sourceEndSeconds` must be greater than `sourceStartSeconds`.
|
|
- `timelineEndSeconds` must be greater than `timelineStartSeconds`.
|
|
- Timeline decisions should be ordered and non-overlapping.
|
|
- `playbackSpeed` should usually stay near `1.0` unless the plan intentionally uses slow motion or speed ramp style.
|
|
- `audioCues` and `voiceover` must be present, even if they are empty arrays.
|
|
|
|
## 7. Add Optional Music, Voiceover, And SFX
|
|
|
|
The renderer can mix optional WAV files into the final output.
|
|
|
|
Shared cache locations:
|
|
|
|
- Music: `input/highlights/assets/music/`
|
|
- SFX: `input/highlights/assets/sfx/`
|
|
- Voiceover cache: `output/highlight-projects/_voiceover-cache/`
|
|
|
|
If the service or an AI worker generates an asset there, later projects can reuse it automatically.
|
|
|
|
Place optional assets here before rendering:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/audio/music.wav
|
|
output/edit-projects/porsche-session-001/audio/voiceover.wav
|
|
output/edit-projects/porsche-session-001/audio/sfx/engine-rev.wav
|
|
```
|
|
|
|
For SFX cues, the `assetKey` is the filename without `.wav`.
|
|
|
|
Example:
|
|
|
|
```json
|
|
{
|
|
"type": "sfx",
|
|
"assetKey": "engine-rev",
|
|
"timelineStartSeconds": 5.2,
|
|
"timelineEndSeconds": 6.4,
|
|
"gainDb": -3.0,
|
|
"notes": "Short engine accent on acceleration shot."
|
|
}
|
|
```
|
|
|
|
For narration, the default `noop` voiceover provider writes a script file but does not synthesize audio. If the plan includes `voiceover` lines, record or generate the narration and place it here before rendering:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/audio/voiceover.wav
|
|
```
|
|
|
|
## 8. Render The Final Video
|
|
|
|
By default, the local profile has:
|
|
|
|
```yaml
|
|
auto-render-when-plan-appears: false
|
|
```
|
|
|
|
That means you manually trigger rendering after the AI writes `inbox/edit-plan.json`:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render
|
|
```
|
|
|
|
If you set `VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=true`, the service renders automatically after it detects and validates:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
|
```
|
|
|
|
By default, automatic rendering is still gated by human review. To approve a validated plan for automatic rendering, create:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/inbox/approved.flag
|
|
```
|
|
|
|
On the next inbox scan, the service renders the already imported `edit-plan.json`. To disable the approval gate for local experiments, set:
|
|
|
|
```bash
|
|
VIDEO_EDITING_LOCAL_DIRECTOR_REQUIRE_APPROVAL=false
|
|
```
|
|
|
|
## 9. Find The Edited Video
|
|
|
|
After rendering completes, the final output is:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/final.mp4
|
|
```
|
|
|
|
The renderer also publishes each final rendered segment clip that was concatenated into the full video:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0001.mp4
|
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0002.mp4
|
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0003.mp4
|
|
```
|
|
|
|
The render manifest is:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/render-manifest.json
|
|
```
|
|
|
|
The QA report is:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/qa-report.json
|
|
```
|
|
|
|
The manifest is useful for debugging because it records the selected clips, rendered segment clip paths, output path, duration, and FFmpeg command summaries.
|
|
|
|
The QA report is useful for acceptance because it records:
|
|
|
|
- final output existence
|
|
- rendered segment clip existence
|
|
- duration consistency with the edit timeline
|
|
- required asset resolution
|
|
- text overlay timeline and placement safety
|
|
- audio mastering presence
|
|
- FFmpeg command completion
|
|
- black frame probe result
|
|
- long silence probe result
|
|
- audio clipping probe result
|
|
|
|
Check the final file with ffprobe:
|
|
|
|
```bash
|
|
ffprobe -v error \
|
|
-show_entries format=duration \
|
|
-show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
|
|
-of json \
|
|
output/edit-projects/porsche-session-001/final.mp4
|
|
```
|
|
|
|
## 10. Review The Result
|
|
|
|
Review these points before accepting the edited video:
|
|
|
|
- `final.mp4` exists.
|
|
- Duration is close to the requested target duration.
|
|
- The opening shot is strong and starts cleanly.
|
|
- There are no black frames at the beginning or end.
|
|
- Cut timing feels intentional.
|
|
- Music, voiceover, and SFX are present if configured.
|
|
- `render-manifest.json` does not contain invalid clip references.
|
|
- `qa-report.json` has no failed `ERROR` checks.
|
|
|
|
Current renderer limitation:
|
|
|
|
- Crossfade-style transitions are represented as paired boundary fades in the concat renderer. They are not true overlapping FFmpeg `xfade` transitions yet.
|
|
|
|
## 11. Troubleshooting
|
|
|
|
No project appears in `output/edit-projects`:
|
|
|
|
- Confirm the service is running with the `cinematic-editing-local` profile.
|
|
- Confirm the source folder is under `input/editing/source`.
|
|
- Confirm the source folder is not still named `.tmp`, `.part`, `.download`, or `.processing`.
|
|
- Confirm the folder contains at least one valid video file.
|
|
- Check logs for `event=local_director_project_claimed`.
|
|
|
|
Analysis fails:
|
|
|
|
- Run `ffprobe` manually against one source clip.
|
|
- Confirm the input file is not still being copied.
|
|
- Move bad files out of the project folder and try again.
|
|
- Check `input/editing/rejected` for rejected project folders.
|
|
|
|
The AI cannot decide from thumbnails:
|
|
|
|
- Increase `VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP`.
|
|
- Keep `VIDEO_EDITING_PROXY_ENABLED=true`.
|
|
- Ask the AI to inspect the generated proxy clips.
|
|
- Delete the failed project output and rerun analysis if you intentionally changed analysis settings.
|
|
|
|
`edit-plan.json` is rejected:
|
|
|
|
- Confirm every `clipId` exists in `analysis.json`.
|
|
- Confirm source timestamps are inside each clip duration.
|
|
- Confirm timeline timestamps are ordered and non-overlapping.
|
|
- Confirm the JSON has no markdown wrapper.
|
|
- Confirm `audioCues` and `voiceover` are arrays, even when empty.
|
|
|
|
Render fails:
|
|
|
|
- Check application logs for `event=edit_render_failed`.
|
|
- Check `render-manifest.json` if it exists.
|
|
- Check `qa-report.json` if it exists.
|
|
- Confirm all referenced source clips are still available under the project.
|
|
- Temporarily remove music, SFX, and voiceover assets to isolate video rendering.
|
|
|
|
QA report has failed checks:
|
|
|
|
- `output_exists`: render did not publish `final.mp4`; check application logs.
|
|
- `duration_matches_timeline`: inspect `edit-plan.json` for timeline gaps or invalid final timestamp.
|
|
- `required_assets_resolved`: add the missing SFX or voiceover file, or remove the cue from the edit plan.
|
|
- `text_overlays_safe`: adjust overlay placement or timeline.
|
|
- `black_frames`: inspect the affected render and replace black source sections or trim the decision.
|
|
- `long_silence`: add music, reduce silent sections, or confirm intentional silence.
|
|
- `audio_clipping`: lower music, SFX, or voiceover gain and render again.
|
|
|
|
Final video has no audio:
|
|
|
|
- Confirm the source clips have audio.
|
|
- Confirm `audio/music.wav` and `audio/voiceover.wav` are valid WAV files if used.
|
|
- Confirm SFX cue `assetKey` values match files under `audio/sfx`.
|
|
- Check the audio mix command in `render-manifest.json`.
|
|
|
|
Rendering is too slow:
|
|
|
|
- Lower `VIDEO_EDITING_OUTPUT_WIDTH` and `VIDEO_EDITING_OUTPUT_HEIGHT`.
|
|
- Lower `VIDEO_EDITING_VIDEO_BITRATE`.
|
|
- Keep the first edit shorter while testing.
|
|
- Add music and voiceover only after the video-only render works.
|
|
|
|
## 12. End-To-End Quick Path
|
|
|
|
Use this condensed checklist when the service is already working:
|
|
|
|
1. Start the service:
|
|
|
|
```bash
|
|
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
|
```
|
|
|
|
2. Create the source project:
|
|
|
|
```text
|
|
input/editing/source/porsche-session-001/
|
|
```
|
|
|
|
3. Put all source clips in that folder.
|
|
|
|
Alternative: put the clips directly in `input/editing/source`; the service will create a `source-clips` edit project.
|
|
|
|
4. Wait for:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/ai-director-prompt.md
|
|
output/edit-projects/porsche-session-001/analysis.json
|
|
```
|
|
|
|
5. Run Codex or Claude in:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/
|
|
```
|
|
|
|
6. Tell the AI:
|
|
|
|
```md
|
|
Read `ai-director-prompt.md`, `analysis.json`, contact sheets, thumbnails, and proxies. Write the cinematic edit plan to `inbox/edit-plan.json`. Do not render video.
|
|
```
|
|
|
|
7. Add optional audio assets under:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/audio/
|
|
```
|
|
|
|
8. Render:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render
|
|
```
|
|
|
|
9. Open the edited video:
|
|
|
|
```text
|
|
output/edit-projects/porsche-session-001/final.mp4
|
|
```
|