forked from jsl/video_editing_poc
Add cinematic editing operator runbook
This commit is contained in:
JSLMPR
parent
9b56e89b4c
commit
2ec2b9c37f
|
|
@ -0,0 +1,469 @@
|
|||
# Cinematic Editing Runbook
|
||||
|
||||
This guide explains how to turn a folder of video clips into one edited cinematic MP4 using the local video editing service and an AI director such as Codex or Claude.
|
||||
|
||||
The service does the heavy video work with FFmpeg. The AI director reviews generated thumbnails, contact sheets, metadata, and optional proxies, then writes an `edit-plan.json`. The service validates that plan and renders the final video.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install these tools on the machine running the service:
|
||||
|
||||
- Java 21
|
||||
- Maven
|
||||
- FFmpeg
|
||||
- ffprobe
|
||||
- Codex, Claude, or another AI instance that can read local files and images
|
||||
|
||||
Verify the tools:
|
||||
|
||||
```bash
|
||||
java -version
|
||||
mvn -version
|
||||
ffmpeg -version
|
||||
ffprobe -version
|
||||
```
|
||||
|
||||
## 2. Start The Service
|
||||
|
||||
From the repository root, start the service with the local cinematic editing profile:
|
||||
|
||||
```bash
|
||||
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
||||
```
|
||||
|
||||
This profile disables the normal 8-second folder scheduler and enables the cinematic editing workflow.
|
||||
|
||||
Expected startup behavior:
|
||||
|
||||
- The service creates `input/editing/source`.
|
||||
- The service creates `input/editing/working`.
|
||||
- The service creates `input/editing/processed`.
|
||||
- The service creates `input/editing/rejected`.
|
||||
- The service creates `output/edit-projects`.
|
||||
- The service logs that local director mode is enabled.
|
||||
|
||||
Useful config defaults:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
editing:
|
||||
project-directory: ./output/edit-projects
|
||||
thumbnail-count-per-clip: 5
|
||||
proxy-enabled: true
|
||||
target-duration-seconds: 60
|
||||
output-width: 1920
|
||||
output-height: 1080
|
||||
output-frame-rate: 30
|
||||
video-bitrate: 12000k
|
||||
audio-bitrate: 192k
|
||||
local-director:
|
||||
source-directory: ./input/editing/source
|
||||
poll-interval-ms: 5000
|
||||
auto-render-when-plan-appears: false
|
||||
```
|
||||
|
||||
## 3. Prepare A Source Project Folder
|
||||
|
||||
You can use either a project folder or loose clips directly under `input/editing/source`.
|
||||
|
||||
The clearest layout is one folder per edit project under `input/editing/source`.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
input/editing/source/porsche-session-001/
|
||||
```
|
||||
|
||||
Put your source clips in that folder:
|
||||
|
||||
```text
|
||||
input/editing/source/porsche-session-001/clip_00000.mp4
|
||||
input/editing/source/porsche-session-001/clip_00001.mp4
|
||||
input/editing/source/porsche-session-001/clip_00002.mp4
|
||||
```
|
||||
|
||||
For the Porsche example with 35 existing 8-second clips, put all 35 files in the same project folder:
|
||||
|
||||
```text
|
||||
input/editing/source/porsche-session-001/clip_00000.mp4
|
||||
input/editing/source/porsche-session-001/clip_00001.mp4
|
||||
...
|
||||
input/editing/source/porsche-session-001/clip_00034.mp4
|
||||
```
|
||||
|
||||
The scheduler also accepts loose video clips directly in `input/editing/source`:
|
||||
|
||||
```text
|
||||
input/editing/source/clip_00000.mp4
|
||||
input/editing/source/clip_00001.mp4
|
||||
input/editing/source/clip_00002.mp4
|
||||
```
|
||||
|
||||
When loose clips are found and no project folder is waiting, the service claims them as one generated project named `source-clips`.
|
||||
|
||||
Recommended flow for large files or many clips:
|
||||
|
||||
1. Copy the folder as `porsche-session-001.tmp`.
|
||||
2. Wait until all clips are fully copied.
|
||||
3. Rename the folder to `porsche-session-001`.
|
||||
|
||||
The scheduler ignores temporary project folders ending in `.tmp`, `.part`, `.download`, or `.processing`.
|
||||
|
||||
## 4. Wait For Service Analysis
|
||||
|
||||
The local director scheduler scans every 5 seconds by default.
|
||||
|
||||
When it finds the project folder, it moves the folder through this lifecycle:
|
||||
|
||||
```text
|
||||
input/editing/source/porsche-session-001
|
||||
input/editing/working/porsche-session-001
|
||||
input/editing/processed/porsche-session-001
|
||||
```
|
||||
|
||||
After analysis succeeds, the generated project appears here:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/
|
||||
```
|
||||
|
||||
Expected generated files and folders:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/project.json
|
||||
output/edit-projects/porsche-session-001/analysis.json
|
||||
output/edit-projects/porsche-session-001/ai-director-prompt.md
|
||||
output/edit-projects/porsche-session-001/director-readme.md
|
||||
output/edit-projects/porsche-session-001/contact-sheets/
|
||||
output/edit-projects/porsche-session-001/thumbnails/
|
||||
output/edit-projects/porsche-session-001/proxies/
|
||||
output/edit-projects/porsche-session-001/inbox/
|
||||
```
|
||||
|
||||
The expected project status at this point is:
|
||||
|
||||
```text
|
||||
WAITING_FOR_DIRECTOR
|
||||
```
|
||||
|
||||
Check project state through the API:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/edit-projects/porsche-session-001
|
||||
```
|
||||
|
||||
## 5. Ask Codex Or Claude To Act As Director
|
||||
|
||||
Open Codex, Claude, or the selected AI instance in this folder:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/
|
||||
```
|
||||
|
||||
Model choice matters mostly for creative judgment, not rendering. Use a stronger model when the footage is ambiguous, repetitive, emotionally important, or needs a premium commercial feel. Use a cheaper model when the generated thumbnails/contact sheets are clear and the edit style is straightforward.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- Use a stronger model for the first director pass on high-value edits, because it chooses shots, pacing, voiceover, text overlays, and audio moments.
|
||||
- Use a cheaper model for JSON cleanup, schema repair, or small plan revisions after the creative direction is already clear.
|
||||
- Do not ask any model to process full-resolution video directly unless necessary. Let the service provide thumbnails, contact sheets, metadata, and proxies.
|
||||
|
||||
Give the AI this prompt:
|
||||
|
||||
```md
|
||||
You are the AI director for this local edit project.
|
||||
|
||||
Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips.
|
||||
```
|
||||
|
||||
The AI director should inspect:
|
||||
|
||||
- `ai-director-prompt.md`
|
||||
- `analysis.json`
|
||||
- `contact-sheets/*.jpg`
|
||||
- `thumbnails/*.jpg`
|
||||
- `proxies/*.mp4` if more visual context is needed
|
||||
|
||||
The AI director must write this file:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
||||
```
|
||||
|
||||
Do not put markdown fences around the JSON. The file must contain strict JSON only.
|
||||
|
||||
## 6. What The Edit Plan Must Contain
|
||||
|
||||
The generated `edit-plan.json` must match the service schema.
|
||||
|
||||
Minimal shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"style": "cinematic-porsche-promo",
|
||||
"targetDurationSeconds": 60,
|
||||
"renderProfile": "mp4-h264-aac-1080p",
|
||||
"summary": "A cinematic Porsche promo with a strong opening, detail shots, driving energy, and a final hero shot.",
|
||||
"decisions": [
|
||||
{
|
||||
"clipId": "clip_00000",
|
||||
"sourceStartSeconds": 0.0,
|
||||
"sourceEndSeconds": 3.5,
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": 3.5,
|
||||
"transitionIn": "fade",
|
||||
"transitionOut": "cut",
|
||||
"playbackSpeed": 1.0,
|
||||
"visualTreatment": "warm high-contrast cinematic grade",
|
||||
"reason": "Opening hero angle establishes the car immediately."
|
||||
}
|
||||
],
|
||||
"audioCues": [],
|
||||
"voiceover": []
|
||||
}
|
||||
```
|
||||
|
||||
Rules the AI director must follow:
|
||||
|
||||
- Every `clipId` must exist in `analysis.json`.
|
||||
- `sourceStartSeconds` and `sourceEndSeconds` must be inside the source clip duration.
|
||||
- `sourceEndSeconds` must be greater than `sourceStartSeconds`.
|
||||
- `timelineEndSeconds` must be greater than `timelineStartSeconds`.
|
||||
- Timeline decisions should be ordered and non-overlapping.
|
||||
- `playbackSpeed` should usually stay near `1.0` unless the plan intentionally uses slow motion or speed ramp style.
|
||||
- `audioCues` and `voiceover` must be present, even if they are empty arrays.
|
||||
|
||||
## 7. Add Optional Music, Voiceover, And SFX
|
||||
|
||||
The renderer can mix optional WAV files into the final output.
|
||||
|
||||
Place optional assets here before rendering:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/audio/music.wav
|
||||
output/edit-projects/porsche-session-001/audio/voiceover.wav
|
||||
output/edit-projects/porsche-session-001/audio/sfx/engine-rev.wav
|
||||
```
|
||||
|
||||
For SFX cues, the `assetKey` is the filename without `.wav`.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "sfx",
|
||||
"assetKey": "engine-rev",
|
||||
"timelineStartSeconds": 5.2,
|
||||
"timelineEndSeconds": 6.4,
|
||||
"gainDb": -3.0,
|
||||
"notes": "Short engine accent on acceleration shot."
|
||||
}
|
||||
```
|
||||
|
||||
For narration, the default `noop` voiceover provider writes a script file but does not synthesize audio. If the plan includes `voiceover` lines, record or generate the narration and place it here before rendering:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/audio/voiceover.wav
|
||||
```
|
||||
|
||||
## 8. Render The Final Video
|
||||
|
||||
By default, the local profile has:
|
||||
|
||||
```yaml
|
||||
auto-render-when-plan-appears: false
|
||||
```
|
||||
|
||||
That means you manually trigger rendering after the AI writes `inbox/edit-plan.json`:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render
|
||||
```
|
||||
|
||||
If you set `VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=true`, the service renders automatically after it detects and validates:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/inbox/edit-plan.json
|
||||
```
|
||||
|
||||
## 9. Find The Edited Video
|
||||
|
||||
After rendering completes, the final output is:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/final.mp4
|
||||
```
|
||||
|
||||
The render manifest is:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/render-manifest.json
|
||||
```
|
||||
|
||||
The QA report is:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/qa-report.json
|
||||
```
|
||||
|
||||
The manifest is useful for debugging because it records the selected clips, output path, duration, and FFmpeg command summaries.
|
||||
|
||||
The QA report is useful for acceptance because it records:
|
||||
|
||||
- final output existence
|
||||
- duration consistency with the edit timeline
|
||||
- required asset resolution
|
||||
- text overlay timeline and placement safety
|
||||
- audio mastering presence
|
||||
- FFmpeg command completion
|
||||
- black frame probe result
|
||||
- long silence probe result
|
||||
- audio clipping probe result
|
||||
|
||||
Check the final file with ffprobe:
|
||||
|
||||
```bash
|
||||
ffprobe -v error \
|
||||
-show_entries format=duration \
|
||||
-show_entries stream=codec_type,codec_name,width,height,r_frame_rate \
|
||||
-of json \
|
||||
output/edit-projects/porsche-session-001/final.mp4
|
||||
```
|
||||
|
||||
## 10. Review The Result
|
||||
|
||||
Review these points before accepting the edited video:
|
||||
|
||||
- `final.mp4` exists.
|
||||
- Duration is close to the requested target duration.
|
||||
- The opening shot is strong and starts cleanly.
|
||||
- There are no black frames at the beginning or end.
|
||||
- Cut timing feels intentional.
|
||||
- Music, voiceover, and SFX are present if configured.
|
||||
- `render-manifest.json` does not contain invalid clip references.
|
||||
- `qa-report.json` has no failed `ERROR` checks.
|
||||
|
||||
Current renderer limitation:
|
||||
|
||||
- Crossfade-style transitions are represented as paired boundary fades in the concat renderer. They are not true overlapping FFmpeg `xfade` transitions yet.
|
||||
|
||||
## 11. Troubleshooting
|
||||
|
||||
No project appears in `output/edit-projects`:
|
||||
|
||||
- Confirm the service is running with the `cinematic-editing-local` profile.
|
||||
- Confirm the source folder is under `input/editing/source`.
|
||||
- Confirm the source folder is not still named `.tmp`, `.part`, `.download`, or `.processing`.
|
||||
- Confirm the folder contains at least one valid video file.
|
||||
- Check logs for `event=local_director_project_claimed`.
|
||||
|
||||
Analysis fails:
|
||||
|
||||
- Run `ffprobe` manually against one source clip.
|
||||
- Confirm the input file is not still being copied.
|
||||
- Move bad files out of the project folder and try again.
|
||||
- Check `input/editing/rejected` for rejected project folders.
|
||||
|
||||
The AI cannot decide from thumbnails:
|
||||
|
||||
- Increase `VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP`.
|
||||
- Keep `VIDEO_EDITING_PROXY_ENABLED=true`.
|
||||
- Ask the AI to inspect the generated proxy clips.
|
||||
- Delete the failed project output and rerun analysis if you intentionally changed analysis settings.
|
||||
|
||||
`edit-plan.json` is rejected:
|
||||
|
||||
- Confirm every `clipId` exists in `analysis.json`.
|
||||
- Confirm source timestamps are inside each clip duration.
|
||||
- Confirm timeline timestamps are ordered and non-overlapping.
|
||||
- Confirm the JSON has no markdown wrapper.
|
||||
- Confirm `audioCues` and `voiceover` are arrays, even when empty.
|
||||
|
||||
Render fails:
|
||||
|
||||
- Check application logs for `event=edit_render_failed`.
|
||||
- Check `render-manifest.json` if it exists.
|
||||
- Check `qa-report.json` if it exists.
|
||||
- Confirm all referenced source clips are still available under the project.
|
||||
- Temporarily remove music, SFX, and voiceover assets to isolate video rendering.
|
||||
|
||||
QA report has failed checks:
|
||||
|
||||
- `output_exists`: render did not publish `final.mp4`; check application logs.
|
||||
- `duration_matches_timeline`: inspect `edit-plan.json` for timeline gaps or invalid final timestamp.
|
||||
- `required_assets_resolved`: add the missing SFX or voiceover file, or remove the cue from the edit plan.
|
||||
- `text_overlays_safe`: adjust overlay placement or timeline.
|
||||
- `black_frames`: inspect the affected render and replace black source sections or trim the decision.
|
||||
- `long_silence`: add music, reduce silent sections, or confirm intentional silence.
|
||||
- `audio_clipping`: lower music, SFX, or voiceover gain and render again.
|
||||
|
||||
Final video has no audio:
|
||||
|
||||
- Confirm the source clips have audio.
|
||||
- Confirm `audio/music.wav` and `audio/voiceover.wav` are valid WAV files if used.
|
||||
- Confirm SFX cue `assetKey` values match files under `audio/sfx`.
|
||||
- Check the audio mix command in `render-manifest.json`.
|
||||
|
||||
Rendering is too slow:
|
||||
|
||||
- Lower `VIDEO_EDITING_OUTPUT_WIDTH` and `VIDEO_EDITING_OUTPUT_HEIGHT`.
|
||||
- Lower `VIDEO_EDITING_VIDEO_BITRATE`.
|
||||
- Keep the first edit shorter while testing.
|
||||
- Add music and voiceover only after the video-only render works.
|
||||
|
||||
## 12. End-To-End Quick Path
|
||||
|
||||
Use this condensed checklist when the service is already working:
|
||||
|
||||
1. Start the service:
|
||||
|
||||
```bash
|
||||
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
|
||||
```
|
||||
|
||||
2. Create the source project:
|
||||
|
||||
```text
|
||||
input/editing/source/porsche-session-001/
|
||||
```
|
||||
|
||||
3. Put all source clips in that folder.
|
||||
|
||||
Alternative: put the clips directly in `input/editing/source`; the service will create a `source-clips` edit project.
|
||||
|
||||
4. Wait for:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/ai-director-prompt.md
|
||||
output/edit-projects/porsche-session-001/analysis.json
|
||||
```
|
||||
|
||||
5. Run Codex or Claude in:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/
|
||||
```
|
||||
|
||||
6. Tell the AI:
|
||||
|
||||
```md
|
||||
Read `ai-director-prompt.md`, `analysis.json`, contact sheets, thumbnails, and proxies. Write the cinematic edit plan to `inbox/edit-plan.json`. Do not render video.
|
||||
```
|
||||
|
||||
7. Add optional audio assets under:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/audio/
|
||||
```
|
||||
|
||||
8. Render:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render
|
||||
```
|
||||
|
||||
9. Open the edited video:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/final.mp4
|
||||
```
|
||||
|
|
@ -311,7 +311,8 @@ The plan should be strict JSON so a cheaper model or deterministic renderer can
|
|||
- [x] Milestone 13 progress: Renderer now applies timed text overlays, records resolved/planned edit assets in the render manifest, adds dynamic punch-in crop motion, ducks music under voiceover, and normalizes final mix loudness with configurable mastering values.
|
||||
- [x] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters.
|
||||
- [x] Milestone 14 progress: Renderer now writes `qa-report.json` with structural checks, asset resolution checks, overlay safety checks, audio mastering checks, FFmpeg command completion checks, and FFmpeg probe checks for black frames, long silence, and audio clipping.
|
||||
- [ ] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips.
|
||||
- [x] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips.
|
||||
- [x] Milestone 15 progress: Added `docs/cinematic-editing-runbook.md` with local setup, source placement, AI director model guidance, edit-plan review, rendering, final output lookup, QA report review, and troubleshooting.
|
||||
- [ ] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content.
|
||||
- [ ] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size.
|
||||
- [ ] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering.
|
||||
|
|
|
|||
Loading…
Reference in New Issue