From 2ec2b9c37f655258bef19215d3112a5b4eb9a399 Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Sat, 11 Jul 2026 01:26:46 +0200 Subject: [PATCH] Add cinematic editing operator runbook --- docs/cinematic-editing-runbook.md | 469 ++++++++++++++++++ ...uction-cinematic-highlight-editing-plan.md | 3 +- 2 files changed, 471 insertions(+), 1 deletion(-) create mode 100644 docs/cinematic-editing-runbook.md diff --git a/docs/cinematic-editing-runbook.md b/docs/cinematic-editing-runbook.md new file mode 100644 index 0000000..406f1cc --- /dev/null +++ b/docs/cinematic-editing-runbook.md @@ -0,0 +1,469 @@ +# Cinematic Editing Runbook + +This guide explains how to turn a folder of video clips into one edited cinematic MP4 using the local video editing service and an AI director such as Codex or Claude. + +The service does the heavy video work with FFmpeg. The AI director reviews generated thumbnails, contact sheets, metadata, and optional proxies, then writes an `edit-plan.json`. The service validates that plan and renders the final video. + +## 1. Prerequisites + +Install these tools on the machine running the service: + +- Java 21 +- Maven +- FFmpeg +- ffprobe +- Codex, Claude, or another AI instance that can read local files and images + +Verify the tools: + +```bash +java -version +mvn -version +ffmpeg -version +ffprobe -version +``` + +## 2. Start The Service + +From the repository root, start the service with the local cinematic editing profile: + +```bash +mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local +``` + +This profile disables the normal 8-second folder scheduler and enables the cinematic editing workflow. + +Expected startup behavior: + +- The service creates `input/editing/source`. +- The service creates `input/editing/working`. +- The service creates `input/editing/processed`. +- The service creates `input/editing/rejected`. +- The service creates `output/edit-projects`. +- The service logs that local director mode is enabled. + +Useful config defaults: + +```yaml +video-clipping: + editing: + project-directory: ./output/edit-projects + thumbnail-count-per-clip: 5 + proxy-enabled: true + target-duration-seconds: 60 + output-width: 1920 + output-height: 1080 + output-frame-rate: 30 + video-bitrate: 12000k + audio-bitrate: 192k + local-director: + source-directory: ./input/editing/source + poll-interval-ms: 5000 + auto-render-when-plan-appears: false +``` + +## 3. Prepare A Source Project Folder + +You can use either a project folder or loose clips directly under `input/editing/source`. + +The clearest layout is one folder per edit project under `input/editing/source`. + +Example: + +```text +input/editing/source/porsche-session-001/ +``` + +Put your source clips in that folder: + +```text +input/editing/source/porsche-session-001/clip_00000.mp4 +input/editing/source/porsche-session-001/clip_00001.mp4 +input/editing/source/porsche-session-001/clip_00002.mp4 +``` + +For the Porsche example with 35 existing 8-second clips, put all 35 files in the same project folder: + +```text +input/editing/source/porsche-session-001/clip_00000.mp4 +input/editing/source/porsche-session-001/clip_00001.mp4 +... +input/editing/source/porsche-session-001/clip_00034.mp4 +``` + +The scheduler also accepts loose video clips directly in `input/editing/source`: + +```text +input/editing/source/clip_00000.mp4 +input/editing/source/clip_00001.mp4 +input/editing/source/clip_00002.mp4 +``` + +When loose clips are found and no project folder is waiting, the service claims them as one generated project named `source-clips`. + +Recommended flow for large files or many clips: + +1. Copy the folder as `porsche-session-001.tmp`. +2. Wait until all clips are fully copied. +3. Rename the folder to `porsche-session-001`. + +The scheduler ignores temporary project folders ending in `.tmp`, `.part`, `.download`, or `.processing`. + +## 4. Wait For Service Analysis + +The local director scheduler scans every 5 seconds by default. + +When it finds the project folder, it moves the folder through this lifecycle: + +```text +input/editing/source/porsche-session-001 +input/editing/working/porsche-session-001 +input/editing/processed/porsche-session-001 +``` + +After analysis succeeds, the generated project appears here: + +```text +output/edit-projects/porsche-session-001/ +``` + +Expected generated files and folders: + +```text +output/edit-projects/porsche-session-001/project.json +output/edit-projects/porsche-session-001/analysis.json +output/edit-projects/porsche-session-001/ai-director-prompt.md +output/edit-projects/porsche-session-001/director-readme.md +output/edit-projects/porsche-session-001/contact-sheets/ +output/edit-projects/porsche-session-001/thumbnails/ +output/edit-projects/porsche-session-001/proxies/ +output/edit-projects/porsche-session-001/inbox/ +``` + +The expected project status at this point is: + +```text +WAITING_FOR_DIRECTOR +``` + +Check project state through the API: + +```bash +curl http://localhost:8080/v1/edit-projects/porsche-session-001 +``` + +## 5. Ask Codex Or Claude To Act As Director + +Open Codex, Claude, or the selected AI instance in this folder: + +```text +output/edit-projects/porsche-session-001/ +``` + +Model choice matters mostly for creative judgment, not rendering. Use a stronger model when the footage is ambiguous, repetitive, emotionally important, or needs a premium commercial feel. Use a cheaper model when the generated thumbnails/contact sheets are clear and the edit style is straightforward. + +Recommended split: + +- Use a stronger model for the first director pass on high-value edits, because it chooses shots, pacing, voiceover, text overlays, and audio moments. +- Use a cheaper model for JSON cleanup, schema repair, or small plan revisions after the creative direction is already clear. +- Do not ask any model to process full-resolution video directly unless necessary. Let the service provide thumbnails, contact sheets, metadata, and proxies. + +Give the AI this prompt: + +```md +You are the AI director for this local edit project. + +Read `ai-director-prompt.md`, `analysis.json`, and the generated contact sheets and thumbnails. Create a cinematic Porsche promo edit plan. Write strict JSON to `inbox/edit-plan.json`. Do not render video and do not modify the source clips. +``` + +The AI director should inspect: + +- `ai-director-prompt.md` +- `analysis.json` +- `contact-sheets/*.jpg` +- `thumbnails/*.jpg` +- `proxies/*.mp4` if more visual context is needed + +The AI director must write this file: + +```text +output/edit-projects/porsche-session-001/inbox/edit-plan.json +``` + +Do not put markdown fences around the JSON. The file must contain strict JSON only. + +## 6. What The Edit Plan Must Contain + +The generated `edit-plan.json` must match the service schema. + +Minimal shape: + +```json +{ + "style": "cinematic-porsche-promo", + "targetDurationSeconds": 60, + "renderProfile": "mp4-h264-aac-1080p", + "summary": "A cinematic Porsche promo with a strong opening, detail shots, driving energy, and a final hero shot.", + "decisions": [ + { + "clipId": "clip_00000", + "sourceStartSeconds": 0.0, + "sourceEndSeconds": 3.5, + "timelineStartSeconds": 0.0, + "timelineEndSeconds": 3.5, + "transitionIn": "fade", + "transitionOut": "cut", + "playbackSpeed": 1.0, + "visualTreatment": "warm high-contrast cinematic grade", + "reason": "Opening hero angle establishes the car immediately." + } + ], + "audioCues": [], + "voiceover": [] +} +``` + +Rules the AI director must follow: + +- Every `clipId` must exist in `analysis.json`. +- `sourceStartSeconds` and `sourceEndSeconds` must be inside the source clip duration. +- `sourceEndSeconds` must be greater than `sourceStartSeconds`. +- `timelineEndSeconds` must be greater than `timelineStartSeconds`. +- Timeline decisions should be ordered and non-overlapping. +- `playbackSpeed` should usually stay near `1.0` unless the plan intentionally uses slow motion or speed ramp style. +- `audioCues` and `voiceover` must be present, even if they are empty arrays. + +## 7. Add Optional Music, Voiceover, And SFX + +The renderer can mix optional WAV files into the final output. + +Place optional assets here before rendering: + +```text +output/edit-projects/porsche-session-001/audio/music.wav +output/edit-projects/porsche-session-001/audio/voiceover.wav +output/edit-projects/porsche-session-001/audio/sfx/engine-rev.wav +``` + +For SFX cues, the `assetKey` is the filename without `.wav`. + +Example: + +```json +{ + "type": "sfx", + "assetKey": "engine-rev", + "timelineStartSeconds": 5.2, + "timelineEndSeconds": 6.4, + "gainDb": -3.0, + "notes": "Short engine accent on acceleration shot." +} +``` + +For narration, the default `noop` voiceover provider writes a script file but does not synthesize audio. If the plan includes `voiceover` lines, record or generate the narration and place it here before rendering: + +```text +output/edit-projects/porsche-session-001/audio/voiceover.wav +``` + +## 8. Render The Final Video + +By default, the local profile has: + +```yaml +auto-render-when-plan-appears: false +``` + +That means you manually trigger rendering after the AI writes `inbox/edit-plan.json`: + +```bash +curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render +``` + +If you set `VIDEO_EDITING_LOCAL_DIRECTOR_AUTO_RENDER=true`, the service renders automatically after it detects and validates: + +```text +output/edit-projects/porsche-session-001/inbox/edit-plan.json +``` + +## 9. Find The Edited Video + +After rendering completes, the final output is: + +```text +output/edit-projects/porsche-session-001/final.mp4 +``` + +The render manifest is: + +```text +output/edit-projects/porsche-session-001/render-manifest.json +``` + +The QA report is: + +```text +output/edit-projects/porsche-session-001/qa-report.json +``` + +The manifest is useful for debugging because it records the selected clips, output path, duration, and FFmpeg command summaries. + +The QA report is useful for acceptance because it records: + +- final output existence +- duration consistency with the edit timeline +- required asset resolution +- text overlay timeline and placement safety +- audio mastering presence +- FFmpeg command completion +- black frame probe result +- long silence probe result +- audio clipping probe result + +Check the final file with ffprobe: + +```bash +ffprobe -v error \ + -show_entries format=duration \ + -show_entries stream=codec_type,codec_name,width,height,r_frame_rate \ + -of json \ + output/edit-projects/porsche-session-001/final.mp4 +``` + +## 10. Review The Result + +Review these points before accepting the edited video: + +- `final.mp4` exists. +- Duration is close to the requested target duration. +- The opening shot is strong and starts cleanly. +- There are no black frames at the beginning or end. +- Cut timing feels intentional. +- Music, voiceover, and SFX are present if configured. +- `render-manifest.json` does not contain invalid clip references. +- `qa-report.json` has no failed `ERROR` checks. + +Current renderer limitation: + +- Crossfade-style transitions are represented as paired boundary fades in the concat renderer. They are not true overlapping FFmpeg `xfade` transitions yet. + +## 11. Troubleshooting + +No project appears in `output/edit-projects`: + +- Confirm the service is running with the `cinematic-editing-local` profile. +- Confirm the source folder is under `input/editing/source`. +- Confirm the source folder is not still named `.tmp`, `.part`, `.download`, or `.processing`. +- Confirm the folder contains at least one valid video file. +- Check logs for `event=local_director_project_claimed`. + +Analysis fails: + +- Run `ffprobe` manually against one source clip. +- Confirm the input file is not still being copied. +- Move bad files out of the project folder and try again. +- Check `input/editing/rejected` for rejected project folders. + +The AI cannot decide from thumbnails: + +- Increase `VIDEO_EDITING_THUMBNAIL_COUNT_PER_CLIP`. +- Keep `VIDEO_EDITING_PROXY_ENABLED=true`. +- Ask the AI to inspect the generated proxy clips. +- Delete the failed project output and rerun analysis if you intentionally changed analysis settings. + +`edit-plan.json` is rejected: + +- Confirm every `clipId` exists in `analysis.json`. +- Confirm source timestamps are inside each clip duration. +- Confirm timeline timestamps are ordered and non-overlapping. +- Confirm the JSON has no markdown wrapper. +- Confirm `audioCues` and `voiceover` are arrays, even when empty. + +Render fails: + +- Check application logs for `event=edit_render_failed`. +- Check `render-manifest.json` if it exists. +- Check `qa-report.json` if it exists. +- Confirm all referenced source clips are still available under the project. +- Temporarily remove music, SFX, and voiceover assets to isolate video rendering. + +QA report has failed checks: + +- `output_exists`: render did not publish `final.mp4`; check application logs. +- `duration_matches_timeline`: inspect `edit-plan.json` for timeline gaps or invalid final timestamp. +- `required_assets_resolved`: add the missing SFX or voiceover file, or remove the cue from the edit plan. +- `text_overlays_safe`: adjust overlay placement or timeline. +- `black_frames`: inspect the affected render and replace black source sections or trim the decision. +- `long_silence`: add music, reduce silent sections, or confirm intentional silence. +- `audio_clipping`: lower music, SFX, or voiceover gain and render again. + +Final video has no audio: + +- Confirm the source clips have audio. +- Confirm `audio/music.wav` and `audio/voiceover.wav` are valid WAV files if used. +- Confirm SFX cue `assetKey` values match files under `audio/sfx`. +- Check the audio mix command in `render-manifest.json`. + +Rendering is too slow: + +- Lower `VIDEO_EDITING_OUTPUT_WIDTH` and `VIDEO_EDITING_OUTPUT_HEIGHT`. +- Lower `VIDEO_EDITING_VIDEO_BITRATE`. +- Keep the first edit shorter while testing. +- Add music and voiceover only after the video-only render works. + +## 12. End-To-End Quick Path + +Use this condensed checklist when the service is already working: + +1. Start the service: + +```bash +mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local +``` + +2. Create the source project: + +```text +input/editing/source/porsche-session-001/ +``` + +3. Put all source clips in that folder. + +Alternative: put the clips directly in `input/editing/source`; the service will create a `source-clips` edit project. + +4. Wait for: + +```text +output/edit-projects/porsche-session-001/ai-director-prompt.md +output/edit-projects/porsche-session-001/analysis.json +``` + +5. Run Codex or Claude in: + +```text +output/edit-projects/porsche-session-001/ +``` + +6. Tell the AI: + +```md +Read `ai-director-prompt.md`, `analysis.json`, contact sheets, thumbnails, and proxies. Write the cinematic edit plan to `inbox/edit-plan.json`. Do not render video. +``` + +7. Add optional audio assets under: + +```text +output/edit-projects/porsche-session-001/audio/ +``` + +8. Render: + +```bash +curl -X POST http://localhost:8080/v1/edit-projects/porsche-session-001:render +``` + +9. Open the edited video: + +```text +output/edit-projects/porsche-session-001/final.mp4 +``` diff --git a/docs/production-cinematic-highlight-editing-plan.md b/docs/production-cinematic-highlight-editing-plan.md index 1c3c1e9..03ff095 100644 --- a/docs/production-cinematic-highlight-editing-plan.md +++ b/docs/production-cinematic-highlight-editing-plan.md @@ -311,7 +311,8 @@ The plan should be strict JSON so a cheaper model or deterministic renderer can - [x] Milestone 13 progress: Renderer now applies timed text overlays, records resolved/planned edit assets in the render manifest, adds dynamic punch-in crop motion, ducks music under voiceover, and normalizes final mix loudness with configurable mastering values. - [x] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters. - [x] Milestone 14 progress: Renderer now writes `qa-report.json` with structural checks, asset resolution checks, overlay safety checks, audio mastering checks, FFmpeg command completion checks, and FFmpeg probe checks for black frames, long silence, and audio clipping. -- [ ] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips. +- [x] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips. +- [x] Milestone 15 progress: Added `docs/cinematic-editing-runbook.md` with local setup, source placement, AI director model guidance, edit-plan review, rendering, final output lookup, QA report review, and troubleshooting. - [ ] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content. - [ ] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size. - [ ] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering.