video_editing_poc/docs/production-cinematic-highlight-editing-plan.md

# Production Cinematic Highlight Editing Plan

## Goal

Build a local-first workflow where a user drops one source video into a source folder and the service produces multiple production-ready cinematic highlight edits. Each highlight must be adapted to the detected content type, such as family vlog, food vlog, car vlog, or a generic fallback category.

The target output is not just a technically valid MP4. The output must feel intentionally directed: strong shot selection, a clear mini-story, pacing matched to music, fitting voiceover, sound effects, cinematic visual treatment, tasteful text overlays, and a final audio mix that sounds finished.

## Research Findings

- Recent highlight-detection research points toward multimodal narrative understanding, not simple duration slicing. A production system should combine shot segmentation, visual analysis, speech/transcript analysis, audio events, pacing, and narrative role detection.
- Audio matters for highlight detection. Interesting moments are often signaled by speech emphasis, engine sound, laughter, applause, cooking sounds, silence, music changes, or loudness peaks.
- Shot and scene detection should be treated as a core primitive. The service should first split the source video into meaningful temporal units before asking an AI director to choose edits.
- Modern editing tools increasingly use transcript search, visual search, beat detection, montage generation, auto-cropping, and music synchronization. The service should expose similar building blocks internally instead of relying on a prompt alone.
- FFmpeg is suitable as the deterministic render backend, but the current renderer must be expanded. Production-level edits need real filter graphs for speed ramps, transitions, stabilization, text, color, grain, vignettes, audio ducking, loudness normalization, SFX placement, and final mastering.

Research sources:

- https://arxiv.org/abs/2507.10495
- https://arxiv.org/abs/2401.10075
- https://arxiv.org/abs/2107.09612
- https://arxiv.org/abs/2306.11918
- https://en.wikipedia.org/wiki/Shot_transition_detection
- https://en.wikipedia.org/wiki/Video_content_analysis
- https://en.wikipedia.org/wiki/Film_editing
- https://en.wikipedia.org/wiki/FFmpeg
- https://www.digitalcameraworld.com/cameras/video-cameras/these-are-the-new-ai-powered-features-coming-to-final-cut-pro-and-imovie

## Target Workflow

1. User places one source video in the configured local source folder.
2. Scheduler detects the video and creates a highlight project.
3. Service probes the source with `ffprobe` and creates stable low-resolution proxy media.
4. Service extracts thumbnails, contact sheets, audio waveform data, audio stems when possible, and transcript data.
5. Service performs shot and scene segmentation.
6. Service classifies the content category.
7. Service generates highlight candidates from visual, audio, transcript, and motion signals.
8. AI director reviews the compact analysis artifacts and creates a strict edit plan.
9. Asset providers create or select voiceover, music, SFX, captions, and overlay text.
10. Renderer executes the edit plan with FFmpeg and produces final highlight MP4s.
11. QA checks validate technical quality and creative completeness.
12. Final clips, manifests, storyboards, and logs are written to the output folder.

## Proposed Folder Contract

Input:

```text
input/highlights/source/<source-video-file>
```

Output:

```text
output/highlight-projects/<project-id>/
  project.json
  source/
  analysis/
    ffprobe.json
    shots.json
    scenes.json
    transcript.json
    audio-events.json
    visual-analysis.json
    category.json
    highlight-candidates.json
  director/
    director-brief.md
    director-prompt.md
    edit-plan.json
    storyboard.md
  assets/
    voiceover/
    music/
    sfx/
    overlays/
  highlights/
    <highlight-id>/
      final.mp4
      preview.mp4
      render-manifest.json
      qa-report.json
```

## Content Categories

### Family Vlog

Creative intent:

- Warm, emotional, human, memory-focused.
- Preserve natural faces, voices, laughter, and real moments.
- Avoid over-editing or aggressive transitions.
- Use gentle music, soft cinematic color, light film grain, and meaningful text overlays.

Highlight signals:

- Smiles, hugs, reactions, children, group moments, travel reveal moments, laughter.
- Speech with emotional words or important context.
- Stable shots with faces and clear composition.

Voiceover style:

- Personal, warm, reflective, short.
- Should sound like a memory being narrated, not an advertisement.

### Food Vlog

Creative intent:

- Sensory, rhythmic, appetizing, tactile.
- Make ingredients, textures, steam, plating, and reactions feel premium.
- Sync cuts to cooking sounds and music beats.

Highlight signals:

- Close-ups of ingredients, cutting, sizzling, steam, pouring, plating, first bite, reaction.
- Strong texture, color, macro framing, hand movement, kitchen action.

Voiceover style:

- Descriptive and sensory.
- Mention texture, aroma, craft, freshness, and payoff.

### Car Vlog

Creative intent:

- Heroic, powerful, premium, dynamic.
- Emphasize stance, motion, engine sound, details, acceleration, road presence, reflections, wheels, badges, lights, interior, and reveal shots.
- Use speed ramps, bass hits, whooshes, cinematic grade, controlled text overlays, and strong music.

Highlight signals:

- Car fully visible, front/rear three-quarter angles, wheels turning, badge/detail shots, engine/exhaust audio, road motion, camera movement, dramatic lighting.
- Avoid repeated static hood shots unless used as setup.

Voiceover style:

- Confident, cinematic, concise.
- Should praise the vehicle specifically based on observed details, not use generic supercar phrases.

### Generic Vlog Fallback

Creative intent:

- Clear story arc, clean pacing, tasteful grade, and context-aware overlays.
- Use the best available moments even when the category is uncertain.

Highlight signals:

- High motion, strong composition, faces, speech peaks, laughter, visual novelty, scene changes.

## AI Director Responsibilities

The AI director should not render the video. It should produce a validated edit decision plan that the renderer can execute.

Required AI director inputs:

- Source metadata: duration, resolution, frame rate, codec, audio streams.
- Shot list with timestamps, representative thumbnails, motion score, blur score, brightness score, face/object labels, and transcript references.
- Audio analysis: speech segments, silence, peaks, loudness, detected audio events, music/noise/speech classification.
- Category classification with confidence and reasons.
- Highlight candidates with ranked evidence.
- Available music, SFX, voice, font, LUT, and transition capabilities.

Required AI director output:

- Highlight list with target duration and category.
- Timeline edit decisions with exact source timestamps.
- Music mood, energy curve, and beat-sync points.
- SFX cues with timestamps and purpose.
- Voiceover script split into timed lines.
- Text overlay copy, timing, animation, placement, and safe-area rules.
- Visual treatment: grade, transitions, speed ramps, stabilization, crop, punch-ins, grain, vignette, glow, blur, and overlays.
- Render profile: aspect ratio, resolution, FPS policy, codec, CRF or bitrate mode, loudness target.

## Renderer Requirements

The renderer must support these production features before the output can be called cinematic:

- Precise trims using source timestamps.
- Crossfades, match cuts, hard cuts, whip transitions, fade to black, and dip transitions.
- Speed ramps and short impact pauses.
- Dynamic crop and punch-in keyframes for static shots.
- Stabilization option for shaky footage.
- Category-specific color grade chains or LUT support.
- Film grain, vignette, sharpening, glow, blur, and selective contrast.
- Text overlays with font, position, animation timing, stroke, shadow, and safe area.
- Caption and subtitle rendering.
- Music bed placement with fade in/out.
- Voiceover placement with sidechain ducking.
- SFX placement for whooshes, impacts, risers, camera clicks, engine hits, cooking sounds, and ambient layers.
- Audio loudness normalization to a configured target.
- Final render manifest with all commands, source segments, assets, and durations.

## Asset Provider Requirements

The system needs explicit providers instead of pretending a prompt can create finished audio by itself.

- Voiceover provider: local TTS or remote TTS, configurable voice per category, output WAV.
- Music provider: licensed local library first, optional generated music provider later.
- SFX provider: licensed local SFX library with tags and category mapping.
- Font provider: local fonts with brand/category mapping.
- LUT provider: local LUT files or named FFmpeg grade presets.
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
- Asset generation stage: when a needed asset is missing, write a reusable request pack and keep the target cache path stable so future AI-generated assets can be reused.

Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.

## Highlight Scoring Model

Each candidate segment should be scored with a weighted model:

```text
score =
  visual_interest
  + motion_energy
  + audio_interest
  + transcript_importance
  + category_relevance
  + composition_quality
  + novelty
  - blur_penalty
  - darkness_penalty
  - repetition_penalty
  - unusable_audio_penalty
```

Category-specific examples:

- Family vlog should boost faces, laughter, dialogue, emotional reactions, and group moments.
- Food vlog should boost close-ups, texture, cooking actions, plating, steam, and bite reactions.
- Car vlog should boost full-car hero shots, motion, wheels, engine audio, badges, roads, and dramatic lighting.

## Edit Plan Schema

The plan should be strict JSON so a cheaper model or deterministic renderer can execute it.

```json
{
  "projectId": "example-project",
  "sourceVideo": "input/highlights/source/example.mp4",
  "contentCategory": "car_vlog",
  "highlights": [
    {
      "id": "hero-reveal-001",
      "title": "The Reveal",
      "targetDurationSeconds": 35,
      "aspectRatio": "16:9",
      "renderProfile": "cinematic_4k_h264",
      "creativeIntent": "Premium Porsche reveal with dynamic pacing and confident narration.",
      "timeline": [
        {
          "type": "video",
          "sourceStart": "00:01:12.400",
          "sourceEnd": "00:01:16.900",
          "timelineStart": "00:00:00.000",
          "effects": ["stabilize", "cinematic_warm_contrast", "slow_push_in"],
          "transitionOut": "bass_hit_cut"
        }
      ],
      "voiceover": [
        {
          "text": "This is not just a drive. It is precision, presence, and control.",
          "timelineStart": "00:00:04.200",
          "voice": "confident_cinematic_male",
          "delivery": "calm premium"
        }
      ],
      "music": {
        "mood": "premium cinematic electronic",
        "energyCurve": "slow_build_to_impact",
        "assetId": "music/premium-drive-90bpm.wav"
      },
      "sfx": [
        {
          "assetId": "sfx/bass-hit-01.wav",
          "timelineStart": "00:00:07.800",
          "gainDb": -6
        }
      ],
      "overlays": [
        {
          "text": "PRECISION IN MOTION",
          "timelineStart": "00:00:08.000",
          "durationSeconds": 2.5,
          "placement": "lower_left_safe",
          "animation": "fade_slide_up"
        }
      ]
    }
  ]
}
```

## Milestones

- [x] Milestone 1: Define the cinematic highlight project domain model, folder contract, and persisted manifests.
- [x] Milestone 1 progress: Added a dedicated highlight project domain, standard folder contract, manifest model, configurable `output/highlight-projects` root, filesystem store, and tests for directory creation, JSON round trips, and path safety.
- [x] Milestone 2: Add a local scheduler that accepts exactly one source video at a time from the highlight source folder.
- [x] Milestone 2 progress: Added a configurable highlight source scheduler and directory initializer for `input/highlights/source`, `working`, `processed`, and `rejected`; it claims one valid source video per scan, creates a highlight project and manifest, copies the source into the project folder, and leaves later videos for later scans.
- [x] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
- [x] Milestone 3 progress: Added highlight source analysis for the new project contract, reusing FFprobe inspection, frame extraction, contact sheet generation, proxy generation, and adding waveform image generation under `analysis/`; the highlight scheduler now runs analysis immediately after creating a project.
- [x] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
- [x] Milestone 4 progress: Added configurable FFmpeg scene detection for highlight sources, persisted `analysis/scene-segments.json`, attached shot segments to `source-analysis.json`, and logged shot segment counts when the highlight scheduler finishes analysis.
- [x] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
- [x] Milestone 5 progress: Added deterministic FFmpeg audio analysis for mean/max volume, silence detection, missing-audio handling, persisted `analysis/audio-analysis.json`, and source-analysis embedding. Non-silent sections are labeled `unclassified_audio` until a speech/music/noise classifier or ASR provider is added.
- [x] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
- [x] Milestone 6 progress: Added persisted `analysis/visual-analysis.json` and source-analysis embedding with blur, exposure, motion, composition, representative thumbnail, face-presence, and object-label fields. Current implementation uses deterministic metadata, thumbnail, and scene-density heuristics; face/object recognition is explicitly marked heuristic until a CV model is connected.
- [x] Milestone 6 local CV extension: Added a configurable `local-cv` visual-analysis provider that posts source metadata, thumbnails, and shot segments to a local HTTP CV worker, persists the returned normalized visual analysis, and falls back to heuristic analysis when configured. See `docs/local-cv-visual-analysis-guide.md`.
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
- [x] Milestone 10: Implement edit plan validation so impossible timestamps, missing assets, invalid overlays, and unsupported effects fail before render.
- [x] Milestone 11: Implement asset provider interfaces for voiceover, music, SFX, fonts, and LUTs.
- [x] Milestone 12: Add a local licensed asset library with category tags and deterministic asset selection.
- [x] Milestone 13: Implement renderer v2 with transitions, speed ramps, dynamic crops, overlays, visual effects, audio ducking, loudness normalization, and render manifests.
- [x] Milestone 13 progress: Renderer now applies timed text overlays, records resolved/planned edit assets in the render manifest, adds dynamic punch-in crop motion, ducks music under voiceover, and normalizes final mix loudness with configurable mastering values.
- [x] Milestone 14: Implement QA checks for black frames, silence, clipping, missing assets, duration mismatch, unsafe text placement, and failed FFmpeg filters.
- [x] Milestone 14 progress: Renderer now writes `qa-report.json` with structural checks, asset resolution checks, overlay safety checks, audio mastering checks, FFmpeg command completion checks, and FFmpeg probe checks for black frames, long silence, and audio clipping.
- [x] Milestone 15: Add an operator runbook for placing a video, running the service locally, choosing a model, reviewing the director plan, and finding final clips.
- [x] Milestone 15 progress: Added `docs/cinematic-editing-runbook.md` with local setup, source placement, AI director model guidance, edit-plan review, rendering, final output lookup, QA report review, and troubleshooting.
- [x] Milestone 16: Add integration tests with small fixture videos for family, food, car, and generic content.
- [x] Milestone 16 progress: Added FFmpeg-backed local director integration coverage for car, food, family, and generic fixture projects, verifying category artifacts, highlight candidates, and director prompts.
- [x] Milestone 17: Add benchmark metrics for analysis time, render time, token usage, asset generation cost, and final output size.
- [x] Milestone 17 progress: Added Micrometer benchmark summaries for analysis clip/error counts, render output duration, render output size, AI director prompt token estimate, and asset generation cost baseline.
- [x] Milestone 18: Add a human review mode where the user can approve or edit the AI director plan before rendering.
- [x] Milestone 18 progress: Added a configurable local director approval gate so imported plans wait for `inbox/approved.flag` before automatic rendering, with profile defaults, runbook guidance, and scanner tests.

## Testing Strategy

- Unit tests for category scoring, candidate ranking, schema validation, folder discovery, and manifest writing.
- Integration tests with short fixture videos that verify each category can produce at least one valid highlight plan.
- Renderer tests that verify FFmpeg commands include expected filters for transitions, overlays, color treatment, music, SFX, and voiceover.
- Golden-file tests for AI prompt inputs and expected JSON schema shape.
- End-to-end local test that places one video in the source folder and verifies final clips, manifests, storyboards, and QA reports are created.
- Negative tests for corrupt input video, unsupported codec, empty video, missing audio, missing asset, invalid timestamp, and renderer failure.

## Step-By-Step Local How-To Guide

This guide describes the intended operator workflow after the milestones are implemented.

### 1. Prepare The Machine

Install and verify required binaries:

```bash
ffmpeg -version
ffprobe -version
```

Optional but recommended tools:

```bash
python3 --version
```

Recommended future optional components:

- PySceneDetect for higher-quality scene detection.
- Whisper or another ASR provider for transcript-based editing.
- A configured TTS provider for voiceover.
- A licensed local music and SFX library.

### 2. Configure The Application

Set the local cinematic highlight folders in `application.yml`:

```yaml
video-clipping:
  cinematic-highlights:
    enabled: true
    source-folder: input/highlights/source
    output-folder: output/highlight-projects
    scheduler-interval: 10s
    max-active-projects: 1
    default-aspect-ratio: "16:9"
    default-render-profile: cinematic_1080p_h264
```

Configure the AI director model separately from the renderer:

```yaml
video-clipping:
  cinematic-highlights:
    director:
      enabled: true
      model: gpt-5
      allow-cheaper-model: true
      cheaper-model: gpt-5-mini
      require-human-approval-before-render: true
```

Configure local assets:

```yaml
video-clipping:
  cinematic-highlights:
    assets:
      music-folder: input/highlights/assets/music
      sfx-folder: input/highlights/assets/sfx
      fonts-folder: input/highlights/assets/fonts
      luts-folder: input/highlights/assets/luts
      voiceover-folder: output/highlight-projects/_voiceover-cache
```

### 3. Create Folders

Create the required local folders:

```bash
mkdir -p input/highlights/source
mkdir -p input/highlights/assets/music
mkdir -p input/highlights/assets/sfx
mkdir -p input/highlights/assets/fonts
mkdir -p input/highlights/assets/luts
mkdir -p output/highlight-projects
```

### 4. Add The Source Video

Place exactly one source video in the source folder:

```text
input/highlights/source/my-porsche-drive.mp4
```

The scheduler should process one source video at a time. If multiple videos are present, it should process the first valid file, finish it, then move to the next one.

Invalid files should be ignored and logged with the reason.

### 5. Start The Application

Run the Spring Boot service:

```bash
./mvnw spring-boot:run
```

Expected startup logs:

```text
Cinematic highlight scheduler started
Watching source folder: input/highlights/source
Output folder: output/highlight-projects
Scheduler interval: 10s
Max active projects: 1
```

### 6. Verify Project Creation

After the scheduler picks up the source file, verify a project folder exists:

```text
output/highlight-projects/<project-id>/
```

Expected early files:

```text
project.json
analysis/ffprobe.json
analysis/shots.json
analysis/scenes.json
analysis/category.json
director/director-brief.md
director/director-prompt.md
```

Expected logs:

```text
Started cinematic highlight project projectId=<project-id> source=my-porsche-drive.mp4
Generated proxy video
Extracted representative thumbnails
Detected shots and scenes
Classified content category category=car_vlog confidence=...
Generated highlight candidates count=...
```

### 7. Review The AI Director Prompt

Open the director prompt:

```text
output/highlight-projects/<project-id>/director/director-prompt.md
```

Check that it includes:

- Source metadata.
- Category classification.
- Contact sheet or thumbnail references.
- Candidate segments with timestamps.
- Available renderer effects.
- Available music, SFX, voice, font, and LUT assets.
- Strict JSON schema requirements.
- Category-specific creative direction.

If the prompt is generic, the final edit will probably be generic. The prompt must force concrete choices: exact cuts, specific overlays, voiceover lines, SFX timing, music mood, and visual treatment.

### 8. Execute The AI Director

Run the configured AI director against the prompt. The director must output:

```text
output/highlight-projects/<project-id>/director/edit-plan.json
```

If human approval is enabled, inspect the edit plan before rendering.

Minimum review checklist:

- The selected shots are not repetitive.
- Each highlight has a clear beginning, middle, and ending.
- The edit uses the correct category style.
- Voiceover text is specific to the observed video.
- Music and SFX cues are not placeholders.
- Text overlays are tasteful and not overused.
- All referenced timestamps are inside the source duration.
- All referenced assets exist.

### 9. Approve Or Correct The Edit Plan

If the plan is weak, edit or regenerate it before rendering.

Common fixes:

- Replace generic voiceover with concrete lines about the actual footage.
- Remove boring shots or repeated angles.
- Add stronger openers and endings.
- Add music beat cues.
- Add SFX only where they support a cut or reveal.
- Add overlay text only where it improves the scene.
- Adjust target duration if the highlight feels too slow.

When approved, write:

```text
output/highlight-projects/<project-id>/director/approved.flag
```

### 10. Render The Highlights

The renderer consumes:

```text
output/highlight-projects/<project-id>/director/edit-plan.json
```

Expected renderer output:

```text
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0001.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0002.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0003.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
```

Expected logs:

```text
Rendering highlight projectId=<project-id> highlightId=<highlight-id>
Applying video trims count=...
Applying transitions count=...
Applying visual effects count=...
Mixing music asset=...
Mixing voiceover lines=...
Mixing SFX count=...
Completed highlight render duration=... output=...
```

### 11. Review The Final Clips

For each `final.mp4`, verify:

- The first three seconds are interesting.
- The edit has intentional pacing.
- The selected shots fit the content type.
- Music fits the scene and is audible.
- Voiceover is clear and not robotic or generic.
- SFX support the edit without becoming noisy.
- Text overlays are readable and not cheap-looking.
- Color grade improves the footage without crushing detail.
- Audio is not clipping and does not have long silent sections.

If the final video is boring, inspect in this order:

1. `analysis/highlight-candidates.json`: weak source moments or wrong scoring.
2. `director/edit-plan.json`: weak AI choices, generic voiceover, no effects.
3. `render-manifest.json`: renderer ignored effects or missing assets.
4. `qa-report.json`: technical failures or missing audio/effects.

### 12. Find The Final Result

Final highlight videos are stored here:

```text
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
```

The full audit trail is stored beside the final video:

```text
render-manifest.json
qa-report.json
```

The creative plan is stored here:

```text
output/highlight-projects/<project-id>/director/storyboard.md
output/highlight-projects/<project-id>/director/edit-plan.json
```

### 13. Clean Up After A Run

After confirming the output is good, move the source video out of the source folder or let the scheduler move it to a processed folder:

```text
input/highlights/processed/<source-video-file>
```

Failed inputs should move to:

```text
input/highlights/failed/<source-video-file>
```

The scheduler must not repeatedly process the same source video.

## Definition Of Done

A highlight export is production-ready only when:

- The selected shots match the detected content category.
- The edit has a clear beginning, middle, and ending.
- Music is present, fitting, and timed to key cuts.
- Voiceover is present when requested and sounds appropriate for the category.
- SFX are present where they add impact and are not randomly overused.
- Text overlays are readable, visually appealing, correctly timed, and safe-area compliant.
- Visual treatment is visible but not destructive.
- Audio is normalized and free of clipping.
- Output includes a render manifest, edit plan, storyboard, and QA report.
- The final MP4 can be played independently without relying on temporary files.

## Key Risk

The previous boring output is expected if the system only creates a weak prompt and then renders simple trims. Cinematic output requires three things working together: richer multimodal analysis, a strict category-aware AI director plan, and a renderer that can actually execute music, SFX, voiceover, overlays, transitions, and visual effects.