video_editing_poc/docs/highlight-rendering-gap-closure-plan.md

Highlight Rendering Gap Closure Plan

Problem

The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.

Observed current output:

output/highlight-projects/<project-id>/
  analysis/
    source-analysis.json
    visual-analysis.json
    audio-analysis.json
    scene-segments.json
    frames/*.jpg
    contact-sheets/*.jpg
    proxies/*.mp4
  source/<source-video>
  highlights/

The highlights/ directory is empty because there is no connected pipeline that converts analysis/source-analysis.json into renderable highlight plans and then renders highlights/<highlight-id>/final.mp4.

Goal

When a user drops one source video into input/highlights/source, the service must eventually produce one or more rendered highlight videos:

output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json

The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.

Key Design Decision

Do not reuse FfmpegEditRenderer directly for highlight projects as-is.

Reason:

• FfmpegEditRenderer expects the older edit-project contract: analysis.json, edit-plan.json, final.mp4 at project root, and EditProjectService.
• Highlight projects use a different contract: analysis/source-analysis.json, director/, and highlights/<highlight-id>/.
• Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.

Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as EditPlan, EditDecision, AudioCue, VoiceoverLine, TextOverlay, RenderManifest, and RenderQaReport.

Target Flow

1. HighlightSourceScheduler claims one source video.
2. HighlightSourceAnalyzer creates analysis artifacts.
3. New HighlightCandidateGenerator creates ranked highlight candidates from scene, audio, and visual analysis.
4. New HighlightEditPlanGenerator creates one deterministic EditPlan per selected candidate.
5. New HighlightRenderer renders each plan into highlights/<highlight-id>/.
6. New QA step probes each rendered MP4 and writes qa-report.json.
7. Scheduler logs event=highlight_flow_completed with final output count and output paths.

Milestones

• Milestone 1: Persist highlight candidates for highlight-source projects.
• Milestone 2: Generate deterministic highlight edit plans from candidates.
• Milestone 3: Implement a highlight-project renderer that writes into highlights/<highlight-id>/.
• Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
• Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
• Milestone 6: Add full rich logs for every highlight rendering step and a final highlight_flow_completed log.
• Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
• Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.

Milestone Details

Milestone 1: Persist Highlight Candidates

Add a component:

HighlightCandidateGenerator

Input:

HighlightSourceAnalysis

Output:

analysis/highlight-candidates.json

Candidate rules for the first version:

• Use scene-segments.json as the primary temporal units.
• If only one scene exists, split the source into overlapping 8-20 second windows.
• Score candidates using available fields:
• visualAnalysis.motionScore
• visualAnalysis.compositionScore
• visualAnalysis.blurScore
• visualAnalysis.exposureScore
• audioAnalysis.sections
• scene duration and position
• Prefer candidates between 8 and 35 seconds.
• Avoid first/last black or silent areas when audio data suggests silence.
• Limit initial output to top 3 candidates.

Required logs:

event=highlight_candidates_started project_id=...
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...

Tests:

• Generates candidates from multiple scene segments.
• Falls back to fixed windows when only one scene exists.
• Rejects or downranks invalid, too-short, too-long, or blurry candidates.
• Persists analysis/highlight-candidates.json.

Milestone 2: Generate Deterministic Edit Plans

Add a component:

HighlightEditPlanGenerator

Input:

HighlightSourceAnalysis
List<HighlightCandidate>

Output per candidate:

highlights/<highlight-id>/edit-plan.json
highlights/<highlight-id>/storyboard.md

First version plan rules:

• Create one highlight per selected candidate.
• Use 3-6 edit decisions per highlight.
• For a short candidate, use the candidate range as one segment.
• For a longer candidate, split into smaller cuts with simple pacing.
• Use deterministic category style:
• car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
• family-like/faces: warm memory style, softer overlays.
• food-like/close action: rhythmic sensory style.
• generic: clean cinematic montage.
• Add optional text overlays only when safe.
• Add voiceover text as script metadata, but do not block render when no TTS provider exists.
• Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.

Required logs:

event=highlight_edit_plan_started project_id=... candidate_id=...
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...

Tests:

• Produces valid EditPlan timestamps inside source duration.
• Produces stable output for the same analysis.
• Creates storyboard markdown.
• Does not require external AI.

Milestone 3: Implement Highlight Renderer

Add:

HighlightRenderer
FfmpegHighlightRenderer

Input:

projectId
highlightId
highlights/<highlight-id>/edit-plan.json
analysis/source-analysis.json

Output:

highlights/<highlight-id>/final.mp4
highlights/<highlight-id>/preview.mp4
highlights/<highlight-id>/render-manifest.json
highlights/<highlight-id>/qa-report.json

Renderer requirements for first version:

• Trim from the original source video using exact timestamps.
• Concatenate selected segments.
• Apply simple visual treatment using FFmpeg filters:
• scale/pad to configured output size
• optional contrast/saturation/sharpen
• optional vignette or fade
• render overlays with drawtext when fonts are available or use default FFmpeg font fallback.
• Mix source audio.
• Optionally mix music/SFX/voiceover when local assets exist.
• Normalize output audio loudness when audio exists.
• Produce preview.mp4 as a lower-resolution copy or direct transcode.
• Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.

Important:

• Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
• Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.

Required logs:

event=highlight_render_started project_id=... highlight_id=...
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...

Tests:

• Builds expected FFmpeg trim commands.
• Builds concat and final render commands.
• Writes final output path under highlights/<highlight-id>/.
• Writes render manifest.
• Handles missing optional assets.
• Fails on invalid source timestamps.

Milestone 4: Local Asset Fallbacks

Current rendered highlights must work even with no licensed asset library.

Add fallback behavior:

• If no music asset exists, keep source audio.
• If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
• If no TTS provider exists, write voiceover script to assets/voiceover/voiceover-script.txt and skip voiceover audio.
• If no LUT exists, use named FFmpeg color preset.
• If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.

Required logs:

event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset

Tests:

• Render plan with no local assets still produces final.mp4.
• Manifest records skipped optional assets.
• Voiceover script file is still created.

Milestone 5: Wire Into Scheduler

Update HighlightSourceScheduler.createProject after analysis:

analysis = analyzer.analyze(projectId)
candidates = candidateGenerator.generate(projectId, analysis)
plans = planGenerator.generate(projectId, analysis, candidates)
renderResults = renderer.render(projectId, plans)
move source to processed
log flow complete

Configuration:

video-clipping:
  editing:
    highlight-scheduler:
      render-enabled: true
      max-highlights-per-source: 3
      highlight-min-duration-seconds: 8
      highlight-max-duration-seconds: 35
      require-director-approval: false

If require-director-approval=true, the scheduler should stop after writing plans and wait for an approval flag before rendering.

Required logs:

event=highlight_flow_started scan_id=... project_id=... source_file=...
event=highlight_flow_analysis_completed scan_id=... project_id=...
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...

Tests:

• Scheduler produces at least one highlights/<highlight-id>/final.mp4 from a valid fixture video.
• Scheduler leaves no source video in source or working after success.
• Scheduler moves failed source to rejected and logs failure.
• Rendering can be disabled for analysis-only mode.

Milestone 6: Rich End-To-End Logging

Add a single correlation ID:

flow_id=<project-id>:<scan-id>

Every log in the highlight path should include:

• scan_id
• project_id
• highlight_id when applicable
• flow_id
• elapsed_ms for completed steps

The final success log must be:

event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...

The final failure log must be:

event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...

Tests:

• Use a log capture test to assert highlight_flow_completed is emitted on success.
• Use a failure test to assert highlight_flow_failed includes the failed step.

Milestone 7: Tests And Coverage

Add focused unit tests:

• HighlightCandidateGeneratorTest
• HighlightEditPlanGeneratorTest
• FfmpegHighlightRendererTest
• HighlightRenderManifestTest
• HighlightSourceSchedulerRenderFlowTest

Add integration test:

HighlightRenderingIntegrationTest

Requirements:

• Create a tiny FFmpeg fixture video.
• Place it in a temp highlight source folder.
• Run the scheduler once.
• Assert:
• analysis/source-analysis.json exists.
• analysis/highlight-candidates.json exists.
• highlights/<highlight-id>/edit-plan.json exists.
• highlights/<highlight-id>/final.mp4 exists.
• highlights/<highlight-id>/render-manifest.json exists.
• highlights/<highlight-id>/qa-report.json exists.
• highlight_flow_completed log exists.

Milestone 8: Runbook Update

Update:

docs/cinematic-editing-runbook.md
docs/production-cinematic-highlight-editing-plan.md

Add:

• Exact output location for final highlight videos.
• How to tell whether the project is analysis-only or rendered.
• How to enable/disable automatic rendering.
• How to inspect highlight-candidates.json.
• How to inspect per-highlight edit-plan.json.
• How to inspect render-manifest.json and qa-report.json.
• Troubleshooting table for empty highlights/.

Implementation Order

1. Implement HighlightCandidateGenerator and persist analysis/highlight-candidates.json.
2. Implement HighlightEditPlanGenerator and write per-highlight plans/storyboards.
3. Implement FfmpegHighlightRenderer with minimal no-asset render.
4. Wire renderer behind highlight-scheduler.render-enabled.
5. Add final highlight_flow_completed and highlight_flow_failed logs.
6. Add integration test proving final.mp4 is created.
7. Add optional assets, overlays, audio mix, and QA improvements.
8. Update runbooks.

Definition Of Done

• Dropping one valid source video into input/highlights/source produces at least one rendered MP4 under output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4.
• The source file is moved to input/highlights/processed.
• The project contains analysis/highlight-candidates.json.
• Each rendered highlight contains edit-plan.json, storyboard.md, render-manifest.json, and qa-report.json.
• Logs contain highlight_flow_completed with final output paths.
• Missing optional assets do not prevent a basic highlight render.
• Full Maven tests pass.