# Highlight Rendering Gap Closure Plan ## Problem The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos. Observed current output: ```text output/highlight-projects// analysis/ source-analysis.json visual-analysis.json audio-analysis.json scene-segments.json frames/*.jpg contact-sheets/*.jpg proxies/*.mp4 source/ highlights/ ``` The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights//final.mp4`. ## Goal When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos: ```text output/highlight-projects//highlights//final.mp4 output/highlight-projects//highlights//preview.mp4 output/highlight-projects//highlights//edit-plan.json output/highlight-projects//highlights//render-manifest.json output/highlight-projects//highlights//qa-report.json ``` The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance. ## Key Design Decision Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is. Reason: - `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`. - Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights//`. - Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion. Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`. ## Target Flow 1. `HighlightSourceScheduler` claims one source video. 2. `HighlightSourceAnalyzer` creates analysis artifacts. 3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis. 4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate. 5. New `HighlightRenderer` renders each plan into `highlights//`. 6. New QA step probes each rendered MP4 and writes `qa-report.json`. 7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths. ## Milestones - [ ] Milestone 1: Persist highlight candidates for highlight-source projects. - [ ] Milestone 2: Generate deterministic highlight edit plans from candidates. - [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights//`. - [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets. - [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow. - [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log. - [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output. - [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs. ## Milestone Details ### Milestone 1: Persist Highlight Candidates Add a component: ```text HighlightCandidateGenerator ``` Input: ```text HighlightSourceAnalysis ``` Output: ```text analysis/highlight-candidates.json ``` Candidate rules for the first version: - Use `scene-segments.json` as the primary temporal units. - If only one scene exists, split the source into overlapping 8-20 second windows. - Score candidates using available fields: - `visualAnalysis.motionScore` - `visualAnalysis.compositionScore` - `visualAnalysis.blurScore` - `visualAnalysis.exposureScore` - `audioAnalysis.sections` - scene duration and position - Prefer candidates between 8 and 35 seconds. - Avoid first/last black or silent areas when audio data suggests silence. - Limit initial output to top 3 candidates. Required logs: ```text event=highlight_candidates_started project_id=... event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=... event=highlight_candidates_completed project_id=... count=... elapsed_ms=... ``` Tests: - Generates candidates from multiple scene segments. - Falls back to fixed windows when only one scene exists. - Rejects or downranks invalid, too-short, too-long, or blurry candidates. - Persists `analysis/highlight-candidates.json`. ### Milestone 2: Generate Deterministic Edit Plans Add a component: ```text HighlightEditPlanGenerator ``` Input: ```text HighlightSourceAnalysis List ``` Output per candidate: ```text highlights//edit-plan.json highlights//storyboard.md ``` First version plan rules: - Create one highlight per selected candidate. - Use 3-6 edit decisions per highlight. - For a short candidate, use the candidate range as one segment. - For a longer candidate, split into smaller cuts with simple pacing. - Use deterministic category style: - car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist. - family-like/faces: warm memory style, softer overlays. - food-like/close action: rhythmic sensory style. - generic: clean cinematic montage. - Add optional text overlays only when safe. - Add voiceover text as script metadata, but do not block render when no TTS provider exists. - Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets. Required logs: ```text event=highlight_edit_plan_started project_id=... candidate_id=... event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=... event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=... ``` Tests: - Produces valid `EditPlan` timestamps inside source duration. - Produces stable output for the same analysis. - Creates storyboard markdown. - Does not require external AI. ### Milestone 3: Implement Highlight Renderer Add: ```text HighlightRenderer FfmpegHighlightRenderer ``` Input: ```text projectId highlightId highlights//edit-plan.json analysis/source-analysis.json ``` Output: ```text highlights//final.mp4 highlights//preview.mp4 highlights//render-manifest.json highlights//qa-report.json ``` Renderer requirements for first version: - Trim from the original source video using exact timestamps. - Concatenate selected segments. - Apply simple visual treatment using FFmpeg filters: - scale/pad to configured output size - optional contrast/saturation/sharpen - optional vignette or fade - render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback. - Mix source audio. - Optionally mix music/SFX/voiceover when local assets exist. - Normalize output audio loudness when audio exists. - Produce `preview.mp4` as a lower-resolution copy or direct transcode. - Write render manifest with commands, decisions, output paths, assets, duration, and exit codes. Important: - Missing music, SFX, voiceover, LUT, or font assets must not block a basic render. - Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed. Required logs: ```text event=highlight_render_started project_id=... highlight_id=... event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=... event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=... event=highlight_render_concat_started project_id=... highlight_id=... segments=... event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=... event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=... event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=... event=highlight_render_failed project_id=... highlight_id=... error_type=... message=... ``` Tests: - Builds expected FFmpeg trim commands. - Builds concat and final render commands. - Writes final output path under `highlights//`. - Writes render manifest. - Handles missing optional assets. - Fails on invalid source timestamps. ### Milestone 4: Local Asset Fallbacks Current rendered highlights must work even with no licensed asset library. Add fallback behavior: - If no music asset exists, keep source audio. - If no SFX asset exists, skip SFX cues and record skipped assets in manifest. - If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio. - If no LUT exists, use named FFmpeg color preset. - If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font. Required logs: ```text event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=... event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset ``` Tests: - Render plan with no local assets still produces `final.mp4`. - Manifest records skipped optional assets. - Voiceover script file is still created. ### Milestone 5: Wire Into Scheduler Update `HighlightSourceScheduler.createProject` after analysis: ```text analysis = analyzer.analyze(projectId) candidates = candidateGenerator.generate(projectId, analysis) plans = planGenerator.generate(projectId, analysis, candidates) renderResults = renderer.render(projectId, plans) move source to processed log flow complete ``` Configuration: ```yaml video-clipping: editing: highlight-scheduler: render-enabled: true max-highlights-per-source: 3 highlight-min-duration-seconds: 8 highlight-max-duration-seconds: 35 require-director-approval: false ``` If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering. Required logs: ```text event=highlight_flow_started scan_id=... project_id=... source_file=... event=highlight_flow_analysis_completed scan_id=... project_id=... event=highlight_flow_candidates_completed scan_id=... project_id=... count=... event=highlight_flow_plans_completed scan_id=... project_id=... count=... event=highlight_flow_renders_completed scan_id=... project_id=... count=... event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=... ``` Tests: - Scheduler produces at least one `highlights//final.mp4` from a valid fixture video. - Scheduler leaves no source video in `source` or `working` after success. - Scheduler moves failed source to rejected and logs failure. - Rendering can be disabled for analysis-only mode. ### Milestone 6: Rich End-To-End Logging Add a single correlation ID: ```text flow_id=: ``` Every log in the highlight path should include: - `scan_id` - `project_id` - `highlight_id` when applicable - `flow_id` - `elapsed_ms` for completed steps The final success log must be: ```text event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=... ``` The final failure log must be: ```text event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=... ``` Tests: - Use a log capture test to assert `highlight_flow_completed` is emitted on success. - Use a failure test to assert `highlight_flow_failed` includes the failed step. ### Milestone 7: Tests And Coverage Add focused unit tests: - `HighlightCandidateGeneratorTest` - `HighlightEditPlanGeneratorTest` - `FfmpegHighlightRendererTest` - `HighlightRenderManifestTest` - `HighlightSourceSchedulerRenderFlowTest` Add integration test: ```text HighlightRenderingIntegrationTest ``` Requirements: - Create a tiny FFmpeg fixture video. - Place it in a temp highlight source folder. - Run the scheduler once. - Assert: - `analysis/source-analysis.json` exists. - `analysis/highlight-candidates.json` exists. - `highlights//edit-plan.json` exists. - `highlights//final.mp4` exists. - `highlights//render-manifest.json` exists. - `highlights//qa-report.json` exists. - `highlight_flow_completed` log exists. ### Milestone 8: Runbook Update Update: ```text docs/cinematic-editing-runbook.md docs/production-cinematic-highlight-editing-plan.md ``` Add: - Exact output location for final highlight videos. - How to tell whether the project is analysis-only or rendered. - How to enable/disable automatic rendering. - How to inspect `highlight-candidates.json`. - How to inspect per-highlight `edit-plan.json`. - How to inspect `render-manifest.json` and `qa-report.json`. - Troubleshooting table for empty `highlights/`. ## Implementation Order 1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`. 2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards. 3. Implement `FfmpegHighlightRenderer` with minimal no-asset render. 4. Wire renderer behind `highlight-scheduler.render-enabled`. 5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs. 6. Add integration test proving `final.mp4` is created. 7. Add optional assets, overlays, audio mix, and QA improvements. 8. Update runbooks. ## Definition Of Done - Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects//highlights//final.mp4`. - The source file is moved to `input/highlights/processed`. - The project contains `analysis/highlight-candidates.json`. - Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`. - Logs contain `highlight_flow_completed` with final output paths. - Missing optional assets do not prevent a basic highlight render. - Full Maven tests pass.