From 80eca92d5628292d05aa0f638059ddc718d2f69b Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Sat, 11 Jul 2026 17:26:36 +0200 Subject: [PATCH] Add highlight rendering gap closure plan --- docs/highlight-rendering-gap-closure-plan.md | 417 +++++++++++++++++++ 1 file changed, 417 insertions(+) create mode 100644 docs/highlight-rendering-gap-closure-plan.md diff --git a/docs/highlight-rendering-gap-closure-plan.md b/docs/highlight-rendering-gap-closure-plan.md new file mode 100644 index 0000000..2d6b82e --- /dev/null +++ b/docs/highlight-rendering-gap-closure-plan.md @@ -0,0 +1,417 @@ +# Highlight Rendering Gap Closure Plan + +## Problem + +The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos. + +Observed current output: + +```text +output/highlight-projects// + analysis/ + source-analysis.json + visual-analysis.json + audio-analysis.json + scene-segments.json + frames/*.jpg + contact-sheets/*.jpg + proxies/*.mp4 + source/ + highlights/ +``` + +The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights//final.mp4`. + +## Goal + +When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos: + +```text +output/highlight-projects//highlights//final.mp4 +output/highlight-projects//highlights//preview.mp4 +output/highlight-projects//highlights//edit-plan.json +output/highlight-projects//highlights//render-manifest.json +output/highlight-projects//highlights//qa-report.json +``` + +The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance. + +## Key Design Decision + +Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is. + +Reason: + +- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`. +- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights//`. +- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion. + +Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`. + +## Target Flow + +1. `HighlightSourceScheduler` claims one source video. +2. `HighlightSourceAnalyzer` creates analysis artifacts. +3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis. +4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate. +5. New `HighlightRenderer` renders each plan into `highlights//`. +6. New QA step probes each rendered MP4 and writes `qa-report.json`. +7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths. + +## Milestones + +- [ ] Milestone 1: Persist highlight candidates for highlight-source projects. +- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates. +- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights//`. +- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets. +- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow. +- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log. +- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output. +- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs. + +## Milestone Details + +### Milestone 1: Persist Highlight Candidates + +Add a component: + +```text +HighlightCandidateGenerator +``` + +Input: + +```text +HighlightSourceAnalysis +``` + +Output: + +```text +analysis/highlight-candidates.json +``` + +Candidate rules for the first version: + +- Use `scene-segments.json` as the primary temporal units. +- If only one scene exists, split the source into overlapping 8-20 second windows. +- Score candidates using available fields: +- `visualAnalysis.motionScore` +- `visualAnalysis.compositionScore` +- `visualAnalysis.blurScore` +- `visualAnalysis.exposureScore` +- `audioAnalysis.sections` +- scene duration and position +- Prefer candidates between 8 and 35 seconds. +- Avoid first/last black or silent areas when audio data suggests silence. +- Limit initial output to top 3 candidates. + +Required logs: + +```text +event=highlight_candidates_started project_id=... +event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=... +event=highlight_candidates_completed project_id=... count=... elapsed_ms=... +``` + +Tests: + +- Generates candidates from multiple scene segments. +- Falls back to fixed windows when only one scene exists. +- Rejects or downranks invalid, too-short, too-long, or blurry candidates. +- Persists `analysis/highlight-candidates.json`. + +### Milestone 2: Generate Deterministic Edit Plans + +Add a component: + +```text +HighlightEditPlanGenerator +``` + +Input: + +```text +HighlightSourceAnalysis +List +``` + +Output per candidate: + +```text +highlights//edit-plan.json +highlights//storyboard.md +``` + +First version plan rules: + +- Create one highlight per selected candidate. +- Use 3-6 edit decisions per highlight. +- For a short candidate, use the candidate range as one segment. +- For a longer candidate, split into smaller cuts with simple pacing. +- Use deterministic category style: +- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist. +- family-like/faces: warm memory style, softer overlays. +- food-like/close action: rhythmic sensory style. +- generic: clean cinematic montage. +- Add optional text overlays only when safe. +- Add voiceover text as script metadata, but do not block render when no TTS provider exists. +- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets. + +Required logs: + +```text +event=highlight_edit_plan_started project_id=... candidate_id=... +event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=... +event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=... +``` + +Tests: + +- Produces valid `EditPlan` timestamps inside source duration. +- Produces stable output for the same analysis. +- Creates storyboard markdown. +- Does not require external AI. + +### Milestone 3: Implement Highlight Renderer + +Add: + +```text +HighlightRenderer +FfmpegHighlightRenderer +``` + +Input: + +```text +projectId +highlightId +highlights//edit-plan.json +analysis/source-analysis.json +``` + +Output: + +```text +highlights//final.mp4 +highlights//preview.mp4 +highlights//render-manifest.json +highlights//qa-report.json +``` + +Renderer requirements for first version: + +- Trim from the original source video using exact timestamps. +- Concatenate selected segments. +- Apply simple visual treatment using FFmpeg filters: +- scale/pad to configured output size +- optional contrast/saturation/sharpen +- optional vignette or fade +- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback. +- Mix source audio. +- Optionally mix music/SFX/voiceover when local assets exist. +- Normalize output audio loudness when audio exists. +- Produce `preview.mp4` as a lower-resolution copy or direct transcode. +- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes. + +Important: + +- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render. +- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed. + +Required logs: + +```text +event=highlight_render_started project_id=... highlight_id=... +event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=... +event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=... +event=highlight_render_concat_started project_id=... highlight_id=... segments=... +event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=... +event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=... +event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=... +event=highlight_render_failed project_id=... highlight_id=... error_type=... message=... +``` + +Tests: + +- Builds expected FFmpeg trim commands. +- Builds concat and final render commands. +- Writes final output path under `highlights//`. +- Writes render manifest. +- Handles missing optional assets. +- Fails on invalid source timestamps. + +### Milestone 4: Local Asset Fallbacks + +Current rendered highlights must work even with no licensed asset library. + +Add fallback behavior: + +- If no music asset exists, keep source audio. +- If no SFX asset exists, skip SFX cues and record skipped assets in manifest. +- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio. +- If no LUT exists, use named FFmpeg color preset. +- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font. + +Required logs: + +```text +event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=... +event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset +``` + +Tests: + +- Render plan with no local assets still produces `final.mp4`. +- Manifest records skipped optional assets. +- Voiceover script file is still created. + +### Milestone 5: Wire Into Scheduler + +Update `HighlightSourceScheduler.createProject` after analysis: + +```text +analysis = analyzer.analyze(projectId) +candidates = candidateGenerator.generate(projectId, analysis) +plans = planGenerator.generate(projectId, analysis, candidates) +renderResults = renderer.render(projectId, plans) +move source to processed +log flow complete +``` + +Configuration: + +```yaml +video-clipping: + editing: + highlight-scheduler: + render-enabled: true + max-highlights-per-source: 3 + highlight-min-duration-seconds: 8 + highlight-max-duration-seconds: 35 + require-director-approval: false +``` + +If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering. + +Required logs: + +```text +event=highlight_flow_started scan_id=... project_id=... source_file=... +event=highlight_flow_analysis_completed scan_id=... project_id=... +event=highlight_flow_candidates_completed scan_id=... project_id=... count=... +event=highlight_flow_plans_completed scan_id=... project_id=... count=... +event=highlight_flow_renders_completed scan_id=... project_id=... count=... +event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=... +``` + +Tests: + +- Scheduler produces at least one `highlights//final.mp4` from a valid fixture video. +- Scheduler leaves no source video in `source` or `working` after success. +- Scheduler moves failed source to rejected and logs failure. +- Rendering can be disabled for analysis-only mode. + +### Milestone 6: Rich End-To-End Logging + +Add a single correlation ID: + +```text +flow_id=: +``` + +Every log in the highlight path should include: + +- `scan_id` +- `project_id` +- `highlight_id` when applicable +- `flow_id` +- `elapsed_ms` for completed steps + +The final success log must be: + +```text +event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=... +``` + +The final failure log must be: + +```text +event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=... +``` + +Tests: + +- Use a log capture test to assert `highlight_flow_completed` is emitted on success. +- Use a failure test to assert `highlight_flow_failed` includes the failed step. + +### Milestone 7: Tests And Coverage + +Add focused unit tests: + +- `HighlightCandidateGeneratorTest` +- `HighlightEditPlanGeneratorTest` +- `FfmpegHighlightRendererTest` +- `HighlightRenderManifestTest` +- `HighlightSourceSchedulerRenderFlowTest` + +Add integration test: + +```text +HighlightRenderingIntegrationTest +``` + +Requirements: + +- Create a tiny FFmpeg fixture video. +- Place it in a temp highlight source folder. +- Run the scheduler once. +- Assert: +- `analysis/source-analysis.json` exists. +- `analysis/highlight-candidates.json` exists. +- `highlights//edit-plan.json` exists. +- `highlights//final.mp4` exists. +- `highlights//render-manifest.json` exists. +- `highlights//qa-report.json` exists. +- `highlight_flow_completed` log exists. + +### Milestone 8: Runbook Update + +Update: + +```text +docs/cinematic-editing-runbook.md +docs/production-cinematic-highlight-editing-plan.md +``` + +Add: + +- Exact output location for final highlight videos. +- How to tell whether the project is analysis-only or rendered. +- How to enable/disable automatic rendering. +- How to inspect `highlight-candidates.json`. +- How to inspect per-highlight `edit-plan.json`. +- How to inspect `render-manifest.json` and `qa-report.json`. +- Troubleshooting table for empty `highlights/`. + +## Implementation Order + +1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`. +2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards. +3. Implement `FfmpegHighlightRenderer` with minimal no-asset render. +4. Wire renderer behind `highlight-scheduler.render-enabled`. +5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs. +6. Add integration test proving `final.mp4` is created. +7. Add optional assets, overlays, audio mix, and QA improvements. +8. Update runbooks. + +## Definition Of Done + +- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects//highlights//final.mp4`. +- The source file is moved to `input/highlights/processed`. +- The project contains `analysis/highlight-candidates.json`. +- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`. +- Logs contain `highlight_flow_completed` with final output paths. +- Missing optional assets do not prevent a basic highlight render. +- Full Maven tests pass.