forked from jsl/video_editing_poc
418 lines
14 KiB
Markdown
418 lines
14 KiB
Markdown
# Highlight Rendering Gap Closure Plan
|
|
|
|
## Problem
|
|
|
|
The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.
|
|
|
|
Observed current output:
|
|
|
|
```text
|
|
output/highlight-projects/<project-id>/
|
|
analysis/
|
|
source-analysis.json
|
|
visual-analysis.json
|
|
audio-analysis.json
|
|
scene-segments.json
|
|
frames/*.jpg
|
|
contact-sheets/*.jpg
|
|
proxies/*.mp4
|
|
source/<source-video>
|
|
highlights/
|
|
```
|
|
|
|
The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights/<highlight-id>/final.mp4`.
|
|
|
|
## Goal
|
|
|
|
When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos:
|
|
|
|
```text
|
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
|
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
|
```
|
|
|
|
The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.
|
|
|
|
## Key Design Decision
|
|
|
|
Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is.
|
|
|
|
Reason:
|
|
|
|
- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`.
|
|
- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights/<highlight-id>/`.
|
|
- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.
|
|
|
|
Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`.
|
|
|
|
## Target Flow
|
|
|
|
1. `HighlightSourceScheduler` claims one source video.
|
|
2. `HighlightSourceAnalyzer` creates analysis artifacts.
|
|
3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis.
|
|
4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate.
|
|
5. New `HighlightRenderer` renders each plan into `highlights/<highlight-id>/`.
|
|
6. New QA step probes each rendered MP4 and writes `qa-report.json`.
|
|
7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths.
|
|
|
|
## Milestones
|
|
|
|
- [ ] Milestone 1: Persist highlight candidates for highlight-source projects.
|
|
- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates.
|
|
- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights/<highlight-id>/`.
|
|
- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
|
|
- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
|
|
- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log.
|
|
- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
|
|
- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.
|
|
|
|
## Milestone Details
|
|
|
|
### Milestone 1: Persist Highlight Candidates
|
|
|
|
Add a component:
|
|
|
|
```text
|
|
HighlightCandidateGenerator
|
|
```
|
|
|
|
Input:
|
|
|
|
```text
|
|
HighlightSourceAnalysis
|
|
```
|
|
|
|
Output:
|
|
|
|
```text
|
|
analysis/highlight-candidates.json
|
|
```
|
|
|
|
Candidate rules for the first version:
|
|
|
|
- Use `scene-segments.json` as the primary temporal units.
|
|
- If only one scene exists, split the source into overlapping 8-20 second windows.
|
|
- Score candidates using available fields:
|
|
- `visualAnalysis.motionScore`
|
|
- `visualAnalysis.compositionScore`
|
|
- `visualAnalysis.blurScore`
|
|
- `visualAnalysis.exposureScore`
|
|
- `audioAnalysis.sections`
|
|
- scene duration and position
|
|
- Prefer candidates between 8 and 35 seconds.
|
|
- Avoid first/last black or silent areas when audio data suggests silence.
|
|
- Limit initial output to top 3 candidates.
|
|
|
|
Required logs:
|
|
|
|
```text
|
|
event=highlight_candidates_started project_id=...
|
|
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
|
|
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Generates candidates from multiple scene segments.
|
|
- Falls back to fixed windows when only one scene exists.
|
|
- Rejects or downranks invalid, too-short, too-long, or blurry candidates.
|
|
- Persists `analysis/highlight-candidates.json`.
|
|
|
|
### Milestone 2: Generate Deterministic Edit Plans
|
|
|
|
Add a component:
|
|
|
|
```text
|
|
HighlightEditPlanGenerator
|
|
```
|
|
|
|
Input:
|
|
|
|
```text
|
|
HighlightSourceAnalysis
|
|
List<HighlightCandidate>
|
|
```
|
|
|
|
Output per candidate:
|
|
|
|
```text
|
|
highlights/<highlight-id>/edit-plan.json
|
|
highlights/<highlight-id>/storyboard.md
|
|
```
|
|
|
|
First version plan rules:
|
|
|
|
- Create one highlight per selected candidate.
|
|
- Use 3-6 edit decisions per highlight.
|
|
- For a short candidate, use the candidate range as one segment.
|
|
- For a longer candidate, split into smaller cuts with simple pacing.
|
|
- Use deterministic category style:
|
|
- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
|
|
- family-like/faces: warm memory style, softer overlays.
|
|
- food-like/close action: rhythmic sensory style.
|
|
- generic: clean cinematic montage.
|
|
- Add optional text overlays only when safe.
|
|
- Add voiceover text as script metadata, but do not block render when no TTS provider exists.
|
|
- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.
|
|
|
|
Required logs:
|
|
|
|
```text
|
|
event=highlight_edit_plan_started project_id=... candidate_id=...
|
|
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
|
|
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Produces valid `EditPlan` timestamps inside source duration.
|
|
- Produces stable output for the same analysis.
|
|
- Creates storyboard markdown.
|
|
- Does not require external AI.
|
|
|
|
### Milestone 3: Implement Highlight Renderer
|
|
|
|
Add:
|
|
|
|
```text
|
|
HighlightRenderer
|
|
FfmpegHighlightRenderer
|
|
```
|
|
|
|
Input:
|
|
|
|
```text
|
|
projectId
|
|
highlightId
|
|
highlights/<highlight-id>/edit-plan.json
|
|
analysis/source-analysis.json
|
|
```
|
|
|
|
Output:
|
|
|
|
```text
|
|
highlights/<highlight-id>/final.mp4
|
|
highlights/<highlight-id>/preview.mp4
|
|
highlights/<highlight-id>/render-manifest.json
|
|
highlights/<highlight-id>/qa-report.json
|
|
```
|
|
|
|
Renderer requirements for first version:
|
|
|
|
- Trim from the original source video using exact timestamps.
|
|
- Concatenate selected segments.
|
|
- Apply simple visual treatment using FFmpeg filters:
|
|
- scale/pad to configured output size
|
|
- optional contrast/saturation/sharpen
|
|
- optional vignette or fade
|
|
- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback.
|
|
- Mix source audio.
|
|
- Optionally mix music/SFX/voiceover when local assets exist.
|
|
- Normalize output audio loudness when audio exists.
|
|
- Produce `preview.mp4` as a lower-resolution copy or direct transcode.
|
|
- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.
|
|
|
|
Important:
|
|
|
|
- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
|
|
- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.
|
|
|
|
Required logs:
|
|
|
|
```text
|
|
event=highlight_render_started project_id=... highlight_id=...
|
|
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
|
|
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
|
|
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
|
|
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
|
|
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
|
|
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
|
|
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Builds expected FFmpeg trim commands.
|
|
- Builds concat and final render commands.
|
|
- Writes final output path under `highlights/<highlight-id>/`.
|
|
- Writes render manifest.
|
|
- Handles missing optional assets.
|
|
- Fails on invalid source timestamps.
|
|
|
|
### Milestone 4: Local Asset Fallbacks
|
|
|
|
Current rendered highlights must work even with no licensed asset library.
|
|
|
|
Add fallback behavior:
|
|
|
|
- If no music asset exists, keep source audio.
|
|
- If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
|
|
- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio.
|
|
- If no LUT exists, use named FFmpeg color preset.
|
|
- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.
|
|
|
|
Required logs:
|
|
|
|
```text
|
|
event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
|
|
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Render plan with no local assets still produces `final.mp4`.
|
|
- Manifest records skipped optional assets.
|
|
- Voiceover script file is still created.
|
|
|
|
### Milestone 5: Wire Into Scheduler
|
|
|
|
Update `HighlightSourceScheduler.createProject` after analysis:
|
|
|
|
```text
|
|
analysis = analyzer.analyze(projectId)
|
|
candidates = candidateGenerator.generate(projectId, analysis)
|
|
plans = planGenerator.generate(projectId, analysis, candidates)
|
|
renderResults = renderer.render(projectId, plans)
|
|
move source to processed
|
|
log flow complete
|
|
```
|
|
|
|
Configuration:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
highlight-scheduler:
|
|
render-enabled: true
|
|
max-highlights-per-source: 3
|
|
highlight-min-duration-seconds: 8
|
|
highlight-max-duration-seconds: 35
|
|
require-director-approval: false
|
|
```
|
|
|
|
If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering.
|
|
|
|
Required logs:
|
|
|
|
```text
|
|
event=highlight_flow_started scan_id=... project_id=... source_file=...
|
|
event=highlight_flow_analysis_completed scan_id=... project_id=...
|
|
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
|
|
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
|
|
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
|
|
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Scheduler produces at least one `highlights/<highlight-id>/final.mp4` from a valid fixture video.
|
|
- Scheduler leaves no source video in `source` or `working` after success.
|
|
- Scheduler moves failed source to rejected and logs failure.
|
|
- Rendering can be disabled for analysis-only mode.
|
|
|
|
### Milestone 6: Rich End-To-End Logging
|
|
|
|
Add a single correlation ID:
|
|
|
|
```text
|
|
flow_id=<project-id>:<scan-id>
|
|
```
|
|
|
|
Every log in the highlight path should include:
|
|
|
|
- `scan_id`
|
|
- `project_id`
|
|
- `highlight_id` when applicable
|
|
- `flow_id`
|
|
- `elapsed_ms` for completed steps
|
|
|
|
The final success log must be:
|
|
|
|
```text
|
|
event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...
|
|
```
|
|
|
|
The final failure log must be:
|
|
|
|
```text
|
|
event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...
|
|
```
|
|
|
|
Tests:
|
|
|
|
- Use a log capture test to assert `highlight_flow_completed` is emitted on success.
|
|
- Use a failure test to assert `highlight_flow_failed` includes the failed step.
|
|
|
|
### Milestone 7: Tests And Coverage
|
|
|
|
Add focused unit tests:
|
|
|
|
- `HighlightCandidateGeneratorTest`
|
|
- `HighlightEditPlanGeneratorTest`
|
|
- `FfmpegHighlightRendererTest`
|
|
- `HighlightRenderManifestTest`
|
|
- `HighlightSourceSchedulerRenderFlowTest`
|
|
|
|
Add integration test:
|
|
|
|
```text
|
|
HighlightRenderingIntegrationTest
|
|
```
|
|
|
|
Requirements:
|
|
|
|
- Create a tiny FFmpeg fixture video.
|
|
- Place it in a temp highlight source folder.
|
|
- Run the scheduler once.
|
|
- Assert:
|
|
- `analysis/source-analysis.json` exists.
|
|
- `analysis/highlight-candidates.json` exists.
|
|
- `highlights/<highlight-id>/edit-plan.json` exists.
|
|
- `highlights/<highlight-id>/final.mp4` exists.
|
|
- `highlights/<highlight-id>/render-manifest.json` exists.
|
|
- `highlights/<highlight-id>/qa-report.json` exists.
|
|
- `highlight_flow_completed` log exists.
|
|
|
|
### Milestone 8: Runbook Update
|
|
|
|
Update:
|
|
|
|
```text
|
|
docs/cinematic-editing-runbook.md
|
|
docs/production-cinematic-highlight-editing-plan.md
|
|
```
|
|
|
|
Add:
|
|
|
|
- Exact output location for final highlight videos.
|
|
- How to tell whether the project is analysis-only or rendered.
|
|
- How to enable/disable automatic rendering.
|
|
- How to inspect `highlight-candidates.json`.
|
|
- How to inspect per-highlight `edit-plan.json`.
|
|
- How to inspect `render-manifest.json` and `qa-report.json`.
|
|
- Troubleshooting table for empty `highlights/`.
|
|
|
|
## Implementation Order
|
|
|
|
1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`.
|
|
2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards.
|
|
3. Implement `FfmpegHighlightRenderer` with minimal no-asset render.
|
|
4. Wire renderer behind `highlight-scheduler.render-enabled`.
|
|
5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs.
|
|
6. Add integration test proving `final.mp4` is created.
|
|
7. Add optional assets, overlays, audio mix, and QA improvements.
|
|
8. Update runbooks.
|
|
|
|
## Definition Of Done
|
|
|
|
- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4`.
|
|
- The source file is moved to `input/highlights/processed`.
|
|
- The project contains `analysis/highlight-candidates.json`.
|
|
- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`.
|
|
- Logs contain `highlight_flow_completed` with final output paths.
|
|
- Missing optional assets do not prevent a basic highlight render.
|
|
- Full Maven tests pass.
|