Add highlight rendering gap closure plan
This commit is contained in:
JSLMPR
parent
8195b58552
commit
80eca92d56
|
|
@ -0,0 +1,417 @@
|
|||
# Highlight Rendering Gap Closure Plan
|
||||
|
||||
## Problem
|
||||
|
||||
The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.
|
||||
|
||||
Observed current output:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/
|
||||
analysis/
|
||||
source-analysis.json
|
||||
visual-analysis.json
|
||||
audio-analysis.json
|
||||
scene-segments.json
|
||||
frames/*.jpg
|
||||
contact-sheets/*.jpg
|
||||
proxies/*.mp4
|
||||
source/<source-video>
|
||||
highlights/
|
||||
```
|
||||
|
||||
The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights/<highlight-id>/final.mp4`.
|
||||
|
||||
## Goal
|
||||
|
||||
When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||
```
|
||||
|
||||
The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.
|
||||
|
||||
## Key Design Decision
|
||||
|
||||
Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is.
|
||||
|
||||
Reason:
|
||||
|
||||
- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`.
|
||||
- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights/<highlight-id>/`.
|
||||
- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.
|
||||
|
||||
Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`.
|
||||
|
||||
## Target Flow
|
||||
|
||||
1. `HighlightSourceScheduler` claims one source video.
|
||||
2. `HighlightSourceAnalyzer` creates analysis artifacts.
|
||||
3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis.
|
||||
4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate.
|
||||
5. New `HighlightRenderer` renders each plan into `highlights/<highlight-id>/`.
|
||||
6. New QA step probes each rendered MP4 and writes `qa-report.json`.
|
||||
7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths.
|
||||
|
||||
## Milestones
|
||||
|
||||
- [ ] Milestone 1: Persist highlight candidates for highlight-source projects.
|
||||
- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates.
|
||||
- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights/<highlight-id>/`.
|
||||
- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
|
||||
- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
|
||||
- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log.
|
||||
- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
|
||||
- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.
|
||||
|
||||
## Milestone Details
|
||||
|
||||
### Milestone 1: Persist Highlight Candidates
|
||||
|
||||
Add a component:
|
||||
|
||||
```text
|
||||
HighlightCandidateGenerator
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
HighlightSourceAnalysis
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
analysis/highlight-candidates.json
|
||||
```
|
||||
|
||||
Candidate rules for the first version:
|
||||
|
||||
- Use `scene-segments.json` as the primary temporal units.
|
||||
- If only one scene exists, split the source into overlapping 8-20 second windows.
|
||||
- Score candidates using available fields:
|
||||
- `visualAnalysis.motionScore`
|
||||
- `visualAnalysis.compositionScore`
|
||||
- `visualAnalysis.blurScore`
|
||||
- `visualAnalysis.exposureScore`
|
||||
- `audioAnalysis.sections`
|
||||
- scene duration and position
|
||||
- Prefer candidates between 8 and 35 seconds.
|
||||
- Avoid first/last black or silent areas when audio data suggests silence.
|
||||
- Limit initial output to top 3 candidates.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_candidates_started project_id=...
|
||||
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
|
||||
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Generates candidates from multiple scene segments.
|
||||
- Falls back to fixed windows when only one scene exists.
|
||||
- Rejects or downranks invalid, too-short, too-long, or blurry candidates.
|
||||
- Persists `analysis/highlight-candidates.json`.
|
||||
|
||||
### Milestone 2: Generate Deterministic Edit Plans
|
||||
|
||||
Add a component:
|
||||
|
||||
```text
|
||||
HighlightEditPlanGenerator
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
HighlightSourceAnalysis
|
||||
List<HighlightCandidate>
|
||||
```
|
||||
|
||||
Output per candidate:
|
||||
|
||||
```text
|
||||
highlights/<highlight-id>/edit-plan.json
|
||||
highlights/<highlight-id>/storyboard.md
|
||||
```
|
||||
|
||||
First version plan rules:
|
||||
|
||||
- Create one highlight per selected candidate.
|
||||
- Use 3-6 edit decisions per highlight.
|
||||
- For a short candidate, use the candidate range as one segment.
|
||||
- For a longer candidate, split into smaller cuts with simple pacing.
|
||||
- Use deterministic category style:
|
||||
- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
|
||||
- family-like/faces: warm memory style, softer overlays.
|
||||
- food-like/close action: rhythmic sensory style.
|
||||
- generic: clean cinematic montage.
|
||||
- Add optional text overlays only when safe.
|
||||
- Add voiceover text as script metadata, but do not block render when no TTS provider exists.
|
||||
- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_edit_plan_started project_id=... candidate_id=...
|
||||
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
|
||||
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Produces valid `EditPlan` timestamps inside source duration.
|
||||
- Produces stable output for the same analysis.
|
||||
- Creates storyboard markdown.
|
||||
- Does not require external AI.
|
||||
|
||||
### Milestone 3: Implement Highlight Renderer
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
HighlightRenderer
|
||||
FfmpegHighlightRenderer
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
projectId
|
||||
highlightId
|
||||
highlights/<highlight-id>/edit-plan.json
|
||||
analysis/source-analysis.json
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
highlights/<highlight-id>/final.mp4
|
||||
highlights/<highlight-id>/preview.mp4
|
||||
highlights/<highlight-id>/render-manifest.json
|
||||
highlights/<highlight-id>/qa-report.json
|
||||
```
|
||||
|
||||
Renderer requirements for first version:
|
||||
|
||||
- Trim from the original source video using exact timestamps.
|
||||
- Concatenate selected segments.
|
||||
- Apply simple visual treatment using FFmpeg filters:
|
||||
- scale/pad to configured output size
|
||||
- optional contrast/saturation/sharpen
|
||||
- optional vignette or fade
|
||||
- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback.
|
||||
- Mix source audio.
|
||||
- Optionally mix music/SFX/voiceover when local assets exist.
|
||||
- Normalize output audio loudness when audio exists.
|
||||
- Produce `preview.mp4` as a lower-resolution copy or direct transcode.
|
||||
- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.
|
||||
|
||||
Important:
|
||||
|
||||
- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
|
||||
- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_render_started project_id=... highlight_id=...
|
||||
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
|
||||
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
|
||||
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
|
||||
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
|
||||
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
|
||||
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
|
||||
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Builds expected FFmpeg trim commands.
|
||||
- Builds concat and final render commands.
|
||||
- Writes final output path under `highlights/<highlight-id>/`.
|
||||
- Writes render manifest.
|
||||
- Handles missing optional assets.
|
||||
- Fails on invalid source timestamps.
|
||||
|
||||
### Milestone 4: Local Asset Fallbacks
|
||||
|
||||
Current rendered highlights must work even with no licensed asset library.
|
||||
|
||||
Add fallback behavior:
|
||||
|
||||
- If no music asset exists, keep source audio.
|
||||
- If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
|
||||
- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio.
|
||||
- If no LUT exists, use named FFmpeg color preset.
|
||||
- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
|
||||
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Render plan with no local assets still produces `final.mp4`.
|
||||
- Manifest records skipped optional assets.
|
||||
- Voiceover script file is still created.
|
||||
|
||||
### Milestone 5: Wire Into Scheduler
|
||||
|
||||
Update `HighlightSourceScheduler.createProject` after analysis:
|
||||
|
||||
```text
|
||||
analysis = analyzer.analyze(projectId)
|
||||
candidates = candidateGenerator.generate(projectId, analysis)
|
||||
plans = planGenerator.generate(projectId, analysis, candidates)
|
||||
renderResults = renderer.render(projectId, plans)
|
||||
move source to processed
|
||||
log flow complete
|
||||
```
|
||||
|
||||
Configuration:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
editing:
|
||||
highlight-scheduler:
|
||||
render-enabled: true
|
||||
max-highlights-per-source: 3
|
||||
highlight-min-duration-seconds: 8
|
||||
highlight-max-duration-seconds: 35
|
||||
require-director-approval: false
|
||||
```
|
||||
|
||||
If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_flow_started scan_id=... project_id=... source_file=...
|
||||
event=highlight_flow_analysis_completed scan_id=... project_id=...
|
||||
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Scheduler produces at least one `highlights/<highlight-id>/final.mp4` from a valid fixture video.
|
||||
- Scheduler leaves no source video in `source` or `working` after success.
|
||||
- Scheduler moves failed source to rejected and logs failure.
|
||||
- Rendering can be disabled for analysis-only mode.
|
||||
|
||||
### Milestone 6: Rich End-To-End Logging
|
||||
|
||||
Add a single correlation ID:
|
||||
|
||||
```text
|
||||
flow_id=<project-id>:<scan-id>
|
||||
```
|
||||
|
||||
Every log in the highlight path should include:
|
||||
|
||||
- `scan_id`
|
||||
- `project_id`
|
||||
- `highlight_id` when applicable
|
||||
- `flow_id`
|
||||
- `elapsed_ms` for completed steps
|
||||
|
||||
The final success log must be:
|
||||
|
||||
```text
|
||||
event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...
|
||||
```
|
||||
|
||||
The final failure log must be:
|
||||
|
||||
```text
|
||||
event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Use a log capture test to assert `highlight_flow_completed` is emitted on success.
|
||||
- Use a failure test to assert `highlight_flow_failed` includes the failed step.
|
||||
|
||||
### Milestone 7: Tests And Coverage
|
||||
|
||||
Add focused unit tests:
|
||||
|
||||
- `HighlightCandidateGeneratorTest`
|
||||
- `HighlightEditPlanGeneratorTest`
|
||||
- `FfmpegHighlightRendererTest`
|
||||
- `HighlightRenderManifestTest`
|
||||
- `HighlightSourceSchedulerRenderFlowTest`
|
||||
|
||||
Add integration test:
|
||||
|
||||
```text
|
||||
HighlightRenderingIntegrationTest
|
||||
```
|
||||
|
||||
Requirements:
|
||||
|
||||
- Create a tiny FFmpeg fixture video.
|
||||
- Place it in a temp highlight source folder.
|
||||
- Run the scheduler once.
|
||||
- Assert:
|
||||
- `analysis/source-analysis.json` exists.
|
||||
- `analysis/highlight-candidates.json` exists.
|
||||
- `highlights/<highlight-id>/edit-plan.json` exists.
|
||||
- `highlights/<highlight-id>/final.mp4` exists.
|
||||
- `highlights/<highlight-id>/render-manifest.json` exists.
|
||||
- `highlights/<highlight-id>/qa-report.json` exists.
|
||||
- `highlight_flow_completed` log exists.
|
||||
|
||||
### Milestone 8: Runbook Update
|
||||
|
||||
Update:
|
||||
|
||||
```text
|
||||
docs/cinematic-editing-runbook.md
|
||||
docs/production-cinematic-highlight-editing-plan.md
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
- Exact output location for final highlight videos.
|
||||
- How to tell whether the project is analysis-only or rendered.
|
||||
- How to enable/disable automatic rendering.
|
||||
- How to inspect `highlight-candidates.json`.
|
||||
- How to inspect per-highlight `edit-plan.json`.
|
||||
- How to inspect `render-manifest.json` and `qa-report.json`.
|
||||
- Troubleshooting table for empty `highlights/`.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`.
|
||||
2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards.
|
||||
3. Implement `FfmpegHighlightRenderer` with minimal no-asset render.
|
||||
4. Wire renderer behind `highlight-scheduler.render-enabled`.
|
||||
5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs.
|
||||
6. Add integration test proving `final.mp4` is created.
|
||||
7. Add optional assets, overlays, audio mix, and QA improvements.
|
||||
8. Update runbooks.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4`.
|
||||
- The source file is moved to `input/highlights/processed`.
|
||||
- The project contains `analysis/highlight-candidates.json`.
|
||||
- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`.
|
||||
- Logs contain `highlight_flow_completed` with final output paths.
|
||||
- Missing optional assets do not prevent a basic highlight render.
|
||||
- Full Maven tests pass.
|
||||
Loading…
Reference in New Issue