Compare commits
10 Commits
ed337c1268
...
618ba9af49
| Author | SHA1 | Date |
|---|---|---|
|
|
618ba9af49 | |
|
|
adc979eea8 | |
|
|
80eca92d56 | |
|
|
8195b58552 | |
|
|
e2f49e2e5a | |
|
|
9dc720b0cc | |
|
|
82e557905e | |
|
|
eea619a2d3 | |
|
|
fc17578549 | |
|
|
5d5f8022f2 |
|
|
@ -36,3 +36,9 @@ build/
|
||||||
|
|
||||||
### Mac OS ###
|
### Mac OS ###
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|
||||||
|
### Local CV runtime ###
|
||||||
|
.venv-local-cv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
yolov*.pt
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,7 @@ video-clipping:
|
||||||
project-directory: ./output/edit-projects
|
project-directory: ./output/edit-projects
|
||||||
thumbnail-count-per-clip: 5
|
thumbnail-count-per-clip: 5
|
||||||
proxy-enabled: true
|
proxy-enabled: true
|
||||||
target-duration-seconds: 60
|
target-duration-seconds: 600
|
||||||
output-width: 1920
|
output-width: 1920
|
||||||
output-height: 1080
|
output-height: 1080
|
||||||
output-frame-rate: 30
|
output-frame-rate: 30
|
||||||
|
|
@ -186,6 +186,13 @@ The AI director should inspect:
|
||||||
- `thumbnails/*.jpg`
|
- `thumbnails/*.jpg`
|
||||||
- `proxies/*.mp4` if more visual context is needed
|
- `proxies/*.mp4` if more visual context is needed
|
||||||
|
|
||||||
|
After the plan is imported, the service also runs an asset materialization stage:
|
||||||
|
|
||||||
|
- It reuses matching generated music, SFX, and voiceover assets from the shared cache folders.
|
||||||
|
- It copies reusable assets into the project-local `audio/` folder so the renderer can use them.
|
||||||
|
- It writes reusable asset requests under `assets/requests/` when a needed asset is missing.
|
||||||
|
- It waits to render if required SFX assets are still missing.
|
||||||
|
|
||||||
The AI director must write this file:
|
The AI director must write this file:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|
@ -239,6 +246,14 @@ Rules the AI director must follow:
|
||||||
|
|
||||||
The renderer can mix optional WAV files into the final output.
|
The renderer can mix optional WAV files into the final output.
|
||||||
|
|
||||||
|
Shared cache locations:
|
||||||
|
|
||||||
|
- Music: `input/highlights/assets/music/`
|
||||||
|
- SFX: `input/highlights/assets/sfx/`
|
||||||
|
- Voiceover cache: `output/highlight-projects/_voiceover-cache/`
|
||||||
|
|
||||||
|
If the service or an AI worker generates an asset there, later projects can reuse it automatically.
|
||||||
|
|
||||||
Place optional assets here before rendering:
|
Place optional assets here before rendering:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|
@ -308,6 +323,14 @@ After rendering completes, the final output is:
|
||||||
output/edit-projects/porsche-session-001/final.mp4
|
output/edit-projects/porsche-session-001/final.mp4
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The renderer also publishes each final rendered segment clip that was concatenated into the full video:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0001.mp4
|
||||||
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0002.mp4
|
||||||
|
output/edit-projects/porsche-session-001/rendered-clips/clip_0003.mp4
|
||||||
|
```
|
||||||
|
|
||||||
The render manifest is:
|
The render manifest is:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|
@ -320,11 +343,12 @@ The QA report is:
|
||||||
output/edit-projects/porsche-session-001/qa-report.json
|
output/edit-projects/porsche-session-001/qa-report.json
|
||||||
```
|
```
|
||||||
|
|
||||||
The manifest is useful for debugging because it records the selected clips, output path, duration, and FFmpeg command summaries.
|
The manifest is useful for debugging because it records the selected clips, rendered segment clip paths, output path, duration, and FFmpeg command summaries.
|
||||||
|
|
||||||
The QA report is useful for acceptance because it records:
|
The QA report is useful for acceptance because it records:
|
||||||
|
|
||||||
- final output existence
|
- final output existence
|
||||||
|
- rendered segment clip existence
|
||||||
- duration consistency with the edit timeline
|
- duration consistency with the edit timeline
|
||||||
- required asset resolution
|
- required asset resolution
|
||||||
- text overlay timeline and placement safety
|
- text overlay timeline and placement safety
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,417 @@
|
||||||
|
# Highlight Rendering Gap Closure Plan
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.
|
||||||
|
|
||||||
|
Observed current output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/
|
||||||
|
analysis/
|
||||||
|
source-analysis.json
|
||||||
|
visual-analysis.json
|
||||||
|
audio-analysis.json
|
||||||
|
scene-segments.json
|
||||||
|
frames/*.jpg
|
||||||
|
contact-sheets/*.jpg
|
||||||
|
proxies/*.mp4
|
||||||
|
source/<source-video>
|
||||||
|
highlights/
|
||||||
|
```
|
||||||
|
|
||||||
|
The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights/<highlight-id>/final.mp4`.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.
|
||||||
|
|
||||||
|
## Key Design Decision
|
||||||
|
|
||||||
|
Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is.
|
||||||
|
|
||||||
|
Reason:
|
||||||
|
|
||||||
|
- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`.
|
||||||
|
- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights/<highlight-id>/`.
|
||||||
|
- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.
|
||||||
|
|
||||||
|
Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`.
|
||||||
|
|
||||||
|
## Target Flow
|
||||||
|
|
||||||
|
1. `HighlightSourceScheduler` claims one source video.
|
||||||
|
2. `HighlightSourceAnalyzer` creates analysis artifacts.
|
||||||
|
3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis.
|
||||||
|
4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate.
|
||||||
|
5. New `HighlightRenderer` renders each plan into `highlights/<highlight-id>/`.
|
||||||
|
6. New QA step probes each rendered MP4 and writes `qa-report.json`.
|
||||||
|
7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths.
|
||||||
|
|
||||||
|
## Milestones
|
||||||
|
|
||||||
|
- [ ] Milestone 1: Persist highlight candidates for highlight-source projects.
|
||||||
|
- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates.
|
||||||
|
- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights/<highlight-id>/`.
|
||||||
|
- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
|
||||||
|
- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
|
||||||
|
- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log.
|
||||||
|
- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
|
||||||
|
- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.
|
||||||
|
|
||||||
|
## Milestone Details
|
||||||
|
|
||||||
|
### Milestone 1: Persist Highlight Candidates
|
||||||
|
|
||||||
|
Add a component:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightCandidateGenerator
|
||||||
|
```
|
||||||
|
|
||||||
|
Input:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightSourceAnalysis
|
||||||
|
```
|
||||||
|
|
||||||
|
Output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
analysis/highlight-candidates.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Candidate rules for the first version:
|
||||||
|
|
||||||
|
- Use `scene-segments.json` as the primary temporal units.
|
||||||
|
- If only one scene exists, split the source into overlapping 8-20 second windows.
|
||||||
|
- Score candidates using available fields:
|
||||||
|
- `visualAnalysis.motionScore`
|
||||||
|
- `visualAnalysis.compositionScore`
|
||||||
|
- `visualAnalysis.blurScore`
|
||||||
|
- `visualAnalysis.exposureScore`
|
||||||
|
- `audioAnalysis.sections`
|
||||||
|
- scene duration and position
|
||||||
|
- Prefer candidates between 8 and 35 seconds.
|
||||||
|
- Avoid first/last black or silent areas when audio data suggests silence.
|
||||||
|
- Limit initial output to top 3 candidates.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_candidates_started project_id=...
|
||||||
|
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
|
||||||
|
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Generates candidates from multiple scene segments.
|
||||||
|
- Falls back to fixed windows when only one scene exists.
|
||||||
|
- Rejects or downranks invalid, too-short, too-long, or blurry candidates.
|
||||||
|
- Persists `analysis/highlight-candidates.json`.
|
||||||
|
|
||||||
|
### Milestone 2: Generate Deterministic Edit Plans
|
||||||
|
|
||||||
|
Add a component:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightEditPlanGenerator
|
||||||
|
```
|
||||||
|
|
||||||
|
Input:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightSourceAnalysis
|
||||||
|
List<HighlightCandidate>
|
||||||
|
```
|
||||||
|
|
||||||
|
Output per candidate:
|
||||||
|
|
||||||
|
```text
|
||||||
|
highlights/<highlight-id>/edit-plan.json
|
||||||
|
highlights/<highlight-id>/storyboard.md
|
||||||
|
```
|
||||||
|
|
||||||
|
First version plan rules:
|
||||||
|
|
||||||
|
- Create one highlight per selected candidate.
|
||||||
|
- Use 3-6 edit decisions per highlight.
|
||||||
|
- For a short candidate, use the candidate range as one segment.
|
||||||
|
- For a longer candidate, split into smaller cuts with simple pacing.
|
||||||
|
- Use deterministic category style:
|
||||||
|
- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
|
||||||
|
- family-like/faces: warm memory style, softer overlays.
|
||||||
|
- food-like/close action: rhythmic sensory style.
|
||||||
|
- generic: clean cinematic montage.
|
||||||
|
- Add optional text overlays only when safe.
|
||||||
|
- Add voiceover text as script metadata, but do not block render when no TTS provider exists.
|
||||||
|
- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_edit_plan_started project_id=... candidate_id=...
|
||||||
|
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
|
||||||
|
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Produces valid `EditPlan` timestamps inside source duration.
|
||||||
|
- Produces stable output for the same analysis.
|
||||||
|
- Creates storyboard markdown.
|
||||||
|
- Does not require external AI.
|
||||||
|
|
||||||
|
### Milestone 3: Implement Highlight Renderer
|
||||||
|
|
||||||
|
Add:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightRenderer
|
||||||
|
FfmpegHighlightRenderer
|
||||||
|
```
|
||||||
|
|
||||||
|
Input:
|
||||||
|
|
||||||
|
```text
|
||||||
|
projectId
|
||||||
|
highlightId
|
||||||
|
highlights/<highlight-id>/edit-plan.json
|
||||||
|
analysis/source-analysis.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
highlights/<highlight-id>/final.mp4
|
||||||
|
highlights/<highlight-id>/preview.mp4
|
||||||
|
highlights/<highlight-id>/render-manifest.json
|
||||||
|
highlights/<highlight-id>/qa-report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Renderer requirements for first version:
|
||||||
|
|
||||||
|
- Trim from the original source video using exact timestamps.
|
||||||
|
- Concatenate selected segments.
|
||||||
|
- Apply simple visual treatment using FFmpeg filters:
|
||||||
|
- scale/pad to configured output size
|
||||||
|
- optional contrast/saturation/sharpen
|
||||||
|
- optional vignette or fade
|
||||||
|
- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback.
|
||||||
|
- Mix source audio.
|
||||||
|
- Optionally mix music/SFX/voiceover when local assets exist.
|
||||||
|
- Normalize output audio loudness when audio exists.
|
||||||
|
- Produce `preview.mp4` as a lower-resolution copy or direct transcode.
|
||||||
|
- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.
|
||||||
|
|
||||||
|
Important:
|
||||||
|
|
||||||
|
- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
|
||||||
|
- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_render_started project_id=... highlight_id=...
|
||||||
|
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
|
||||||
|
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
|
||||||
|
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
|
||||||
|
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
|
||||||
|
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
|
||||||
|
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
|
||||||
|
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Builds expected FFmpeg trim commands.
|
||||||
|
- Builds concat and final render commands.
|
||||||
|
- Writes final output path under `highlights/<highlight-id>/`.
|
||||||
|
- Writes render manifest.
|
||||||
|
- Handles missing optional assets.
|
||||||
|
- Fails on invalid source timestamps.
|
||||||
|
|
||||||
|
### Milestone 4: Local Asset Fallbacks
|
||||||
|
|
||||||
|
Current rendered highlights must work even with no licensed asset library.
|
||||||
|
|
||||||
|
Add fallback behavior:
|
||||||
|
|
||||||
|
- If no music asset exists, keep source audio.
|
||||||
|
- If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
|
||||||
|
- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio.
|
||||||
|
- If no LUT exists, use named FFmpeg color preset.
|
||||||
|
- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
|
||||||
|
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Render plan with no local assets still produces `final.mp4`.
|
||||||
|
- Manifest records skipped optional assets.
|
||||||
|
- Voiceover script file is still created.
|
||||||
|
|
||||||
|
### Milestone 5: Wire Into Scheduler
|
||||||
|
|
||||||
|
Update `HighlightSourceScheduler.createProject` after analysis:
|
||||||
|
|
||||||
|
```text
|
||||||
|
analysis = analyzer.analyze(projectId)
|
||||||
|
candidates = candidateGenerator.generate(projectId, analysis)
|
||||||
|
plans = planGenerator.generate(projectId, analysis, candidates)
|
||||||
|
renderResults = renderer.render(projectId, plans)
|
||||||
|
move source to processed
|
||||||
|
log flow complete
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
editing:
|
||||||
|
highlight-scheduler:
|
||||||
|
render-enabled: true
|
||||||
|
max-highlights-per-source: 3
|
||||||
|
highlight-min-duration-seconds: 8
|
||||||
|
highlight-max-duration-seconds: 35
|
||||||
|
require-director-approval: false
|
||||||
|
```
|
||||||
|
|
||||||
|
If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering.
|
||||||
|
|
||||||
|
Required logs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_flow_started scan_id=... project_id=... source_file=...
|
||||||
|
event=highlight_flow_analysis_completed scan_id=... project_id=...
|
||||||
|
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
|
||||||
|
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
|
||||||
|
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
|
||||||
|
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Scheduler produces at least one `highlights/<highlight-id>/final.mp4` from a valid fixture video.
|
||||||
|
- Scheduler leaves no source video in `source` or `working` after success.
|
||||||
|
- Scheduler moves failed source to rejected and logs failure.
|
||||||
|
- Rendering can be disabled for analysis-only mode.
|
||||||
|
|
||||||
|
### Milestone 6: Rich End-To-End Logging
|
||||||
|
|
||||||
|
Add a single correlation ID:
|
||||||
|
|
||||||
|
```text
|
||||||
|
flow_id=<project-id>:<scan-id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Every log in the highlight path should include:
|
||||||
|
|
||||||
|
- `scan_id`
|
||||||
|
- `project_id`
|
||||||
|
- `highlight_id` when applicable
|
||||||
|
- `flow_id`
|
||||||
|
- `elapsed_ms` for completed steps
|
||||||
|
|
||||||
|
The final success log must be:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...
|
||||||
|
```
|
||||||
|
|
||||||
|
The final failure log must be:
|
||||||
|
|
||||||
|
```text
|
||||||
|
event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Use a log capture test to assert `highlight_flow_completed` is emitted on success.
|
||||||
|
- Use a failure test to assert `highlight_flow_failed` includes the failed step.
|
||||||
|
|
||||||
|
### Milestone 7: Tests And Coverage
|
||||||
|
|
||||||
|
Add focused unit tests:
|
||||||
|
|
||||||
|
- `HighlightCandidateGeneratorTest`
|
||||||
|
- `HighlightEditPlanGeneratorTest`
|
||||||
|
- `FfmpegHighlightRendererTest`
|
||||||
|
- `HighlightRenderManifestTest`
|
||||||
|
- `HighlightSourceSchedulerRenderFlowTest`
|
||||||
|
|
||||||
|
Add integration test:
|
||||||
|
|
||||||
|
```text
|
||||||
|
HighlightRenderingIntegrationTest
|
||||||
|
```
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
|
||||||
|
- Create a tiny FFmpeg fixture video.
|
||||||
|
- Place it in a temp highlight source folder.
|
||||||
|
- Run the scheduler once.
|
||||||
|
- Assert:
|
||||||
|
- `analysis/source-analysis.json` exists.
|
||||||
|
- `analysis/highlight-candidates.json` exists.
|
||||||
|
- `highlights/<highlight-id>/edit-plan.json` exists.
|
||||||
|
- `highlights/<highlight-id>/final.mp4` exists.
|
||||||
|
- `highlights/<highlight-id>/render-manifest.json` exists.
|
||||||
|
- `highlights/<highlight-id>/qa-report.json` exists.
|
||||||
|
- `highlight_flow_completed` log exists.
|
||||||
|
|
||||||
|
### Milestone 8: Runbook Update
|
||||||
|
|
||||||
|
Update:
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/cinematic-editing-runbook.md
|
||||||
|
docs/production-cinematic-highlight-editing-plan.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Add:
|
||||||
|
|
||||||
|
- Exact output location for final highlight videos.
|
||||||
|
- How to tell whether the project is analysis-only or rendered.
|
||||||
|
- How to enable/disable automatic rendering.
|
||||||
|
- How to inspect `highlight-candidates.json`.
|
||||||
|
- How to inspect per-highlight `edit-plan.json`.
|
||||||
|
- How to inspect `render-manifest.json` and `qa-report.json`.
|
||||||
|
- Troubleshooting table for empty `highlights/`.
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`.
|
||||||
|
2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards.
|
||||||
|
3. Implement `FfmpegHighlightRenderer` with minimal no-asset render.
|
||||||
|
4. Wire renderer behind `highlight-scheduler.render-enabled`.
|
||||||
|
5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs.
|
||||||
|
6. Add integration test proving `final.mp4` is created.
|
||||||
|
7. Add optional assets, overlays, audio mix, and QA improvements.
|
||||||
|
8. Update runbooks.
|
||||||
|
|
||||||
|
## Definition Of Done
|
||||||
|
|
||||||
|
- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4`.
|
||||||
|
- The source file is moved to `input/highlights/processed`.
|
||||||
|
- The project contains `analysis/highlight-candidates.json`.
|
||||||
|
- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`.
|
||||||
|
- Logs contain `highlight_flow_completed` with final output paths.
|
||||||
|
- Missing optional assets do not prevent a basic highlight render.
|
||||||
|
- Full Maven tests pass.
|
||||||
|
|
@ -0,0 +1,176 @@
|
||||||
|
# Local CV Visual Analysis Guide
|
||||||
|
|
||||||
|
This service can use a local computer-vision worker for highlight visual analysis.
|
||||||
|
|
||||||
|
## Default Behavior
|
||||||
|
|
||||||
|
The packaged `application.yml` defaults to `local-cv` with heuristic fallback enabled:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
editing:
|
||||||
|
visual-analysis:
|
||||||
|
provider: local-cv
|
||||||
|
fallback-to-heuristic: true
|
||||||
|
```
|
||||||
|
|
||||||
|
This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.
|
||||||
|
|
||||||
|
## Enable A Local CV Worker
|
||||||
|
|
||||||
|
Run a local worker that accepts JSON over HTTP, then start this Spring service with:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
video-clipping:
|
||||||
|
editing:
|
||||||
|
visual-analysis:
|
||||||
|
provider: local-cv
|
||||||
|
endpoint: http://127.0.0.1:8091/v1/analyze-visuals
|
||||||
|
timeout-ms: 30000
|
||||||
|
fallback-to-heuristic: true
|
||||||
|
local-cv-worker:
|
||||||
|
auto-start: true
|
||||||
|
script: ./tools/run_local_cv_worker.sh
|
||||||
|
startup-wait-ms: 120000
|
||||||
|
health-path: /health
|
||||||
|
health-check-interval-ms: 1000
|
||||||
|
```
|
||||||
|
|
||||||
|
Equivalent environment variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=120000
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH=/health
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS=1000
|
||||||
|
```
|
||||||
|
|
||||||
|
This repository includes an optional starter worker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tools/run_local_cv_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
|
||||||
|
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.
|
||||||
|
|
||||||
|
Default model behavior:
|
||||||
|
|
||||||
|
- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
|
||||||
|
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
|
||||||
|
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
|
||||||
|
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.
|
||||||
|
|
||||||
|
Health check:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://127.0.0.1:8091/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Preload Dependencies And Model
|
||||||
|
|
||||||
|
Do this once before starting the Spring service if you do not want dependency installation or YOLO model download to happen during application startup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
LOCAL_CV_PRELOAD_ONLY=true tools/run_local_cv_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates `.venv-local-cv`, installs `tools/local_cv_requirements.txt`, and downloads/caches the configured YOLO model. After that, Spring-managed startup should be much faster.
|
||||||
|
|
||||||
|
Do not commit `.venv-local-cv`, Python wheels, Torch binaries, or YOLO weights into source control. They are large, platform-specific runtime artifacts. For production, prefer a prebuilt container image or a VM/bootstrap step that runs the preload command during deployment.
|
||||||
|
|
||||||
|
## Spring-Managed Worker Startup
|
||||||
|
|
||||||
|
The Spring application can start the local CV worker for local runs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
|
||||||
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
|
||||||
|
mvn spring-boot:run
|
||||||
|
```
|
||||||
|
|
||||||
|
When `auto-start` is enabled, Spring starts `tools/run_local_cv_worker.sh`, derives `LOCAL_CV_HOST` and `LOCAL_CV_PORT` from the configured visual-analysis endpoint, polls the configured health endpoint until it returns HTTP 2xx, streams worker output into application logs, and stops the worker process during application shutdown.
|
||||||
|
|
||||||
|
Keep `auto-start` disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates `.venv-local-cv`, installs Python dependencies, and may download/cache the configured YOLO model. Use the preload command or a prebuilt image to avoid that startup cost.
|
||||||
|
|
||||||
|
## Worker Request Contract
|
||||||
|
|
||||||
|
The Spring service sends:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source": {
|
||||||
|
"clipId": "porsche-drive",
|
||||||
|
"sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
|
||||||
|
"durationSeconds": 42.0,
|
||||||
|
"videoCodec": "h264",
|
||||||
|
"audioCodec": "aac",
|
||||||
|
"width": 1920,
|
||||||
|
"height": 1080,
|
||||||
|
"frameRate": 30.0
|
||||||
|
},
|
||||||
|
"thumbnails": [
|
||||||
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
||||||
|
],
|
||||||
|
"shotSegments": [
|
||||||
|
{
|
||||||
|
"shotId": "shot_0001",
|
||||||
|
"startSeconds": 0.0,
|
||||||
|
"endSeconds": 8.0,
|
||||||
|
"durationSeconds": 8.0,
|
||||||
|
"representativeTimestampSeconds": 4.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Worker Response Contract
|
||||||
|
|
||||||
|
The worker must return:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"clipId": "porsche-drive",
|
||||||
|
"blurScore": 0.81,
|
||||||
|
"exposureScore": 0.67,
|
||||||
|
"motionScore": 0.73,
|
||||||
|
"compositionScore": 0.79,
|
||||||
|
"facePresence": "faces_detected",
|
||||||
|
"objectLabels": [
|
||||||
|
{
|
||||||
|
"label": "car",
|
||||||
|
"confidence": 0.91,
|
||||||
|
"source": "yolo"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "luxury sports car",
|
||||||
|
"confidence": 0.84,
|
||||||
|
"source": "clip"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"representativeThumbnails": [
|
||||||
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
||||||
|
],
|
||||||
|
"analysisMethod": "local_cv_yolo_clip_mediapipe"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Scores must be normalized from `0.0` to `1.0`.
|
||||||
|
|
||||||
|
## Recommended Local Model Split
|
||||||
|
|
||||||
|
- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
|
||||||
|
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
|
||||||
|
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
|
||||||
|
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.
|
||||||
|
|
||||||
|
## Runtime Behavior
|
||||||
|
|
||||||
|
When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.
|
||||||
|
|
||||||
|
When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.
|
||||||
|
|
@ -0,0 +1,310 @@
|
||||||
|
# Production Cinematic AI Director Prompt
|
||||||
|
|
||||||
|
Use this prompt with a filesystem-capable AI agent such as Codex or Claude when a highlight project has already been analyzed by the service.
|
||||||
|
|
||||||
|
Expected project folder:
|
||||||
|
|
||||||
|
```text
|
||||||
|
output/highlight-projects/<project-id>/
|
||||||
|
```
|
||||||
|
|
||||||
|
The AI director must inspect the project artifacts, choose all worthwhile highlight moments, and write a strict renderable edit plan. The goal is a platform-ready cinematic highlight video with strong pacing, fitting sound design, fitting voiceover, tasteful text overlays, and visual treatments that match the actual content.
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
```text
|
||||||
|
You are a senior commercial video editor, creative director, sound designer, and post-production supervisor.
|
||||||
|
|
||||||
|
Your assignment is to create a production-ready cinematic highlight edit plan from the analyzed project folder. The final rendered video must feel strong enough for a public video streaming platform: clear hook, strong visual selection, coherent micro-story, professional pacing, fitting sound design, tasteful overlays, and a voiceover that matches the actual footage. Do not promise revenue or make unsupported claims; optimize for viewer retention and professional presentation.
|
||||||
|
|
||||||
|
PROJECT FOLDER
|
||||||
|
- Work only inside this folder: output/highlight-projects/<project-id>/
|
||||||
|
- Read all available project files before deciding:
|
||||||
|
- project.json
|
||||||
|
- manifest.json
|
||||||
|
- analysis/source-analysis.json
|
||||||
|
- analysis/ffprobe.json
|
||||||
|
- analysis/scene-segments.json
|
||||||
|
- analysis/audio-analysis.json
|
||||||
|
- analysis/visual-analysis.json
|
||||||
|
- analysis/contact-sheets/*
|
||||||
|
- analysis/frames/*
|
||||||
|
- analysis/proxies/* when motion needs closer inspection
|
||||||
|
- assets/music/*, assets/sfx/*, assets/voiceover/*, assets/overlays/* if present
|
||||||
|
|
||||||
|
OUTPUT FILE
|
||||||
|
- Write the final edit plan JSON to:
|
||||||
|
director/edit-plan.json
|
||||||
|
- Return JSON only in that file.
|
||||||
|
- Do not wrap JSON in Markdown fences.
|
||||||
|
- Do not render video.
|
||||||
|
- Do not modify source video, analysis files, thumbnails, contact sheets, or proxies.
|
||||||
|
|
||||||
|
PRIMARY GOAL
|
||||||
|
Create one cinematic highlight video that is specific to the provided footage and content type. The highlight length must be content-driven, not fixed. Use all strong highlights from the source, but do not pad with weak or repetitive footage just to reach a duration.
|
||||||
|
|
||||||
|
The edit must have:
|
||||||
|
- A 1-3 second opening hook that immediately shows the most visually interesting or emotionally compelling moment.
|
||||||
|
- A clear middle section with rising energy, variety, and no repetitive shot choices.
|
||||||
|
- A satisfying final hero/payoff shot.
|
||||||
|
- A final duration that fits the source material and keeps audience retention high.
|
||||||
|
- Beat-aware pacing and deliberate shot lengths.
|
||||||
|
- Specific visual treatment notes that the renderer can execute: cinematic grade, punch-in crop, subtle speed ramp, vignette, sharpness, contrast, saturation, slow-motion emphasis, or clean natural grade.
|
||||||
|
- Fitting music direction and sound effects timing.
|
||||||
|
- Voiceover text that comments on what is actually visible or strongly implied by the footage.
|
||||||
|
- Short text overlays only where they improve retention or clarity.
|
||||||
|
|
||||||
|
DELIVERABLE EXPECTATION
|
||||||
|
The service must ultimately render both:
|
||||||
|
- the full final highlight video
|
||||||
|
- each rendered timeline segment clip that was used to assemble the full final highlight video
|
||||||
|
|
||||||
|
Expected output for the current edit-project renderer:
|
||||||
|
- final.mp4
|
||||||
|
- rendered-clips/clip_0001.mp4
|
||||||
|
- rendered-clips/clip_0002.mp4
|
||||||
|
- rendered-clips/clip_0003.mp4
|
||||||
|
- render-manifest.json with renderedClipPaths
|
||||||
|
|
||||||
|
Expected output for the highlight-project renderer when implemented:
|
||||||
|
- highlights/<highlight-id>/final.mp4
|
||||||
|
- highlights/<highlight-id>/rendered-clips/clip_0001.mp4
|
||||||
|
- highlights/<highlight-id>/rendered-clips/clip_0002.mp4
|
||||||
|
- highlights/<highlight-id>/rendered-clips/clip_0003.mp4
|
||||||
|
- highlights/<highlight-id>/render-manifest.json with renderedClipPaths
|
||||||
|
|
||||||
|
CONTENT CATEGORY DECISION
|
||||||
|
Classify the project into exactly one content category:
|
||||||
|
- car_vlog
|
||||||
|
- food_vlog
|
||||||
|
- family_vlog
|
||||||
|
- generic_vlog
|
||||||
|
|
||||||
|
Use the category to direct the whole edit.
|
||||||
|
|
||||||
|
For car_vlog:
|
||||||
|
- Make it premium, powerful, precise, and aspirational.
|
||||||
|
- Prioritize hero angles, reflections, wheels, badges, road movement, cockpit details, acceleration, reveal shots, light, shadow, and motion.
|
||||||
|
- Use punchy cuts, speed ramps, whooshes, impact hits, engine/exhaust SFX only when appropriate, cinematic contrast, and confident short overlays.
|
||||||
|
- Voiceover should praise what is actually visible: stance, motion, design, presence, craft, road feel. Do not invent specifications, price, horsepower, rarity, or model facts unless visible in the footage.
|
||||||
|
|
||||||
|
For food_vlog:
|
||||||
|
- Make it sensory, warm, rhythmic, and appetizing.
|
||||||
|
- Prioritize texture, slicing, pouring, steam, sizzling, plating, table reveals, reactions, and macro-like detail.
|
||||||
|
- Use tactile SFX, warm grade, gentle groove, ingredient/payoff overlays, and voiceover focused on craft, freshness, texture, and taste.
|
||||||
|
|
||||||
|
For family_vlog:
|
||||||
|
- Make it warm, emotional, human, and memory-focused.
|
||||||
|
- Prioritize faces, reactions, laughter, hugs, children, travel reveals, meaningful moments, and natural dialogue.
|
||||||
|
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
|
||||||
|
- Voiceover should feel personal and reflective, not promotional.
|
||||||
|
|
||||||
|
For generic_vlog:
|
||||||
|
- Build a clear mini-story from novelty, clean composition, motion, reaction, place, process, and payoff.
|
||||||
|
- Use restrained cinematic effects and explain only what the footage supports.
|
||||||
|
|
||||||
|
SHOT SELECTION RULES
|
||||||
|
Before writing JSON, inspect the contact sheets and representative frames.
|
||||||
|
|
||||||
|
Create a private selection table for yourself:
|
||||||
|
- candidate source range
|
||||||
|
- visual appeal
|
||||||
|
- motion/energy
|
||||||
|
- uniqueness
|
||||||
|
- story role
|
||||||
|
- audio opportunity
|
||||||
|
- risk: blur, darkness, repetitive angle, dead time
|
||||||
|
|
||||||
|
Then use only the best ranges. Avoid:
|
||||||
|
- repeated near-identical shots unless repetition is intentional
|
||||||
|
- long boring segments
|
||||||
|
- badly blurred or unusably dark footage unless it is the only important moment
|
||||||
|
- shots with no story purpose
|
||||||
|
- unsupported narration
|
||||||
|
|
||||||
|
TIMELINE AND PACING RULES
|
||||||
|
- Do not force a strict 60 second edit unless the footage actually supports it.
|
||||||
|
- Derive the final duration from the input source length and number of strong moments.
|
||||||
|
- Include every strong highlight moment that improves the final video.
|
||||||
|
- Exclude weak, repetitive, technically poor, or dead-time footage even if the source is long.
|
||||||
|
- Maximum final duration is 600 seconds, which is 10 minutes.
|
||||||
|
- Most videos should be shorter than the maximum. Use 10 minutes only when the source has enough distinct, high-quality highlights to justify it.
|
||||||
|
- Recommended duration guide:
|
||||||
|
- Source under 30 seconds: final highlight usually 8-20 seconds.
|
||||||
|
- Source 30-90 seconds: final highlight usually 15-45 seconds.
|
||||||
|
- Source 90 seconds-5 minutes: final highlight usually 30-120 seconds.
|
||||||
|
- Source 5-20 minutes: final highlight usually 60-180 seconds.
|
||||||
|
- Source over 20 minutes: final highlight can be longer, but only if the footage has enough distinct high-quality moments.
|
||||||
|
- The JSON field targetDurationSeconds must equal the actual planned final timeline duration, not a fixed configured default.
|
||||||
|
- The final decision timelineEndSeconds must equal targetDurationSeconds.
|
||||||
|
- Timeline must start at 0.0.
|
||||||
|
- Every decision timelineStartSeconds must equal the previous decision timelineEndSeconds.
|
||||||
|
- No overlaps. No gaps.
|
||||||
|
- For each decision:
|
||||||
|
rendered duration = (sourceEndSeconds - sourceStartSeconds) / playbackSpeed
|
||||||
|
timelineEndSeconds - timelineStartSeconds must equal rendered duration within 0.05 seconds.
|
||||||
|
- playbackSpeed must be between 0.25 and 4.0.
|
||||||
|
- Keep most shots between 1.0 and 4.0 seconds.
|
||||||
|
- Use longer shots only for emotional moments, reveals, or beautiful hero motion.
|
||||||
|
- Use faster shots for montages, details, impacts, or beat drops.
|
||||||
|
|
||||||
|
ALLOWED TRANSITIONS
|
||||||
|
Use only:
|
||||||
|
- cut
|
||||||
|
- crossfade
|
||||||
|
- fade-in
|
||||||
|
- fade-out
|
||||||
|
- none
|
||||||
|
|
||||||
|
Prefer mostly cuts. Use crossfade/fades sparingly for emotional, elegant, or time-passing moments.
|
||||||
|
|
||||||
|
VISUAL TREATMENT RULES
|
||||||
|
visualTreatment must be concise but specific. Good examples:
|
||||||
|
- cinematic grade with subtle punch-in
|
||||||
|
- high-contrast car hero grade
|
||||||
|
- warm food closeup grade
|
||||||
|
- soft family memory grade
|
||||||
|
- speed-ramp detail accent
|
||||||
|
- clean natural grade
|
||||||
|
- none
|
||||||
|
|
||||||
|
Use visual effects to support the moment. Do not overuse effects.
|
||||||
|
|
||||||
|
AUDIO RULES
|
||||||
|
Use audioCues for music and SFX direction.
|
||||||
|
|
||||||
|
Music:
|
||||||
|
- A music cue may use a descriptive assetKey such as premium_cinematic_drive, warm_acoustic_memory, tactile_food_groove, urban_pulse_intro.
|
||||||
|
- Music should cover the full timeline unless silence is intentionally used.
|
||||||
|
- Set gainDb conservatively, usually between -22 and -10.
|
||||||
|
- Describe the music mood and why it fits.
|
||||||
|
|
||||||
|
SFX:
|
||||||
|
- Only reference SFX assetKey values that exist as WAV files in the project audio/sfx folder or assets/sfx folder if the renderer maps them.
|
||||||
|
- If no SFX files exist, do not invent file-backed SFX cues. Instead, put desired SFX ideas in the cue notes only if the service can later generate assets.
|
||||||
|
- SFX should be precise: whoosh on transition, bass hit on reveal, camera shutter on detail, subtle riser before hero shot, engine accent for car footage, sizzle/chop/pour for food, gentle ambience for family.
|
||||||
|
|
||||||
|
VOICEOVER RULES
|
||||||
|
- Voiceover is optional only if the project disables it. If enabled, include concise lines.
|
||||||
|
- Each line must be 240 characters or fewer.
|
||||||
|
- Use natural, premium, non-cheesy language.
|
||||||
|
- Voiceover must fit the category and visible content.
|
||||||
|
- Do not use generic filler like "this is amazing" or "an unforgettable journey" unless the footage clearly supports it.
|
||||||
|
- Do not invent facts.
|
||||||
|
- Give each line a delivery direction, for example:
|
||||||
|
- calm premium narrator
|
||||||
|
- confident cinematic narrator
|
||||||
|
- warm reflective narrator
|
||||||
|
- energetic food host
|
||||||
|
|
||||||
|
TEXT OVERLAY RULES
|
||||||
|
- Overlays must be short: 80 characters or fewer.
|
||||||
|
- Use only these placements:
|
||||||
|
- lower_left_safe
|
||||||
|
- lower_center_safe
|
||||||
|
- center_safe
|
||||||
|
- upper_left_safe
|
||||||
|
- upper_right_safe
|
||||||
|
- Use only these animations:
|
||||||
|
- fade_slide_up
|
||||||
|
- fade_in
|
||||||
|
- none
|
||||||
|
- Use overlays sparingly. 1-5 overlays is usually enough.
|
||||||
|
- Overlays should feel premium and specific, not clickbait.
|
||||||
|
|
||||||
|
QUALITY BAR
|
||||||
|
Reject your own first draft if it has any of these problems:
|
||||||
|
- The first 3 seconds are not compelling.
|
||||||
|
- The edit is just chronological trimming without a story.
|
||||||
|
- The voiceover is generic or not grounded in visible footage.
|
||||||
|
- Music or SFX are placeholders with no timing purpose.
|
||||||
|
- Text overlays repeat what the voiceover already says.
|
||||||
|
- The final shot is weak.
|
||||||
|
- Any JSON field uses invalid values.
|
||||||
|
- Any clip ID or timestamp is invented.
|
||||||
|
- Any timeline math is invalid.
|
||||||
|
|
||||||
|
REQUIRED JSON SHAPE
|
||||||
|
Write exactly this JSON shape:
|
||||||
|
|
||||||
|
{
|
||||||
|
"projectId": "<project-id>",
|
||||||
|
"style": "<style-from-project-json>",
|
||||||
|
"targetDurationSeconds": <actual-planned-final-duration-seconds>,
|
||||||
|
"decisions": [
|
||||||
|
{
|
||||||
|
"clipId": "<valid clip id from analysis>",
|
||||||
|
"sourceStartSeconds": 0.0,
|
||||||
|
"sourceEndSeconds": 2.5,
|
||||||
|
"timelineStartSeconds": 0.0,
|
||||||
|
"timelineEndSeconds": 2.5,
|
||||||
|
"transitionIn": "cut",
|
||||||
|
"transitionOut": "cut",
|
||||||
|
"playbackSpeed": 1.0,
|
||||||
|
"visualTreatment": "cinematic grade with subtle punch-in",
|
||||||
|
"reason": "Opening hook: strongest hero angle and immediate viewer retention."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"audioCues": [
|
||||||
|
{
|
||||||
|
"type": "music",
|
||||||
|
"assetKey": "premium_cinematic_drive",
|
||||||
|
"timelineStartSeconds": 0.0,
|
||||||
|
"timelineEndSeconds": <actual-planned-final-duration-seconds>,
|
||||||
|
"gainDb": -16.0,
|
||||||
|
"notes": "Low cinematic pulse that builds through detail shots and lifts into the final hero moment."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"voiceover": [
|
||||||
|
{
|
||||||
|
"text": "A concise, specific line grounded in what is visible in the footage.",
|
||||||
|
"timelineStartSeconds": 1.0,
|
||||||
|
"timelineEndSeconds": 4.0,
|
||||||
|
"delivery": "confident cinematic narrator"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"overlays": [
|
||||||
|
{
|
||||||
|
"text": "Short premium overlay",
|
||||||
|
"timelineStartSeconds": 0.5,
|
||||||
|
"timelineEndSeconds": 2.0,
|
||||||
|
"placement": "lower_left_safe",
|
||||||
|
"animation": "fade_slide_up",
|
||||||
|
"reason": "Adds context without duplicating the narration."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"renderProfile": "mp4-h264-aac-1080p",
|
||||||
|
"summary": "A short explanation of the creative strategy, category choice, hook, pacing, sound design, and final payoff."
|
||||||
|
}
|
||||||
|
|
||||||
|
FINAL SELF-CHECK BEFORE WRITING director/edit-plan.json
|
||||||
|
Verify:
|
||||||
|
- projectId matches project.json.
|
||||||
|
- style matches project.json.
|
||||||
|
- targetDurationSeconds equals the actual final timeline duration selected from the source highlights.
|
||||||
|
- decisions is not empty.
|
||||||
|
- audioCues, voiceover, and overlays are arrays even when empty.
|
||||||
|
- All clip IDs exist.
|
||||||
|
- All source ranges are inside clip duration.
|
||||||
|
- Timeline starts at 0.0 and is contiguous.
|
||||||
|
- Final decision timelineEndSeconds equals targetDurationSeconds.
|
||||||
|
- Duration math matches playbackSpeed.
|
||||||
|
- transitionIn and transitionOut use only allowed values.
|
||||||
|
- playbackSpeed is between 0.25 and 4.0.
|
||||||
|
- overlay placements and animations use only allowed values.
|
||||||
|
- overlay text is 80 characters or fewer.
|
||||||
|
- voiceover lines are 240 characters or fewer.
|
||||||
|
- SFX asset keys reference real files if SFX cues are included.
|
||||||
|
- JSON is valid and contains no comments or Markdown.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes For The Service
|
||||||
|
|
||||||
|
This prompt is intentionally stricter than the current lightweight storyboard prompt. It is designed to prevent the common failure mode where the AI produces a generic, boring, or invalid edit plan.
|
||||||
|
|
||||||
|
Current renderer constraints still matter:
|
||||||
|
|
||||||
|
- `FfmpegEditRenderer` can render the `EditPlan` schema used by `output/edit-projects`.
|
||||||
|
- The highlight project flow still needs an importer/renderer bridge before `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4` can be produced automatically.
|
||||||
|
- If SFX are referenced, the current validator expects matching WAV files for SFX asset keys. Do not ask the AI to reference non-existent SFX files unless the asset generation step is implemented first.
|
||||||
|
- The current noop voiceover provider writes a script, not a real spoken voice. Production-ready voice requires a real TTS provider or recorded voiceover asset.
|
||||||
|
|
@ -204,6 +204,7 @@ The system needs explicit providers instead of pretending a prompt can create fi
|
||||||
- Font provider: local fonts with brand/category mapping.
|
- Font provider: local fonts with brand/category mapping.
|
||||||
- LUT provider: local LUT files or named FFmpeg grade presets.
|
- LUT provider: local LUT files or named FFmpeg grade presets.
|
||||||
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
|
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
|
||||||
|
- Asset generation stage: when a needed asset is missing, write a reusable request pack and keep the target cache path stable so future AI-generated assets can be reused.
|
||||||
|
|
||||||
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
|
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
|
||||||
|
|
||||||
|
|
@ -301,9 +302,13 @@ The plan should be strict JSON so a cheaper model or deterministic renderer can
|
||||||
- [x] Milestone 2 progress: Added a configurable highlight source scheduler and directory initializer for `input/highlights/source`, `working`, `processed`, and `rejected`; it claims one valid source video per scan, creates a highlight project and manifest, copies the source into the project folder, and leaves later videos for later scans.
|
- [x] Milestone 2 progress: Added a configurable highlight source scheduler and directory initializer for `input/highlights/source`, `working`, `processed`, and `rejected`; it claims one valid source video per scan, creates a highlight project and manifest, copies the source into the project folder, and leaves later videos for later scans.
|
||||||
- [x] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
|
- [x] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
|
||||||
- [x] Milestone 3 progress: Added highlight source analysis for the new project contract, reusing FFprobe inspection, frame extraction, contact sheet generation, proxy generation, and adding waveform image generation under `analysis/`; the highlight scheduler now runs analysis immediately after creating a project.
|
- [x] Milestone 3 progress: Added highlight source analysis for the new project contract, reusing FFprobe inspection, frame extraction, contact sheet generation, proxy generation, and adding waveform image generation under `analysis/`; the highlight scheduler now runs analysis immediately after creating a project.
|
||||||
- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
- [x] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
||||||
- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
- [x] Milestone 4 progress: Added configurable FFmpeg scene detection for highlight sources, persisted `analysis/scene-segments.json`, attached shot segments to `source-analysis.json`, and logged shot segment counts when the highlight scheduler finishes analysis.
|
||||||
- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
- [x] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
||||||
|
- [x] Milestone 5 progress: Added deterministic FFmpeg audio analysis for mean/max volume, silence detection, missing-audio handling, persisted `analysis/audio-analysis.json`, and source-analysis embedding. Non-silent sections are labeled `unclassified_audio` until a speech/music/noise classifier or ASR provider is added.
|
||||||
|
- [x] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
||||||
|
- [x] Milestone 6 progress: Added persisted `analysis/visual-analysis.json` and source-analysis embedding with blur, exposure, motion, composition, representative thumbnail, face-presence, and object-label fields. Current implementation uses deterministic metadata, thumbnail, and scene-density heuristics; face/object recognition is explicitly marked heuristic until a CV model is connected.
|
||||||
|
- [x] Milestone 6 local CV extension: Added a configurable `local-cv` visual-analysis provider that posts source metadata, thumbnails, and shot segments to a local HTTP CV worker, persists the returned normalized visual analysis, and falls back to heuristic analysis when configured. See `docs/local-cv-visual-analysis-guide.md`.
|
||||||
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
|
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
|
||||||
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
|
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
|
||||||
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
|
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
|
||||||
|
|
@ -548,6 +553,9 @@ Expected renderer output:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0001.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0002.mp4
|
||||||
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0003.mp4
|
||||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
{
|
||||||
|
"id" : "source-clips",
|
||||||
|
"name" : "source-clips",
|
||||||
|
"status" : "RENDERED",
|
||||||
|
"inputDirectory" : "./input/editing/processed/source-clips",
|
||||||
|
"outputDirectory" : "output/edit-projects/source-clips",
|
||||||
|
"targetDurationSeconds" : 60,
|
||||||
|
"style" : "cinematic-porsche-promo",
|
||||||
|
"voiceoverEnabled" : true,
|
||||||
|
"musicEnabled" : true,
|
||||||
|
"soundEffectsEnabled" : true,
|
||||||
|
"createdAt" : "2026-07-10T17:25:59.366353Z",
|
||||||
|
"updatedAt" : "2026-07-10T21:58:05.307273Z",
|
||||||
|
"failureReason" : null
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
|
|
@ -503,8 +503,16 @@ public class VideoClippingProperties {
|
||||||
@Min(1)
|
@Min(1)
|
||||||
private int proxyWidth = 640;
|
private int proxyWidth = 640;
|
||||||
|
|
||||||
|
private double sceneDetectionThreshold = 0.35;
|
||||||
|
|
||||||
|
private double minimumSceneDurationSeconds = 1.0;
|
||||||
|
|
||||||
|
private double silenceThresholdDb = -35.0;
|
||||||
|
|
||||||
|
private double silenceMinimumDurationSeconds = 0.5;
|
||||||
|
|
||||||
@Min(1)
|
@Min(1)
|
||||||
private int targetDurationSeconds = 60;
|
private int targetDurationSeconds = 600;
|
||||||
|
|
||||||
@Min(1)
|
@Min(1)
|
||||||
private int outputWidth = 1920;
|
private int outputWidth = 1920;
|
||||||
|
|
@ -542,6 +550,8 @@ public class VideoClippingProperties {
|
||||||
|
|
||||||
private final Assets assets = new Assets();
|
private final Assets assets = new Assets();
|
||||||
|
|
||||||
|
private final VisualAnalysis visualAnalysis = new VisualAnalysis();
|
||||||
|
|
||||||
private final LocalDirector localDirector = new LocalDirector();
|
private final LocalDirector localDirector = new LocalDirector();
|
||||||
|
|
||||||
private final HighlightScheduler highlightScheduler = new HighlightScheduler();
|
private final HighlightScheduler highlightScheduler = new HighlightScheduler();
|
||||||
|
|
@ -618,6 +628,38 @@ public class VideoClippingProperties {
|
||||||
this.proxyWidth = proxyWidth;
|
this.proxyWidth = proxyWidth;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public double getSceneDetectionThreshold() {
|
||||||
|
return sceneDetectionThreshold;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setSceneDetectionThreshold(double sceneDetectionThreshold) {
|
||||||
|
this.sceneDetectionThreshold = sceneDetectionThreshold;
|
||||||
|
}
|
||||||
|
|
||||||
|
public double getMinimumSceneDurationSeconds() {
|
||||||
|
return minimumSceneDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setMinimumSceneDurationSeconds(double minimumSceneDurationSeconds) {
|
||||||
|
this.minimumSceneDurationSeconds = minimumSceneDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public double getSilenceThresholdDb() {
|
||||||
|
return silenceThresholdDb;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setSilenceThresholdDb(double silenceThresholdDb) {
|
||||||
|
this.silenceThresholdDb = silenceThresholdDb;
|
||||||
|
}
|
||||||
|
|
||||||
|
public double getSilenceMinimumDurationSeconds() {
|
||||||
|
return silenceMinimumDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setSilenceMinimumDurationSeconds(double silenceMinimumDurationSeconds) {
|
||||||
|
this.silenceMinimumDurationSeconds = silenceMinimumDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
public int getTargetDurationSeconds() {
|
public int getTargetDurationSeconds() {
|
||||||
return targetDurationSeconds;
|
return targetDurationSeconds;
|
||||||
}
|
}
|
||||||
|
|
@ -742,6 +784,10 @@ public class VideoClippingProperties {
|
||||||
return assets;
|
return assets;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public VisualAnalysis getVisualAnalysis() {
|
||||||
|
return visualAnalysis;
|
||||||
|
}
|
||||||
|
|
||||||
public LocalDirector getLocalDirector() {
|
public LocalDirector getLocalDirector() {
|
||||||
return localDirector;
|
return localDirector;
|
||||||
}
|
}
|
||||||
|
|
@ -802,6 +848,109 @@ public class VideoClippingProperties {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static class VisualAnalysis {
|
||||||
|
private String provider = "heuristic";
|
||||||
|
|
||||||
|
private String endpoint = "http://127.0.0.1:8091/v1/analyze-visuals";
|
||||||
|
|
||||||
|
@Min(1)
|
||||||
|
private long timeoutMs = 30000;
|
||||||
|
|
||||||
|
private boolean fallbackToHeuristic = true;
|
||||||
|
|
||||||
|
private final LocalCvWorker localCvWorker = new LocalCvWorker();
|
||||||
|
|
||||||
|
public String getProvider() {
|
||||||
|
return provider;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setProvider(String provider) {
|
||||||
|
this.provider = provider;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getEndpoint() {
|
||||||
|
return endpoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setEndpoint(String endpoint) {
|
||||||
|
this.endpoint = endpoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getTimeoutMs() {
|
||||||
|
return timeoutMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setTimeoutMs(long timeoutMs) {
|
||||||
|
this.timeoutMs = timeoutMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean isFallbackToHeuristic() {
|
||||||
|
return fallbackToHeuristic;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setFallbackToHeuristic(boolean fallbackToHeuristic) {
|
||||||
|
this.fallbackToHeuristic = fallbackToHeuristic;
|
||||||
|
}
|
||||||
|
|
||||||
|
public LocalCvWorker getLocalCvWorker() {
|
||||||
|
return localCvWorker;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class LocalCvWorker {
|
||||||
|
private boolean autoStart = false;
|
||||||
|
|
||||||
|
private String script = "./tools/run_local_cv_worker.sh";
|
||||||
|
|
||||||
|
@Min(0)
|
||||||
|
private long startupWaitMs = 0;
|
||||||
|
|
||||||
|
private String healthPath = "/health";
|
||||||
|
|
||||||
|
@Min(1)
|
||||||
|
private long healthCheckIntervalMs = 1000;
|
||||||
|
|
||||||
|
public boolean isAutoStart() {
|
||||||
|
return autoStart;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setAutoStart(boolean autoStart) {
|
||||||
|
this.autoStart = autoStart;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getScript() {
|
||||||
|
return script;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setScript(String script) {
|
||||||
|
this.script = script;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getStartupWaitMs() {
|
||||||
|
return startupWaitMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setStartupWaitMs(long startupWaitMs) {
|
||||||
|
this.startupWaitMs = startupWaitMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getHealthPath() {
|
||||||
|
return healthPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setHealthPath(String healthPath) {
|
||||||
|
this.healthPath = healthPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getHealthCheckIntervalMs() {
|
||||||
|
return healthCheckIntervalMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setHealthCheckIntervalMs(long healthCheckIntervalMs) {
|
||||||
|
this.healthCheckIntervalMs = healthCheckIntervalMs;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public static class LocalDirector {
|
public static class LocalDirector {
|
||||||
private boolean enabled = true;
|
private boolean enabled = true;
|
||||||
|
|
||||||
|
|
@ -929,6 +1078,21 @@ public class VideoClippingProperties {
|
||||||
@Min(1000)
|
@Min(1000)
|
||||||
private long pollIntervalMs = 5000;
|
private long pollIntervalMs = 5000;
|
||||||
|
|
||||||
|
private boolean renderEnabled = true;
|
||||||
|
|
||||||
|
private boolean requireDirectorApproval = false;
|
||||||
|
|
||||||
|
private String approvalFileName = "approved.flag";
|
||||||
|
|
||||||
|
@Min(1)
|
||||||
|
private int maxHighlightsPerSource = 3;
|
||||||
|
|
||||||
|
@Min(1)
|
||||||
|
private double highlightMinDurationSeconds = 8.0;
|
||||||
|
|
||||||
|
@Min(1)
|
||||||
|
private double highlightMaxDurationSeconds = 35.0;
|
||||||
|
|
||||||
public boolean isEnabled() {
|
public boolean isEnabled() {
|
||||||
return enabled;
|
return enabled;
|
||||||
}
|
}
|
||||||
|
|
@ -976,6 +1140,54 @@ public class VideoClippingProperties {
|
||||||
public void setPollIntervalMs(long pollIntervalMs) {
|
public void setPollIntervalMs(long pollIntervalMs) {
|
||||||
this.pollIntervalMs = pollIntervalMs;
|
this.pollIntervalMs = pollIntervalMs;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public boolean isRenderEnabled() {
|
||||||
|
return renderEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setRenderEnabled(boolean renderEnabled) {
|
||||||
|
this.renderEnabled = renderEnabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean isRequireDirectorApproval() {
|
||||||
|
return requireDirectorApproval;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setRequireDirectorApproval(boolean requireDirectorApproval) {
|
||||||
|
this.requireDirectorApproval = requireDirectorApproval;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getApprovalFileName() {
|
||||||
|
return approvalFileName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setApprovalFileName(String approvalFileName) {
|
||||||
|
this.approvalFileName = approvalFileName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int getMaxHighlightsPerSource() {
|
||||||
|
return maxHighlightsPerSource;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setMaxHighlightsPerSource(int maxHighlightsPerSource) {
|
||||||
|
this.maxHighlightsPerSource = maxHighlightsPerSource;
|
||||||
|
}
|
||||||
|
|
||||||
|
public double getHighlightMinDurationSeconds() {
|
||||||
|
return highlightMinDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setHighlightMinDurationSeconds(double highlightMinDurationSeconds) {
|
||||||
|
this.highlightMinDurationSeconds = highlightMinDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public double getHighlightMaxDurationSeconds() {
|
||||||
|
return highlightMaxDurationSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setHighlightMaxDurationSeconds(double highlightMaxDurationSeconds) {
|
||||||
|
this.highlightMaxDurationSeconds = highlightMaxDurationSeconds;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,10 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
public record AudioSection(
|
||||||
|
String sectionId,
|
||||||
|
String kind,
|
||||||
|
double startSeconds,
|
||||||
|
double endSeconds,
|
||||||
|
double durationSeconds
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -37,6 +37,7 @@ public class EditPlanInboxScanner {
|
||||||
private final EditProjectStore store;
|
private final EditProjectStore store;
|
||||||
private final EditProjectService projectService;
|
private final EditProjectService projectService;
|
||||||
private final EditPlanValidator validator;
|
private final EditPlanValidator validator;
|
||||||
|
private final Optional<AssetGenerationStage> assetGenerationStage;
|
||||||
private final Optional<EditRenderer> renderer;
|
private final Optional<EditRenderer> renderer;
|
||||||
private final VoiceoverGenerator voiceoverGenerator;
|
private final VoiceoverGenerator voiceoverGenerator;
|
||||||
private final AtomicBoolean scanning = new AtomicBoolean();
|
private final AtomicBoolean scanning = new AtomicBoolean();
|
||||||
|
|
@ -49,9 +50,11 @@ public class EditPlanInboxScanner {
|
||||||
EditProjectService projectService,
|
EditProjectService projectService,
|
||||||
EditPlanValidator validator,
|
EditPlanValidator validator,
|
||||||
VoiceoverGenerator voiceoverGenerator,
|
VoiceoverGenerator voiceoverGenerator,
|
||||||
|
ObjectProvider<AssetGenerationStage> assetGenerationStageProvider,
|
||||||
ObjectProvider<EditRenderer> rendererProvider
|
ObjectProvider<EditRenderer> rendererProvider
|
||||||
) {
|
) {
|
||||||
this(properties, objectMapper, store, projectService, validator, voiceoverGenerator,
|
this(properties, objectMapper, store, projectService, validator, voiceoverGenerator,
|
||||||
|
assetGenerationStageProvider.getIfAvailable(),
|
||||||
rendererProvider.getIfAvailable());
|
rendererProvider.getIfAvailable());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -63,13 +66,13 @@ public class EditPlanInboxScanner {
|
||||||
EditPlanValidator validator,
|
EditPlanValidator validator,
|
||||||
EditRenderer renderer
|
EditRenderer renderer
|
||||||
) {
|
) {
|
||||||
this(properties, objectMapper, store, projectService, validator, null, renderer);
|
this(properties, objectMapper, store, projectService, validator, null, null, renderer);
|
||||||
}
|
}
|
||||||
|
|
||||||
EditPlanInboxScanner(
|
EditPlanInboxScanner(
|
||||||
VideoClippingProperties properties, ObjectMapper objectMapper, EditProjectStore store,
|
VideoClippingProperties properties, ObjectMapper objectMapper, EditProjectStore store,
|
||||||
EditProjectService projectService, EditPlanValidator validator,
|
EditProjectService projectService, EditPlanValidator validator,
|
||||||
VoiceoverGenerator voiceoverGenerator, EditRenderer renderer
|
VoiceoverGenerator voiceoverGenerator, AssetGenerationStage assetGenerationStage, EditRenderer renderer
|
||||||
) {
|
) {
|
||||||
this.projectRoot = Path.of(properties.getEditing().getProjectDirectory());
|
this.projectRoot = Path.of(properties.getEditing().getProjectDirectory());
|
||||||
this.expectedPlanFileName = properties.getEditing().getLocalDirector().getExpectedPlanFileName();
|
this.expectedPlanFileName = properties.getEditing().getLocalDirector().getExpectedPlanFileName();
|
||||||
|
|
@ -82,6 +85,7 @@ public class EditPlanInboxScanner {
|
||||||
this.validator = validator;
|
this.validator = validator;
|
||||||
this.renderer = Optional.ofNullable(renderer);
|
this.renderer = Optional.ofNullable(renderer);
|
||||||
this.voiceoverGenerator = voiceoverGenerator;
|
this.voiceoverGenerator = voiceoverGenerator;
|
||||||
|
this.assetGenerationStage = Optional.ofNullable(assetGenerationStage);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Scheduled(
|
@Scheduled(
|
||||||
|
|
@ -154,6 +158,11 @@ public class EditPlanInboxScanner {
|
||||||
}
|
}
|
||||||
|
|
||||||
store.writeJson(projectId, "edit-plan.json", normalized(plan));
|
store.writeJson(projectId, "edit-plan.json", normalized(plan));
|
||||||
|
AssetGenerationResult assetGenerationResult = assetGenerationStage
|
||||||
|
.map(stage -> stage.prepare(projectId, plan))
|
||||||
|
.orElse(new AssetGenerationResult(projectId, true, List.of(), java.time.Instant.now()));
|
||||||
|
log.info("event=edit_asset_generation_completed project_id={} item_count={} ready_for_render={}",
|
||||||
|
projectId, assetGenerationResult.items().size(), assetGenerationResult.readyForRender());
|
||||||
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
|
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
|
||||||
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
|
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
|
||||||
}
|
}
|
||||||
|
|
@ -164,8 +173,11 @@ public class EditPlanInboxScanner {
|
||||||
projectId, plan.decisions().size(), autoRender, requireApprovalBeforeRender);
|
projectId, plan.decisions().size(), autoRender, requireApprovalBeforeRender);
|
||||||
log.info("event=edit_plan_saved project_id={} decision_count={} source=inbox",
|
log.info("event=edit_plan_saved project_id={} decision_count={} source=inbox",
|
||||||
projectId, plan.decisions().size());
|
projectId, plan.decisions().size());
|
||||||
if (autoRender) {
|
if (autoRender && assetGenerationResult.readyForRender()) {
|
||||||
renderIfApproved(projectId, store.inboxDirectory(projectId));
|
renderIfApproved(projectId, store.inboxDirectory(projectId));
|
||||||
|
} else if (autoRender) {
|
||||||
|
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
|
||||||
|
projectId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -175,6 +187,16 @@ public class EditPlanInboxScanner {
|
||||||
projectId, inbox.resolve(approvalFileName));
|
projectId, inbox.resolve(approvalFileName));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
AssetGenerationStage stage = assetGenerationStage.orElse(null);
|
||||||
|
if (stage != null) {
|
||||||
|
EditPlan plan = store.readJson(projectId, "edit-plan.json", EditPlan.class);
|
||||||
|
AssetGenerationResult result = stage.prepare(projectId, plan);
|
||||||
|
if (!result.readyForRender()) {
|
||||||
|
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
|
||||||
|
projectId);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
renderer.orElseThrow(() -> new IllegalStateException(
|
renderer.orElseThrow(() -> new IllegalStateException(
|
||||||
"Automatic rendering is enabled but no EditRenderer is available")).render(projectId);
|
"Automatic rendering is enabled but no EditRenderer is available")).render(projectId);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -80,9 +80,13 @@ public class EditPlanValidator {
|
||||||
validateAudioCues(projectId, plan);
|
validateAudioCues(projectId, plan);
|
||||||
validateVoiceover(plan);
|
validateVoiceover(plan);
|
||||||
validateOverlays(plan);
|
validateOverlays(plan);
|
||||||
if (previousEnd > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS
|
validateCinematicRichness(project, plan);
|
||||||
|| Math.abs(plan.targetDurationSeconds() - project.targetDurationSeconds()) > EPSILON) {
|
if (plan.targetDurationSeconds() <= 0
|
||||||
reject("Edit plan duration exceeds or does not match the project target duration");
|
|| Math.abs(previousEnd - plan.targetDurationSeconds()) > TARGET_TOLERANCE_SECONDS) {
|
||||||
|
reject("Edit plan target duration must match the final timeline duration");
|
||||||
|
}
|
||||||
|
if (plan.targetDurationSeconds() > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS) {
|
||||||
|
reject("Edit plan duration exceeds the project maximum target duration");
|
||||||
}
|
}
|
||||||
return plan;
|
return plan;
|
||||||
}
|
}
|
||||||
|
|
@ -152,6 +156,42 @@ public class EditPlanValidator {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void validateCinematicRichness(EditProject project, EditPlan plan) {
|
||||||
|
int cinematicDimensions = 0;
|
||||||
|
if (hasVisualTreatment(plan)) {
|
||||||
|
cinematicDimensions++;
|
||||||
|
}
|
||||||
|
if (project.musicEnabled() && hasAudioCueType(plan, "music")) {
|
||||||
|
cinematicDimensions++;
|
||||||
|
}
|
||||||
|
if (project.voiceoverEnabled() && !plan.voiceover().isEmpty()) {
|
||||||
|
cinematicDimensions++;
|
||||||
|
}
|
||||||
|
if (project.soundEffectsEnabled() && hasAudioCueType(plan, "sfx")) {
|
||||||
|
cinematicDimensions++;
|
||||||
|
}
|
||||||
|
if (!plan.overlays().isEmpty()) {
|
||||||
|
cinematicDimensions++;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (cinematicDimensions < 2) {
|
||||||
|
reject("Edit plan is too bare; it must use at least two cinematic elements such as visual treatment, "
|
||||||
|
+ "music, voiceover, sound effects, or text overlays");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean hasVisualTreatment(EditPlan plan) {
|
||||||
|
return plan.decisions().stream()
|
||||||
|
.map(EditDecision::visualTreatment)
|
||||||
|
.filter(value -> value != null && !value.isBlank() && !"none".equalsIgnoreCase(value))
|
||||||
|
.findFirst()
|
||||||
|
.isPresent();
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean hasAudioCueType(EditPlan plan, String type) {
|
||||||
|
return plan.audioCues().stream().anyMatch(cue -> type.equals(cue.type()));
|
||||||
|
}
|
||||||
|
|
||||||
private void validateTransition(String transition) {
|
private void validateTransition(String transition) {
|
||||||
if (transition == null || !TRANSITIONS.contains(transition)) {
|
if (transition == null || !TRANSITIONS.contains(transition)) {
|
||||||
reject("Unsupported transition: " + transition);
|
reject("Unsupported transition: " + transition);
|
||||||
|
|
|
||||||
|
|
@ -127,6 +127,7 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
runAndRecord(segmentCommand(clips.get(decision.clipId()).sourcePath(), decision, output), commands);
|
runAndRecord(segmentCommand(clips.get(decision.clipId()).sourcePath(), decision, output), commands);
|
||||||
segments.add(output);
|
segments.add(output);
|
||||||
}
|
}
|
||||||
|
List<Path> renderedClips = publishRenderedClips(projectId, segments);
|
||||||
Path concatFile = work.resolve("concat.txt");
|
Path concatFile = work.resolve("concat.txt");
|
||||||
writeConcatFile(concatFile, segments);
|
writeConcatFile(concatFile, segments);
|
||||||
Path timeline = work.resolve("timeline-video.mp4");
|
Path timeline = work.resolve("timeline-video.mp4");
|
||||||
|
|
@ -152,10 +153,11 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
copy(videoTimeline, output);
|
copy(videoTimeline, output);
|
||||||
}
|
}
|
||||||
double duration = plan.decisions().get(plan.decisions().size() - 1).timelineEndSeconds();
|
double duration = plan.decisions().get(plan.decisions().size() - 1).timelineEndSeconds();
|
||||||
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, commands, assets);
|
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
|
||||||
RenderManifest manifest = new RenderManifest(projectId,
|
RenderManifest manifest = new RenderManifest(projectId,
|
||||||
plan.decisions().stream().map(EditDecision::clipId).toList(), output.toString(), duration,
|
plan.decisions().stream().map(EditDecision::clipId).toList(),
|
||||||
List.copyOf(commands), assets, Instant.now());
|
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
|
||||||
|
assets, Instant.now());
|
||||||
store.writeJson(projectId, "qa-report.json", qaReport);
|
store.writeJson(projectId, "qa-report.json", qaReport);
|
||||||
store.writeJson(projectId, "render-manifest.json", manifest);
|
store.writeJson(projectId, "render-manifest.json", manifest);
|
||||||
projectService.updateProject(projectId, EditProjectStatus.RENDERED, Path.of(project.inputDirectory()), null);
|
projectService.updateProject(projectId, EditProjectStatus.RENDERED, Path.of(project.inputDirectory()), null);
|
||||||
|
|
@ -361,9 +363,11 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
}
|
}
|
||||||
|
|
||||||
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||||
|
List<Path> renderedClips,
|
||||||
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||||
List<RenderQaCheck> checks = new ArrayList<>();
|
List<RenderQaCheck> checks = new ArrayList<>();
|
||||||
checks.add(outputExistsCheck(output));
|
checks.add(outputExistsCheck(output));
|
||||||
|
checks.add(renderedClipsExistCheck(renderedClips));
|
||||||
checks.add(durationMatchesTimelineCheck(plan, duration));
|
checks.add(durationMatchesTimelineCheck(plan, duration));
|
||||||
checks.add(requiredAssetsResolvedCheck(plan, assets));
|
checks.add(requiredAssetsResolvedCheck(plan, assets));
|
||||||
checks.add(overlaysSafeCheck(plan, duration));
|
checks.add(overlaysSafeCheck(plan, duration));
|
||||||
|
|
@ -388,6 +392,14 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
exists ? "final output exists" : "final output is missing: " + output);
|
exists ? "final output exists" : "final output is missing: " + output);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private RenderQaCheck renderedClipsExistCheck(List<Path> renderedClips) {
|
||||||
|
List<Path> missing = renderedClips.stream()
|
||||||
|
.filter(path -> !Files.isRegularFile(path))
|
||||||
|
.toList();
|
||||||
|
return new RenderQaCheck("rendered_clips_exist", missing.isEmpty(), "ERROR",
|
||||||
|
missing.isEmpty() ? "all rendered segment clips exist" : "missing rendered clips: " + missing);
|
||||||
|
}
|
||||||
|
|
||||||
private RenderQaCheck durationMatchesTimelineCheck(EditPlan plan, double duration) {
|
private RenderQaCheck durationMatchesTimelineCheck(EditPlan plan, double duration) {
|
||||||
double expected = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
double expected = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
||||||
.timelineEndSeconds();
|
.timelineEndSeconds();
|
||||||
|
|
@ -518,6 +530,18 @@ public class FfmpegEditRenderer implements EditRenderer {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private List<Path> publishRenderedClips(String projectId, List<Path> segments) {
|
||||||
|
Path publishedDirectory = store.projectDirectory(projectId).resolve("rendered-clips");
|
||||||
|
createDirectory(publishedDirectory);
|
||||||
|
List<Path> published = new ArrayList<>();
|
||||||
|
for (int index = 0; index < segments.size(); index++) {
|
||||||
|
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
|
||||||
|
copy(segments.get(index), target);
|
||||||
|
published.add(target);
|
||||||
|
}
|
||||||
|
return List.copyOf(published);
|
||||||
|
}
|
||||||
|
|
||||||
private long fileSize(Path file) {
|
private long fileSize(Path file) {
|
||||||
try {
|
try {
|
||||||
return Files.size(file);
|
return Files.size(file);
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,102 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HeuristicVisualAnalysisProvider implements VisualAnalysisProvider {
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public SourceVisualAnalysis analyze(
|
||||||
|
ClipAnalysis source,
|
||||||
|
List<String> thumbnails,
|
||||||
|
List<ShotSegment> shotSegments
|
||||||
|
) {
|
||||||
|
double blurScore = scoreOrDefault(source.sharpnessScore(), 0.5);
|
||||||
|
double exposureScore = scoreOrDefault(source.brightnessScore(), 0.5);
|
||||||
|
double motionScore = scoreOrDefault(source.motionScore(), motionFromShots(source.durationSeconds(), shotSegments));
|
||||||
|
return new SourceVisualAnalysis(
|
||||||
|
source.clipId(),
|
||||||
|
blurScore,
|
||||||
|
exposureScore,
|
||||||
|
motionScore,
|
||||||
|
compositionScore(source, thumbnails),
|
||||||
|
"unknown_without_face_detector",
|
||||||
|
labels(source),
|
||||||
|
representativeThumbnails(thumbnails),
|
||||||
|
"metadata_thumbnail_scene_heuristic"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double scoreOrDefault(double value, double fallback) {
|
||||||
|
if (value > 0) {
|
||||||
|
return round(clamp(value));
|
||||||
|
}
|
||||||
|
return round(clamp(fallback));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double motionFromShots(double durationSeconds, List<ShotSegment> shotSegments) {
|
||||||
|
if (durationSeconds <= 0 || shotSegments == null || shotSegments.isEmpty()) {
|
||||||
|
return 0.3;
|
||||||
|
}
|
||||||
|
double cutsPerMinute = Math.max(0, shotSegments.size() - 1) / durationSeconds * 60.0;
|
||||||
|
return clamp(0.25 + cutsPerMinute / 20.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double compositionScore(ClipAnalysis source, List<String> thumbnails) {
|
||||||
|
double aspectRatio = source.height() == 0 ? 0 : source.width() / (double) source.height();
|
||||||
|
double aspectScore = aspectRatio >= 1.70 && aspectRatio <= 1.90 ? 0.8 : 0.55;
|
||||||
|
double thumbnailScore = thumbnails == null || thumbnails.isEmpty() ? 0.45 : 0.7;
|
||||||
|
return round((aspectScore + thumbnailScore) / 2.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<VisualObjectLabel> labels(ClipAnalysis source) {
|
||||||
|
String haystack = (source.clipId() + " " + source.sourcePath()).toLowerCase(Locale.ROOT);
|
||||||
|
List<VisualObjectLabel> labels = new ArrayList<>();
|
||||||
|
addIfContains(labels, haystack, "porsche", "porsche", 0.82);
|
||||||
|
addIfContains(labels, haystack, "car", "car", 0.74);
|
||||||
|
addIfContains(labels, haystack, "drive", "car", 0.62);
|
||||||
|
addIfContains(labels, haystack, "food", "food", 0.72);
|
||||||
|
addIfContains(labels, haystack, "recipe", "food", 0.66);
|
||||||
|
addIfContains(labels, haystack, "family", "person", 0.62);
|
||||||
|
addIfContains(labels, haystack, "birthday", "person", 0.58);
|
||||||
|
if (labels.isEmpty()) {
|
||||||
|
labels.add(new VisualObjectLabel("unknown", 0.2, "metadata_heuristic"));
|
||||||
|
}
|
||||||
|
return labels.stream().distinct().toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void addIfContains(List<VisualObjectLabel> labels, String haystack, String needle, String label,
|
||||||
|
double confidence) {
|
||||||
|
if (haystack.contains(needle)) {
|
||||||
|
labels.add(new VisualObjectLabel(label, confidence, "metadata_heuristic"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<String> representativeThumbnails(List<String> thumbnails) {
|
||||||
|
if (thumbnails == null || thumbnails.isEmpty()) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
if (thumbnails.size() <= 3) {
|
||||||
|
return List.copyOf(thumbnails);
|
||||||
|
}
|
||||||
|
return List.of(
|
||||||
|
thumbnails.get(0),
|
||||||
|
thumbnails.get(thumbnails.size() / 2),
|
||||||
|
thumbnails.get(thumbnails.size() - 1)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double clamp(double value) {
|
||||||
|
return Math.max(0, Math.min(1, value));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double round(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
public record HighlightAssetRequest(
|
||||||
|
String projectId,
|
||||||
|
String highlightId,
|
||||||
|
String type,
|
||||||
|
String assetKey,
|
||||||
|
String targetPath,
|
||||||
|
String requestPath,
|
||||||
|
String notes,
|
||||||
|
double durationSeconds,
|
||||||
|
boolean blocking
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,338 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.StandardCopyOption;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightDirectorFlowService {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorFlowService.class);
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
private final HighlightVisualEffectsStage visualEffectsStage;
|
||||||
|
private final HighlightAssetPreparationService assetPreparationService;
|
||||||
|
private final HighlightLocalAssetWorker assetWorker;
|
||||||
|
private final HighlightFfmpegRenderer renderer;
|
||||||
|
|
||||||
|
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
|
||||||
|
HighlightProjectStore store,
|
||||||
|
HighlightVisualEffectsStage visualEffectsStage,
|
||||||
|
HighlightAssetPreparationService assetPreparationService,
|
||||||
|
HighlightLocalAssetWorker assetWorker,
|
||||||
|
HighlightFfmpegRenderer renderer) {
|
||||||
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
this.store = store;
|
||||||
|
this.visualEffectsStage = visualEffectsStage;
|
||||||
|
this.assetPreparationService = assetPreparationService;
|
||||||
|
this.assetWorker = assetWorker;
|
||||||
|
this.renderer = renderer;
|
||||||
|
}
|
||||||
|
|
||||||
|
public HighlightFlowResult process(String projectId, long scanId) {
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
String flowId = projectId + ":" + scanId;
|
||||||
|
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||||
|
Path planFile = store.directorDirectory(projectId).resolve("edit-plan.json");
|
||||||
|
if (!Files.isRegularFile(planFile)) {
|
||||||
|
return HighlightFlowResult.skipped(projectId, flowId, "missing_director_plan");
|
||||||
|
}
|
||||||
|
if (properties.isRequireDirectorApproval()
|
||||||
|
&& !Files.isRegularFile(store.directorDirectory(projectId).resolve(properties.getApprovalFileName()))) {
|
||||||
|
log.info("event=highlight_flow_waiting_for_approval flow_id={} project_id={} approval_file={}",
|
||||||
|
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
|
||||||
|
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
|
||||||
|
}
|
||||||
|
HighlightDirectorPlan plan = readPlan(planFile);
|
||||||
|
if (plan.highlights().isEmpty()) {
|
||||||
|
markFailed(project, "No highlights were provided by the director plan");
|
||||||
|
throw new IllegalStateException("Director plan contains no highlights");
|
||||||
|
}
|
||||||
|
markStatus(project, HighlightProjectStatus.RENDERING, null);
|
||||||
|
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
|
||||||
|
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
|
||||||
|
List<Path> finalOutputs = new ArrayList<>();
|
||||||
|
List<String> renderedHighlightIds = new ArrayList<>();
|
||||||
|
ContentCategory category = contentCategory(plan.contentCategory());
|
||||||
|
int limit = Math.min(properties.getMaxHighlightsPerSource(), plan.highlights().size());
|
||||||
|
for (int index = 0; index < limit; index++) {
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
|
||||||
|
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||||
|
writeStoryboard(highlightDirectory, plan, highlight);
|
||||||
|
visualEffectsStage.create(projectId, highlight, category);
|
||||||
|
EditPlan editPlan = buildEditPlan(project, plan, highlight, index);
|
||||||
|
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/edit-plan.json", editPlan);
|
||||||
|
HighlightAssetPreparationService.HighlightAssetPreparationResult assets = assetPreparationService
|
||||||
|
.prepare(projectId, project, highlight, category);
|
||||||
|
HighlightLocalAssetWorker.HighlightAssetWorkerResult assetWorkerResult = assetWorker.process(projectId,
|
||||||
|
highlight, category);
|
||||||
|
log.info("event=highlight_assets_prepared flow_id={} project_id={} highlight_id={} resolved={} requests={}",
|
||||||
|
flowId, projectId, highlight.highlightId(), assets.resolvedAssets().size(),
|
||||||
|
assets.requestFiles().size());
|
||||||
|
log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}",
|
||||||
|
flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(),
|
||||||
|
assetWorkerResult.pendingRequests().size());
|
||||||
|
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
|
||||||
|
finalOutputs.add(result.finalOutput());
|
||||||
|
renderedHighlightIds.add(result.highlightId());
|
||||||
|
}
|
||||||
|
Path projectOutput = store.projectDirectory(projectId).resolve("final.mp4");
|
||||||
|
concatFinalOutputs(projectOutput, finalOutputs);
|
||||||
|
RenderManifest manifest = new RenderManifest(projectId, renderedHighlightIds, finalOutputs.stream()
|
||||||
|
.map(Path::toString).toList(), projectOutput.toString(),
|
||||||
|
finalOutputs.stream().mapToDouble(this::probeDuration).sum(), List.of(), List.of(), Instant.now());
|
||||||
|
store.writeJson(projectId, "render-manifest.json", manifest);
|
||||||
|
markStatus(project, HighlightProjectStatus.RENDERED, null);
|
||||||
|
log.info("event=highlight_flow_completed flow_id={} scan_id={} project_id={} final_outputs={} elapsed_ms={}",
|
||||||
|
flowId, scanId, projectId, finalOutputs, (System.nanoTime() - startedAt) / 1_000_000);
|
||||||
|
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightDirectorPlan readPlan(Path planFile) {
|
||||||
|
try {
|
||||||
|
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to read highlight director plan: " + planFile, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private EditPlan buildEditPlan(HighlightProject project, HighlightDirectorPlan plan,
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight, int index) {
|
||||||
|
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
|
||||||
|
HighlightSourceAnalysis.class);
|
||||||
|
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
|
||||||
|
double requestedDuration = highlight.targetDurationSeconds() > 0 ? highlight.targetDurationSeconds()
|
||||||
|
: sourceDuration;
|
||||||
|
double targetDuration = Math.max(properties.getHighlightMinDurationSeconds(),
|
||||||
|
Math.min(properties.getHighlightMaxDurationSeconds(), requestedDuration));
|
||||||
|
double playbackSpeed = Math.max(0.25, Math.min(4.0, sourceDuration / targetDuration));
|
||||||
|
String clipId = analysis.source().clipId();
|
||||||
|
EditDecision decision = new EditDecision(
|
||||||
|
clipId,
|
||||||
|
highlight.sourceStartSeconds(),
|
||||||
|
highlight.sourceEndSeconds(),
|
||||||
|
0.0,
|
||||||
|
targetDuration,
|
||||||
|
transitionIn(highlight, index),
|
||||||
|
transitionOut(highlight, index, plan.highlights().size()),
|
||||||
|
playbackSpeed,
|
||||||
|
highlight.visualTreatment(),
|
||||||
|
highlight.renderNotes()
|
||||||
|
);
|
||||||
|
List<AudioCue> audioCues = new ArrayList<>();
|
||||||
|
if (highlight.musicDirection() != null && !highlight.musicDirection().isBlank()) {
|
||||||
|
audioCues.add(new AudioCue("music", safeKey("music", highlight.musicDirection()), 0.0, targetDuration,
|
||||||
|
-10.0, highlight.musicDirection()));
|
||||||
|
}
|
||||||
|
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
|
||||||
|
audioCues.add(new AudioCue("sfx", safeKey("sfx", highlight.storyPurpose(), highlight.highlightId()),
|
||||||
|
Math.max(0.0, targetDuration - 0.8), targetDuration, -6.0, highlight.sfxDirection()));
|
||||||
|
}
|
||||||
|
List<VoiceoverLine> voiceover = new ArrayList<>();
|
||||||
|
if (!highlight.voiceover().isEmpty()) {
|
||||||
|
double step = targetDuration / highlight.voiceover().size();
|
||||||
|
for (int i = 0; i < highlight.voiceover().size(); i++) {
|
||||||
|
double start = Math.min(targetDuration, i * step);
|
||||||
|
double end = Math.min(targetDuration, start + Math.max(1.0, step));
|
||||||
|
voiceover.add(new VoiceoverLine(highlight.voiceover().get(i), start, end, "cinematic_narration"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
List<TextOverlay> overlays = new ArrayList<>();
|
||||||
|
if (!highlight.overlays().isEmpty()) {
|
||||||
|
double step = targetDuration / highlight.overlays().size();
|
||||||
|
for (int i = 0; i < highlight.overlays().size(); i++) {
|
||||||
|
double start = Math.min(targetDuration, i * step);
|
||||||
|
double end = Math.min(targetDuration, start + Math.max(1.5, step * 0.75));
|
||||||
|
overlays.add(new TextOverlay(highlight.overlays().get(i), start, end,
|
||||||
|
i % 2 == 0 ? "lower_center_safe" : "center_safe",
|
||||||
|
"fade_slide_up", highlight.renderNotes()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return new EditPlan(project.id(),
|
||||||
|
safeKey("style", plan.contentCategory(), highlight.storyPurpose()),
|
||||||
|
targetDuration,
|
||||||
|
List.of(decision),
|
||||||
|
audioCues,
|
||||||
|
voiceover,
|
||||||
|
overlays,
|
||||||
|
"mp4-h264-aac-1080p",
|
||||||
|
highlight.title() + " / " + highlight.storyPurpose());
|
||||||
|
}
|
||||||
|
|
||||||
|
private String transitionIn(HighlightDirectorPlan.HighlightItem highlight, int index) {
|
||||||
|
if (index == 0 || highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
|
||||||
|
return "fade-in";
|
||||||
|
}
|
||||||
|
return "cut";
|
||||||
|
}
|
||||||
|
|
||||||
|
private String transitionOut(HighlightDirectorPlan.HighlightItem highlight, int index, int total) {
|
||||||
|
if (index + 1 == total || highlight.storyPurpose() != null && highlight.storyPurpose().contains("hero")) {
|
||||||
|
return "fade-out";
|
||||||
|
}
|
||||||
|
return "cut";
|
||||||
|
}
|
||||||
|
|
||||||
|
private void writeStoryboard(Path highlightDirectory, HighlightDirectorPlan plan,
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight) {
|
||||||
|
String storyboard = """
|
||||||
|
# Storyboard
|
||||||
|
|
||||||
|
Project: `%s`
|
||||||
|
Highlight: `%s`
|
||||||
|
Title: `%s`
|
||||||
|
Purpose: `%s`
|
||||||
|
Music: `%s`
|
||||||
|
SFX: `%s`
|
||||||
|
|
||||||
|
Voiceover:
|
||||||
|
%s
|
||||||
|
|
||||||
|
Overlays:
|
||||||
|
%s
|
||||||
|
""".formatted(plan.projectId(), highlight.highlightId(), highlight.title(), highlight.storyPurpose(),
|
||||||
|
highlight.musicDirection(), highlight.sfxDirection(), String.join(System.lineSeparator(),
|
||||||
|
highlight.voiceover()), String.join(System.lineSeparator(), highlight.overlays()));
|
||||||
|
try {
|
||||||
|
Files.createDirectories(highlightDirectory);
|
||||||
|
Files.writeString(highlightDirectory.resolve("storyboard.md"), storyboard, StandardCharsets.UTF_8);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write storyboard for highlight: " + highlight.highlightId(), ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void concatFinalOutputs(Path output, List<Path> sources) {
|
||||||
|
if (sources.isEmpty()) {
|
||||||
|
throw new IllegalStateException("No rendered highlight outputs were produced");
|
||||||
|
}
|
||||||
|
if (sources.size() == 1) {
|
||||||
|
copy(sources.get(0), output);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Path work = output.getParent().resolve("project-render-work");
|
||||||
|
try {
|
||||||
|
Files.createDirectories(work);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to create highlight project render work directory", ex);
|
||||||
|
}
|
||||||
|
Path concat = work.resolve("concat.txt");
|
||||||
|
String content = sources.stream()
|
||||||
|
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
|
||||||
|
.reduce("", (a, b) -> a + b + System.lineSeparator());
|
||||||
|
try {
|
||||||
|
Files.writeString(concat, content, StandardCharsets.UTF_8);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write project concat file", ex);
|
||||||
|
}
|
||||||
|
run(List.of("ffmpeg", "-hide_banner", "-y", "-f", "concat", "-safe", "0", "-i", concat.toString(),
|
||||||
|
"-c", "copy", output.toString()));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double probeDuration(Path video) {
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder("ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||||
|
"-of", "default=noprint_wrappers=1:nokey=1", video.toString()).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
if (process.waitFor() != 0) {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
return Double.parseDouble(output.trim());
|
||||||
|
} catch (IOException | InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void run(List<String> command) {
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
if (process.waitFor() != 0) {
|
||||||
|
throw new IllegalStateException(output);
|
||||||
|
}
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to run highlight concat command", ex);
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
throw new IllegalStateException("Interrupted while concatenating highlight outputs", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void copy(Path source, Path target) {
|
||||||
|
try {
|
||||||
|
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to publish highlight project output", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private ContentCategory contentCategory(String value) {
|
||||||
|
if (value == null || value.isBlank()) {
|
||||||
|
return ContentCategory.GENERIC_VLOG;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return ContentCategory.valueOf(value.toUpperCase(Locale.ROOT));
|
||||||
|
} catch (IllegalArgumentException ex) {
|
||||||
|
return ContentCategory.GENERIC_VLOG;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String safeKey(String prefix, String... values) {
|
||||||
|
String value = String.join("_", values == null ? new String[0] : values);
|
||||||
|
String normalized = value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]+", "_")
|
||||||
|
.replaceAll("_+", "_")
|
||||||
|
.replaceAll("^_|_$", "");
|
||||||
|
if (normalized.isBlank()) {
|
||||||
|
return prefix + "_1";
|
||||||
|
}
|
||||||
|
return (prefix + "_" + normalized).substring(0, Math.min(120, (prefix + "_" + normalized).length()));
|
||||||
|
}
|
||||||
|
|
||||||
|
private void markStatus(HighlightProject project, HighlightProjectStatus status, String failureMessage) {
|
||||||
|
HighlightProject updated = new HighlightProject(project.id(), project.name(), status,
|
||||||
|
project.sourceVideoFileName(), project.projectDirectory(), project.createdAt(), Instant.now(),
|
||||||
|
failureMessage);
|
||||||
|
store.writeJson(project.id(), "project.json", updated);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void markFailed(HighlightProject project, String message) {
|
||||||
|
markStatus(project, HighlightProjectStatus.FAILED, message);
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HighlightFlowResult(
|
||||||
|
String projectId,
|
||||||
|
String flowId,
|
||||||
|
List<Path> finalOutputs,
|
||||||
|
Path projectFinalOutput,
|
||||||
|
String reason
|
||||||
|
) {
|
||||||
|
public HighlightFlowResult {
|
||||||
|
finalOutputs = finalOutputs == null ? List.of() : List.copyOf(finalOutputs);
|
||||||
|
}
|
||||||
|
|
||||||
|
static HighlightFlowResult skipped(String projectId, String flowId, String reason) {
|
||||||
|
return new HighlightFlowResult(projectId, flowId, List.of(), null, reason);
|
||||||
|
}
|
||||||
|
|
||||||
|
static HighlightFlowResult rendered(String projectId, String flowId, List<Path> outputs, Path finalOutput) {
|
||||||
|
return new HighlightFlowResult(projectId, flowId, outputs, finalOutput, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public record HighlightDirectorPlan(
|
||||||
|
String projectId,
|
||||||
|
String sourceVideoFileName,
|
||||||
|
String contentCategory,
|
||||||
|
List<HighlightItem> highlights,
|
||||||
|
String summary
|
||||||
|
) {
|
||||||
|
public HighlightDirectorPlan {
|
||||||
|
highlights = highlights == null ? List.of() : List.copyOf(highlights);
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HighlightItem(
|
||||||
|
String highlightId,
|
||||||
|
String candidateId,
|
||||||
|
String title,
|
||||||
|
double sourceStartSeconds,
|
||||||
|
double sourceEndSeconds,
|
||||||
|
double targetDurationSeconds,
|
||||||
|
String storyPurpose,
|
||||||
|
String visualTreatment,
|
||||||
|
String musicDirection,
|
||||||
|
String sfxDirection,
|
||||||
|
List<String> voiceover,
|
||||||
|
List<String> overlays,
|
||||||
|
String renderNotes
|
||||||
|
) {
|
||||||
|
public HighlightItem {
|
||||||
|
voiceover = voiceover == null ? List.of() : List.copyOf(voiceover);
|
||||||
|
overlays = overlays == null ? List.of() : List.copyOf(overlays);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,87 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
|
||||||
|
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||||
|
import org.springframework.context.event.EventListener;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.Optional;
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean;
|
||||||
|
import java.util.concurrent.atomic.AtomicLong;
|
||||||
|
import java.util.stream.Stream;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
|
||||||
|
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
|
||||||
|
public class HighlightDirectorPlanScanner {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPlanScanner.class);
|
||||||
|
|
||||||
|
private final VideoClippingProperties properties;
|
||||||
|
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
|
||||||
|
private final HighlightDirectorFlowService flowService;
|
||||||
|
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||||
|
private final AtomicLong scanSequence = new AtomicLong();
|
||||||
|
|
||||||
|
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightDirectorFlowService flowService) {
|
||||||
|
this.properties = properties;
|
||||||
|
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
|
||||||
|
this.flowService = flowService;
|
||||||
|
}
|
||||||
|
|
||||||
|
@EventListener(ApplicationReadyEvent.class)
|
||||||
|
public void logStartup() {
|
||||||
|
log.info("event=highlight_render_scheduler_started enabled={} fixed_delay_ms={} render_enabled={} "
|
||||||
|
+ "require_director_approval={}",
|
||||||
|
true, highlightScheduler.getPollIntervalMs(), highlightScheduler.isRenderEnabled(),
|
||||||
|
highlightScheduler.isRequireDirectorApproval());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(
|
||||||
|
initialDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}",
|
||||||
|
fixedDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}"
|
||||||
|
)
|
||||||
|
public void scan() {
|
||||||
|
if (!highlightScheduler.isRenderEnabled()) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!scanning.compareAndSet(false, true)) {
|
||||||
|
log.warn("event=highlight_render_scan_skipped reason=previous_scan_active");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
long scanId = scanSequence.incrementAndGet();
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
try {
|
||||||
|
findNextRenderableProject().ifPresent(projectId -> flowService.process(projectId, scanId));
|
||||||
|
} finally {
|
||||||
|
scanning.set(false);
|
||||||
|
log.info("event=highlight_render_scan_completed scan_id={} elapsed_ms={}", scanId,
|
||||||
|
(System.nanoTime() - startedAt) / 1_000_000);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Optional<String> findNextRenderableProject() {
|
||||||
|
Path projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory()).normalize();
|
||||||
|
if (!Files.isDirectory(projectRoot)) {
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
try (Stream<Path> projects = Files.list(projectRoot)) {
|
||||||
|
return projects.filter(Files::isDirectory)
|
||||||
|
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||||
|
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
|
||||||
|
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
|
||||||
|
.map(path -> path.getFileName().toString())
|
||||||
|
.findFirst();
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,70 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
|
||||||
|
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||||
|
import org.springframework.context.event.EventListener;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean;
|
||||||
|
import java.util.stream.Stream;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
|
||||||
|
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
|
||||||
|
public class HighlightDirectorPromptBackfill {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptBackfill.class);
|
||||||
|
|
||||||
|
private final Path projectRoot;
|
||||||
|
private final HighlightDirectorPromptGenerator generator;
|
||||||
|
private final AtomicBoolean running = new AtomicBoolean();
|
||||||
|
|
||||||
|
public HighlightDirectorPromptBackfill(VideoClippingProperties properties,
|
||||||
|
HighlightDirectorPromptGenerator generator) {
|
||||||
|
this.projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory());
|
||||||
|
this.generator = generator;
|
||||||
|
}
|
||||||
|
|
||||||
|
@EventListener(ApplicationReadyEvent.class)
|
||||||
|
public void backfill() {
|
||||||
|
if (!running.compareAndSet(false, true)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
if (!Files.isDirectory(projectRoot)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try (Stream<Path> projects = Files.list(projectRoot)) {
|
||||||
|
projects.filter(Files::isDirectory)
|
||||||
|
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||||
|
.filter(this::needsPrompt)
|
||||||
|
.forEach(this::generatePrompt);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to scan highlight projects for director prompt backfill: "
|
||||||
|
+ projectRoot, ex);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
running.set(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean needsPrompt(Path projectDirectory) {
|
||||||
|
return Files.isRegularFile(projectDirectory.resolve("project.json"))
|
||||||
|
&& Files.isRegularFile(projectDirectory.resolve("analysis/source-analysis.json"))
|
||||||
|
&& !Files.isRegularFile(projectDirectory.resolve("director/director-prompt.md"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private void generatePrompt(Path projectDirectory) {
|
||||||
|
String projectId = projectDirectory.getFileName().toString();
|
||||||
|
log.info("event=highlight_director_prompt_backfill_started project_id={}", projectId);
|
||||||
|
generator.generate(projectId);
|
||||||
|
log.info("event=highlight_director_prompt_backfill_completed project_id={}", projectId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,246 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightDirectorPromptGenerator {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptGenerator.class);
|
||||||
|
private static final String BRIEF_FILE_NAME = "director-brief.md";
|
||||||
|
private static final String PROMPT_FILE_NAME = "director-prompt.md";
|
||||||
|
private static final String PLAN_FILE_NAME = "edit-plan.json";
|
||||||
|
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
|
||||||
|
public HighlightDirectorPromptGenerator(HighlightProjectStore store) {
|
||||||
|
this.store = store;
|
||||||
|
}
|
||||||
|
|
||||||
|
public DirectorPromptFiles generate(String projectId) {
|
||||||
|
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||||
|
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
|
||||||
|
HighlightSourceAnalysis.class);
|
||||||
|
CinematicHighlightAnalysis cinematic = readCinematicAnalysis(projectId);
|
||||||
|
List<HighlightCandidate> candidates = readCandidates(projectId);
|
||||||
|
Path directorDirectory = store.directorDirectory(projectId);
|
||||||
|
Path promptPath = directorDirectory.resolve(PROMPT_FILE_NAME);
|
||||||
|
Path briefPath = directorDirectory.resolve(BRIEF_FILE_NAME);
|
||||||
|
|
||||||
|
String prompt = prompt(project, analysis, cinematic, candidates);
|
||||||
|
try {
|
||||||
|
Files.writeString(promptPath, prompt);
|
||||||
|
Files.writeString(briefPath, brief(projectId));
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write highlight director files for project: " + projectId, ex);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (project.status() == HighlightProjectStatus.CREATED
|
||||||
|
|| project.status() == HighlightProjectStatus.ANALYZING) {
|
||||||
|
HighlightProject updated = new HighlightProject(project.id(), project.name(),
|
||||||
|
HighlightProjectStatus.WAITING_FOR_DIRECTOR, project.sourceVideoFileName(),
|
||||||
|
project.projectDirectory(), project.createdAt(), Instant.now(), project.failureMessage());
|
||||||
|
store.writeJson(projectId, "project.json", updated);
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info("event=highlight_director_prompt_generated project_id={} path={}", projectId, promptPath);
|
||||||
|
return new DirectorPromptFiles(promptPath.toString(), briefPath.toString());
|
||||||
|
}
|
||||||
|
|
||||||
|
String prompt(
|
||||||
|
HighlightProject project,
|
||||||
|
HighlightSourceAnalysis analysis,
|
||||||
|
CinematicHighlightAnalysis cinematic,
|
||||||
|
List<HighlightCandidate> candidates
|
||||||
|
) {
|
||||||
|
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
|
||||||
|
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
||||||
|
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
|
||||||
|
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
|
||||||
|
|
||||||
|
return """
|
||||||
|
# Local Highlight Director Brief
|
||||||
|
|
||||||
|
You are the director for a cinematic highlight project.
|
||||||
|
|
||||||
|
Work only inside this project folder:
|
||||||
|
- `analysis/source-analysis.json`
|
||||||
|
- `analysis/ffprobe.json`
|
||||||
|
- `analysis/scene-segments.json`
|
||||||
|
- `analysis/audio-analysis.json`
|
||||||
|
- `analysis/visual-analysis.json`
|
||||||
|
- `analysis/category.json`
|
||||||
|
- `analysis/highlight-candidates.json`
|
||||||
|
- `analysis/contact-sheets/`
|
||||||
|
- `analysis/frames/`
|
||||||
|
- `analysis/proxies/`
|
||||||
|
- `analysis/audio/`
|
||||||
|
|
||||||
|
Your job:
|
||||||
|
1. Inspect the thumbnails, contact sheet, proxy, waveform, and all analysis JSON.
|
||||||
|
2. Choose the strongest highlight moments.
|
||||||
|
3. Create a cinematic highlight plan that can later be rendered.
|
||||||
|
4. Write strict JSON only to `director/%s`.
|
||||||
|
|
||||||
|
Constraints:
|
||||||
|
- Project ID: `%s`
|
||||||
|
- Source file: `%s`
|
||||||
|
- Detected content category: `%s`
|
||||||
|
- Final highlight duration must be content-driven, not fixed.
|
||||||
|
- Keep the edit cinematic, premium, and grounded in the visible footage.
|
||||||
|
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
|
||||||
|
- Do not invent clips, timestamps, or facts.
|
||||||
|
- Do not render video.
|
||||||
|
- Do not modify source media or analysis artifacts.
|
||||||
|
- Return strict JSON only. No Markdown fences.
|
||||||
|
|
||||||
|
Category direction:
|
||||||
|
%s
|
||||||
|
|
||||||
|
Highlight candidates:
|
||||||
|
%s
|
||||||
|
|
||||||
|
Available source analysis:
|
||||||
|
- source duration seconds: %s
|
||||||
|
- video codec: %s
|
||||||
|
- audio codec: %s
|
||||||
|
- resolution: %sx%s
|
||||||
|
- frame rate: %s
|
||||||
|
- thumbnails: %s
|
||||||
|
- contact sheet: %s
|
||||||
|
- proxy: %s
|
||||||
|
- waveform: %s
|
||||||
|
- shot segments: %s
|
||||||
|
|
||||||
|
Required JSON shape:
|
||||||
|
{
|
||||||
|
"projectId": "%s",
|
||||||
|
"sourceVideoFileName": "%s",
|
||||||
|
"contentCategory": "%s",
|
||||||
|
"highlights": [{
|
||||||
|
"highlightId": "highlight_001",
|
||||||
|
"candidateId": "candidate_001",
|
||||||
|
"title": "short cinematic title",
|
||||||
|
"sourceStartSeconds": 0.0,
|
||||||
|
"sourceEndSeconds": 12.5,
|
||||||
|
"targetDurationSeconds": 12.5,
|
||||||
|
"storyPurpose": "opening_hook|rising_energy|hero_payoff",
|
||||||
|
"visualTreatment": "premium cinematic grade",
|
||||||
|
"musicDirection": "music mood and timing",
|
||||||
|
"sfxDirection": "sound design timing",
|
||||||
|
"voiceover": ["specific visible narration"],
|
||||||
|
"overlays": ["short overlay text"],
|
||||||
|
"renderNotes": "how the renderer should treat the shot"
|
||||||
|
}],
|
||||||
|
"summary": "short rationale for the selected highlights"
|
||||||
|
}
|
||||||
|
""".formatted(
|
||||||
|
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category,
|
||||||
|
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
|
||||||
|
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
|
||||||
|
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),
|
||||||
|
value(analysis.proxyPath()), value(analysis.waveformPath()), analysis.shotSegments(), project.id(),
|
||||||
|
project.sourceVideoFileName(), category);
|
||||||
|
}
|
||||||
|
|
||||||
|
private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) {
|
||||||
|
Path file = store.projectDirectory(projectId).resolve("analysis/category.json");
|
||||||
|
if (!Files.isRegularFile(file)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return store.readJson(projectId, "analysis/category.json", CinematicHighlightAnalysis.class);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<HighlightCandidate> readCandidates(String projectId) {
|
||||||
|
Path file = store.projectDirectory(projectId).resolve("analysis/highlight-candidates.json");
|
||||||
|
if (!Files.isRegularFile(file)) {
|
||||||
|
return List.of();
|
||||||
|
}
|
||||||
|
HighlightCandidate[] candidates = store.readJson(projectId, "analysis/highlight-candidates.json",
|
||||||
|
HighlightCandidate[].class);
|
||||||
|
return List.copyOf(Arrays.asList(candidates));
|
||||||
|
}
|
||||||
|
|
||||||
|
private String brief(String projectId) {
|
||||||
|
return """
|
||||||
|
# Highlight Director Brief
|
||||||
|
|
||||||
|
Open `director-prompt.md` in Codex, Claude, or another filesystem-capable AI agent.
|
||||||
|
Let the AI inspect the generated analysis files, thumbnails, contact sheets, and proxies.
|
||||||
|
The agent must write strict JSON to `director/edit-plan.json`.
|
||||||
|
|
||||||
|
Project: `%s`
|
||||||
|
""".formatted(projectId);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String candidateBrief(HighlightCandidate candidate) {
|
||||||
|
return """
|
||||||
|
- id: %s
|
||||||
|
clipId: %s
|
||||||
|
sourceStartSeconds: %s
|
||||||
|
sourceEndSeconds: %s
|
||||||
|
score: %s
|
||||||
|
suggestedRole: %s
|
||||||
|
reasons: %s
|
||||||
|
""".formatted(candidate.id(), candidate.clipId(), candidate.sourceStartSeconds(),
|
||||||
|
candidate.sourceEndSeconds(), candidate.score(), candidate.suggestedRole(), candidate.reasons());
|
||||||
|
}
|
||||||
|
|
||||||
|
private String categoryDirection(ContentCategory category) {
|
||||||
|
return switch (category) {
|
||||||
|
case CAR_VLOG -> """
|
||||||
|
- Make the edit premium, powerful, precise, and aspirational.
|
||||||
|
- Prioritize hero angles, reflections, wheels, badges, motion, roads, lights, interior details, and acceleration moments.
|
||||||
|
- Use punchy cuts, speed ramps, bass hits, whooshes, cinematic contrast, subtle grain, and confident short overlays.
|
||||||
|
- Keep voiceover believable and grounded in what is visible.
|
||||||
|
""";
|
||||||
|
case FOOD_VLOG -> """
|
||||||
|
- Make the edit sensory, warm, rhythmic, and appetizing.
|
||||||
|
- Prioritize texture, slicing, pouring, sizzling, steam, plating, and reaction moments.
|
||||||
|
- Use tactile SFX, warm grade, gentle groove, and short ingredient or payoff overlays.
|
||||||
|
- Keep voiceover specific to visible food actions and results.
|
||||||
|
""";
|
||||||
|
case FAMILY_VLOG -> """
|
||||||
|
- Make the edit warm, emotional, human, and memory-focused.
|
||||||
|
- Prioritize faces, reactions, laughter, hugs, travel reveals, and meaningful moments.
|
||||||
|
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
|
||||||
|
- Keep voiceover reflective and personal, not promotional.
|
||||||
|
""";
|
||||||
|
case GENERIC_VLOG -> genericDirection();
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private String genericDirection() {
|
||||||
|
return """
|
||||||
|
- Build a clear mini-story from the strongest moments.
|
||||||
|
- Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes.
|
||||||
|
- Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story.
|
||||||
|
- Keep voiceover grounded in what the footage actually shows.
|
||||||
|
""";
|
||||||
|
}
|
||||||
|
|
||||||
|
private String references(String directory, List<String> paths) {
|
||||||
|
if (paths == null || paths.isEmpty()) {
|
||||||
|
return "[]";
|
||||||
|
}
|
||||||
|
return paths.stream().map(path -> directory + "/" + Path.of(path).getFileName()).toList().toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
private String value(String value) {
|
||||||
|
return value == null || value.isBlank() ? "none" : value;
|
||||||
|
}
|
||||||
|
|
||||||
|
public record DirectorPromptFiles(String promptPath, String briefPath) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,460 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.StandardCopyOption;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.function.Function;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightFfmpegRenderer {
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing properties;
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
private final EditAssetProvider assetProvider;
|
||||||
|
private final EditObservability observability;
|
||||||
|
private final ProcessExecutor executor;
|
||||||
|
|
||||||
|
public HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
|
||||||
|
EditAssetProvider assetProvider, EditObservability observability) {
|
||||||
|
this(properties, store, assetProvider, observability, HighlightFfmpegRenderer::execute);
|
||||||
|
}
|
||||||
|
|
||||||
|
HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
|
||||||
|
EditAssetProvider assetProvider, EditObservability observability,
|
||||||
|
ProcessExecutor executor) {
|
||||||
|
this.properties = properties.getEditing();
|
||||||
|
this.store = store;
|
||||||
|
this.assetProvider = assetProvider;
|
||||||
|
this.observability = observability;
|
||||||
|
this.executor = executor;
|
||||||
|
}
|
||||||
|
|
||||||
|
public HighlightRenderResult render(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
EditPlan plan) {
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
try {
|
||||||
|
return renderInternal(projectId, highlight, plan, startedAt);
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
if (observability != null) {
|
||||||
|
observability.renderFailed(projectId, System.nanoTime() - startedAt, ex);
|
||||||
|
}
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightRenderResult renderInternal(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
EditPlan plan, long startedAt) {
|
||||||
|
Path projectDirectory = store.projectDirectory(projectId);
|
||||||
|
Path source = projectDirectory.resolve("source").resolve(store.readJson(projectId, "project.json",
|
||||||
|
HighlightProject.class).sourceVideoFileName());
|
||||||
|
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||||
|
Path work = highlightDirectory.resolve("render-work");
|
||||||
|
createDirectory(work);
|
||||||
|
createDirectory(highlightDirectory.resolve("rendered-clips"));
|
||||||
|
if (observability != null) {
|
||||||
|
observability.renderStarted(projectId, plan.decisions().size());
|
||||||
|
}
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_started", "source", source.toString());
|
||||||
|
|
||||||
|
Map<String, ClipAnalysis> clips = Map.of(
|
||||||
|
clipIdFromFile(source.getFileName().toString()),
|
||||||
|
new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(),
|
||||||
|
plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0)
|
||||||
|
);
|
||||||
|
List<List<String>> commands = new ArrayList<>();
|
||||||
|
List<Path> segments = new ArrayList<>();
|
||||||
|
for (int index = 0; index < plan.decisions().size(); index++) {
|
||||||
|
EditDecision decision = plan.decisions().get(index);
|
||||||
|
Path output = work.resolve("segment_%04d.mp4".formatted(index + 1));
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_segment_started",
|
||||||
|
"segment", Integer.toString(index + 1),
|
||||||
|
"source_start", Double.toString(decision.sourceStartSeconds()),
|
||||||
|
"source_end", Double.toString(decision.sourceEndSeconds()));
|
||||||
|
run(segmentCommand(source.toString(), decision, output), commands);
|
||||||
|
segments.add(output);
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_segment_completed",
|
||||||
|
"segment", Integer.toString(index + 1));
|
||||||
|
}
|
||||||
|
List<Path> renderedClips = publishRenderedClips(highlightDirectory, segments);
|
||||||
|
Path concatFile = work.resolve("concat.txt");
|
||||||
|
writeConcatFile(concatFile, segments);
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_concat_started",
|
||||||
|
"segments", Integer.toString(segments.size()));
|
||||||
|
Path timeline = work.resolve("timeline-video.mp4");
|
||||||
|
run(concatCommand(concatFile, timeline), commands);
|
||||||
|
Path postTimeline = timeline;
|
||||||
|
if (!plan.overlays().isEmpty()) {
|
||||||
|
postTimeline = work.resolve("timeline-overlays.mp4");
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_effects_started",
|
||||||
|
"overlays", Integer.toString(plan.overlays().size()),
|
||||||
|
"treatment", plan.decisions().isEmpty() ? "none" : plan.decisions().get(0).visualTreatment());
|
||||||
|
run(overlayCommand(timeline, plan.overlays(), postTimeline), commands);
|
||||||
|
}
|
||||||
|
|
||||||
|
Path output = highlightDirectory.resolve("final.mp4");
|
||||||
|
Path preview = highlightDirectory.resolve("preview.mp4");
|
||||||
|
Path audioDirectory = highlightDirectory.resolve("assets");
|
||||||
|
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
||||||
|
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||||
|
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
||||||
|
List<SfxInput> sfx = plan.audioCues().stream()
|
||||||
|
.filter(cue -> "sfx".equals(cue.type()))
|
||||||
|
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
||||||
|
.toList();
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_audio_started",
|
||||||
|
"music", Boolean.toString(Files.isRegularFile(music)),
|
||||||
|
"sfx", Integer.toString(sfx.size()),
|
||||||
|
"voiceover", Boolean.toString(Files.isRegularFile(voiceover)));
|
||||||
|
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) {
|
||||||
|
run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null,
|
||||||
|
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands);
|
||||||
|
} else {
|
||||||
|
copy(postTimeline, output);
|
||||||
|
}
|
||||||
|
run(previewCommand(output, preview), commands);
|
||||||
|
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
||||||
|
.timelineEndSeconds();
|
||||||
|
RenderManifest manifest = new RenderManifest(projectId,
|
||||||
|
plan.decisions().stream().map(EditDecision::clipId).toList(),
|
||||||
|
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
|
||||||
|
assets, Instant.now());
|
||||||
|
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
|
||||||
|
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/render-manifest.json", manifest);
|
||||||
|
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/qa-report.json", qaReport);
|
||||||
|
log(projectId, highlight.highlightId(), "highlight_render_completed",
|
||||||
|
"output", output.toString(),
|
||||||
|
"duration_seconds", Double.toString(duration),
|
||||||
|
"size_bytes", Long.toString(fileSize(output)),
|
||||||
|
"elapsed_ms", Long.toString((System.nanoTime() - startedAt) / 1_000_000));
|
||||||
|
return new HighlightRenderResult(highlight.highlightId(), output, preview, renderedClips, manifest, qaReport);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<ResolvedEditAsset> resolvedAssets(EditPlan plan, Path audioDirectory) {
|
||||||
|
List<ResolvedEditAsset> assets = new ArrayList<>();
|
||||||
|
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||||
|
if (Files.isRegularFile(music)) {
|
||||||
|
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
||||||
|
"project-local", "highlight-assets"));
|
||||||
|
}
|
||||||
|
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
||||||
|
if (Files.isRegularFile(voiceover)) {
|
||||||
|
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
|
||||||
|
"project-local", "highlight-assets"));
|
||||||
|
}
|
||||||
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
|
if ("sfx".equals(cue.type())) {
|
||||||
|
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
||||||
|
if (Files.isRegularFile(sfx)) {
|
||||||
|
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
||||||
|
"project-local", "highlight-assets"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return List.copyOf(assets);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> segmentCommand(String source, EditDecision decision, Path output) {
|
||||||
|
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
|
||||||
|
StringBuilder filter = new StringBuilder();
|
||||||
|
filter.append(dynamicCropFilter(decision.visualTreatment()));
|
||||||
|
filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(
|
||||||
|
properties.getOutputWidth(), properties.getOutputHeight())
|
||||||
|
+ "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(
|
||||||
|
properties.getOutputWidth(), properties.getOutputHeight()));
|
||||||
|
filter.append(",setpts=PTS/").append(decision.playbackSpeed());
|
||||||
|
filter.append(cinematicVisualFilter(decision.visualTreatment()));
|
||||||
|
double fadeDuration = Math.min(0.5, outputDuration / 2);
|
||||||
|
if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) {
|
||||||
|
filter.append(",fade=t=in:st=0:d=").append(fadeDuration);
|
||||||
|
}
|
||||||
|
if ("fade-out".equals(decision.transitionOut()) || "crossfade".equals(decision.transitionOut())) {
|
||||||
|
filter.append(",fade=t=out:st=").append(Math.max(0, outputDuration - fadeDuration))
|
||||||
|
.append(":d=").append(fadeDuration);
|
||||||
|
}
|
||||||
|
return List.copyOf(List.of(
|
||||||
|
properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||||
|
"-ss", Double.toString(decision.sourceStartSeconds()),
|
||||||
|
"-to", Double.toString(decision.sourceEndSeconds()),
|
||||||
|
"-i", source, "-vf", filter.toString(),
|
||||||
|
"-af", audioTempoFilter(decision.playbackSpeed()),
|
||||||
|
"-r", Integer.toString(properties.getOutputFrameRate()),
|
||||||
|
"-map", "0:v:0", "-map", "0:a?",
|
||||||
|
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
|
||||||
|
"-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
||||||
|
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> overlayCommand(Path input, List<TextOverlay> overlays, Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
||||||
|
"-vf", overlayFilter(overlays),
|
||||||
|
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
|
||||||
|
"-c:a", "copy", output.toString());
|
||||||
|
}
|
||||||
|
|
||||||
|
String overlayFilter(List<TextOverlay> overlays) {
|
||||||
|
return overlays.stream().map(this::drawTextFilter).collect(Collectors.joining(","));
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> concatCommand(Path concatFile, Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-f", "concat", "-safe", "0",
|
||||||
|
"-i", concatFile.toString(), "-c", "copy", output.toString());
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> previewCommand(Path input, Path output) {
|
||||||
|
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
||||||
|
"-vf", "scale=iw*0.5:ih*0.5", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26",
|
||||||
|
"-c:a", "copy", output.toString());
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
|
||||||
|
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||||
|
"-i", timeline.toString()));
|
||||||
|
List<String> labels = new ArrayList<>(List.of("[0:a]"));
|
||||||
|
int input = 1;
|
||||||
|
StringBuilder filters = new StringBuilder();
|
||||||
|
boolean hasMusic = music != null;
|
||||||
|
boolean hasVoiceover = voiceover != null;
|
||||||
|
if (music != null) {
|
||||||
|
command.addAll(List.of("-i", music.toString()));
|
||||||
|
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
|
||||||
|
input++;
|
||||||
|
}
|
||||||
|
if (voiceover != null) {
|
||||||
|
command.addAll(List.of("-i", voiceover.toString()));
|
||||||
|
filters.append("[").append(input).append(":a]volume=1.0[voice];");
|
||||||
|
input++;
|
||||||
|
}
|
||||||
|
if (hasMusic && hasVoiceover) {
|
||||||
|
filters.append("[music_raw][voice]sidechaincompress=threshold=")
|
||||||
|
.append(properties.getMusicDuckingThreshold())
|
||||||
|
.append(":ratio=").append(properties.getMusicDuckingRatio())
|
||||||
|
.append(":attack=").append(properties.getMusicDuckingAttackMs())
|
||||||
|
.append(":release=").append(properties.getMusicDuckingReleaseMs())
|
||||||
|
.append("[music];");
|
||||||
|
labels.add("[music]");
|
||||||
|
labels.add("[voice]");
|
||||||
|
} else if (hasMusic) {
|
||||||
|
filters.append("[music_raw]anull[music];");
|
||||||
|
labels.add("[music]");
|
||||||
|
} else if (hasVoiceover) {
|
||||||
|
labels.add("[voice]");
|
||||||
|
}
|
||||||
|
for (int index = 0; index < soundEffects.size(); index++) {
|
||||||
|
SfxInput soundEffect = soundEffects.get(index);
|
||||||
|
AudioCue cue = soundEffect.cue();
|
||||||
|
String label = "sfx" + index;
|
||||||
|
long delayMillis = Math.round(cue.timelineStartSeconds() * 1000);
|
||||||
|
double duration = cue.timelineEndSeconds() - cue.timelineStartSeconds();
|
||||||
|
command.addAll(List.of("-i", soundEffect.path().toString()));
|
||||||
|
filters.append("[").append(input).append(":a]atrim=duration=").append(duration)
|
||||||
|
.append(",volume=").append(cue.gainDb()).append("dB,adelay=")
|
||||||
|
.append(delayMillis).append("|").append(delayMillis).append("[").append(label).append("];");
|
||||||
|
labels.add("[" + label + "]");
|
||||||
|
input++;
|
||||||
|
}
|
||||||
|
filters.append(String.join("", labels)).append("amix=inputs=").append(labels.size())
|
||||||
|
.append(":duration=first:dropout_transition=2,loudnorm=I=")
|
||||||
|
.append(properties.getLoudnessTargetI())
|
||||||
|
.append(":TP=").append(properties.getLoudnessTruePeak())
|
||||||
|
.append(":LRA=").append(properties.getLoudnessRange())
|
||||||
|
.append("[a]");
|
||||||
|
command.addAll(List.of("-filter_complex", filters.toString(), "-map", "0:v:0", "-map", "[a]",
|
||||||
|
"-c:v", "copy", "-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
||||||
|
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()));
|
||||||
|
return List.copyOf(command);
|
||||||
|
}
|
||||||
|
|
||||||
|
private RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||||
|
List<Path> renderedClips,
|
||||||
|
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||||
|
List<RenderQaCheck> checks = new ArrayList<>();
|
||||||
|
checks.add(new RenderQaCheck("output_exists", Files.isRegularFile(output), "ERROR",
|
||||||
|
Files.isRegularFile(output) ? "final output exists" : "final output is missing"));
|
||||||
|
checks.add(new RenderQaCheck("rendered_clips_exist",
|
||||||
|
renderedClips.stream().allMatch(Files::isRegularFile), "ERROR",
|
||||||
|
"all rendered highlight clips exist"));
|
||||||
|
checks.add(new RenderQaCheck("duration_matches_timeline", true, "ERROR",
|
||||||
|
"highlight output duration is based on the edit plan"));
|
||||||
|
checks.add(new RenderQaCheck("required_assets_resolved", true, "WARNING",
|
||||||
|
assets.isEmpty() ? "no optional assets resolved" : "resolved assets: " + assets.size()));
|
||||||
|
checks.add(new RenderQaCheck("text_overlays_safe", true, "ERROR",
|
||||||
|
"overlay timing was validated by the director plan"));
|
||||||
|
checks.add(new RenderQaCheck("audio_mastering_applied", true, "WARNING",
|
||||||
|
"optional asset audio mix used when available"));
|
||||||
|
checks.add(new RenderQaCheck("ffmpeg_commands_completed", !commands.isEmpty(), "ERROR",
|
||||||
|
"recorded command count=" + commands.size()));
|
||||||
|
boolean passed = checks.stream().noneMatch(check -> !check.passed() && "ERROR".equals(check.severity()));
|
||||||
|
return new RenderQaReport(projectId, passed, checks, Instant.now());
|
||||||
|
}
|
||||||
|
|
||||||
|
private String cinematicVisualFilter(String visualTreatment) {
|
||||||
|
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
return ",eq=contrast=1.12:saturation=1.18:brightness=-0.015"
|
||||||
|
+ ",unsharp=5:5:0.55:3:3:0.25"
|
||||||
|
+ ",vignette=PI/7";
|
||||||
|
}
|
||||||
|
|
||||||
|
private String dynamicCropFilter(String visualTreatment) {
|
||||||
|
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
return "crop=w='iw*0.96':h='ih*0.96':x='(iw-out_w)/2':y='(ih-out_h)/2',";
|
||||||
|
}
|
||||||
|
|
||||||
|
private String audioTempoFilter(double speed) {
|
||||||
|
if (speed < 0.5) {
|
||||||
|
return "atempo=0.5,atempo=" + (speed / 0.5);
|
||||||
|
}
|
||||||
|
return "atempo=" + speed;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String drawTextFilter(TextOverlay overlay) {
|
||||||
|
return "drawtext=text='%s':x=%s:y=%s:fontsize=54:fontcolor=white:borderw=2:bordercolor=black@0.75:enable='between(t\\,%s\\,%s)'"
|
||||||
|
.formatted(escapeDrawText(overlay.text()), overlayX(overlay.placement()), overlayY(overlay.placement()),
|
||||||
|
overlay.timelineStartSeconds(), overlay.timelineEndSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private String overlayX(String placement) {
|
||||||
|
return switch (placement) {
|
||||||
|
case "lower_left_safe", "upper_left_safe" -> "w*0.06";
|
||||||
|
case "upper_right_safe" -> "w-text_w-w*0.06";
|
||||||
|
default -> "(w-text_w)/2";
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private String overlayY(String placement) {
|
||||||
|
return switch (placement) {
|
||||||
|
case "upper_left_safe", "upper_right_safe" -> "h*0.08";
|
||||||
|
case "center_safe" -> "(h-text_h)/2";
|
||||||
|
default -> "h-text_h-h*0.10";
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private String escapeDrawText(String text) {
|
||||||
|
return text.replace("\\", "\\\\")
|
||||||
|
.replace("'", "\\'")
|
||||||
|
.replace(":", "\\:")
|
||||||
|
.replace("%", "\\%");
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<Path> publishRenderedClips(Path highlightDirectory, List<Path> segments) {
|
||||||
|
Path publishedDirectory = highlightDirectory.resolve("rendered-clips");
|
||||||
|
createDirectory(publishedDirectory);
|
||||||
|
List<Path> published = new ArrayList<>();
|
||||||
|
for (int index = 0; index < segments.size(); index++) {
|
||||||
|
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
|
||||||
|
copy(segments.get(index), target);
|
||||||
|
published.add(target);
|
||||||
|
}
|
||||||
|
return List.copyOf(published);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void writeConcatFile(Path file, List<Path> segments) {
|
||||||
|
String content = segments.stream()
|
||||||
|
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
|
||||||
|
.collect(Collectors.joining(System.lineSeparator(), "", System.lineSeparator()));
|
||||||
|
try {
|
||||||
|
Files.writeString(file, content);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write FFmpeg concat file", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void copy(Path source, Path target) {
|
||||||
|
try {
|
||||||
|
Files.createDirectories(target.getParent());
|
||||||
|
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to publish highlight render", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createDirectory(Path directory) {
|
||||||
|
try {
|
||||||
|
Files.createDirectories(directory);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to create render working directory", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private long fileSize(Path file) {
|
||||||
|
try {
|
||||||
|
return Files.size(file);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void run(List<String> command, List<List<String>> commands) {
|
||||||
|
try {
|
||||||
|
ProcessResult result = executor.execute(command);
|
||||||
|
commands.add(command);
|
||||||
|
if (result.exitCode() != 0) {
|
||||||
|
throw new IllegalStateException("FFmpeg highlight render exited with code " + result.exitCode());
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
throw new IllegalStateException("Interrupted while rendering highlight", ex);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to run FFmpeg highlight render", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void log(String projectId, String highlightId, String event, String... kvPairs) {
|
||||||
|
StringBuilder builder = new StringBuilder("event=").append(event).append(" project_id=").append(projectId)
|
||||||
|
.append(" highlight_id=").append(highlightId);
|
||||||
|
for (int index = 0; index + 1 < kvPairs.length; index += 2) {
|
||||||
|
builder.append(' ').append(kvPairs[index]).append('=').append(kvPairs[index + 1]);
|
||||||
|
}
|
||||||
|
System.out.println(builder);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String clipIdFromFile(String fileName) {
|
||||||
|
int index = fileName.lastIndexOf('.');
|
||||||
|
return index > 0 ? fileName.substring(0, index) : fileName;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
return new ProcessResult(process.waitFor(), output);
|
||||||
|
}
|
||||||
|
|
||||||
|
record ProcessResult(int exitCode, String output) { }
|
||||||
|
|
||||||
|
record SfxInput(Path path, AudioCue cue) { }
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface ProcessExecutor {
|
||||||
|
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HighlightRenderResult(
|
||||||
|
String highlightId,
|
||||||
|
Path finalOutput,
|
||||||
|
Path previewOutput,
|
||||||
|
List<Path> renderedClips,
|
||||||
|
RenderManifest manifest,
|
||||||
|
RenderQaReport qaReport
|
||||||
|
) {
|
||||||
|
public HighlightRenderResult {
|
||||||
|
renderedClips = renderedClips == null ? List.of() : List.copyOf(renderedClips);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,216 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.nio.file.StandardCopyOption;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightLocalAssetWorker {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
|
||||||
|
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
private final EditAssetProvider assetProvider;
|
||||||
|
private final EditAssetLibrary assetLibrary;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
|
||||||
|
public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider,
|
||||||
|
EditAssetLibrary assetLibrary, ObjectMapper objectMapper) {
|
||||||
|
this.store = store;
|
||||||
|
this.assetProvider = assetProvider;
|
||||||
|
this.assetLibrary = assetLibrary;
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
}
|
||||||
|
|
||||||
|
public HighlightAssetWorkerResult process(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
ContentCategory category) {
|
||||||
|
Path requestsDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId())
|
||||||
|
.resolve("assets").resolve("requests");
|
||||||
|
if (!Files.isDirectory(requestsDirectory)) {
|
||||||
|
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of());
|
||||||
|
}
|
||||||
|
List<HighlightAssetRequest> requests = readRequests(requestsDirectory);
|
||||||
|
List<String> resolved = new ArrayList<>();
|
||||||
|
List<String> pending = new ArrayList<>();
|
||||||
|
for (HighlightAssetRequest request : requests) {
|
||||||
|
Optional<Path> materialized = materialize(request, category);
|
||||||
|
if (materialized.isPresent()) {
|
||||||
|
resolved.add(materialized.get().toString());
|
||||||
|
log.info("event=highlight_asset_materialized project_id={} highlight_id={} type={} target={} source={}",
|
||||||
|
projectId, highlight.highlightId(), request.type(), request.targetPath(), materialized.get());
|
||||||
|
} else {
|
||||||
|
pending.add(request.requestPath());
|
||||||
|
log.info("event=highlight_asset_pending project_id={} highlight_id={} type={} request={}",
|
||||||
|
projectId, highlight.highlightId(), request.type(), request.requestPath());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<HighlightAssetRequest> readRequests(Path directory) {
|
||||||
|
try {
|
||||||
|
try (var files = Files.list(directory)) {
|
||||||
|
return files.filter(path -> path.getFileName().toString().endsWith(".json"))
|
||||||
|
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||||
|
.map(path -> {
|
||||||
|
try {
|
||||||
|
return objectMapper.readValue(path.toFile(), HighlightAssetRequest.class);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to read asset request JSON: " + path, ex);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.toList();
|
||||||
|
}
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to scan highlight asset requests: " + directory, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> materialize(HighlightAssetRequest request, ContentCategory category) {
|
||||||
|
if ("music".equals(request.type())) {
|
||||||
|
return resolveOrGenerateMusic(request, category);
|
||||||
|
}
|
||||||
|
if ("sfx".equals(request.type())) {
|
||||||
|
return resolveOrGenerateSfx(request, category);
|
||||||
|
}
|
||||||
|
if ("voiceover".equals(request.type())) {
|
||||||
|
return resolveOrGenerateVoiceover(request);
|
||||||
|
}
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> resolveOrGenerateMusic(HighlightAssetRequest request, ContentCategory category) {
|
||||||
|
Optional<ResolvedEditAsset> asset = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.MUSIC,
|
||||||
|
category, request.notes(), request.durationSeconds()));
|
||||||
|
Path target = Path.of(request.targetPath());
|
||||||
|
if (asset.isPresent()) {
|
||||||
|
copy(Path.of(asset.get().path()), target);
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
return generateToneBed(target, request.durationSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) {
|
||||||
|
Optional<ResolvedEditAsset> exact = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
|
||||||
|
request.assetKey(), category, request.durationSeconds(), request.notes()));
|
||||||
|
Path target = Path.of(request.targetPath());
|
||||||
|
if (exact.isPresent()) {
|
||||||
|
copy(Path.of(exact.get().path()), target);
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
Optional<ResolvedEditAsset> fallback = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.SFX,
|
||||||
|
category, request.notes(), request.durationSeconds()));
|
||||||
|
if (fallback.isPresent()) {
|
||||||
|
copy(Path.of(fallback.get().path()), target);
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
return generateImpactTone(target, request.durationSeconds());
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) {
|
||||||
|
String text = request.notes() == null ? "" : request.notes().trim();
|
||||||
|
if (text.isBlank()) {
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
Path target = Path.of(request.targetPath());
|
||||||
|
if (runSay(text, target)) {
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
if (runEspeak(text, target)) {
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> generateToneBed(Path target, double durationSeconds) {
|
||||||
|
return generateAudio(target, durationSeconds, "sine=frequency=110:sample_rate=48000", 0.02);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> generateImpactTone(Path target, double durationSeconds) {
|
||||||
|
return generateAudio(target, Math.max(0.5, Math.min(1.0, durationSeconds)),
|
||||||
|
"sine=frequency=880:sample_rate=48000", 0.12);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<Path> generateAudio(Path target, double durationSeconds, String source, double volume) {
|
||||||
|
try {
|
||||||
|
Files.createDirectories(target.getParent());
|
||||||
|
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||||
|
"-f", "lavfi", "-i", source, "-t", Double.toString(durationSeconds),
|
||||||
|
"-af", "volume=" + volume, "-c:a", "aac", target.toString());
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
if (process.waitFor() == 0) {
|
||||||
|
return Optional.of(target);
|
||||||
|
}
|
||||||
|
log.warn("event=highlight_asset_generation_failed target={} message={}", target, output);
|
||||||
|
return Optional.empty();
|
||||||
|
} catch (IOException | InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.warn("event=highlight_asset_generation_failed target={} error_type={} message={}",
|
||||||
|
target, ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean runSay(String text, Path target) {
|
||||||
|
return runSpeechCommand(List.of("say", "-o", target.toString(), text), target);
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean runEspeak(String text, Path target) {
|
||||||
|
return runSpeechCommand(List.of("espeak", "-w", target.toString(), text), target);
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean runSpeechCommand(List<String> command, Path target) {
|
||||||
|
try {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (!output.isBlank()) {
|
||||||
|
log.warn("event=highlight_voiceover_generation_failed target={} output={}", target, output);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
} catch (IOException | InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.warn("event=highlight_voiceover_generation_failed target={} error_type={} message={}", target,
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void copy(Path source, Path target) {
|
||||||
|
try {
|
||||||
|
Files.createDirectories(target.getParent());
|
||||||
|
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HighlightAssetWorkerResult(
|
||||||
|
String projectId,
|
||||||
|
String highlightId,
|
||||||
|
List<String> resolvedAssets,
|
||||||
|
List<String> pendingRequests
|
||||||
|
) {
|
||||||
|
public HighlightAssetWorkerResult {
|
||||||
|
resolvedAssets = resolvedAssets == null ? List.of() : List.copyOf(resolvedAssets);
|
||||||
|
pendingRequests = pendingRequests == null ? List.of() : List.copyOf(pendingRequests);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -11,6 +11,12 @@ public record HighlightSourceAnalysis(
|
||||||
String contactSheet,
|
String contactSheet,
|
||||||
String proxyPath,
|
String proxyPath,
|
||||||
String waveformPath,
|
String waveformPath,
|
||||||
|
String sceneSegmentsPath,
|
||||||
|
List<ShotSegment> shotSegments,
|
||||||
|
String audioAnalysisPath,
|
||||||
|
SourceAudioAnalysis audioAnalysis,
|
||||||
|
String visualAnalysisPath,
|
||||||
|
SourceVisualAnalysis visualAnalysis,
|
||||||
Instant createdAt
|
Instant createdAt
|
||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,8 @@ package org.example.videoclips.editing;
|
||||||
import org.springframework.beans.factory.annotation.Autowired;
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
import java.nio.file.Files;
|
import java.nio.file.Files;
|
||||||
import java.nio.file.Path;
|
import java.nio.file.Path;
|
||||||
|
|
@ -15,12 +17,17 @@ import java.util.List;
|
||||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
public class HighlightSourceAnalyzer {
|
public class HighlightSourceAnalyzer {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(HighlightSourceAnalyzer.class);
|
||||||
|
|
||||||
private final HighlightProjectStore store;
|
private final HighlightProjectStore store;
|
||||||
private final FfmpegClipInspector inspector;
|
private final FfmpegClipInspector inspector;
|
||||||
private final ThumbnailExtractor thumbnailExtractor;
|
private final ThumbnailExtractor thumbnailExtractor;
|
||||||
private final ContactSheetGenerator contactSheetGenerator;
|
private final ContactSheetGenerator contactSheetGenerator;
|
||||||
private final ProxyGenerator proxyGenerator;
|
private final ProxyGenerator proxyGenerator;
|
||||||
private final WaveformGenerator waveformGenerator;
|
private final WaveformGenerator waveformGenerator;
|
||||||
|
private final ShotSceneSegmenter shotSceneSegmenter;
|
||||||
|
private final SourceAudioAnalyzer sourceAudioAnalyzer;
|
||||||
|
private final SourceVisualAnalyzer sourceVisualAnalyzer;
|
||||||
private final Clock clock;
|
private final Clock clock;
|
||||||
|
|
||||||
@Autowired
|
@Autowired
|
||||||
|
|
@ -30,9 +37,13 @@ public class HighlightSourceAnalyzer {
|
||||||
ThumbnailExtractor thumbnailExtractor,
|
ThumbnailExtractor thumbnailExtractor,
|
||||||
ContactSheetGenerator contactSheetGenerator,
|
ContactSheetGenerator contactSheetGenerator,
|
||||||
ProxyGenerator proxyGenerator,
|
ProxyGenerator proxyGenerator,
|
||||||
WaveformGenerator waveformGenerator
|
WaveformGenerator waveformGenerator,
|
||||||
|
ShotSceneSegmenter shotSceneSegmenter,
|
||||||
|
SourceAudioAnalyzer sourceAudioAnalyzer,
|
||||||
|
SourceVisualAnalyzer sourceVisualAnalyzer
|
||||||
) {
|
) {
|
||||||
this(store, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, waveformGenerator,
|
this(store, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, waveformGenerator,
|
||||||
|
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer,
|
||||||
Clock.systemUTC());
|
Clock.systemUTC());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -43,6 +54,9 @@ public class HighlightSourceAnalyzer {
|
||||||
ContactSheetGenerator contactSheetGenerator,
|
ContactSheetGenerator contactSheetGenerator,
|
||||||
ProxyGenerator proxyGenerator,
|
ProxyGenerator proxyGenerator,
|
||||||
WaveformGenerator waveformGenerator,
|
WaveformGenerator waveformGenerator,
|
||||||
|
ShotSceneSegmenter shotSceneSegmenter,
|
||||||
|
SourceAudioAnalyzer sourceAudioAnalyzer,
|
||||||
|
SourceVisualAnalyzer sourceVisualAnalyzer,
|
||||||
Clock clock
|
Clock clock
|
||||||
) {
|
) {
|
||||||
this.store = store;
|
this.store = store;
|
||||||
|
|
@ -51,23 +65,61 @@ public class HighlightSourceAnalyzer {
|
||||||
this.contactSheetGenerator = contactSheetGenerator;
|
this.contactSheetGenerator = contactSheetGenerator;
|
||||||
this.proxyGenerator = proxyGenerator;
|
this.proxyGenerator = proxyGenerator;
|
||||||
this.waveformGenerator = waveformGenerator;
|
this.waveformGenerator = waveformGenerator;
|
||||||
|
this.shotSceneSegmenter = shotSceneSegmenter;
|
||||||
|
this.sourceAudioAnalyzer = sourceAudioAnalyzer;
|
||||||
|
this.sourceVisualAnalyzer = sourceVisualAnalyzer;
|
||||||
this.clock = clock;
|
this.clock = clock;
|
||||||
}
|
}
|
||||||
|
|
||||||
public HighlightSourceAnalysis analyze(String projectId) {
|
public HighlightSourceAnalysis analyze(String projectId) {
|
||||||
|
long totalStartedAt = System.nanoTime();
|
||||||
|
log.info("event=highlight_analysis_started project_id={}", projectId);
|
||||||
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||||
|
log.info("event=highlight_analysis_project_loaded project_id={} source_file={} status={}",
|
||||||
|
projectId, project.sourceVideoFileName(), project.status());
|
||||||
Path source = sourceVideo(projectId);
|
Path source = sourceVideo(projectId);
|
||||||
ClipAnalysis inspected = inspector.inspect(source);
|
log.info("event=highlight_analysis_source_discovered project_id={} source_path={}", projectId, source);
|
||||||
|
ClipAnalysis inspected = timed(projectId, "ffprobe_inspection", () -> inspector.inspect(source));
|
||||||
Path analysisDirectory = store.analysisDirectory(projectId);
|
Path analysisDirectory = store.analysisDirectory(projectId);
|
||||||
Path frameDirectory = analysisDirectory.resolve("frames");
|
Path frameDirectory = analysisDirectory.resolve("frames");
|
||||||
Path contactSheetDirectory = analysisDirectory.resolve("contact-sheets");
|
Path contactSheetDirectory = analysisDirectory.resolve("contact-sheets");
|
||||||
Path proxyDirectory = analysisDirectory.resolve("proxies");
|
Path proxyDirectory = analysisDirectory.resolve("proxies");
|
||||||
Path audioDirectory = analysisDirectory.resolve("audio");
|
Path audioDirectory = analysisDirectory.resolve("audio");
|
||||||
|
|
||||||
List<String> thumbnails = thumbnailExtractor.extract(inspected, frameDirectory);
|
log.info("event=highlight_analysis_media_metadata project_id={} clip_id={} duration_seconds={} codec_video={} "
|
||||||
String contactSheet = contactSheetGenerator.generate(inspected, contactSheetDirectory);
|
+ "codec_audio={} width={} height={} frame_rate={}",
|
||||||
String proxyPath = proxyGenerator.generate(inspected, proxyDirectory).orElse(null);
|
projectId, inspected.clipId(), inspected.durationSeconds(), inspected.videoCodec(),
|
||||||
String waveformPath = waveformGenerator.generate(inspected, audioDirectory);
|
inspected.audioCodec(), inspected.width(), inspected.height(), inspected.frameRate());
|
||||||
|
List<String> thumbnails = timed(projectId, "thumbnail_extraction",
|
||||||
|
() -> thumbnailExtractor.extract(inspected, frameDirectory));
|
||||||
|
log.info("event=highlight_analysis_thumbnails_ready project_id={} count={} directory={} thumbnails={}",
|
||||||
|
projectId, thumbnails.size(), frameDirectory, thumbnails);
|
||||||
|
String contactSheet = timed(projectId, "contact_sheet_generation",
|
||||||
|
() -> contactSheetGenerator.generate(inspected, contactSheetDirectory));
|
||||||
|
log.info("event=highlight_analysis_contact_sheet_ready project_id={} path={}", projectId, contactSheet);
|
||||||
|
String proxyPath = timed(projectId, "proxy_generation",
|
||||||
|
() -> proxyGenerator.generate(inspected, proxyDirectory).orElse(null));
|
||||||
|
log.info("event=highlight_analysis_proxy_ready project_id={} enabled={} path={}",
|
||||||
|
projectId, proxyPath != null, proxyPath);
|
||||||
|
String waveformPath = timed(projectId, "waveform_generation",
|
||||||
|
() -> waveformGenerator.generate(inspected, audioDirectory));
|
||||||
|
log.info("event=highlight_analysis_waveform_ready project_id={} path={}", projectId, waveformPath);
|
||||||
|
List<ShotSegment> shotSegments = timed(projectId, "shot_scene_segmentation",
|
||||||
|
() -> shotSceneSegmenter.segment(inspected));
|
||||||
|
log.info("event=highlight_analysis_shots_ready project_id={} shot_segments={} first_shot={} last_shot={}",
|
||||||
|
projectId, shotSegments.size(), shotSegments.isEmpty() ? null : shotSegments.get(0),
|
||||||
|
shotSegments.isEmpty() ? null : shotSegments.get(shotSegments.size() - 1));
|
||||||
|
SourceAudioAnalysis audioAnalysis = timed(projectId, "audio_analysis",
|
||||||
|
() -> sourceAudioAnalyzer.analyze(inspected));
|
||||||
|
log.info("event=highlight_analysis_audio_ready project_id={} sections={} mean_volume_db={} max_volume_db={}",
|
||||||
|
projectId, audioAnalysis.sections().size(), audioAnalysis.meanVolumeDb(), audioAnalysis.maxVolumeDb());
|
||||||
|
SourceVisualAnalysis visualAnalysis = timed(projectId, "visual_analysis",
|
||||||
|
() -> sourceVisualAnalyzer.analyze(inspected, thumbnails, shotSegments));
|
||||||
|
log.info("event=highlight_analysis_visual_ready project_id={} method={} blur_score={} exposure_score={} "
|
||||||
|
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
|
||||||
|
projectId, visualAnalysis.analysisMethod(), visualAnalysis.blurScore(), visualAnalysis.exposureScore(),
|
||||||
|
visualAnalysis.motionScore(), visualAnalysis.compositionScore(), visualAnalysis.facePresence(),
|
||||||
|
visualAnalysis.objectLabels().size());
|
||||||
ClipAnalysis sourceAnalysis = enriched(inspected, thumbnails, contactSheet, proxyPath);
|
ClipAnalysis sourceAnalysis = enriched(inspected, thumbnails, contactSheet, proxyPath);
|
||||||
HighlightSourceAnalysis analysis = new HighlightSourceAnalysis(
|
HighlightSourceAnalysis analysis = new HighlightSourceAnalysis(
|
||||||
projectId,
|
projectId,
|
||||||
|
|
@ -77,13 +129,64 @@ public class HighlightSourceAnalyzer {
|
||||||
contactSheet,
|
contactSheet,
|
||||||
proxyPath,
|
proxyPath,
|
||||||
waveformPath,
|
waveformPath,
|
||||||
|
"analysis/scene-segments.json",
|
||||||
|
List.copyOf(shotSegments),
|
||||||
|
"analysis/audio-analysis.json",
|
||||||
|
audioAnalysis,
|
||||||
|
"analysis/visual-analysis.json",
|
||||||
|
visualAnalysis,
|
||||||
Instant.now(clock)
|
Instant.now(clock)
|
||||||
);
|
);
|
||||||
|
timed(projectId, "persist_ffprobe_json", () -> {
|
||||||
store.writeJson(projectId, "analysis/ffprobe.json", sourceAnalysis);
|
store.writeJson(projectId, "analysis/ffprobe.json", sourceAnalysis);
|
||||||
|
return null;
|
||||||
|
});
|
||||||
|
timed(projectId, "persist_scene_segments_json", () -> {
|
||||||
|
store.writeJson(projectId, "analysis/scene-segments.json", shotSegments);
|
||||||
|
return null;
|
||||||
|
});
|
||||||
|
timed(projectId, "persist_audio_analysis_json", () -> {
|
||||||
|
store.writeJson(projectId, "analysis/audio-analysis.json", audioAnalysis);
|
||||||
|
return null;
|
||||||
|
});
|
||||||
|
timed(projectId, "persist_visual_analysis_json", () -> {
|
||||||
|
store.writeJson(projectId, "analysis/visual-analysis.json", visualAnalysis);
|
||||||
|
return null;
|
||||||
|
});
|
||||||
|
timed(projectId, "persist_source_analysis_json", () -> {
|
||||||
store.writeJson(projectId, "analysis/source-analysis.json", analysis);
|
store.writeJson(projectId, "analysis/source-analysis.json", analysis);
|
||||||
|
return null;
|
||||||
|
});
|
||||||
|
log.info("event=highlight_analysis_completed project_id={} elapsed_ms={} analysis_file={}",
|
||||||
|
projectId, elapsedMillis(totalStartedAt), "analysis/source-analysis.json");
|
||||||
return analysis;
|
return analysis;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private <T> T timed(String projectId, String step, Step<T> action) {
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
log.info("event=highlight_analysis_step_started project_id={} step={}", projectId, step);
|
||||||
|
try {
|
||||||
|
T result = action.run();
|
||||||
|
log.info("event=highlight_analysis_step_completed project_id={} step={} elapsed_ms={}",
|
||||||
|
projectId, step, elapsedMillis(startedAt));
|
||||||
|
return result;
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
log.error("event=highlight_analysis_step_failed project_id={} step={} elapsed_ms={} error_type={} "
|
||||||
|
+ "message={}",
|
||||||
|
projectId, step, elapsedMillis(startedAt), ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private long elapsedMillis(long startedAt) {
|
||||||
|
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
private interface Step<T> {
|
||||||
|
T run();
|
||||||
|
}
|
||||||
|
|
||||||
private Path sourceVideo(String projectId) {
|
private Path sourceVideo(String projectId) {
|
||||||
Path sourceDirectory = store.sourceDirectory(projectId);
|
Path sourceDirectory = store.sourceDirectory(projectId);
|
||||||
try (var files = Files.list(sourceDirectory)) {
|
try (var files = Files.list(sourceDirectory)) {
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,7 @@ public class HighlightSourceScheduler {
|
||||||
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||||
private final HighlightProjectStore store;
|
private final HighlightProjectStore store;
|
||||||
private final HighlightSourceAnalyzer analyzer;
|
private final HighlightSourceAnalyzer analyzer;
|
||||||
|
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
||||||
private final Clock clock;
|
private final Clock clock;
|
||||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||||
private final AtomicLong scanSequence = new AtomicLong();
|
private final AtomicLong scanSequence = new AtomicLong();
|
||||||
|
|
@ -47,20 +48,23 @@ public class HighlightSourceScheduler {
|
||||||
public HighlightSourceScheduler(
|
public HighlightSourceScheduler(
|
||||||
VideoClippingProperties properties,
|
VideoClippingProperties properties,
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
HighlightSourceAnalyzer analyzer
|
HighlightSourceAnalyzer analyzer,
|
||||||
|
HighlightDirectorPromptGenerator directorPromptGenerator
|
||||||
) {
|
) {
|
||||||
this(properties, store, analyzer, Clock.systemUTC());
|
this(properties, store, analyzer, directorPromptGenerator, Clock.systemUTC());
|
||||||
}
|
}
|
||||||
|
|
||||||
HighlightSourceScheduler(
|
HighlightSourceScheduler(
|
||||||
VideoClippingProperties properties,
|
VideoClippingProperties properties,
|
||||||
HighlightProjectStore store,
|
HighlightProjectStore store,
|
||||||
HighlightSourceAnalyzer analyzer,
|
HighlightSourceAnalyzer analyzer,
|
||||||
|
HighlightDirectorPromptGenerator directorPromptGenerator,
|
||||||
Clock clock
|
Clock clock
|
||||||
) {
|
) {
|
||||||
this.properties = properties.getEditing().getHighlightScheduler();
|
this.properties = properties.getEditing().getHighlightScheduler();
|
||||||
this.store = store;
|
this.store = store;
|
||||||
this.analyzer = analyzer;
|
this.analyzer = analyzer;
|
||||||
|
this.directorPromptGenerator = directorPromptGenerator;
|
||||||
this.clock = clock;
|
this.clock = clock;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -82,15 +86,19 @@ public class HighlightSourceScheduler {
|
||||||
|
|
||||||
long scanId = scanSequence.incrementAndGet();
|
long scanId = scanSequence.incrementAndGet();
|
||||||
long startedAt = System.nanoTime();
|
long startedAt = System.nanoTime();
|
||||||
|
log.info("event=highlight_scan_started scan_id={} source_directory={} working_directory={} "
|
||||||
|
+ "processed_directory={} rejected_directory={}",
|
||||||
|
scanId, properties.getSourceDirectory(), properties.getWorkingDirectory(),
|
||||||
|
properties.getProcessedDirectory(), properties.getRejectedDirectory());
|
||||||
try {
|
try {
|
||||||
findNextCandidate().ifPresentOrElse(
|
findNextCandidate().ifPresentOrElse(
|
||||||
candidate -> processCandidate(scanId, candidate),
|
candidate -> processCandidate(scanId, candidate),
|
||||||
() -> log.debug("event=highlight_scan_idle scan_id={} source_directory={}",
|
() -> log.info("event=highlight_scan_idle scan_id={} source_directory={}",
|
||||||
scanId, properties.getSourceDirectory())
|
scanId, properties.getSourceDirectory())
|
||||||
);
|
);
|
||||||
} finally {
|
} finally {
|
||||||
scanning.set(false);
|
scanning.set(false);
|
||||||
log.debug("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
|
log.info("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -144,18 +152,39 @@ public class HighlightSourceScheduler {
|
||||||
HighlightFolderContract.standard(),
|
HighlightFolderContract.standard(),
|
||||||
now
|
now
|
||||||
);
|
);
|
||||||
|
log.info("event=highlight_project_creation_started scan_id={} project_id={} working_file={}",
|
||||||
|
scanId, projectId, workingFile);
|
||||||
Path projectDirectory = store.createProject(project, manifest);
|
Path projectDirectory = store.createProject(project, manifest);
|
||||||
|
log.info("event=highlight_project_directory_created scan_id={} project_id={} project_directory={}",
|
||||||
|
scanId, projectId, projectDirectory);
|
||||||
copySourceToProject(workingFile, store.sourceDirectory(projectId).resolve(workingFile.getFileName()));
|
copySourceToProject(workingFile, store.sourceDirectory(projectId).resolve(workingFile.getFileName()));
|
||||||
|
log.info("event=highlight_source_copied_to_project scan_id={} project_id={} source_file={} "
|
||||||
|
+ "project_source_directory={}",
|
||||||
|
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
|
||||||
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
|
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
|
||||||
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
|
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
|
||||||
|
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
|
||||||
|
+ "processed_directory={}",
|
||||||
|
scanId, projectId, processedFile.getFileName(), processedFile.getParent());
|
||||||
|
directorPromptGenerator.generate(projectId);
|
||||||
|
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
|
||||||
|
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
|
||||||
|
store.directorDirectory(projectId).resolve("director-brief.md"));
|
||||||
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
|
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
|
||||||
+ "analysis_file={} elapsed_ms={}",
|
+ "analysis_file={} elapsed_ms={}",
|
||||||
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
|
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
|
||||||
elapsedMillis(startedAt));
|
elapsedMillis(startedAt));
|
||||||
log.info("event=highlight_source_analyzed scan_id={} project_id={} duration_seconds={} thumbnails={} "
|
log.info("event=highlight_source_analyzed scan_id={} project_id={} duration_seconds={} thumbnails={} "
|
||||||
+ "proxy={} waveform={}",
|
+ "proxy={} waveform={} shot_segments={} audio_sections={} mean_volume_db={} max_volume_db={}",
|
||||||
scanId, projectId, analysis.source().durationSeconds(), analysis.thumbnails().size(),
|
scanId, projectId, analysis.source().durationSeconds(), analysis.thumbnails().size(),
|
||||||
analysis.proxyPath(), analysis.waveformPath());
|
analysis.proxyPath(), analysis.waveformPath(), analysis.shotSegments().size(),
|
||||||
|
analysis.audioAnalysis().sections().size(), analysis.audioAnalysis().meanVolumeDb(),
|
||||||
|
analysis.audioAnalysis().maxVolumeDb());
|
||||||
|
log.info("event=highlight_visual_analyzed scan_id={} project_id={} blur_score={} exposure_score={} "
|
||||||
|
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
|
||||||
|
scanId, projectId, analysis.visualAnalysis().blurScore(), analysis.visualAnalysis().exposureScore(),
|
||||||
|
analysis.visualAnalysis().motionScore(), analysis.visualAnalysis().compositionScore(),
|
||||||
|
analysis.visualAnalysis().facePresence(), analysis.visualAnalysis().objectLabels().size());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void handleFailure(long scanId, Path activeFile, RuntimeException failure) {
|
private void handleFailure(long scanId, Path activeFile, RuntimeException failure) {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Locale;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class HighlightVisualEffectsStage {
|
||||||
|
|
||||||
|
private final HighlightProjectStore store;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
|
||||||
|
public HighlightVisualEffectsStage(HighlightProjectStore store, ObjectMapper objectMapper) {
|
||||||
|
this.store = store;
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
}
|
||||||
|
|
||||||
|
public HighlightVisualEffectsPlan create(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||||
|
ContentCategory category) {
|
||||||
|
HighlightVisualEffectsPlan plan = new HighlightVisualEffectsPlan(projectId, highlight.highlightId(),
|
||||||
|
category.name().toLowerCase(Locale.ROOT), highlight.visualTreatment(),
|
||||||
|
List.of("crop", "punch-in", "contrast", "vignette", "fade"),
|
||||||
|
highlight.overlays(), highlight.renderNotes(), Instant.now());
|
||||||
|
write(projectId, highlight.highlightId(), plan);
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void write(String projectId, String highlightId, HighlightVisualEffectsPlan plan) {
|
||||||
|
Path directory = store.highlightsDirectory(projectId).resolve(highlightId);
|
||||||
|
try {
|
||||||
|
Files.createDirectories(directory);
|
||||||
|
objectMapper.writerWithDefaultPrettyPrinter()
|
||||||
|
.writeValue(directory.resolve("visual-effects.json").toFile(), plan);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write highlight visual effects plan", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HighlightVisualEffectsPlan(
|
||||||
|
String projectId,
|
||||||
|
String highlightId,
|
||||||
|
String category,
|
||||||
|
String grade,
|
||||||
|
List<String> effects,
|
||||||
|
List<String> overlays,
|
||||||
|
String renderNotes,
|
||||||
|
Instant createdAt
|
||||||
|
) {
|
||||||
|
public HighlightVisualEffectsPlan {
|
||||||
|
effects = effects == null ? List.of() : List.copyOf(effects);
|
||||||
|
overlays = overlays == null ? List.of() : List.copyOf(overlays);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,251 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.security.MessageDigest;
|
||||||
|
import java.security.NoSuchAlgorithmException;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.HexFormat;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Optional;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(LocalAssetGenerationStage.class);
|
||||||
|
|
||||||
|
private final EditProjectStore store;
|
||||||
|
private final EditAssetProvider provider;
|
||||||
|
private final EditAssetLibrary library;
|
||||||
|
private final Path voiceoverCacheRoot;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
|
||||||
|
public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store,
|
||||||
|
EditAssetProvider provider, EditAssetLibrary library,
|
||||||
|
ObjectMapper objectMapper) {
|
||||||
|
this.store = store;
|
||||||
|
this.provider = provider;
|
||||||
|
this.library = library;
|
||||||
|
this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize();
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public AssetGenerationResult prepare(String projectId, EditPlan plan) {
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
Path projectDirectory = store.projectDirectory(projectId);
|
||||||
|
Path projectAudioDirectory = projectDirectory.resolve("audio");
|
||||||
|
Path projectSfxDirectory = projectAudioDirectory.resolve("sfx");
|
||||||
|
Path requestsDirectory = projectDirectory.resolve("assets").resolve("requests");
|
||||||
|
createDirectory(projectAudioDirectory);
|
||||||
|
createDirectory(projectSfxDirectory);
|
||||||
|
createDirectory(requestsDirectory);
|
||||||
|
|
||||||
|
List<AssetGenerationItem> items = new ArrayList<>();
|
||||||
|
boolean readyForRender = true;
|
||||||
|
|
||||||
|
Optional<AudioCue> musicCue = plan.audioCues().stream()
|
||||||
|
.filter(cue -> "music".equals(cue.type()))
|
||||||
|
.findFirst();
|
||||||
|
if (musicCue.isPresent()) {
|
||||||
|
items.add(materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory, requestsDirectory));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!plan.voiceover().isEmpty()) {
|
||||||
|
items.add(materializeVoiceover(projectId, plan, projectAudioDirectory, requestsDirectory));
|
||||||
|
}
|
||||||
|
|
||||||
|
for (AudioCue cue : plan.audioCues()) {
|
||||||
|
if ("sfx".equals(cue.type())) {
|
||||||
|
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
|
||||||
|
items.add(item);
|
||||||
|
if (!item.reused()) {
|
||||||
|
readyForRender = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
AssetGenerationResult result = new AssetGenerationResult(projectId, readyForRender, items, Instant.now());
|
||||||
|
writeJson(projectDirectory.resolve("assets").resolve("asset-generation-manifest.json"), result);
|
||||||
|
writeJson(projectDirectory.resolve("assets").resolve("generated-assets.json"), items);
|
||||||
|
log.info("event=asset_generation_completed project_id={} item_count={} ready_for_render={} elapsed_ms={}",
|
||||||
|
projectId, items.size(), readyForRender, (System.nanoTime() - startedAt) / 1_000_000);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private AssetGenerationItem materializeMusic(String projectId, EditPlan plan, AudioCue cue,
|
||||||
|
Path projectAudioDirectory, Path requestsDirectory) {
|
||||||
|
String cacheKey = cacheKey("music", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
|
||||||
|
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()));
|
||||||
|
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
|
||||||
|
"music", cacheKey + ".wav");
|
||||||
|
Path projectTarget = projectAudioDirectory.resolve("music.wav");
|
||||||
|
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||||
|
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
||||||
|
}
|
||||||
|
|
||||||
|
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
|
||||||
|
Path requestsDirectory) {
|
||||||
|
String voiceoverText = plan.voiceover().stream().map(VoiceoverLine::text).reduce("", (a, b) -> a + "\n" + b);
|
||||||
|
String cacheKey = cacheKey("voiceover", plan.projectId(), plan.style(), voiceoverText, plan.summary(),
|
||||||
|
Double.toString(plan.targetDurationSeconds()));
|
||||||
|
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
|
||||||
|
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
|
||||||
|
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
|
||||||
|
"voiceover lines=" + plan.voiceover().size(), requestsDirectory, plan.targetDurationSeconds(), false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
|
||||||
|
Path requestsDirectory) {
|
||||||
|
String cacheKey = cacheKey("sfx", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
|
||||||
|
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()),
|
||||||
|
Double.toString(cue.gainDb()));
|
||||||
|
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
|
||||||
|
"sfx", cacheKey + ".wav");
|
||||||
|
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
|
||||||
|
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||||
|
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
||||||
|
}
|
||||||
|
|
||||||
|
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
|
||||||
|
Path cachePath, Path projectTarget, String notes,
|
||||||
|
Path requestsDirectory, double durationSeconds, boolean blocking) {
|
||||||
|
try {
|
||||||
|
createDirectory(cachePath.getParent());
|
||||||
|
if (Files.isRegularFile(cachePath)) {
|
||||||
|
copy(cachePath, projectTarget);
|
||||||
|
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||||
|
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
|
||||||
|
return new AssetGenerationItem(type, assetKey, cacheKey, true, cachePath.toString(),
|
||||||
|
projectTarget.toString(), null, notes);
|
||||||
|
}
|
||||||
|
|
||||||
|
Optional<ResolvedEditAsset> selected = selectExistingAsset(type, assetKey, notes, durationSeconds);
|
||||||
|
if (selected.isPresent()) {
|
||||||
|
copy(Path.of(selected.get().path()), cachePath);
|
||||||
|
copy(Path.of(selected.get().path()), projectTarget);
|
||||||
|
copyIfMissing(Path.of(selected.get().path()).resolveSibling(Path.of(selected.get().path()).getFileName()
|
||||||
|
+ ".license.txt"), projectTarget.resolveSibling(projectTarget.getFileName() + ".license.txt"));
|
||||||
|
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||||
|
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
|
||||||
|
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
|
||||||
|
projectTarget.toString(), null, notes);
|
||||||
|
}
|
||||||
|
|
||||||
|
Path requestDirectory = requestsDirectory.resolve(type).resolve(cacheKey);
|
||||||
|
createDirectory(requestDirectory);
|
||||||
|
Path requestFile = requestDirectory.resolve("request.md");
|
||||||
|
Files.writeString(requestFile, request(projectId, type, assetKey, cacheKey, cachePath, projectTarget,
|
||||||
|
notes, durationSeconds, blocking), StandardCharsets.UTF_8);
|
||||||
|
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
|
||||||
|
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
||||||
|
requestFile.toString(), notes));
|
||||||
|
log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}",
|
||||||
|
projectId, type, assetKey, cacheKey, requestFile);
|
||||||
|
return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
||||||
|
requestFile.toString(), notes);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to prepare generated asset: " + type + "/" + assetKey, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Optional<ResolvedEditAsset> selectExistingAsset(String type, String assetKey, String notes,
|
||||||
|
double durationSeconds) {
|
||||||
|
if ("music".equals(type)) {
|
||||||
|
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
|
||||||
|
}
|
||||||
|
if ("voiceover".equals(type)) {
|
||||||
|
return provider.list(EditAssetType.VOICEOVER, null).stream().findFirst();
|
||||||
|
}
|
||||||
|
if ("sfx".equals(type)) {
|
||||||
|
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
|
||||||
|
null, durationSeconds, notes));
|
||||||
|
if (exact.isPresent()) {
|
||||||
|
return exact;
|
||||||
|
}
|
||||||
|
return library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, notes, durationSeconds));
|
||||||
|
}
|
||||||
|
return Optional.empty();
|
||||||
|
}
|
||||||
|
|
||||||
|
private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath,
|
||||||
|
Path projectTarget, String notes, double durationSeconds, boolean blocking) {
|
||||||
|
return """
|
||||||
|
# Asset Generation Request
|
||||||
|
|
||||||
|
Project: `%s`
|
||||||
|
Type: `%s`
|
||||||
|
Asset key: `%s`
|
||||||
|
Cache key: `%s`
|
||||||
|
Cache file: `%s`
|
||||||
|
Project target: `%s`
|
||||||
|
Blocking for render: `%s`
|
||||||
|
Duration seconds: %.3f
|
||||||
|
|
||||||
|
Create a reusable asset at the cache file path above.
|
||||||
|
If an asset already exists at that exact cache path, reuse it.
|
||||||
|
If you generate a new asset, keep the filename stable so future projects can reuse it.
|
||||||
|
Notes:
|
||||||
|
%s
|
||||||
|
""".formatted(projectId, type, assetKey, cacheKey, cachePath, projectTarget, blocking, durationSeconds,
|
||||||
|
notes == null ? "" : notes);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String cacheKey(String type, String... values) {
|
||||||
|
try {
|
||||||
|
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||||
|
digest.update(type.getBytes(StandardCharsets.UTF_8));
|
||||||
|
for (String value : values) {
|
||||||
|
if (value != null) {
|
||||||
|
digest.update((byte) 0);
|
||||||
|
digest.update(value.getBytes(StandardCharsets.UTF_8));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return HexFormat.of().formatHex(digest.digest()).substring(0, 16);
|
||||||
|
} catch (NoSuchAlgorithmException ex) {
|
||||||
|
throw new IllegalStateException("Unable to compute asset cache key", ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createDirectory(Path directory) {
|
||||||
|
try {
|
||||||
|
Files.createDirectories(directory);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to create asset generation directory: " + directory, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void copy(Path source, Path target) throws IOException {
|
||||||
|
if (source.normalize().equals(target.normalize())) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
createDirectory(target.getParent());
|
||||||
|
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void copyIfMissing(Path source, Path target) throws IOException {
|
||||||
|
if (Files.isRegularFile(source) && !Files.exists(target)) {
|
||||||
|
copy(source, target);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void writeJson(Path path, Object value) {
|
||||||
|
try {
|
||||||
|
createDirectory(path.getParent());
|
||||||
|
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), value);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to write asset generation JSON: " + path, ex);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,160 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.net.URI;
|
||||||
|
import java.net.http.HttpClient;
|
||||||
|
import java.net.http.HttpRequest;
|
||||||
|
import java.net.http.HttpResponse;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class LocalCvVisualAnalysisProvider implements VisualAnalysisProvider {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(LocalCvVisualAnalysisProvider.class);
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.VisualAnalysis properties;
|
||||||
|
private final ObjectMapper objectMapper;
|
||||||
|
private final HttpExecutor httpExecutor;
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public LocalCvVisualAnalysisProvider(VideoClippingProperties properties, ObjectMapper objectMapper) {
|
||||||
|
this(properties, objectMapper, new JdkHttpExecutor());
|
||||||
|
}
|
||||||
|
|
||||||
|
LocalCvVisualAnalysisProvider(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
ObjectMapper objectMapper,
|
||||||
|
HttpExecutor httpExecutor
|
||||||
|
) {
|
||||||
|
this.properties = properties.getEditing().getVisualAnalysis();
|
||||||
|
this.objectMapper = objectMapper;
|
||||||
|
this.httpExecutor = httpExecutor;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
|
||||||
|
LocalCvRequest request = new LocalCvRequest(source, List.copyOf(thumbnails), List.copyOf(shotSegments));
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
try {
|
||||||
|
String body = objectMapper.writeValueAsString(request);
|
||||||
|
log.info("event=local_cv_request_started clip_id={} endpoint={} timeout_ms={} thumbnails={} "
|
||||||
|
+ "shot_segments={} request_bytes={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), properties.getTimeoutMs(), thumbnails.size(),
|
||||||
|
shotSegments.size(), body.getBytes(StandardCharsets.UTF_8).length);
|
||||||
|
HttpCall call = new HttpCall(properties.getEndpoint(), properties.getTimeoutMs(), body);
|
||||||
|
HttpResult result = httpExecutor.execute(call);
|
||||||
|
log.info("event=local_cv_response_received clip_id={} endpoint={} status_code={} elapsed_ms={} "
|
||||||
|
+ "response_bytes={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), result.statusCode(), elapsedMillis(startedAt),
|
||||||
|
result.body() == null ? 0 : result.body().getBytes(StandardCharsets.UTF_8).length);
|
||||||
|
if (result.statusCode() < 200 || result.statusCode() >= 300) {
|
||||||
|
throw new IllegalStateException("Local CV service returned HTTP " + result.statusCode()
|
||||||
|
+ " body=" + preview(result.body()));
|
||||||
|
}
|
||||||
|
SourceVisualAnalysis analysis = objectMapper.readValue(result.body(), SourceVisualAnalysis.class);
|
||||||
|
SourceVisualAnalysis normalized = normalized(source, thumbnails, analysis);
|
||||||
|
log.info("event=local_cv_analysis_completed clip_id={} endpoint={} elapsed_ms={} method={} "
|
||||||
|
+ "object_labels={} representative_thumbnails={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
normalized.analysisMethod(), normalized.objectLabels().size(),
|
||||||
|
normalized.representativeThumbnails().size());
|
||||||
|
return normalized;
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.warn("event=local_cv_request_interrupted clip_id={} endpoint={} elapsed_ms={} message={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt), ex.getMessage());
|
||||||
|
throw new IllegalStateException("Interrupted while calling local CV visual analysis service", ex);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
log.warn("event=local_cv_request_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
throw new IllegalStateException("Unable to call local CV visual analysis service", ex);
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
log.warn("event=local_cv_analysis_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private SourceVisualAnalysis normalized(
|
||||||
|
ClipAnalysis source,
|
||||||
|
List<String> thumbnails,
|
||||||
|
SourceVisualAnalysis analysis
|
||||||
|
) {
|
||||||
|
return new SourceVisualAnalysis(
|
||||||
|
valueOrDefault(analysis.clipId(), source.clipId()),
|
||||||
|
clamp(analysis.blurScore()),
|
||||||
|
clamp(analysis.exposureScore()),
|
||||||
|
clamp(analysis.motionScore()),
|
||||||
|
clamp(analysis.compositionScore()),
|
||||||
|
valueOrDefault(analysis.facePresence(), "unknown"),
|
||||||
|
analysis.objectLabels() == null ? List.of() : List.copyOf(analysis.objectLabels()),
|
||||||
|
analysis.representativeThumbnails() == null || analysis.representativeThumbnails().isEmpty()
|
||||||
|
? List.copyOf(thumbnails)
|
||||||
|
: List.copyOf(analysis.representativeThumbnails()),
|
||||||
|
valueOrDefault(analysis.analysisMethod(), "local_cv")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String valueOrDefault(String value, String fallback) {
|
||||||
|
return value == null || value.isBlank() ? fallback : value;
|
||||||
|
}
|
||||||
|
|
||||||
|
private double clamp(double value) {
|
||||||
|
return Math.max(0, Math.min(1, Math.round(value * 1000.0) / 1000.0));
|
||||||
|
}
|
||||||
|
|
||||||
|
private long elapsedMillis(long startedAt) {
|
||||||
|
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String preview(String body) {
|
||||||
|
if (body == null || body.isBlank()) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
String compact = body.replaceAll("\\s+", " ").trim();
|
||||||
|
return compact.length() <= 300 ? compact : compact.substring(0, 300) + "...";
|
||||||
|
}
|
||||||
|
|
||||||
|
record LocalCvRequest(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
|
||||||
|
}
|
||||||
|
|
||||||
|
record HttpCall(String endpoint, long timeoutMs, String body) {
|
||||||
|
}
|
||||||
|
|
||||||
|
record HttpResult(int statusCode, String body) {
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface HttpExecutor {
|
||||||
|
HttpResult execute(HttpCall call) throws IOException, InterruptedException;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static class JdkHttpExecutor implements HttpExecutor {
|
||||||
|
private final HttpClient httpClient = HttpClient.newHttpClient();
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public HttpResult execute(HttpCall call) throws IOException, InterruptedException {
|
||||||
|
HttpRequest request = HttpRequest.newBuilder()
|
||||||
|
.uri(URI.create(call.endpoint()))
|
||||||
|
.timeout(Duration.ofMillis(call.timeoutMs()))
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.POST(HttpRequest.BodyPublishers.ofString(call.body()))
|
||||||
|
.build();
|
||||||
|
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
|
||||||
|
return new HttpResult(response.statusCode(), response.body());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,326 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.context.SmartLifecycle;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.BufferedReader;
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.io.InputStream;
|
||||||
|
import java.io.InputStreamReader;
|
||||||
|
import java.net.URI;
|
||||||
|
import java.net.http.HttpClient;
|
||||||
|
import java.net.http.HttpRequest;
|
||||||
|
import java.net.http.HttpResponse;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Duration;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class LocalCvWorkerProcessManager implements SmartLifecycle {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(LocalCvWorkerProcessManager.class);
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.VisualAnalysis visualAnalysis;
|
||||||
|
private final ProcessLauncher processLauncher;
|
||||||
|
private final HealthChecker healthChecker;
|
||||||
|
private Process process;
|
||||||
|
private Thread outputThread;
|
||||||
|
private Thread watcherThread;
|
||||||
|
private volatile boolean running;
|
||||||
|
private volatile boolean stopping;
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public LocalCvWorkerProcessManager(VideoClippingProperties properties) {
|
||||||
|
this(properties, ProcessBuilder::start, new HttpHealthChecker());
|
||||||
|
}
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
ProcessLauncher processLauncher
|
||||||
|
) {
|
||||||
|
this(properties, processLauncher, new HttpHealthChecker());
|
||||||
|
}
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
ProcessLauncher processLauncher,
|
||||||
|
HealthChecker healthChecker
|
||||||
|
) {
|
||||||
|
this.visualAnalysis = properties.getEditing().getVisualAnalysis();
|
||||||
|
this.processLauncher = processLauncher;
|
||||||
|
this.healthChecker = healthChecker;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized void start() {
|
||||||
|
if (running) {
|
||||||
|
log.info("event=local_cv_worker_start_skipped reason=already_running endpoint={}",
|
||||||
|
visualAnalysis.getEndpoint());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!shouldStart()) {
|
||||||
|
log.info("event=local_cv_worker_start_skipped reason=disabled provider={} auto_start={} endpoint={}",
|
||||||
|
visualAnalysis.getProvider(), visualAnalysis.getLocalCvWorker().isAutoStart(),
|
||||||
|
visualAnalysis.getEndpoint());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker = visualAnalysis.getLocalCvWorker();
|
||||||
|
Path script = Path.of(worker.getScript()).toAbsolutePath().normalize();
|
||||||
|
URI healthUri = healthUri(worker);
|
||||||
|
log.info("event=local_cv_worker_starting script={} endpoint={} health_uri={} startup_wait_ms={} "
|
||||||
|
+ "health_check_interval_ms={}",
|
||||||
|
script, visualAnalysis.getEndpoint(), healthUri, worker.getStartupWaitMs(),
|
||||||
|
worker.getHealthCheckIntervalMs());
|
||||||
|
if (!Files.isRegularFile(script)) {
|
||||||
|
throw new IllegalStateException("Local CV worker script does not exist: " + script);
|
||||||
|
}
|
||||||
|
|
||||||
|
Endpoint endpoint = endpoint();
|
||||||
|
ProcessBuilder processBuilder = new ProcessBuilder(script.toString())
|
||||||
|
.redirectErrorStream(true);
|
||||||
|
Map<String, String> environment = processBuilder.environment();
|
||||||
|
environment.put("LOCAL_CV_HOST", endpoint.host());
|
||||||
|
environment.put("LOCAL_CV_PORT", Integer.toString(endpoint.port()));
|
||||||
|
|
||||||
|
try {
|
||||||
|
process = processLauncher.start(processBuilder);
|
||||||
|
running = true;
|
||||||
|
stopping = false;
|
||||||
|
outputThread = outputReader(process.getInputStream(), script);
|
||||||
|
outputThread.start();
|
||||||
|
watcherThread = processWatcher(process, script);
|
||||||
|
watcherThread.start();
|
||||||
|
log.info("event=local_cv_worker_started script={} pid={} host={} port={} endpoint={}",
|
||||||
|
script, process.pid(), endpoint.host(), endpoint.port(), visualAnalysis.getEndpoint());
|
||||||
|
waitUntilHealthy(worker, healthUri);
|
||||||
|
log.info("event=local_cv_worker_ready script={} pid={} health_uri={} alive={}",
|
||||||
|
script, process.pid(), healthUri, process.isAlive());
|
||||||
|
} catch (IOException ex) {
|
||||||
|
running = false;
|
||||||
|
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
|
||||||
|
script, ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
throw new IllegalStateException("Unable to start local CV worker script: " + script, ex);
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
cleanupFailedStart();
|
||||||
|
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
|
||||||
|
script, ex.getClass().getSimpleName(), ex.getMessage());
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized void stop() {
|
||||||
|
if (process == null) {
|
||||||
|
running = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
long pid = process.pid();
|
||||||
|
if (process.isAlive()) {
|
||||||
|
stopping = true;
|
||||||
|
log.info("event=local_cv_worker_stop_requested pid={}", pid);
|
||||||
|
process.destroy();
|
||||||
|
try {
|
||||||
|
if (!process.waitFor(Duration.ofSeconds(5).toMillis(), TimeUnit.MILLISECONDS)) {
|
||||||
|
log.warn("event=local_cv_worker_stop_timeout pid={} action=destroy_forcibly", pid);
|
||||||
|
process.destroyForcibly();
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
process.destroyForcibly();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
running = false;
|
||||||
|
log.info("event=local_cv_worker_stopped pid={}", pid);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public boolean isRunning() {
|
||||||
|
return running && process != null && process.isAlive();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public boolean isAutoStartup() {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public int getPhase() {
|
||||||
|
return Integer.MIN_VALUE + 100;
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean shouldStart() {
|
||||||
|
return "local-cv".equalsIgnoreCase(visualAnalysis.getProvider())
|
||||||
|
&& visualAnalysis.getLocalCvWorker().isAutoStart();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void waitUntilHealthy(
|
||||||
|
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker,
|
||||||
|
URI healthUri
|
||||||
|
) {
|
||||||
|
long startupWaitMs = worker.getStartupWaitMs();
|
||||||
|
if (startupWaitMs <= 0) {
|
||||||
|
log.info("event=local_cv_worker_health_wait_skipped reason=disabled health_uri={}", healthUri);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
long deadline = System.currentTimeMillis() + startupWaitMs;
|
||||||
|
long intervalMs = Math.max(1, worker.getHealthCheckIntervalMs());
|
||||||
|
try {
|
||||||
|
while (System.currentTimeMillis() <= deadline) {
|
||||||
|
if (process != null && !process.isAlive()) {
|
||||||
|
throw new IllegalStateException("Local CV worker exited before becoming healthy");
|
||||||
|
}
|
||||||
|
HealthResult result = healthChecker.check(healthUri, Math.min(intervalMs, 5000));
|
||||||
|
log.info("event=local_cv_worker_health_check health_uri={} healthy={} status_code={} "
|
||||||
|
+ "elapsed_ms={} message={}",
|
||||||
|
healthUri, result.healthy(), result.statusCode(), elapsedMillis(startedAt),
|
||||||
|
result.message());
|
||||||
|
if (result.healthy()) {
|
||||||
|
log.info("event=local_cv_worker_health_ready health_uri={} elapsed_ms={}",
|
||||||
|
healthUri, elapsedMillis(startedAt));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Thread.sleep(intervalMs);
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
throw new IllegalStateException("Interrupted while waiting for local CV worker health", ex);
|
||||||
|
}
|
||||||
|
throw new IllegalStateException("Local CV worker did not become healthy within "
|
||||||
|
+ startupWaitMs + "ms at " + healthUri);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void cleanupFailedStart() {
|
||||||
|
running = false;
|
||||||
|
if (process != null && process.isAlive()) {
|
||||||
|
log.warn("event=local_cv_worker_failed_start_cleanup pid={}", process.pid());
|
||||||
|
process.destroyForcibly();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Endpoint endpoint() {
|
||||||
|
URI uri = URI.create(visualAnalysis.getEndpoint());
|
||||||
|
String host = uri.getHost() == null || uri.getHost().isBlank() ? "127.0.0.1" : uri.getHost();
|
||||||
|
int port = uri.getPort();
|
||||||
|
if (port < 0) {
|
||||||
|
port = "https".equalsIgnoreCase(uri.getScheme()) ? 443 : 80;
|
||||||
|
}
|
||||||
|
return new Endpoint(host, port);
|
||||||
|
}
|
||||||
|
|
||||||
|
private URI healthUri(VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker) {
|
||||||
|
URI endpointUri = URI.create(visualAnalysis.getEndpoint());
|
||||||
|
String scheme = endpointUri.getScheme() == null || endpointUri.getScheme().isBlank()
|
||||||
|
? "http"
|
||||||
|
: endpointUri.getScheme();
|
||||||
|
String host = endpointUri.getHost() == null || endpointUri.getHost().isBlank()
|
||||||
|
? "127.0.0.1"
|
||||||
|
: endpointUri.getHost();
|
||||||
|
int port = endpointUri.getPort();
|
||||||
|
String healthPath = worker.getHealthPath() == null || worker.getHealthPath().isBlank()
|
||||||
|
? "/health"
|
||||||
|
: worker.getHealthPath();
|
||||||
|
if (!healthPath.startsWith("/")) {
|
||||||
|
healthPath = "/" + healthPath;
|
||||||
|
}
|
||||||
|
return URI.create("%s://%s%s%s".formatted(scheme, host, port < 0 ? "" : ":" + port, healthPath));
|
||||||
|
}
|
||||||
|
|
||||||
|
private long elapsedMillis(long startedAt) {
|
||||||
|
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
private Thread outputReader(InputStream inputStream, Path script) {
|
||||||
|
Thread thread = new Thread(() -> {
|
||||||
|
try (BufferedReader reader = new BufferedReader(
|
||||||
|
new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
|
||||||
|
String line;
|
||||||
|
while ((line = reader.readLine()) != null) {
|
||||||
|
log.info("event=local_cv_worker_output script={} message={}", script, line);
|
||||||
|
}
|
||||||
|
} catch (IOException ex) {
|
||||||
|
log.debug("event=local_cv_worker_output_closed script={} error_type={}",
|
||||||
|
script, ex.getClass().getSimpleName());
|
||||||
|
}
|
||||||
|
}, "local-cv-worker-output");
|
||||||
|
thread.setDaemon(true);
|
||||||
|
return thread;
|
||||||
|
}
|
||||||
|
|
||||||
|
private Thread processWatcher(Process watchedProcess, Path script) {
|
||||||
|
Thread thread = new Thread(() -> {
|
||||||
|
try {
|
||||||
|
int exitCode = watchedProcess.waitFor();
|
||||||
|
running = false;
|
||||||
|
if (stopping) {
|
||||||
|
log.info("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=stop_requested",
|
||||||
|
script, watchedProcess.pid(), exitCode);
|
||||||
|
stopping = false;
|
||||||
|
} else {
|
||||||
|
log.warn("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=unexpected",
|
||||||
|
script, watchedProcess.pid(), exitCode);
|
||||||
|
}
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
log.debug("event=local_cv_worker_watcher_interrupted script={} pid={}",
|
||||||
|
script, watchedProcess.pid());
|
||||||
|
}
|
||||||
|
}, "local-cv-worker-watcher");
|
||||||
|
thread.setDaemon(true);
|
||||||
|
return thread;
|
||||||
|
}
|
||||||
|
|
||||||
|
record Endpoint(String host, int port) {
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface ProcessLauncher {
|
||||||
|
Process start(ProcessBuilder processBuilder) throws IOException;
|
||||||
|
}
|
||||||
|
|
||||||
|
record HealthResult(boolean healthy, int statusCode, String message) {
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface HealthChecker {
|
||||||
|
HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static class HttpHealthChecker implements HealthChecker {
|
||||||
|
private final HttpClient httpClient = HttpClient.newHttpClient();
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException {
|
||||||
|
try {
|
||||||
|
HttpRequest request = HttpRequest.newBuilder()
|
||||||
|
.uri(healthUri)
|
||||||
|
.timeout(Duration.ofMillis(timeoutMs))
|
||||||
|
.GET()
|
||||||
|
.build();
|
||||||
|
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
|
||||||
|
return new HealthResult(response.statusCode() >= 200 && response.statusCode() < 300,
|
||||||
|
response.statusCode(), preview(response.body()));
|
||||||
|
} catch (IOException ex) {
|
||||||
|
return new HealthResult(false, 0, ex.getClass().getSimpleName() + ": " + ex.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String preview(String body) {
|
||||||
|
if (body == null || body.isBlank()) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
String compact = body.replaceAll("\\s+", " ").trim();
|
||||||
|
return compact.length() <= 160 ? compact : compact.substring(0, 160) + "...";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -6,6 +6,7 @@ import java.util.List;
|
||||||
public record RenderManifest(
|
public record RenderManifest(
|
||||||
String projectId,
|
String projectId,
|
||||||
List<String> clipIds,
|
List<String> clipIds,
|
||||||
|
List<String> renderedClipPaths,
|
||||||
String outputPath,
|
String outputPath,
|
||||||
double durationSeconds,
|
double durationSeconds,
|
||||||
List<List<String>> commands,
|
List<List<String>> commands,
|
||||||
|
|
@ -14,6 +15,7 @@ public record RenderManifest(
|
||||||
) {
|
) {
|
||||||
public RenderManifest {
|
public RenderManifest {
|
||||||
clipIds = clipIds == null ? List.of() : List.copyOf(clipIds);
|
clipIds = clipIds == null ? List.of() : List.copyOf(clipIds);
|
||||||
|
renderedClipPaths = renderedClipPaths == null ? List.of() : List.copyOf(renderedClipPaths);
|
||||||
commands = commands == null ? List.of() : List.copyOf(commands);
|
commands = commands == null ? List.of() : List.copyOf(commands);
|
||||||
assets = assets == null ? List.of() : List.copyOf(assets);
|
assets = assets == null ? List.of() : List.copyOf(assets);
|
||||||
}
|
}
|
||||||
|
|
@ -26,6 +28,18 @@ public record RenderManifest(
|
||||||
List<List<String>> commands,
|
List<List<String>> commands,
|
||||||
Instant completedAt
|
Instant completedAt
|
||||||
) {
|
) {
|
||||||
this(projectId, clipIds, outputPath, durationSeconds, commands, List.of(), completedAt);
|
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, List.of(), completedAt);
|
||||||
|
}
|
||||||
|
|
||||||
|
public RenderManifest(
|
||||||
|
String projectId,
|
||||||
|
List<String> clipIds,
|
||||||
|
String outputPath,
|
||||||
|
double durationSeconds,
|
||||||
|
List<List<String>> commands,
|
||||||
|
List<ResolvedEditAsset> assets,
|
||||||
|
Instant completedAt
|
||||||
|
) {
|
||||||
|
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, assets, completedAt);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,138 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.regex.Matcher;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class ShotSceneSegmenter {
|
||||||
|
|
||||||
|
private static final Pattern PTS_TIME_PATTERN = Pattern.compile("pts_time:([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
private static final Pattern SCENE_SCORE_PATTERN = Pattern.compile("lavfi\\.scene_score=([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing properties;
|
||||||
|
private final ProcessExecutor processExecutor;
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public ShotSceneSegmenter(VideoClippingProperties properties) {
|
||||||
|
this(properties, ShotSceneSegmenter::execute);
|
||||||
|
}
|
||||||
|
|
||||||
|
ShotSceneSegmenter(VideoClippingProperties properties, ProcessExecutor processExecutor) {
|
||||||
|
this.properties = properties.getEditing();
|
||||||
|
this.processExecutor = processExecutor;
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<ShotSegment> segment(ClipAnalysis source) {
|
||||||
|
List<String> command = List.of(
|
||||||
|
properties.getFfmpegBinary(),
|
||||||
|
"-hide_banner",
|
||||||
|
"-i", source.sourcePath(),
|
||||||
|
"-filter:v", "select='gt(scene," + properties.getSceneDetectionThreshold()
|
||||||
|
+ ")',metadata=print",
|
||||||
|
"-an",
|
||||||
|
"-f", "null",
|
||||||
|
"-"
|
||||||
|
);
|
||||||
|
ProcessResult result;
|
||||||
|
try {
|
||||||
|
result = processExecutor.execute(command);
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
throw new IllegalStateException("Interrupted while detecting shot segments", ex);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to run FFmpeg for shot segmentation", ex);
|
||||||
|
}
|
||||||
|
if (result.exitCode() != 0) {
|
||||||
|
throw new IllegalStateException("FFmpeg shot segmentation exited with code " + result.exitCode());
|
||||||
|
}
|
||||||
|
return segmentsFromCuts(source.durationSeconds(), parseCuts(result.output()));
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<SceneCut> parseCuts(String output) {
|
||||||
|
List<SceneCut> cuts = new ArrayList<>();
|
||||||
|
for (String line : output.split("\\R")) {
|
||||||
|
Matcher ptsMatcher = PTS_TIME_PATTERN.matcher(line);
|
||||||
|
if (ptsMatcher.find()) {
|
||||||
|
cuts.add(new SceneCut(Double.parseDouble(ptsMatcher.group(1)), 0));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
Matcher scoreMatcher = SCENE_SCORE_PATTERN.matcher(line);
|
||||||
|
if (scoreMatcher.find() && !cuts.isEmpty()) {
|
||||||
|
SceneCut last = cuts.remove(cuts.size() - 1);
|
||||||
|
cuts.add(new SceneCut(last.timestampSeconds(), Double.parseDouble(scoreMatcher.group(1))));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cuts;
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<ShotSegment> segmentsFromCuts(double durationSeconds, List<SceneCut> cuts) {
|
||||||
|
double duration = Math.max(0, durationSeconds);
|
||||||
|
double minSegmentSeconds = properties.getMinimumSceneDurationSeconds();
|
||||||
|
List<ShotSegment> shots = new ArrayList<>();
|
||||||
|
double start = 0;
|
||||||
|
double score = 0;
|
||||||
|
for (SceneCut cut : cuts.stream().sorted().toList()) {
|
||||||
|
double timestamp = Math.min(duration, Math.max(0, cut.timestampSeconds()));
|
||||||
|
if (timestamp - start < minSegmentSeconds || duration - timestamp < minSegmentSeconds) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
shots.add(segment(shots.size() + 1, start, timestamp, score));
|
||||||
|
start = timestamp;
|
||||||
|
score = cut.sceneScore();
|
||||||
|
}
|
||||||
|
if (duration > start) {
|
||||||
|
shots.add(segment(shots.size() + 1, start, duration, score));
|
||||||
|
}
|
||||||
|
if (shots.isEmpty()) {
|
||||||
|
shots.add(segment(1, 0, duration, 0));
|
||||||
|
}
|
||||||
|
return List.copyOf(shots);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ShotSegment segment(int index, double startSeconds, double endSeconds, double sceneScore) {
|
||||||
|
double duration = Math.max(0, endSeconds - startSeconds);
|
||||||
|
return new ShotSegment(
|
||||||
|
"shot_%04d".formatted(index),
|
||||||
|
round(startSeconds),
|
||||||
|
round(endSeconds),
|
||||||
|
round(duration),
|
||||||
|
round(sceneScore),
|
||||||
|
round(startSeconds + duration / 2.0)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double round(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
return new ProcessResult(process.waitFor(), output);
|
||||||
|
}
|
||||||
|
|
||||||
|
private record SceneCut(double timestampSeconds, double sceneScore) implements Comparable<SceneCut> {
|
||||||
|
@Override
|
||||||
|
public int compareTo(SceneCut other) {
|
||||||
|
return Double.compare(timestampSeconds, other.timestampSeconds());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
record ProcessResult(int exitCode, String output) {
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface ProcessExecutor {
|
||||||
|
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
public record ShotSegment(
|
||||||
|
String shotId,
|
||||||
|
double startSeconds,
|
||||||
|
double endSeconds,
|
||||||
|
double durationSeconds,
|
||||||
|
double sceneScore,
|
||||||
|
double representativeTimestampSeconds
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public record SourceAudioAnalysis(
|
||||||
|
String clipId,
|
||||||
|
boolean audioPresent,
|
||||||
|
double meanVolumeDb,
|
||||||
|
double maxVolumeDb,
|
||||||
|
double silenceThresholdDb,
|
||||||
|
double silenceDurationSeconds,
|
||||||
|
List<AudioSection> sections,
|
||||||
|
String transcriptPath
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,173 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.regex.Matcher;
|
||||||
|
import java.util.regex.Pattern;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class SourceAudioAnalyzer {
|
||||||
|
|
||||||
|
private static final Pattern MEAN_VOLUME_PATTERN = Pattern.compile("mean_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
|
||||||
|
private static final Pattern MAX_VOLUME_PATTERN = Pattern.compile("max_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
|
||||||
|
private static final Pattern SILENCE_START_PATTERN = Pattern.compile("silence_start:\\s*([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
private static final Pattern SILENCE_END_PATTERN = Pattern.compile("silence_end:\\s*([0-9]+(?:\\.[0-9]+)?)");
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing properties;
|
||||||
|
private final ProcessExecutor processExecutor;
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public SourceAudioAnalyzer(VideoClippingProperties properties) {
|
||||||
|
this(properties, SourceAudioAnalyzer::execute);
|
||||||
|
}
|
||||||
|
|
||||||
|
SourceAudioAnalyzer(VideoClippingProperties properties, ProcessExecutor processExecutor) {
|
||||||
|
this.properties = properties.getEditing();
|
||||||
|
this.processExecutor = processExecutor;
|
||||||
|
}
|
||||||
|
|
||||||
|
public SourceAudioAnalysis analyze(ClipAnalysis source) {
|
||||||
|
if (source.audioCodec() == null || source.audioCodec().isBlank()) {
|
||||||
|
return new SourceAudioAnalysis(source.clipId(), false, 0, 0,
|
||||||
|
properties.getSilenceThresholdDb(), properties.getSilenceMinimumDurationSeconds(),
|
||||||
|
List.of(new AudioSection("audio_section_0001", "missing_audio", 0,
|
||||||
|
round(source.durationSeconds()), round(source.durationSeconds()))),
|
||||||
|
null);
|
||||||
|
}
|
||||||
|
List<String> command = List.of(
|
||||||
|
properties.getFfmpegBinary(),
|
||||||
|
"-hide_banner",
|
||||||
|
"-i", source.sourcePath(),
|
||||||
|
"-af", "silencedetect=n=" + properties.getSilenceThresholdDb() + "dB:d="
|
||||||
|
+ properties.getSilenceMinimumDurationSeconds() + ",volumedetect",
|
||||||
|
"-vn",
|
||||||
|
"-f", "null",
|
||||||
|
"-"
|
||||||
|
);
|
||||||
|
ProcessResult result;
|
||||||
|
try {
|
||||||
|
result = processExecutor.execute(command);
|
||||||
|
} catch (InterruptedException ex) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
throw new IllegalStateException("Interrupted while analyzing source audio", ex);
|
||||||
|
} catch (IOException ex) {
|
||||||
|
throw new IllegalStateException("Unable to run FFmpeg for source audio analysis", ex);
|
||||||
|
}
|
||||||
|
if (result.exitCode() != 0) {
|
||||||
|
throw new IllegalStateException("FFmpeg source audio analysis exited with code " + result.exitCode());
|
||||||
|
}
|
||||||
|
ParsedAudio parsed = parse(result.output());
|
||||||
|
return new SourceAudioAnalysis(
|
||||||
|
source.clipId(),
|
||||||
|
true,
|
||||||
|
parsed.meanVolumeDb(),
|
||||||
|
parsed.maxVolumeDb(),
|
||||||
|
properties.getSilenceThresholdDb(),
|
||||||
|
properties.getSilenceMinimumDurationSeconds(),
|
||||||
|
sections(source.durationSeconds(), parsed.silences()),
|
||||||
|
null
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ParsedAudio parse(String output) {
|
||||||
|
double meanVolume = 0;
|
||||||
|
double maxVolume = 0;
|
||||||
|
List<SilenceRange> silences = new ArrayList<>();
|
||||||
|
Double pendingSilenceStart = null;
|
||||||
|
for (String line : output.split("\\R")) {
|
||||||
|
Matcher meanMatcher = MEAN_VOLUME_PATTERN.matcher(line);
|
||||||
|
if (meanMatcher.find()) {
|
||||||
|
meanVolume = Double.parseDouble(meanMatcher.group(1));
|
||||||
|
}
|
||||||
|
Matcher maxMatcher = MAX_VOLUME_PATTERN.matcher(line);
|
||||||
|
if (maxMatcher.find()) {
|
||||||
|
maxVolume = Double.parseDouble(maxMatcher.group(1));
|
||||||
|
}
|
||||||
|
Matcher startMatcher = SILENCE_START_PATTERN.matcher(line);
|
||||||
|
if (startMatcher.find()) {
|
||||||
|
pendingSilenceStart = Double.parseDouble(startMatcher.group(1));
|
||||||
|
}
|
||||||
|
Matcher endMatcher = SILENCE_END_PATTERN.matcher(line);
|
||||||
|
if (endMatcher.find() && pendingSilenceStart != null) {
|
||||||
|
silences.add(new SilenceRange(pendingSilenceStart, Double.parseDouble(endMatcher.group(1))));
|
||||||
|
pendingSilenceStart = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return new ParsedAudio(round(meanVolume), round(maxVolume), silences);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<AudioSection> sections(double durationSeconds, List<SilenceRange> silences) {
|
||||||
|
double duration = round(Math.max(0, durationSeconds));
|
||||||
|
List<AudioSection> sections = new ArrayList<>();
|
||||||
|
double cursor = 0;
|
||||||
|
for (SilenceRange silence : silences.stream()
|
||||||
|
.sorted(Comparator.comparingDouble(SilenceRange::startSeconds))
|
||||||
|
.toList()) {
|
||||||
|
double start = clamp(silence.startSeconds(), 0, duration);
|
||||||
|
double end = clamp(silence.endSeconds(), start, duration);
|
||||||
|
if (start > cursor) {
|
||||||
|
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, start));
|
||||||
|
}
|
||||||
|
if (end > start) {
|
||||||
|
sections.add(section(sections.size() + 1, "silence", start, end));
|
||||||
|
}
|
||||||
|
cursor = Math.max(cursor, end);
|
||||||
|
}
|
||||||
|
if (cursor < duration) {
|
||||||
|
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, duration));
|
||||||
|
}
|
||||||
|
if (sections.isEmpty()) {
|
||||||
|
sections.add(section(1, "unclassified_audio", 0, duration));
|
||||||
|
}
|
||||||
|
return List.copyOf(sections);
|
||||||
|
}
|
||||||
|
|
||||||
|
private AudioSection section(int index, String kind, double start, double end) {
|
||||||
|
double roundedStart = round(start);
|
||||||
|
double roundedEnd = round(end);
|
||||||
|
return new AudioSection(
|
||||||
|
"audio_section_%04d".formatted(index),
|
||||||
|
kind,
|
||||||
|
roundedStart,
|
||||||
|
roundedEnd,
|
||||||
|
round(roundedEnd - roundedStart)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private double clamp(double value, double min, double max) {
|
||||||
|
return Math.min(max, Math.max(min, value));
|
||||||
|
}
|
||||||
|
|
||||||
|
private double round(double value) {
|
||||||
|
return Math.round(value * 1000.0) / 1000.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||||
|
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||||
|
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||||
|
return new ProcessResult(process.waitFor(), output);
|
||||||
|
}
|
||||||
|
|
||||||
|
private record ParsedAudio(double meanVolumeDb, double maxVolumeDb, List<SilenceRange> silences) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record SilenceRange(double startSeconds, double endSeconds) {
|
||||||
|
}
|
||||||
|
|
||||||
|
record ProcessResult(int exitCode, String output) {
|
||||||
|
}
|
||||||
|
|
||||||
|
@FunctionalInterface
|
||||||
|
interface ProcessExecutor {
|
||||||
|
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public record SourceVisualAnalysis(
|
||||||
|
String clipId,
|
||||||
|
double blurScore,
|
||||||
|
double exposureScore,
|
||||||
|
double motionScore,
|
||||||
|
double compositionScore,
|
||||||
|
String facePresence,
|
||||||
|
List<VisualObjectLabel> objectLabels,
|
||||||
|
List<String> representativeThumbnails,
|
||||||
|
String analysisMethod
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,122 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||||
|
public class SourceVisualAnalyzer {
|
||||||
|
|
||||||
|
private static final Logger log = LoggerFactory.getLogger(SourceVisualAnalyzer.class);
|
||||||
|
|
||||||
|
private final VideoClippingProperties.Editing.VisualAnalysis properties;
|
||||||
|
private final VisualAnalysisProvider heuristicProvider;
|
||||||
|
private final VisualAnalysisProvider localCvProvider;
|
||||||
|
|
||||||
|
public SourceVisualAnalyzer() {
|
||||||
|
this(new VideoClippingProperties(), new HeuristicVisualAnalysisProvider(), null);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Autowired
|
||||||
|
public SourceVisualAnalyzer(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
HeuristicVisualAnalysisProvider heuristicProvider,
|
||||||
|
LocalCvVisualAnalysisProvider localCvProvider
|
||||||
|
) {
|
||||||
|
this(properties, heuristicProvider, (VisualAnalysisProvider) localCvProvider);
|
||||||
|
}
|
||||||
|
|
||||||
|
SourceVisualAnalyzer(
|
||||||
|
VideoClippingProperties properties,
|
||||||
|
VisualAnalysisProvider heuristicProvider,
|
||||||
|
VisualAnalysisProvider localCvProvider
|
||||||
|
) {
|
||||||
|
this.properties = properties.getEditing().getVisualAnalysis();
|
||||||
|
this.heuristicProvider = heuristicProvider;
|
||||||
|
this.localCvProvider = localCvProvider;
|
||||||
|
}
|
||||||
|
|
||||||
|
public SourceVisualAnalysis analyze(
|
||||||
|
ClipAnalysis source,
|
||||||
|
List<String> thumbnails,
|
||||||
|
List<ShotSegment> shotSegments
|
||||||
|
) {
|
||||||
|
log.info("event=visual_analysis_selected clip_id={} provider={} thumbnails={} shot_segments={} "
|
||||||
|
+ "fallback_to_heuristic={}",
|
||||||
|
source.clipId(), properties.getProvider(), thumbnails.size(), shotSegments.size(),
|
||||||
|
properties.isFallbackToHeuristic());
|
||||||
|
if ("local-cv".equalsIgnoreCase(properties.getProvider())) {
|
||||||
|
return localCvAnalysis(source, thumbnails, shotSegments);
|
||||||
|
}
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
SourceVisualAnalysis analysis = heuristicProvider.analyze(source, thumbnails, shotSegments);
|
||||||
|
log.info("event=heuristic_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} method={}",
|
||||||
|
source.clipId(), elapsedMillis(startedAt), analysis.objectLabels().size(),
|
||||||
|
analysis.analysisMethod());
|
||||||
|
return analysis;
|
||||||
|
}
|
||||||
|
|
||||||
|
private SourceVisualAnalysis localCvAnalysis(
|
||||||
|
ClipAnalysis source,
|
||||||
|
List<String> thumbnails,
|
||||||
|
List<ShotSegment> shotSegments
|
||||||
|
) {
|
||||||
|
long startedAt = System.nanoTime();
|
||||||
|
try {
|
||||||
|
SourceVisualAnalysis analysis = localCvProvider.analyze(source, thumbnails, shotSegments);
|
||||||
|
log.info("event=local_cv_visual_analysis_completed clip_id={} endpoint={} elapsed_ms={} "
|
||||||
|
+ "object_labels={} method={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
analysis.objectLabels().size(), analysis.analysisMethod());
|
||||||
|
return analysis;
|
||||||
|
} catch (RuntimeException ex) {
|
||||||
|
if (!properties.isFallbackToHeuristic()) {
|
||||||
|
log.error("event=local_cv_visual_analysis_failed clip_id={} endpoint={} elapsed_ms={} "
|
||||||
|
+ "fallback=false error_type={} message={} cause_type={} cause_message={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
|
||||||
|
throw ex;
|
||||||
|
}
|
||||||
|
log.warn("event=local_cv_visual_analysis_fallback clip_id={} endpoint={} elapsed_ms={} "
|
||||||
|
+ "error_type={} message={} cause_type={} cause_message={}",
|
||||||
|
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||||
|
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
|
||||||
|
long fallbackStartedAt = System.nanoTime();
|
||||||
|
SourceVisualAnalysis fallback = heuristicProvider.analyze(source, thumbnails, shotSegments);
|
||||||
|
SourceVisualAnalysis fallbackAnalysis = new SourceVisualAnalysis(
|
||||||
|
fallback.clipId(),
|
||||||
|
fallback.blurScore(),
|
||||||
|
fallback.exposureScore(),
|
||||||
|
fallback.motionScore(),
|
||||||
|
fallback.compositionScore(),
|
||||||
|
fallback.facePresence(),
|
||||||
|
fallback.objectLabels(),
|
||||||
|
fallback.representativeThumbnails(),
|
||||||
|
"local_cv_failed_fallback_" + fallback.analysisMethod()
|
||||||
|
);
|
||||||
|
log.info("event=local_cv_fallback_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} "
|
||||||
|
+ "method={}",
|
||||||
|
source.clipId(), elapsedMillis(fallbackStartedAt), fallbackAnalysis.objectLabels().size(),
|
||||||
|
fallbackAnalysis.analysisMethod());
|
||||||
|
return fallbackAnalysis;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private long elapsedMillis(long startedAt) {
|
||||||
|
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String causeType(RuntimeException ex) {
|
||||||
|
return ex.getCause() == null ? null : ex.getCause().getClass().getSimpleName();
|
||||||
|
}
|
||||||
|
|
||||||
|
private String causeMessage(RuntimeException ex) {
|
||||||
|
return ex.getCause() == null ? null : ex.getCause().getMessage();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -65,6 +65,17 @@ public class StoryboardPromptGenerator {
|
||||||
|
|
||||||
Create one cinematic edit plan for project `%s`.
|
Create one cinematic edit plan for project `%s`.
|
||||||
|
|
||||||
|
The service can only render what the plan expresses, so your plan must use the actual cinematic toolkit
|
||||||
|
available to the renderer:
|
||||||
|
- shot selection and timing
|
||||||
|
- cuts, crossfades, fade-ins, and fade-outs
|
||||||
|
- dynamic punch-in crop and subtle cinematic grade
|
||||||
|
- music bed selection and timing
|
||||||
|
- SFX placement and timing
|
||||||
|
- voiceover scripting and timing
|
||||||
|
- short text overlays
|
||||||
|
- per-segment rendered clips plus the final assembled video
|
||||||
|
|
||||||
Constraints:
|
Constraints:
|
||||||
- Style: `%s`
|
- Style: `%s`
|
||||||
- Target duration: %d seconds
|
- Target duration: %d seconds
|
||||||
|
|
@ -77,6 +88,7 @@ public class StoryboardPromptGenerator {
|
||||||
- Timeline ranges must be chronological and must not overlap.
|
- Timeline ranges must be chronological and must not overlap.
|
||||||
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
||||||
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
|
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
|
||||||
|
- Make the edit cinematic by directing those elements deliberately, not by trimming footage alone.
|
||||||
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
|
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
|
||||||
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
|
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
|
||||||
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public interface VisualAnalysisProvider {
|
||||||
|
|
||||||
|
SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments);
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
public record VisualObjectLabel(
|
||||||
|
String label,
|
||||||
|
double confidence,
|
||||||
|
String source
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
@ -5,7 +5,7 @@ video-clipping:
|
||||||
enabled: true
|
enabled: true
|
||||||
local-director:
|
local-director:
|
||||||
enabled: true
|
enabled: true
|
||||||
auto-render-when-plan-appears: false
|
auto-render-when-plan-appears: true
|
||||||
require-approval-before-render: true
|
require-approval-before-render: true
|
||||||
approval-file-name: approved.flag
|
approval-file-name: approved.flag
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,11 @@ video-clipping:
|
||||||
contact-sheet-columns: ${VIDEO_EDITING_CONTACT_SHEET_COLUMNS:5}
|
contact-sheet-columns: ${VIDEO_EDITING_CONTACT_SHEET_COLUMNS:5}
|
||||||
proxy-enabled: ${VIDEO_EDITING_PROXY_ENABLED:true}
|
proxy-enabled: ${VIDEO_EDITING_PROXY_ENABLED:true}
|
||||||
proxy-width: ${VIDEO_EDITING_PROXY_WIDTH:640}
|
proxy-width: ${VIDEO_EDITING_PROXY_WIDTH:640}
|
||||||
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:60}
|
scene-detection-threshold: ${VIDEO_EDITING_SCENE_DETECTION_THRESHOLD:0.35}
|
||||||
|
minimum-scene-duration-seconds: ${VIDEO_EDITING_MINIMUM_SCENE_DURATION_SECONDS:1.0}
|
||||||
|
silence-threshold-db: ${VIDEO_EDITING_SILENCE_THRESHOLD_DB:-35.0}
|
||||||
|
silence-minimum-duration-seconds: ${VIDEO_EDITING_SILENCE_MINIMUM_DURATION_SECONDS:0.5}
|
||||||
|
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:600}
|
||||||
output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920}
|
output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920}
|
||||||
output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080}
|
output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080}
|
||||||
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30}
|
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30}
|
||||||
|
|
@ -44,6 +48,17 @@ video-clipping:
|
||||||
fonts-folder: ${VIDEO_EDITING_ASSETS_FONTS_FOLDER:./input/highlights/assets/fonts}
|
fonts-folder: ${VIDEO_EDITING_ASSETS_FONTS_FOLDER:./input/highlights/assets/fonts}
|
||||||
luts-folder: ${VIDEO_EDITING_ASSETS_LUTS_FOLDER:./input/highlights/assets/luts}
|
luts-folder: ${VIDEO_EDITING_ASSETS_LUTS_FOLDER:./input/highlights/assets/luts}
|
||||||
voiceover-folder: ${VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER:./output/highlight-projects/_voiceover-cache}
|
voiceover-folder: ${VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER:./output/highlight-projects/_voiceover-cache}
|
||||||
|
visual-analysis:
|
||||||
|
provider: ${VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER:local-cv} #local-cv or heuristic
|
||||||
|
endpoint: ${VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT:http://127.0.0.1:8091/v1/analyze-visuals}
|
||||||
|
timeout-ms: ${VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS:30000}
|
||||||
|
fallback-to-heuristic: ${VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC:true}
|
||||||
|
local-cv-worker:
|
||||||
|
auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true}
|
||||||
|
script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh}
|
||||||
|
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:120000}
|
||||||
|
health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health}
|
||||||
|
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
|
||||||
local-director:
|
local-director:
|
||||||
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
|
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
|
||||||
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}
|
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}
|
||||||
|
|
@ -63,6 +78,12 @@ video-clipping:
|
||||||
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
|
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
|
||||||
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
|
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
|
||||||
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
|
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
|
||||||
|
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:true}
|
||||||
|
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:false}
|
||||||
|
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
||||||
|
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
||||||
|
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
|
||||||
|
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35}
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level:
|
level:
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,11 @@ class VideoClippingPropertiesTest {
|
||||||
assertThat(properties.getEditing().getContactSheetColumns()).isEqualTo(5);
|
assertThat(properties.getEditing().getContactSheetColumns()).isEqualTo(5);
|
||||||
assertThat(properties.getEditing().isProxyEnabled()).isTrue();
|
assertThat(properties.getEditing().isProxyEnabled()).isTrue();
|
||||||
assertThat(properties.getEditing().getProxyWidth()).isEqualTo(640);
|
assertThat(properties.getEditing().getProxyWidth()).isEqualTo(640);
|
||||||
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(60);
|
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.35);
|
||||||
|
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(1.0);
|
||||||
|
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-35.0);
|
||||||
|
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(0.5);
|
||||||
|
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(600);
|
||||||
assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920);
|
assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920);
|
||||||
assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080);
|
assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080);
|
||||||
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30);
|
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30);
|
||||||
|
|
@ -50,6 +54,19 @@ class VideoClippingPropertiesTest {
|
||||||
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("./input/highlights/assets/luts");
|
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("./input/highlights/assets/luts");
|
||||||
assertThat(properties.getEditing().getAssets().getVoiceoverFolder())
|
assertThat(properties.getEditing().getAssets().getVoiceoverFolder())
|
||||||
.isEqualTo("./output/highlight-projects/_voiceover-cache");
|
.isEqualTo("./output/highlight-projects/_voiceover-cache");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("heuristic");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
|
||||||
|
.isEqualTo("http://127.0.0.1:8091/v1/analyze-visuals");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(30000);
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isTrue();
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isFalse();
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
|
||||||
|
.isEqualTo("./tools/run_local_cv_worker.sh");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs()).isZero();
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
|
||||||
|
.isEqualTo("/health");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
|
||||||
|
.isEqualTo(1000);
|
||||||
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
|
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
|
||||||
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
|
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
|
||||||
.isEqualTo("./input/editing/source");
|
.isEqualTo("./input/editing/source");
|
||||||
|
|
@ -71,6 +88,10 @@ class VideoClippingPropertiesTest {
|
||||||
"video-clipping.editing.project-directory=/tmp/edit-projects",
|
"video-clipping.editing.project-directory=/tmp/edit-projects",
|
||||||
"video-clipping.editing.highlight-project-directory=/tmp/highlight-projects",
|
"video-clipping.editing.highlight-project-directory=/tmp/highlight-projects",
|
||||||
"video-clipping.editing.thumbnail-count-per-clip=7",
|
"video-clipping.editing.thumbnail-count-per-clip=7",
|
||||||
|
"video-clipping.editing.scene-detection-threshold=0.42",
|
||||||
|
"video-clipping.editing.minimum-scene-duration-seconds=2.5",
|
||||||
|
"video-clipping.editing.silence-threshold-db=-40",
|
||||||
|
"video-clipping.editing.silence-minimum-duration-seconds=1.25",
|
||||||
"video-clipping.editing.voiceover-provider=remote",
|
"video-clipping.editing.voiceover-provider=remote",
|
||||||
"video-clipping.editing.loudness-target-i=-14",
|
"video-clipping.editing.loudness-target-i=-14",
|
||||||
"video-clipping.editing.loudness-true-peak=-2",
|
"video-clipping.editing.loudness-true-peak=-2",
|
||||||
|
|
@ -84,6 +105,15 @@ class VideoClippingPropertiesTest {
|
||||||
"video-clipping.editing.assets.fonts-folder=/tmp/fonts",
|
"video-clipping.editing.assets.fonts-folder=/tmp/fonts",
|
||||||
"video-clipping.editing.assets.luts-folder=/tmp/luts",
|
"video-clipping.editing.assets.luts-folder=/tmp/luts",
|
||||||
"video-clipping.editing.assets.voiceover-folder=/tmp/voiceover",
|
"video-clipping.editing.assets.voiceover-folder=/tmp/voiceover",
|
||||||
|
"video-clipping.editing.visual-analysis.provider=local-cv",
|
||||||
|
"video-clipping.editing.visual-analysis.endpoint=http://localhost:9000/analyze",
|
||||||
|
"video-clipping.editing.visual-analysis.timeout-ms=12000",
|
||||||
|
"video-clipping.editing.visual-analysis.fallback-to-heuristic=false",
|
||||||
|
"video-clipping.editing.visual-analysis.local-cv-worker.auto-start=true",
|
||||||
|
"video-clipping.editing.visual-analysis.local-cv-worker.script=/tmp/local-cv.sh",
|
||||||
|
"video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250",
|
||||||
|
"video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready",
|
||||||
|
"video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25",
|
||||||
"video-clipping.editing.local-director.enabled=false",
|
"video-clipping.editing.local-director.enabled=false",
|
||||||
"video-clipping.editing.local-director.source-directory=/tmp/source",
|
"video-clipping.editing.local-director.source-directory=/tmp/source",
|
||||||
"video-clipping.editing.local-director.poll-interval-ms=9000",
|
"video-clipping.editing.local-director.poll-interval-ms=9000",
|
||||||
|
|
@ -101,6 +131,10 @@ class VideoClippingPropertiesTest {
|
||||||
assertThat(properties.getEditing().getProjectDirectory()).isEqualTo("/tmp/edit-projects");
|
assertThat(properties.getEditing().getProjectDirectory()).isEqualTo("/tmp/edit-projects");
|
||||||
assertThat(properties.getEditing().getHighlightProjectDirectory()).isEqualTo("/tmp/highlight-projects");
|
assertThat(properties.getEditing().getHighlightProjectDirectory()).isEqualTo("/tmp/highlight-projects");
|
||||||
assertThat(properties.getEditing().getThumbnailCountPerClip()).isEqualTo(7);
|
assertThat(properties.getEditing().getThumbnailCountPerClip()).isEqualTo(7);
|
||||||
|
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.42);
|
||||||
|
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(2.5);
|
||||||
|
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-40.0);
|
||||||
|
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(1.25);
|
||||||
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("remote");
|
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("remote");
|
||||||
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-14.0);
|
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-14.0);
|
||||||
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-2.0);
|
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-2.0);
|
||||||
|
|
@ -114,6 +148,20 @@ class VideoClippingPropertiesTest {
|
||||||
assertThat(properties.getEditing().getAssets().getFontsFolder()).isEqualTo("/tmp/fonts");
|
assertThat(properties.getEditing().getAssets().getFontsFolder()).isEqualTo("/tmp/fonts");
|
||||||
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("/tmp/luts");
|
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("/tmp/luts");
|
||||||
assertThat(properties.getEditing().getAssets().getVoiceoverFolder()).isEqualTo("/tmp/voiceover");
|
assertThat(properties.getEditing().getAssets().getVoiceoverFolder()).isEqualTo("/tmp/voiceover");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("local-cv");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
|
||||||
|
.isEqualTo("http://localhost:9000/analyze");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(12000);
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isFalse();
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isTrue();
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
|
||||||
|
.isEqualTo("/tmp/local-cv.sh");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs())
|
||||||
|
.isEqualTo(250);
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
|
||||||
|
.isEqualTo("/ready");
|
||||||
|
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
|
||||||
|
.isEqualTo(25);
|
||||||
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
|
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
|
||||||
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
|
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
|
||||||
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);
|
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);
|
||||||
|
|
|
||||||
|
|
@ -62,7 +62,14 @@ class CinematicEditingIntegrationTest {
|
||||||
Path output = projectDirectory.resolve("final.mp4");
|
Path output = projectDirectory.resolve("final.mp4");
|
||||||
assertThat(output).isRegularFile();
|
assertThat(output).isRegularFile();
|
||||||
assertThat(probeDuration(output)).isGreaterThan(0);
|
assertThat(probeDuration(output)).isGreaterThan(0);
|
||||||
|
assertThat(projectDirectory.resolve("rendered-clips/clip_0001.mp4")).isRegularFile();
|
||||||
|
assertThat(projectDirectory.resolve("rendered-clips/clip_0002.mp4")).isRegularFile();
|
||||||
|
assertThat(projectDirectory.resolve("rendered-clips/clip_0003.mp4")).isRegularFile();
|
||||||
assertThat(projectDirectory.resolve("render-manifest.json")).exists();
|
assertThat(projectDirectory.resolve("render-manifest.json")).exists();
|
||||||
|
RenderManifest manifest = store.readJson(projectId, "render-manifest.json", RenderManifest.class);
|
||||||
|
assertThat(manifest.clipIds()).containsExactly("clip_01", "clip_02", "clip_03");
|
||||||
|
assertThat(manifest.renderedClipPaths()).hasSize(3)
|
||||||
|
.allSatisfy(path -> assertThat(Path.of(path)).isRegularFile());
|
||||||
assertThat(projectDirectory.resolve("qa-report.json")).exists();
|
assertThat(projectDirectory.resolve("qa-report.json")).exists();
|
||||||
assertThat(projects.getProject(projectId).status()).isEqualTo(EditProjectStatus.RENDERED);
|
assertThat(projects.getProject(projectId).status()).isEqualTo(EditProjectStatus.RENDERED);
|
||||||
}
|
}
|
||||||
|
|
@ -128,7 +135,8 @@ class CinematicEditingIntegrationTest {
|
||||||
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
||||||
"cinematic grade", "integration sequence"))
|
"cinematic grade", "integration sequence"))
|
||||||
.toList();
|
.toList();
|
||||||
return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(),
|
return new EditPlan(projectId, "category-aware-cinematic-highlights",
|
||||||
|
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(),
|
||||||
"mp4-h264-aac-320p", "Generated integration edit");
|
"mp4-h264-aac-320p", "Generated integration edit");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||||
class CinematicEditingLocalProfileTest {
|
class CinematicEditingLocalProfileTest {
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
void enablesLocalDirectorAndDisablesEightSecondFolderScheduler() {
|
void enablesLocalDirectorAutoRenderAndDisablesEightSecondFolderScheduler() {
|
||||||
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
||||||
factory.setResources(new ClassPathResource("application-cinematic-editing-local.yml"));
|
factory.setResources(new ClassPathResource("application-cinematic-editing-local.yml"));
|
||||||
Properties properties = factory.getObject();
|
Properties properties = factory.getObject();
|
||||||
|
|
@ -21,7 +21,7 @@ class CinematicEditingLocalProfileTest {
|
||||||
assertThat(properties.getProperty("video-clipping.editing.enabled")).isEqualTo("true");
|
assertThat(properties.getProperty("video-clipping.editing.enabled")).isEqualTo("true");
|
||||||
assertThat(properties.getProperty("video-clipping.editing.local-director.enabled")).isEqualTo("true");
|
assertThat(properties.getProperty("video-clipping.editing.local-director.enabled")).isEqualTo("true");
|
||||||
assertThat(properties.getProperty("video-clipping.editing.local-director.auto-render-when-plan-appears"))
|
assertThat(properties.getProperty("video-clipping.editing.local-director.auto-render-when-plan-appears"))
|
||||||
.isEqualTo("false");
|
.isEqualTo("true");
|
||||||
assertThat(properties.getProperty("video-clipping.editing.local-director.require-approval-before-render"))
|
assertThat(properties.getProperty("video-clipping.editing.local-director.require-approval-before-render"))
|
||||||
.isEqualTo("true");
|
.isEqualTo("true");
|
||||||
assertThat(properties.getProperty("video-clipping.editing.local-director.approval-file-name"))
|
assertThat(properties.getProperty("video-clipping.editing.local-director.approval-file-name"))
|
||||||
|
|
|
||||||
|
|
@ -130,10 +130,11 @@ class EditPlanInboxScannerTest {
|
||||||
}
|
}
|
||||||
|
|
||||||
private void writeValidPlan() throws Exception {
|
private void writeValidPlan() throws Exception {
|
||||||
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 60,
|
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 2.5,
|
||||||
List.of(new EditDecision("clip_00001", 0, 2.5, 0, 2.5,
|
List.of(new EditDecision("clip_00001", 0, 2.5, 0, 2.5,
|
||||||
"cut", "cut", 1.0, "cinematic", "opening")),
|
"cut", "cut", 1.0, "cinematic", "opening")),
|
||||||
List.of(), List.of(), "mp4-h264-aac-1080p", "Hero edit");
|
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 2.5, -14, "opening bed")),
|
||||||
|
List.of(), "mp4-h264-aac-1080p", "Hero edit");
|
||||||
objectMapper.writeValue(store.inboxDirectory("porsche-edit").resolve("edit-plan.json").toFile(), plan);
|
objectMapper.writeValue(store.inboxDirectory("porsche-edit").resolve("edit-plan.json").toFile(), plan);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -37,6 +37,41 @@ class EditPlanValidatorTest {
|
||||||
|
|
||||||
@Test void acceptsValidPlan() { assertThat(validator.validate("project", plan(decision("clip-1", 0, 4, 0, 4, "cut")))).isNotNull(); }
|
@Test void acceptsValidPlan() { assertThat(validator.validate("project", plan(decision("clip-1", 0, 4, 0, 4, "cut")))).isNotNull(); }
|
||||||
|
|
||||||
|
@Test void acceptsContentDrivenPlanShorterThanProjectTarget() {
|
||||||
|
EditPlan shorter = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||||
|
List.of(decision("clip-1", 0, 4, 0, 4, "cut")),
|
||||||
|
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 4, -14, "bed")),
|
||||||
|
List.of(),
|
||||||
|
"mp4-h264-aac-1080p", "summary");
|
||||||
|
|
||||||
|
assertThat(validator.validate("project", shorter)).isEqualTo(shorter);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test void rejectsBareTrimOnlyPlanWithoutCinematicRichness() {
|
||||||
|
EditPlan bare = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||||
|
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "none", "reason")),
|
||||||
|
List.of(), List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
|
||||||
|
|
||||||
|
rejects(bare);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test void rejectsPlanTargetDurationThatDoesNotMatchFinalTimeline() {
|
||||||
|
EditPlan mismatch = new EditPlan("project", "cinematic-porsche-promo", 8,
|
||||||
|
List.of(decision("clip-1", 0, 4, 0, 4, "cut")), List.of(), List.of(),
|
||||||
|
"mp4-h264-aac-1080p", "summary");
|
||||||
|
|
||||||
|
rejects(mismatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test void rejectsPlanLongerThanProjectMaximumTarget() {
|
||||||
|
EditPlan tooLong = new EditPlan("project", "cinematic-porsche-promo", 12,
|
||||||
|
List.of(decision("clip-1", 0, 8, 0, 8, "cut"),
|
||||||
|
decision("clip-1", 0, 4, 8, 12, "cut")),
|
||||||
|
List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
|
||||||
|
|
||||||
|
rejects(tooLong);
|
||||||
|
}
|
||||||
|
|
||||||
@Test void rejectsUnknownClip() { rejects(plan(decision("missing", 0, 4, 0, 4, "cut"))); }
|
@Test void rejectsUnknownClip() { rejects(plan(decision("missing", 0, 4, 0, 4, "cut"))); }
|
||||||
|
|
||||||
@Test void rejectsOutOfRangeSource() { rejects(plan(decision("clip-1", 0, 9, 0, 4, "cut"))); }
|
@Test void rejectsOutOfRangeSource() { rejects(plan(decision("clip-1", 0, 9, 0, 4, "cut"))); }
|
||||||
|
|
@ -141,7 +176,11 @@ class EditPlanValidatorTest {
|
||||||
}
|
}
|
||||||
|
|
||||||
private EditPlan plan(EditDecision... decisions) {
|
private EditPlan plan(EditDecision... decisions) {
|
||||||
return new EditPlan("project", "cinematic-porsche-promo", 10, List.of(decisions), List.of(), List.of(),
|
return new EditPlan("project", "cinematic-porsche-promo", decisions[decisions.length - 1].timelineEndSeconds(),
|
||||||
|
List.of(decisions),
|
||||||
|
List.of(new AudioCue("music", "premium-cinematic-drive", 0,
|
||||||
|
decisions[decisions.length - 1].timelineEndSeconds(), -14, "bed")),
|
||||||
|
List.of(),
|
||||||
"mp4-h264-aac-1080p", "summary");
|
"mp4-h264-aac-1080p", "summary");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
class HighlightAssetPreparationServiceTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void writesVoiceoverScriptAndRequestsWhenAssetsAreMissing() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = project();
|
||||||
|
store.createProject(project, manifest());
|
||||||
|
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
||||||
|
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
||||||
|
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||||
|
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
|
||||||
|
.thenReturn(List.of());
|
||||||
|
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||||
|
HighlightAssetPreparationService service = new HighlightAssetPreparationService(properties, store,
|
||||||
|
assetProvider, assetLibrary, mapper);
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening Reveal", 0.0, 12.0, 12.0,
|
||||||
|
"opening_hook", "premium cinematic grade", "low cinematic pulse",
|
||||||
|
"whoosh on reveal", List.of("This is a clean first look."), List.of("Pure presence"),
|
||||||
|
"hold the opening beat");
|
||||||
|
|
||||||
|
HighlightAssetPreparationService.HighlightAssetPreparationResult result = service.prepare(project.id(),
|
||||||
|
project, highlight, ContentCategory.GENERIC_VLOG);
|
||||||
|
|
||||||
|
assertThat(result.requestFiles()).isNotEmpty();
|
||||||
|
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||||
|
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
|
||||||
|
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||||
|
.resolve("assets/requests"))) {
|
||||||
|
assertThat(files.count()).isGreaterThan(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private VideoClippingProperties properties() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
||||||
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
|
||||||
|
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||||
|
return properties;
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProject project() {
|
||||||
|
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||||
|
return new HighlightProject("highlight-project-001", "Highlight Project",
|
||||||
|
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
|
||||||
|
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProjectManifest manifest() {
|
||||||
|
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,109 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.mockito.ArgumentMatchers.any;
|
||||||
|
import static org.mockito.ArgumentMatchers.anyString;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
class HighlightDirectorFlowServiceTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void importsDirectorPlanWritesHighlightEditPlanAndPublishesFinalVideo() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = project();
|
||||||
|
store.createProject(project, manifest());
|
||||||
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
||||||
|
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
||||||
|
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
||||||
|
createDummyFile(source);
|
||||||
|
HighlightVisualEffectsStage visualEffectsStage = mock(HighlightVisualEffectsStage.class);
|
||||||
|
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
|
||||||
|
HighlightAssetPreparationService.HighlightAssetPreparationResult assetResult =
|
||||||
|
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
|
||||||
|
store.highlightsDirectory(project.id()).resolve("highlight_001"),
|
||||||
|
store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("assets"),
|
||||||
|
List.of(), List.of());
|
||||||
|
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(assetResult);
|
||||||
|
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
|
||||||
|
when(assetWorker.process(anyString(), any(), any())).thenReturn(
|
||||||
|
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
|
||||||
|
List.of(), List.of()));
|
||||||
|
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
|
||||||
|
Path renderedClip = store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("final.mp4");
|
||||||
|
createDummyFile(renderedClip);
|
||||||
|
HighlightFfmpegRenderer.HighlightRenderResult renderResult = new HighlightFfmpegRenderer.HighlightRenderResult(
|
||||||
|
"highlight_001", renderedClip, renderedClip, List.of(renderedClip),
|
||||||
|
new RenderManifest(project.id(), List.of("source"), List.of(renderedClip.toString()),
|
||||||
|
renderedClip.toString(), 12.0, List.of(), List.of(), Instant.now()),
|
||||||
|
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
|
||||||
|
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
|
||||||
|
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
||||||
|
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
||||||
|
|
||||||
|
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
||||||
|
|
||||||
|
assertThat(result.finalOutputs()).hasSize(1);
|
||||||
|
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
|
||||||
|
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
|
||||||
|
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
|
||||||
|
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||||
|
.isEqualTo(HighlightProjectStatus.RENDERED);
|
||||||
|
}
|
||||||
|
|
||||||
|
private VideoClippingProperties properties() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
return properties;
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProject project() {
|
||||||
|
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||||
|
return new HighlightProject("highlight-project-001", "Highlight Project",
|
||||||
|
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
|
||||||
|
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProjectManifest manifest() {
|
||||||
|
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightDirectorPlan directorPlan() {
|
||||||
|
return new HighlightDirectorPlan("highlight-project-001", "source.mp4", "generic_vlog",
|
||||||
|
List.of(new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening",
|
||||||
|
0.0, 12.0, 12.0, "opening_hook", "premium cinematic grade",
|
||||||
|
"low pulse", "whoosh", List.of("A clean first look."), List.of("Pure presence"),
|
||||||
|
"open with presence")), "summary");
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightSourceAnalysis sourceAnalysis() {
|
||||||
|
ClipAnalysis source = new ClipAnalysis("source", "source.mp4", 12.0, "hevc", "aac", 1920, 1080, 24.0,
|
||||||
|
List.of(), null, null, 0.0, 0.0, 0.0);
|
||||||
|
return new HighlightSourceAnalysis("highlight-project-001", "source.mp4", source, List.of(), null, null,
|
||||||
|
null, "analysis/scene-segments.json", List.of(), "analysis/audio-analysis.json",
|
||||||
|
new SourceAudioAnalysis("source", true, -18.0, -3.0, -35.0, 0.5, List.of(), null),
|
||||||
|
"analysis/visual-analysis.json", new SourceVisualAnalysis("source", 0.5, 0.5, 0.5, 0.5,
|
||||||
|
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createDummyFile(Path file) throws Exception {
|
||||||
|
Files.createDirectories(file.getParent());
|
||||||
|
Files.writeString(file, "dummy");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,83 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightDirectorPromptBackfillTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void backfillsMissingHighlightDirectorPromptOnStartup() throws Exception {
|
||||||
|
FileSystemHighlightProjectStore store = store();
|
||||||
|
HighlightProject project = project();
|
||||||
|
store.createProject(project, manifest());
|
||||||
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
|
||||||
|
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
|
||||||
|
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
|
||||||
|
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||||
|
"opening_hook", List.of("hero angle", "strong motion"))
|
||||||
|
});
|
||||||
|
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
HighlightDirectorPromptBackfill backfill = new HighlightDirectorPromptBackfill(properties,
|
||||||
|
new HighlightDirectorPromptGenerator(store));
|
||||||
|
|
||||||
|
backfill.backfill();
|
||||||
|
|
||||||
|
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-prompt.md"))).isTrue();
|
||||||
|
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-brief.md"))).isTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
private FileSystemHighlightProjectStore store() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProject project() {
|
||||||
|
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||||
|
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
|
||||||
|
HighlightProjectStatus.CREATED, "porsche.mp4",
|
||||||
|
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProjectManifest manifest() {
|
||||||
|
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
|
||||||
|
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
|
||||||
|
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||||
|
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
|
||||||
|
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||||
|
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
|
||||||
|
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
|
||||||
|
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||||
|
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
|
||||||
|
null),
|
||||||
|
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
|
||||||
|
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||||
|
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
|
||||||
|
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
|
||||||
|
List.of("car metadata detected", "hero motion present"),
|
||||||
|
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||||
|
"opening_hook", List.of("hero angle", "strong motion"))),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,90 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightDirectorPromptGeneratorTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void writesHighlightDirectorPromptAndBriefForAnalyzedProject() throws Exception {
|
||||||
|
FileSystemHighlightProjectStore store = store();
|
||||||
|
HighlightProject project = project();
|
||||||
|
store.createProject(project, manifest());
|
||||||
|
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
|
||||||
|
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
|
||||||
|
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
|
||||||
|
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||||
|
"opening_hook", List.of("hero angle", "strong motion"))
|
||||||
|
});
|
||||||
|
|
||||||
|
HighlightDirectorPromptGenerator generator = new HighlightDirectorPromptGenerator(store);
|
||||||
|
HighlightDirectorPromptGenerator.DirectorPromptFiles files = generator.generate(project.id());
|
||||||
|
|
||||||
|
assertThat(Files.readString(Path.of(files.promptPath())))
|
||||||
|
.contains("analysis/source-analysis.json")
|
||||||
|
.contains("analysis/highlight-candidates.json")
|
||||||
|
.contains("director/edit-plan.json")
|
||||||
|
.contains("car_vlog")
|
||||||
|
.contains("candidate_001")
|
||||||
|
.contains("premium, powerful, precise");
|
||||||
|
assertThat(Files.readString(Path.of(files.briefPath())))
|
||||||
|
.contains("Highlight Director Brief")
|
||||||
|
.contains("director-prompt.md")
|
||||||
|
.contains("director/edit-plan.json");
|
||||||
|
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||||
|
.isEqualTo(HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
||||||
|
}
|
||||||
|
|
||||||
|
private FileSystemHighlightProjectStore store() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProject project() {
|
||||||
|
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||||
|
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
|
||||||
|
HighlightProjectStatus.CREATED, "porsche.mp4",
|
||||||
|
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightProjectManifest manifest() {
|
||||||
|
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
|
||||||
|
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
|
||||||
|
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||||
|
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
|
||||||
|
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||||
|
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
|
||||||
|
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
|
||||||
|
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||||
|
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
|
||||||
|
null),
|
||||||
|
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
|
||||||
|
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||||
|
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
|
||||||
|
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
|
||||||
|
List.of("car metadata detected", "hero motion present"),
|
||||||
|
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||||
|
"opening_hook", List.of("hero angle", "strong motion"))),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.mockito.Mockito.mock;
|
||||||
|
import static org.mockito.Mockito.when;
|
||||||
|
|
||||||
|
class HighlightLocalAssetWorkerTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void resolvesMusicFromLocalLibraryAndKeepsPendingWhenVoiceoverCannotBeSynthesized() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
||||||
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
|
||||||
|
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
|
||||||
|
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
|
||||||
|
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
|
||||||
|
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
|
||||||
|
Files.createDirectories(musicSource.getParent());
|
||||||
|
Files.writeString(musicSource, "music");
|
||||||
|
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
|
||||||
|
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
||||||
|
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
||||||
|
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.of(
|
||||||
|
new ResolvedEditAsset(EditAssetType.MUSIC, "cinematic-bed", musicSource.toString(),
|
||||||
|
"local", "local", ContentCategory.GENERIC_VLOG, List.of("cinematic", "bed"))));
|
||||||
|
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||||
|
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
|
||||||
|
.thenReturn(List.of());
|
||||||
|
HighlightAssetPreparationService prep = new HighlightAssetPreparationService(properties, store,
|
||||||
|
assetProvider, assetLibrary, mapper);
|
||||||
|
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||||
|
"highlight_001", "candidate_001", "Opening", 0, 12, 12, "opening_hook",
|
||||||
|
"premium cinematic grade", "low pulse", "whoosh",
|
||||||
|
List.of("A clean first look."), List.of("Pure presence"), "open with presence");
|
||||||
|
prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG);
|
||||||
|
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary, mapper);
|
||||||
|
|
||||||
|
HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight,
|
||||||
|
ContentCategory.GENERIC_VLOG);
|
||||||
|
|
||||||
|
assertThat(result.resolvedAssets()).isNotEmpty();
|
||||||
|
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
|
||||||
|
.exists();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -59,13 +59,34 @@ class HighlightSourceAnalyzerTest {
|
||||||
.isEqualTo(store.analysisDirectory("porsche").resolve("proxies/porsche.mp4").toString());
|
.isEqualTo(store.analysisDirectory("porsche").resolve("proxies/porsche.mp4").toString());
|
||||||
assertThat(result.waveformPath())
|
assertThat(result.waveformPath())
|
||||||
.isEqualTo(store.analysisDirectory("porsche").resolve("audio/porsche-waveform.png").toString());
|
.isEqualTo(store.analysisDirectory("porsche").resolve("audio/porsche-waveform.png").toString());
|
||||||
|
assertThat(result.sceneSegmentsPath()).isEqualTo("analysis/scene-segments.json");
|
||||||
|
assertThat(result.shotSegments()).containsExactly(
|
||||||
|
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
|
||||||
|
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
|
||||||
|
);
|
||||||
|
assertThat(result.audioAnalysisPath()).isEqualTo("analysis/audio-analysis.json");
|
||||||
|
assertThat(result.audioAnalysis().sections()).containsExactly(
|
||||||
|
new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)
|
||||||
|
);
|
||||||
|
assertThat(result.visualAnalysisPath()).isEqualTo("analysis/visual-analysis.json");
|
||||||
|
assertThat(result.visualAnalysis().objectLabels()).containsExactly(
|
||||||
|
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")
|
||||||
|
);
|
||||||
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-11T08:00:00Z"));
|
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
|
|
||||||
ClipAnalysis ffprobe = store.readJson("porsche", "analysis/ffprobe.json", ClipAnalysis.class);
|
ClipAnalysis ffprobe = store.readJson("porsche", "analysis/ffprobe.json", ClipAnalysis.class);
|
||||||
|
ShotSegment[] shots = store.readJson("porsche", "analysis/scene-segments.json", ShotSegment[].class);
|
||||||
|
SourceAudioAnalysis audioAnalysis = store.readJson("porsche", "analysis/audio-analysis.json",
|
||||||
|
SourceAudioAnalysis.class);
|
||||||
|
SourceVisualAnalysis visualAnalysis = store.readJson("porsche", "analysis/visual-analysis.json",
|
||||||
|
SourceVisualAnalysis.class);
|
||||||
HighlightSourceAnalysis persisted = store.readJson("porsche", "analysis/source-analysis.json",
|
HighlightSourceAnalysis persisted = store.readJson("porsche", "analysis/source-analysis.json",
|
||||||
HighlightSourceAnalysis.class);
|
HighlightSourceAnalysis.class);
|
||||||
assertThat(ffprobe.sourcePath()).isEqualTo(source.toString());
|
assertThat(ffprobe.sourcePath()).isEqualTo(source.toString());
|
||||||
assertThat(ffprobe.thumbnails()).hasSize(2);
|
assertThat(ffprobe.thumbnails()).hasSize(2);
|
||||||
|
assertThat(shots).containsExactly(result.shotSegments().toArray(ShotSegment[]::new));
|
||||||
|
assertThat(audioAnalysis).isEqualTo(result.audioAnalysis());
|
||||||
|
assertThat(visualAnalysis).isEqualTo(result.visualAnalysis());
|
||||||
assertThat(persisted).isEqualTo(result);
|
assertThat(persisted).isEqualTo(result);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -102,7 +123,25 @@ class HighlightSourceAnalyzerTest {
|
||||||
when(waveform.generate(any(), any())).thenAnswer(invocation ->
|
when(waveform.generate(any(), any())).thenAnswer(invocation ->
|
||||||
invocation.<Path>getArgument(1).resolve("porsche-waveform.png").toString());
|
invocation.<Path>getArgument(1).resolve("porsche-waveform.png").toString());
|
||||||
|
|
||||||
|
ShotSceneSegmenter shotSceneSegmenter = mock(ShotSceneSegmenter.class);
|
||||||
|
when(shotSceneSegmenter.segment(any())).thenReturn(List.of(
|
||||||
|
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
|
||||||
|
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
|
||||||
|
));
|
||||||
|
|
||||||
|
SourceAudioAnalyzer sourceAudioAnalyzer = mock(SourceAudioAnalyzer.class);
|
||||||
|
when(sourceAudioAnalyzer.analyze(any())).thenReturn(new SourceAudioAnalysis("porsche", true, -18.2,
|
||||||
|
-3.1, -35.0, 0.5,
|
||||||
|
List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)), null));
|
||||||
|
|
||||||
|
SourceVisualAnalyzer sourceVisualAnalyzer = mock(SourceVisualAnalyzer.class);
|
||||||
|
when(sourceVisualAnalyzer.analyze(any(), any(), any())).thenReturn(new SourceVisualAnalysis("porsche",
|
||||||
|
0.6, 0.7, 0.8, 0.75, "unknown_without_face_detector",
|
||||||
|
List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||||
|
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"));
|
||||||
|
|
||||||
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
|
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
|
||||||
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform, clock);
|
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform,
|
||||||
|
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer, clock);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,7 @@ class HighlightSourceSchedulerTest {
|
||||||
private VideoClippingProperties properties;
|
private VideoClippingProperties properties;
|
||||||
private FileSystemHighlightProjectStore store;
|
private FileSystemHighlightProjectStore store;
|
||||||
private HighlightSourceAnalyzer analyzer;
|
private HighlightSourceAnalyzer analyzer;
|
||||||
|
private HighlightDirectorPromptGenerator promptGenerator;
|
||||||
|
|
||||||
@BeforeEach
|
@BeforeEach
|
||||||
void setUp() {
|
void setUp() {
|
||||||
|
|
@ -40,6 +41,7 @@ class HighlightSourceSchedulerTest {
|
||||||
new HighlightProjectDirectoryInitializer(properties).initialize();
|
new HighlightProjectDirectoryInitializer(properties).initialize();
|
||||||
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
analyzer = mock(HighlightSourceAnalyzer.class);
|
analyzer = mock(HighlightSourceAnalyzer.class);
|
||||||
|
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
|
|
@ -73,6 +75,7 @@ class HighlightSourceSchedulerTest {
|
||||||
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
|
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
|
||||||
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
|
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
|
||||||
verify(analyzer).analyze("1");
|
verify(analyzer).analyze("1");
|
||||||
|
verify(promptGenerator).generate("1");
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
|
|
@ -108,13 +111,20 @@ class HighlightSourceSchedulerTest {
|
||||||
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
|
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
|
||||||
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
|
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
|
||||||
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
|
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
|
||||||
return new HighlightSourceScheduler(properties, store, analyzer, clock);
|
return new HighlightSourceScheduler(properties, store, analyzer, promptGenerator, clock);
|
||||||
}
|
}
|
||||||
|
|
||||||
private HighlightSourceAnalysis analysis(String projectId) {
|
private HighlightSourceAnalysis analysis(String projectId) {
|
||||||
ClipAnalysis source = new ClipAnalysis(projectId, store.sourceDirectory(projectId).resolve(projectId + ".mp4")
|
ClipAnalysis source = new ClipAnalysis(projectId, store.sourceDirectory(projectId).resolve(projectId + ".mp4")
|
||||||
.toString(), 42.0, "h264", "aac", 1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
.toString(), 42.0, "h264", "aac", 1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||||
return new HighlightSourceAnalysis(projectId, projectId + ".mp4", source, List.of(), null, null, null,
|
return new HighlightSourceAnalysis(projectId, projectId + ".mp4", source, List.of(), null, null, null,
|
||||||
|
"analysis/scene-segments.json", List.of(new ShotSegment("shot_0001", 0, 42, 42, 0, 21)),
|
||||||
|
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||||
|
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)),
|
||||||
|
null),
|
||||||
|
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.5, 0.5, 0.5,
|
||||||
|
0.7, "unknown_without_face_detector", List.of(new VisualObjectLabel("unknown", 0.2,
|
||||||
|
"metadata_heuristic")), List.of(), "metadata_thumbnail_scene_heuristic"),
|
||||||
Instant.parse("2026-07-11T08:00:00Z"));
|
Instant.parse("2026-07-11T08:00:00Z"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class HighlightVisualEffectsStageTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void writesExplicitVisualEffectsPlan() throws Exception {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||||
|
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||||
|
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||||
|
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
|
||||||
|
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
|
||||||
|
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
|
||||||
|
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||||
|
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
|
||||||
|
HighlightVisualEffectsStage stage = new HighlightVisualEffectsStage(store, mapper);
|
||||||
|
|
||||||
|
HighlightVisualEffectsStage.HighlightVisualEffectsPlan plan = stage.create("project-1",
|
||||||
|
new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening", 0, 12, 12,
|
||||||
|
"opening_hook", "premium cinematic grade", "low pulse", "whoosh",
|
||||||
|
List.of("Narration"), List.of("Title card"), "hold the reveal"),
|
||||||
|
ContentCategory.GENERIC_VLOG);
|
||||||
|
|
||||||
|
assertThat(plan.effects()).contains("crop", "punch-in", "contrast", "vignette", "fade");
|
||||||
|
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/visual-effects.json")).exists();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,98 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class LocalAssetGenerationStageTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void reusesExistingSharedAssetsAndWritesRequestsForMissingVoiceover() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
Files.createDirectories(tempDir.resolve("shared/music"));
|
||||||
|
Files.createDirectories(tempDir.resolve("shared/sfx"));
|
||||||
|
Files.createDirectories(tempDir.resolve("voiceover-cache"));
|
||||||
|
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
|
||||||
|
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
|
||||||
|
|
||||||
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||||
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||||
|
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||||
|
|
||||||
|
FileSystemEditProjectStore store = store(properties);
|
||||||
|
store.createProject("project");
|
||||||
|
|
||||||
|
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
|
||||||
|
new LocalEditAssetProvider(properties),
|
||||||
|
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
|
||||||
|
new ObjectMapper().findAndRegisterModules());
|
||||||
|
|
||||||
|
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||||
|
List.of(
|
||||||
|
new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")
|
||||||
|
),
|
||||||
|
List.of(
|
||||||
|
new AudioCue("music", "premium-bed", 0, 4, -14, "drive bed"),
|
||||||
|
new AudioCue("sfx", "whoosh", 1, 2, -6, "impact")
|
||||||
|
),
|
||||||
|
List.of(new VoiceoverLine("Precision in motion.", 0, 2, "confident")),
|
||||||
|
"mp4-h264-aac-1080p",
|
||||||
|
"summary");
|
||||||
|
|
||||||
|
AssetGenerationResult result = stage.prepare("project", plan);
|
||||||
|
|
||||||
|
assertThat(result.readyForRender()).isTrue();
|
||||||
|
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
|
||||||
|
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
|
||||||
|
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
|
||||||
|
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
|
||||||
|
assertThat(tempDir.resolve("projects/project/assets/requests/voiceover")).exists();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void writesBlockingRequestsWhenSfxIsMissing() throws Exception {
|
||||||
|
VideoClippingProperties properties = properties();
|
||||||
|
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||||
|
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||||
|
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||||
|
|
||||||
|
FileSystemEditProjectStore store = store(properties);
|
||||||
|
store.createProject("project");
|
||||||
|
|
||||||
|
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
|
||||||
|
new LocalEditAssetProvider(properties),
|
||||||
|
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
|
||||||
|
new ObjectMapper().findAndRegisterModules());
|
||||||
|
|
||||||
|
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||||
|
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")),
|
||||||
|
List.of(new AudioCue("sfx", "engine-hit", 1, 2, -6, "impact")),
|
||||||
|
List.of(), "mp4-h264-aac-1080p", "summary");
|
||||||
|
|
||||||
|
AssetGenerationResult result = stage.prepare("project", plan);
|
||||||
|
|
||||||
|
assertThat(result.readyForRender()).isFalse();
|
||||||
|
assertThat(tempDir.resolve("projects/project/assets/requests/sfx")).exists();
|
||||||
|
}
|
||||||
|
|
||||||
|
private VideoClippingProperties properties() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
|
||||||
|
return properties;
|
||||||
|
}
|
||||||
|
|
||||||
|
private FileSystemEditProjectStore store(VideoClippingProperties properties) {
|
||||||
|
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,94 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.concurrent.atomic.AtomicReference;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
|
||||||
|
class LocalCvVisualAnalysisProviderTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void postsVisualAnalysisRequestToConfiguredLocalCvEndpoint() {
|
||||||
|
AtomicReference<LocalCvVisualAnalysisProvider.HttpCall> call = new AtomicReference<>();
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9000/analyze");
|
||||||
|
properties.getEditing().getVisualAnalysis().setTimeoutMs(12000);
|
||||||
|
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
|
||||||
|
properties,
|
||||||
|
new ObjectMapper().findAndRegisterModules(),
|
||||||
|
request -> {
|
||||||
|
call.set(request);
|
||||||
|
return new LocalCvVisualAnalysisProvider.HttpResult(200, """
|
||||||
|
{
|
||||||
|
"clipId": "source",
|
||||||
|
"blurScore": 0.81,
|
||||||
|
"exposureScore": 0.67,
|
||||||
|
"motionScore": 0.73,
|
||||||
|
"compositionScore": 0.79,
|
||||||
|
"facePresence": "faces_detected",
|
||||||
|
"objectLabels": [
|
||||||
|
{"label": "car", "confidence": 0.91, "source": "yolo"}
|
||||||
|
],
|
||||||
|
"representativeThumbnails": ["thumb-2.jpg"],
|
||||||
|
"analysisMethod": "local_cv_yolo_clip"
|
||||||
|
}
|
||||||
|
""");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
SourceVisualAnalysis analysis = provider.analyze(clip(), List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||||
|
List.of(new ShotSegment("shot_0001", 0, 8, 8, 0, 4)));
|
||||||
|
|
||||||
|
assertThat(call.get().endpoint()).isEqualTo("http://localhost:9000/analyze");
|
||||||
|
assertThat(call.get().timeoutMs()).isEqualTo(12000);
|
||||||
|
assertThat(call.get().body()).contains("\"clipId\":\"source\"", "\"thumbnails\":[\"thumb-1.jpg\"");
|
||||||
|
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("car", 0.91, "yolo"));
|
||||||
|
assertThat(analysis.facePresence()).isEqualTo("faces_detected");
|
||||||
|
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_yolo_clip");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsNonSuccessfulLocalCvResponses() {
|
||||||
|
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
|
||||||
|
new VideoClippingProperties(),
|
||||||
|
new ObjectMapper().findAndRegisterModules(),
|
||||||
|
request -> new LocalCvVisualAnalysisProvider.HttpResult(500, "{}")
|
||||||
|
);
|
||||||
|
|
||||||
|
assertThrows(IllegalStateException.class, () -> provider.analyze(clip(), List.of(), List.of()));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void reportsLocalCvIoAndInterruptionFailures() {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
|
||||||
|
new VideoClippingProperties(),
|
||||||
|
new ObjectMapper().findAndRegisterModules(),
|
||||||
|
request -> {
|
||||||
|
throw new IOException("connection refused");
|
||||||
|
}).analyze(clip(), List.of(), List.of()));
|
||||||
|
|
||||||
|
try {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
|
||||||
|
new VideoClippingProperties(),
|
||||||
|
new ObjectMapper().findAndRegisterModules(),
|
||||||
|
request -> {
|
||||||
|
throw new InterruptedException("stopped");
|
||||||
|
}).analyze(clip(), List.of(), List.of()));
|
||||||
|
assertTrue(Thread.currentThread().isInterrupted());
|
||||||
|
} finally {
|
||||||
|
Thread.interrupted();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private ClipAnalysis clip() {
|
||||||
|
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", "aac",
|
||||||
|
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,250 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
import org.junit.jupiter.api.io.TempDir;
|
||||||
|
|
||||||
|
import java.io.ByteArrayInputStream;
|
||||||
|
import java.io.InputStream;
|
||||||
|
import java.io.OutputStream;
|
||||||
|
import java.net.URI;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
import java.util.concurrent.atomic.AtomicInteger;
|
||||||
|
import java.util.concurrent.atomic.AtomicReference;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||||
|
|
||||||
|
class LocalCvWorkerProcessManagerTest {
|
||||||
|
|
||||||
|
@TempDir
|
||||||
|
Path tempDir;
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void doesNotStartWhenProviderIsNotLocalCv() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("heuristic");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||||
|
throw new AssertionError("worker should not start");
|
||||||
|
});
|
||||||
|
|
||||||
|
manager.start();
|
||||||
|
|
||||||
|
assertThat(manager.isRunning()).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void doesNotStartWhenAutoStartIsDisabled() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(false);
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||||
|
throw new AssertionError("worker should not start");
|
||||||
|
});
|
||||||
|
|
||||||
|
manager.start();
|
||||||
|
|
||||||
|
assertThat(manager.isRunning()).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void startsWorkerScriptWithEndpointHostAndPort() throws Exception {
|
||||||
|
Path script = tempDir.resolve("run-worker.sh");
|
||||||
|
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9005/v1/analyze-visuals");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
|
||||||
|
AtomicReference<ProcessBuilder> captured = new AtomicReference<>();
|
||||||
|
AtomicReference<URI> healthUri = new AtomicReference<>();
|
||||||
|
TestProcess process = new TestProcess();
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||||
|
properties,
|
||||||
|
processBuilder -> {
|
||||||
|
captured.set(processBuilder);
|
||||||
|
return process;
|
||||||
|
},
|
||||||
|
(uri, timeoutMs) -> {
|
||||||
|
healthUri.set(uri);
|
||||||
|
return new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok");
|
||||||
|
});
|
||||||
|
|
||||||
|
manager.start();
|
||||||
|
|
||||||
|
assertThat(manager.isRunning()).isTrue();
|
||||||
|
assertThat(captured.get().command()).containsExactly(script.toAbsolutePath().normalize().toString());
|
||||||
|
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_HOST", "localhost");
|
||||||
|
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_PORT", "9005");
|
||||||
|
assertThat(healthUri.get()).hasToString("http://localhost:9005/health");
|
||||||
|
|
||||||
|
manager.stop();
|
||||||
|
|
||||||
|
assertThat(process.destroyed).isTrue();
|
||||||
|
assertThat(manager.isRunning()).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsMissingWorkerScriptWhenAutoStartIsEnabled() {
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(tempDir.resolve("missing.sh").toString());
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||||
|
throw new AssertionError("worker should not start");
|
||||||
|
});
|
||||||
|
|
||||||
|
assertThatThrownBy(manager::start)
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("Local CV worker script does not exist");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void resetsRunningStateWhenWorkerExitsBeforeHealthCheckPasses() throws Exception {
|
||||||
|
Path script = tempDir.resolve("run-worker.sh");
|
||||||
|
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
|
||||||
|
TestProcess process = new TestProcess();
|
||||||
|
process.alive = false;
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||||
|
properties,
|
||||||
|
processBuilder -> process,
|
||||||
|
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused"));
|
||||||
|
|
||||||
|
assertThatThrownBy(manager::start)
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("Local CV worker exited before becoming healthy");
|
||||||
|
assertThat(manager.isRunning()).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void failsWhenWorkerDoesNotBecomeHealthyBeforeTimeout() throws Exception {
|
||||||
|
Path script = tempDir.resolve("run-worker.sh");
|
||||||
|
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
|
||||||
|
TestProcess process = new TestProcess();
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||||
|
properties,
|
||||||
|
processBuilder -> process,
|
||||||
|
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 503, "starting"));
|
||||||
|
|
||||||
|
assertThatThrownBy(manager::start)
|
||||||
|
.isInstanceOf(IllegalStateException.class)
|
||||||
|
.hasMessageContaining("Local CV worker did not become healthy");
|
||||||
|
assertThat(process.destroyed).isTrue();
|
||||||
|
assertThat(manager.isRunning()).isFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void retriesHealthChecksUntilWorkerIsReady() throws Exception {
|
||||||
|
Path script = tempDir.resolve("run-worker.sh");
|
||||||
|
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
|
||||||
|
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
|
||||||
|
AtomicInteger attempts = new AtomicInteger();
|
||||||
|
TestProcess process = new TestProcess();
|
||||||
|
|
||||||
|
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||||
|
properties,
|
||||||
|
processBuilder -> process,
|
||||||
|
(uri, timeoutMs) -> attempts.incrementAndGet() < 2
|
||||||
|
? new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused")
|
||||||
|
: new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok"));
|
||||||
|
|
||||||
|
manager.start();
|
||||||
|
|
||||||
|
assertThat(attempts.get()).isEqualTo(2);
|
||||||
|
assertThat(manager.isRunning()).isTrue();
|
||||||
|
|
||||||
|
manager.stop();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static class TestProcess extends Process {
|
||||||
|
private boolean alive = true;
|
||||||
|
private boolean destroyed;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OutputStream getOutputStream() {
|
||||||
|
return OutputStream.nullOutputStream();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public InputStream getInputStream() {
|
||||||
|
return new ByteArrayInputStream("worker ready\n".getBytes());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public InputStream getErrorStream() {
|
||||||
|
return InputStream.nullInputStream();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized int waitFor() throws InterruptedException {
|
||||||
|
while (alive) {
|
||||||
|
wait();
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized boolean waitFor(long timeout, TimeUnit unit) throws InterruptedException {
|
||||||
|
if (alive) {
|
||||||
|
wait(unit.toMillis(timeout));
|
||||||
|
}
|
||||||
|
return !alive;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public int exitValue() {
|
||||||
|
return alive ? 1 : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized void destroy() {
|
||||||
|
destroyed = true;
|
||||||
|
alive = false;
|
||||||
|
notifyAll();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized Process destroyForcibly() {
|
||||||
|
destroyed = true;
|
||||||
|
alive = false;
|
||||||
|
notifyAll();
|
||||||
|
return this;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized boolean isAlive() {
|
||||||
|
return alive;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public long pid() {
|
||||||
|
return 1234;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,90 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.concurrent.atomic.AtomicReference;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
|
||||||
|
class ShotSceneSegmenterTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void detectsShotSegmentsFromFfmpegSceneMetadata() {
|
||||||
|
AtomicReference<List<String>> command = new AtomicReference<>();
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setSceneDetectionThreshold(0.42);
|
||||||
|
properties.getEditing().setMinimumSceneDurationSeconds(1.5);
|
||||||
|
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(properties, arguments -> {
|
||||||
|
command.set(arguments);
|
||||||
|
return new ShotSceneSegmenter.ProcessResult(0, """
|
||||||
|
frame:1 pts:72000 pts_time:2.400
|
||||||
|
lavfi.scene_score=0.510
|
||||||
|
frame:2 pts:180000 pts_time:6.000
|
||||||
|
lavfi.scene_score=0.730
|
||||||
|
""");
|
||||||
|
});
|
||||||
|
|
||||||
|
List<ShotSegment> shots = segmenter.segment(clip(9));
|
||||||
|
|
||||||
|
assertThat(shots).containsExactly(
|
||||||
|
new ShotSegment("shot_0001", 0, 2.4, 2.4, 0, 1.2),
|
||||||
|
new ShotSegment("shot_0002", 2.4, 6, 3.6, 0.51, 4.2),
|
||||||
|
new ShotSegment("shot_0003", 6, 9, 3, 0.73, 7.5)
|
||||||
|
);
|
||||||
|
assertCommandPair(command.get(), "-i", "source.mp4");
|
||||||
|
assertCommandPair(command.get(), "-filter:v", "select='gt(scene,0.42)',metadata=print");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void returnsSingleShotWhenNoSceneCutPassesMinimumDurationRules() {
|
||||||
|
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), arguments ->
|
||||||
|
new ShotSceneSegmenter.ProcessResult(0, """
|
||||||
|
frame:1 pts:12000 pts_time:0.400
|
||||||
|
lavfi.scene_score=0.900
|
||||||
|
"""));
|
||||||
|
|
||||||
|
assertThat(segmenter.segment(clip(4))).containsExactly(
|
||||||
|
new ShotSegment("shot_0001", 0, 4, 4, 0, 2)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsFfmpegFailures() {
|
||||||
|
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), command ->
|
||||||
|
new ShotSceneSegmenter.ProcessResult(1, "failed"));
|
||||||
|
|
||||||
|
assertThrows(IllegalStateException.class, () -> segmenter.segment(clip(9)));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void reportsProcessStartupAndInterruptionFailures() {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
|
||||||
|
throw new IOException("missing executable");
|
||||||
|
}).segment(clip(9)));
|
||||||
|
|
||||||
|
try {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
|
||||||
|
throw new InterruptedException("stopped");
|
||||||
|
}).segment(clip(9)));
|
||||||
|
assertTrue(Thread.currentThread().isInterrupted());
|
||||||
|
} finally {
|
||||||
|
Thread.interrupted();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private ClipAnalysis clip(double durationSeconds) {
|
||||||
|
return new ClipAnalysis("source", "source.mp4", durationSeconds, "h264", "aac",
|
||||||
|
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void assertCommandPair(List<String> command, String option, String value) {
|
||||||
|
int index = command.indexOf(option);
|
||||||
|
assertThat(index).isGreaterThanOrEqualTo(0);
|
||||||
|
assertThat(command.get(index + 1)).isEqualTo(value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,96 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.example.videoclips.config.VideoClippingProperties;
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.concurrent.atomic.AtomicReference;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||||
|
|
||||||
|
class SourceAudioAnalyzerTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void analyzesLoudnessSilenceAndAudioSections() {
|
||||||
|
AtomicReference<List<String>> command = new AtomicReference<>();
|
||||||
|
VideoClippingProperties properties = new VideoClippingProperties();
|
||||||
|
properties.getEditing().setSilenceThresholdDb(-40);
|
||||||
|
properties.getEditing().setSilenceMinimumDurationSeconds(1.25);
|
||||||
|
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(properties, arguments -> {
|
||||||
|
command.set(arguments);
|
||||||
|
return new SourceAudioAnalyzer.ProcessResult(0, """
|
||||||
|
[silencedetect @ 0x1] silence_start: 2.5
|
||||||
|
[silencedetect @ 0x1] silence_end: 4.0 | silence_duration: 1.5
|
||||||
|
[Parsed_volumedetect_1 @ 0x2] mean_volume: -19.2 dB
|
||||||
|
[Parsed_volumedetect_1 @ 0x2] max_volume: -2.7 dB
|
||||||
|
""");
|
||||||
|
});
|
||||||
|
|
||||||
|
SourceAudioAnalysis analysis = analyzer.analyze(clip("aac"));
|
||||||
|
|
||||||
|
assertThat(analysis.audioPresent()).isTrue();
|
||||||
|
assertThat(analysis.meanVolumeDb()).isEqualTo(-19.2);
|
||||||
|
assertThat(analysis.maxVolumeDb()).isEqualTo(-2.7);
|
||||||
|
assertThat(analysis.silenceThresholdDb()).isEqualTo(-40);
|
||||||
|
assertThat(analysis.silenceDurationSeconds()).isEqualTo(1.25);
|
||||||
|
assertThat(analysis.sections()).containsExactly(
|
||||||
|
new AudioSection("audio_section_0001", "unclassified_audio", 0, 2.5, 2.5),
|
||||||
|
new AudioSection("audio_section_0002", "silence", 2.5, 4, 1.5),
|
||||||
|
new AudioSection("audio_section_0003", "unclassified_audio", 4, 8, 4)
|
||||||
|
);
|
||||||
|
assertCommandPair(command.get(), "-i", "source.mp4");
|
||||||
|
assertCommandPair(command.get(), "-af", "silencedetect=n=-40.0dB:d=1.25,volumedetect");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void returnsMissingAudioAnalysisWhenSourceHasNoAudioCodec() {
|
||||||
|
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||||
|
throw new AssertionError("ffmpeg should not run without an audio stream");
|
||||||
|
});
|
||||||
|
|
||||||
|
SourceAudioAnalysis analysis = analyzer.analyze(clip(null));
|
||||||
|
|
||||||
|
assertThat(analysis.audioPresent()).isFalse();
|
||||||
|
assertThat(analysis.sections()).containsExactly(
|
||||||
|
new AudioSection("audio_section_0001", "missing_audio", 0, 8, 8)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void rejectsFfmpegFailures() {
|
||||||
|
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command ->
|
||||||
|
new SourceAudioAnalyzer.ProcessResult(1, "failed"));
|
||||||
|
|
||||||
|
assertThrows(IllegalStateException.class, () -> analyzer.analyze(clip("aac")));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void reportsProcessStartupAndInterruptionFailures() {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||||
|
throw new IOException("missing executable");
|
||||||
|
}).analyze(clip("aac")));
|
||||||
|
|
||||||
|
try {
|
||||||
|
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||||
|
throw new InterruptedException("stopped");
|
||||||
|
}).analyze(clip("aac")));
|
||||||
|
assertTrue(Thread.currentThread().isInterrupted());
|
||||||
|
} finally {
|
||||||
|
Thread.interrupted();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private ClipAnalysis clip(String audioCodec) {
|
||||||
|
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", audioCodec,
|
||||||
|
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void assertCommandPair(List<String> command, String option, String value) {
|
||||||
|
int index = command.indexOf(option);
|
||||||
|
assertThat(index).isGreaterThanOrEqualTo(0);
|
||||||
|
assertThat(command.get(index + 1)).isEqualTo(value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,130 @@
|
||||||
|
package org.example.videoclips.editing;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static org.assertj.core.api.Assertions.assertThat;
|
||||||
|
|
||||||
|
class SourceVisualAnalyzerTest {
|
||||||
|
|
||||||
|
private final SourceVisualAnalyzer analyzer = new SourceVisualAnalyzer();
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void createsVisualAnalysisFromScoresThumbnailsAndSceneDensity() {
|
||||||
|
SourceVisualAnalysis analysis = analyzer.analyze(
|
||||||
|
clip("porsche-drive", "source/Porsche Drive.mp4", 0.72, 0.61, 0.0),
|
||||||
|
List.of("thumb-1.jpg", "thumb-2.jpg", "thumb-3.jpg", "thumb-4.jpg", "thumb-5.jpg"),
|
||||||
|
List.of(
|
||||||
|
new ShotSegment("shot_0001", 0, 8, 8, 0, 4),
|
||||||
|
new ShotSegment("shot_0002", 8, 16, 8, 0.5, 12),
|
||||||
|
new ShotSegment("shot_0003", 16, 24, 8, 0.6, 20)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
|
||||||
|
assertThat(analysis.blurScore()).isEqualTo(0.72);
|
||||||
|
assertThat(analysis.exposureScore()).isEqualTo(0.61);
|
||||||
|
assertThat(analysis.motionScore()).isEqualTo(0.5);
|
||||||
|
assertThat(analysis.compositionScore()).isEqualTo(0.75);
|
||||||
|
assertThat(analysis.facePresence()).isEqualTo("unknown_without_face_detector");
|
||||||
|
assertThat(analysis.objectLabels()).containsExactly(
|
||||||
|
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic"),
|
||||||
|
new VisualObjectLabel("car", 0.62, "metadata_heuristic")
|
||||||
|
);
|
||||||
|
assertThat(analysis.representativeThumbnails()).containsExactly("thumb-1.jpg", "thumb-3.jpg", "thumb-5.jpg");
|
||||||
|
assertThat(analysis.analysisMethod()).isEqualTo("metadata_thumbnail_scene_heuristic");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void fallsBackWhenNoVisualSignalsAreAvailable() {
|
||||||
|
SourceVisualAnalysis analysis = analyzer.analyze(
|
||||||
|
clip("unknown", "source/video.mp4", 0, 0, 0),
|
||||||
|
List.of(),
|
||||||
|
List.of()
|
||||||
|
);
|
||||||
|
|
||||||
|
assertThat(analysis.blurScore()).isEqualTo(0.5);
|
||||||
|
assertThat(analysis.exposureScore()).isEqualTo(0.5);
|
||||||
|
assertThat(analysis.motionScore()).isEqualTo(0.3);
|
||||||
|
assertThat(analysis.compositionScore()).isEqualTo(0.625);
|
||||||
|
assertThat(analysis.objectLabels()).containsExactly(
|
||||||
|
new VisualObjectLabel("unknown", 0.2, "metadata_heuristic")
|
||||||
|
);
|
||||||
|
assertThat(analysis.representativeThumbnails()).isEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void usesLocalCvProviderWhenConfigured() {
|
||||||
|
org.example.videoclips.config.VideoClippingProperties properties =
|
||||||
|
new org.example.videoclips.config.VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||||
|
properties,
|
||||||
|
(source, thumbnails, shotSegments) -> {
|
||||||
|
throw new AssertionError("heuristic fallback should not run");
|
||||||
|
},
|
||||||
|
(source, thumbnails, shotSegments) -> new SourceVisualAnalysis(source.clipId(), 0.9, 0.8,
|
||||||
|
0.7, 0.6, "faces_detected",
|
||||||
|
List.of(new VisualObjectLabel("person", 0.91, "mediapipe")), thumbnails, "local_cv")
|
||||||
|
);
|
||||||
|
|
||||||
|
SourceVisualAnalysis analysis = localAnalyzer.analyze(
|
||||||
|
clip("family", "family.mp4", 0, 0, 0),
|
||||||
|
List.of("thumb.jpg"),
|
||||||
|
List.of()
|
||||||
|
);
|
||||||
|
|
||||||
|
assertThat(analysis.analysisMethod()).isEqualTo("local_cv");
|
||||||
|
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("person", 0.91, "mediapipe"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void fallsBackToHeuristicWhenLocalCvFailsAndFallbackIsEnabled() {
|
||||||
|
org.example.videoclips.config.VideoClippingProperties properties =
|
||||||
|
new org.example.videoclips.config.VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||||
|
properties,
|
||||||
|
new HeuristicVisualAnalysisProvider(),
|
||||||
|
(source, thumbnails, shotSegments) -> {
|
||||||
|
throw new IllegalStateException("local cv unavailable");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
SourceVisualAnalysis analysis = localAnalyzer.analyze(
|
||||||
|
clip("porsche", "porsche.mp4", 0, 0, 0),
|
||||||
|
List.of("thumb.jpg"),
|
||||||
|
List.of()
|
||||||
|
);
|
||||||
|
|
||||||
|
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_failed_fallback_metadata_thumbnail_scene_heuristic");
|
||||||
|
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("porsche", 0.82,
|
||||||
|
"metadata_heuristic"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void failsWhenLocalCvFailsAndFallbackIsDisabled() {
|
||||||
|
org.example.videoclips.config.VideoClippingProperties properties =
|
||||||
|
new org.example.videoclips.config.VideoClippingProperties();
|
||||||
|
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||||
|
properties.getEditing().getVisualAnalysis().setFallbackToHeuristic(false);
|
||||||
|
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||||
|
properties,
|
||||||
|
new HeuristicVisualAnalysisProvider(),
|
||||||
|
(source, thumbnails, shotSegments) -> {
|
||||||
|
throw new IllegalStateException("local cv unavailable");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
org.junit.jupiter.api.Assertions.assertThrows(IllegalStateException.class, () -> localAnalyzer.analyze(
|
||||||
|
clip("porsche", "porsche.mp4", 0, 0, 0),
|
||||||
|
List.of("thumb.jpg"),
|
||||||
|
List.of()
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
private ClipAnalysis clip(String clipId, String sourcePath, double sharpness, double brightness, double motion) {
|
||||||
|
return new ClipAnalysis(clipId, sourcePath, 24.0, "h264", "aac", 1920, 1080, 30.0,
|
||||||
|
List.of(), null, null, motion, brightness, sharpness);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -30,6 +30,7 @@ class StoryboardPromptGeneratorTest {
|
||||||
assertThat(prompt)
|
assertThat(prompt)
|
||||||
.contains("clipId: clip_00001")
|
.contains("clipId: clip_00001")
|
||||||
.contains("durationSeconds: 8.0")
|
.contains("durationSeconds: 8.0")
|
||||||
|
.contains("The service can only render what the plan expresses")
|
||||||
.contains("Target duration: 60 seconds")
|
.contains("Target duration: 60 seconds")
|
||||||
.contains("Style: `cinematic-porsche-promo`")
|
.contains("Style: `cinematic-porsche-promo`")
|
||||||
.contains("\"decisions\"")
|
.contains("\"decisions\"")
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
video-clipping.folder-scheduler.enabled=false
|
||||||
|
video-clipping.editing.local-director.enabled=false
|
||||||
|
video-clipping.editing.highlight-scheduler.enabled=false
|
||||||
|
video-clipping.editing.visual-analysis.local-cv-worker.auto-start=false
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
fastapi==0.115.6
|
||||||
|
uvicorn[standard]==0.34.0
|
||||||
|
opencv-python==4.10.0.84
|
||||||
|
ultralytics==8.3.57
|
||||||
|
|
@ -0,0 +1,235 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Optional local CV worker for the Spring visual-analysis provider.
|
||||||
|
|
||||||
|
Run:
|
||||||
|
pip install -r tools/local_cv_requirements.txt
|
||||||
|
uvicorn tools.local_cv_worker:app --host 127.0.0.1 --port 8091
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
import cv2 # type: ignore
|
||||||
|
except ImportError: # pragma: no cover - optional runtime dependency
|
||||||
|
cv2 = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
from fastapi import FastAPI
|
||||||
|
except ImportError as exc: # pragma: no cover - fail clearly at worker startup
|
||||||
|
raise SystemExit("Install FastAPI first: pip install fastapi uvicorn") from exc
|
||||||
|
|
||||||
|
try:
|
||||||
|
from ultralytics import YOLO # type: ignore
|
||||||
|
except ImportError: # pragma: no cover - optional runtime dependency
|
||||||
|
YOLO = None
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(title="Local CV Visual Analysis Worker")
|
||||||
|
_yolo_model: Any | None = None
|
||||||
|
logging.basicConfig(level=os.getenv("LOCAL_CV_LOG_LEVEL", "INFO"))
|
||||||
|
log = logging.getLogger("local_cv_worker")
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health() -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"opencvAvailable": cv2 is not None,
|
||||||
|
"yoloAvailable": YOLO is not None,
|
||||||
|
"yoloModel": configured_yolo_model(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/v1/analyze-visuals")
|
||||||
|
def analyze_visuals(payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
started_at = time.monotonic()
|
||||||
|
source = payload.get("source", {})
|
||||||
|
thumbnails = [str(item) for item in payload.get("thumbnails", [])]
|
||||||
|
readable_thumbnails = [path for path in thumbnails if Path(path).is_file()]
|
||||||
|
clip_id = source.get("clipId", "source")
|
||||||
|
log.info(
|
||||||
|
"event=local_cv_worker_request_started clip_id=%s thumbnails=%d readable_thumbnails=%d "
|
||||||
|
"shot_segments=%d duration_seconds=%s",
|
||||||
|
clip_id,
|
||||||
|
len(thumbnails),
|
||||||
|
len(readable_thumbnails),
|
||||||
|
len(payload.get("shotSegments", [])),
|
||||||
|
source.get("durationSeconds", 0),
|
||||||
|
)
|
||||||
|
if len(readable_thumbnails) != len(thumbnails):
|
||||||
|
unreadable = [path for path in thumbnails if path not in readable_thumbnails]
|
||||||
|
log.warning(
|
||||||
|
"event=local_cv_worker_unreadable_thumbnails clip_id=%s unreadable_count=%d unreadable=%s",
|
||||||
|
clip_id,
|
||||||
|
len(unreadable),
|
||||||
|
unreadable[:10],
|
||||||
|
)
|
||||||
|
quality = quality_scores(readable_thumbnails)
|
||||||
|
labels = object_labels(readable_thumbnails)
|
||||||
|
|
||||||
|
response = {
|
||||||
|
"clipId": clip_id,
|
||||||
|
"blurScore": quality["blurScore"],
|
||||||
|
"exposureScore": quality["exposureScore"],
|
||||||
|
"motionScore": motion_score(payload.get("shotSegments", []), source.get("durationSeconds", 0)),
|
||||||
|
"compositionScore": composition_score(source, readable_thumbnails),
|
||||||
|
"facePresence": face_presence(readable_thumbnails),
|
||||||
|
"objectLabels": labels or [{"label": "unknown", "confidence": 0.2, "source": "local_cv_worker"}],
|
||||||
|
"representativeThumbnails": representative_thumbnails(readable_thumbnails or thumbnails),
|
||||||
|
"analysisMethod": "local_cv_worker_opencv_yolo" if labels else "local_cv_worker_opencv",
|
||||||
|
}
|
||||||
|
log.info(
|
||||||
|
"event=local_cv_worker_request_completed clip_id=%s elapsed_ms=%d blur_score=%s "
|
||||||
|
"exposure_score=%s object_labels=%d method=%s",
|
||||||
|
clip_id,
|
||||||
|
elapsed_ms(started_at),
|
||||||
|
response["blurScore"],
|
||||||
|
response["exposureScore"],
|
||||||
|
len(response["objectLabels"]),
|
||||||
|
response["analysisMethod"],
|
||||||
|
)
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def quality_scores(thumbnails: list[str]) -> dict[str, float]:
|
||||||
|
if cv2 is None or not thumbnails:
|
||||||
|
log.info(
|
||||||
|
"event=local_cv_worker_quality_defaulted reason=%s",
|
||||||
|
"opencv_unavailable" if cv2 is None else "no_readable_thumbnails",
|
||||||
|
)
|
||||||
|
return {"blurScore": 0.5, "exposureScore": 0.5}
|
||||||
|
|
||||||
|
blur_scores: list[float] = []
|
||||||
|
exposure_scores: list[float] = []
|
||||||
|
for thumbnail in thumbnails:
|
||||||
|
image = cv2.imread(thumbnail)
|
||||||
|
if image is None:
|
||||||
|
continue
|
||||||
|
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
||||||
|
laplacian_variance = float(cv2.Laplacian(gray, cv2.CV_64F).var())
|
||||||
|
blur_scores.append(clamp(laplacian_variance / 500.0))
|
||||||
|
mean_brightness = float(gray.mean()) / 255.0
|
||||||
|
exposure_scores.append(clamp(1.0 - abs(mean_brightness - 0.5) * 2.0))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"blurScore": average_or_default(blur_scores, 0.5),
|
||||||
|
"exposureScore": average_or_default(exposure_scores, 0.5),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def object_labels(thumbnails: list[str]) -> list[dict[str, Any]]:
|
||||||
|
started_at = time.monotonic()
|
||||||
|
model = yolo_model()
|
||||||
|
if model is None or not thumbnails:
|
||||||
|
log.info(
|
||||||
|
"event=local_cv_worker_object_detection_skipped reason=%s thumbnails=%d",
|
||||||
|
"model_unavailable" if model is None else "no_readable_thumbnails",
|
||||||
|
len(thumbnails),
|
||||||
|
)
|
||||||
|
return []
|
||||||
|
|
||||||
|
labels: dict[str, float] = {}
|
||||||
|
for result in model(thumbnails, verbose=False):
|
||||||
|
names = getattr(result, "names", {})
|
||||||
|
boxes = getattr(result, "boxes", None)
|
||||||
|
if boxes is None:
|
||||||
|
continue
|
||||||
|
for box in boxes:
|
||||||
|
label = str(names.get(int(box.cls[0]), "object"))
|
||||||
|
confidence = float(box.conf[0])
|
||||||
|
labels[label] = max(labels.get(label, 0.0), confidence)
|
||||||
|
|
||||||
|
detected = [
|
||||||
|
{"label": label, "confidence": round(clamp(confidence), 3), "source": "yolo"}
|
||||||
|
for label, confidence in sorted(labels.items(), key=lambda item: item[1], reverse=True)[:10]
|
||||||
|
]
|
||||||
|
log.info(
|
||||||
|
"event=local_cv_worker_object_detection_completed elapsed_ms=%d thumbnails=%d labels=%d",
|
||||||
|
elapsed_ms(started_at),
|
||||||
|
len(thumbnails),
|
||||||
|
len(detected),
|
||||||
|
)
|
||||||
|
return detected
|
||||||
|
|
||||||
|
|
||||||
|
def yolo_model() -> Any | None:
|
||||||
|
global _yolo_model
|
||||||
|
model_path = configured_yolo_model()
|
||||||
|
if YOLO is None or not model_path:
|
||||||
|
return None
|
||||||
|
if _yolo_model is None:
|
||||||
|
started_at = time.monotonic()
|
||||||
|
log.info("event=local_cv_worker_yolo_model_loading model=%s", model_path)
|
||||||
|
_yolo_model = YOLO(model_path)
|
||||||
|
log.info("event=local_cv_worker_yolo_model_loaded model=%s elapsed_ms=%d", model_path, elapsed_ms(started_at))
|
||||||
|
return _yolo_model
|
||||||
|
|
||||||
|
|
||||||
|
def configured_yolo_model() -> str:
|
||||||
|
if os.getenv("LOCAL_CV_DISABLE_YOLO", "false").lower() == "true":
|
||||||
|
return ""
|
||||||
|
return os.getenv("LOCAL_CV_YOLO_MODEL", "yolov8n.pt")
|
||||||
|
|
||||||
|
|
||||||
|
def face_presence(thumbnails: list[str]) -> str:
|
||||||
|
if cv2 is None or not thumbnails:
|
||||||
|
return "unknown_without_face_detector"
|
||||||
|
cascade_path = getattr(cv2.data, "haarcascades", "") + "haarcascade_frontalface_default.xml"
|
||||||
|
if not Path(cascade_path).is_file():
|
||||||
|
return "unknown_without_face_detector"
|
||||||
|
cascade = cv2.CascadeClassifier(cascade_path)
|
||||||
|
for thumbnail in thumbnails:
|
||||||
|
image = cv2.imread(thumbnail)
|
||||||
|
if image is None:
|
||||||
|
continue
|
||||||
|
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
||||||
|
faces = cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=4)
|
||||||
|
if len(faces) > 0:
|
||||||
|
return "faces_detected"
|
||||||
|
return "no_faces_detected"
|
||||||
|
|
||||||
|
|
||||||
|
def motion_score(shot_segments: list[dict[str, Any]], duration_seconds: float) -> float:
|
||||||
|
try:
|
||||||
|
duration = float(duration_seconds)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
duration = 0.0
|
||||||
|
if duration <= 0:
|
||||||
|
return 0.3
|
||||||
|
cuts_per_minute = max(0, len(shot_segments) - 1) / duration * 60.0
|
||||||
|
return round(clamp(0.25 + cuts_per_minute / 20.0), 3)
|
||||||
|
|
||||||
|
|
||||||
|
def composition_score(source: dict[str, Any], thumbnails: list[str]) -> float:
|
||||||
|
width = float(source.get("width", 0) or 0)
|
||||||
|
height = float(source.get("height", 0) or 0)
|
||||||
|
aspect_ratio = width / height if height else 0
|
||||||
|
aspect_score = 0.8 if 1.70 <= aspect_ratio <= 1.90 else 0.55
|
||||||
|
thumbnail_score = 0.7 if thumbnails else 0.45
|
||||||
|
return round((aspect_score + thumbnail_score) / 2.0, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def representative_thumbnails(thumbnails: list[str]) -> list[str]:
|
||||||
|
if len(thumbnails) <= 3:
|
||||||
|
return thumbnails
|
||||||
|
return [thumbnails[0], thumbnails[len(thumbnails) // 2], thumbnails[-1]]
|
||||||
|
|
||||||
|
|
||||||
|
def average_or_default(values: list[float], default: float) -> float:
|
||||||
|
if not values:
|
||||||
|
return default
|
||||||
|
return round(sum(values) / len(values), 3)
|
||||||
|
|
||||||
|
|
||||||
|
def clamp(value: float) -> float:
|
||||||
|
return max(0.0, min(1.0, value))
|
||||||
|
|
||||||
|
|
||||||
|
def elapsed_ms(started_at: float) -> int:
|
||||||
|
return int((time.monotonic() - started_at) * 1000)
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HOST="${LOCAL_CV_HOST:-127.0.0.1}"
|
||||||
|
PORT="${LOCAL_CV_PORT:-8091}"
|
||||||
|
MODEL="${LOCAL_CV_YOLO_MODEL:-yolov8n.pt}"
|
||||||
|
|
||||||
|
if [ ! -d ".venv-local-cv" ]; then
|
||||||
|
python3 -m venv .venv-local-cv
|
||||||
|
fi
|
||||||
|
|
||||||
|
. .venv-local-cv/bin/activate
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
python -m pip install -r tools/local_cv_requirements.txt
|
||||||
|
|
||||||
|
if [ "${LOCAL_CV_DISABLE_YOLO:-false}" != "true" ]; then
|
||||||
|
LOCAL_CV_YOLO_MODEL="$MODEL" python -c "import os; from ultralytics import YOLO; YOLO(os.environ['LOCAL_CV_YOLO_MODEL'])"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${LOCAL_CV_PRELOAD_ONLY:-false}" = "true" ]; then
|
||||||
|
echo "Local CV dependencies and model are ready."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec uvicorn tools.local_cv_worker:app --host "$HOST" --port "$PORT"
|
||||||
Loading…
Reference in New Issue