Compare commits
10 Commits
ed337c1268
...
618ba9af49
| Author | SHA1 | Date |
|---|---|---|
|
|
618ba9af49 | |
|
|
adc979eea8 | |
|
|
80eca92d56 | |
|
|
8195b58552 | |
|
|
e2f49e2e5a | |
|
|
9dc720b0cc | |
|
|
82e557905e | |
|
|
eea619a2d3 | |
|
|
fc17578549 | |
|
|
5d5f8022f2 |
|
|
@ -36,3 +36,9 @@ build/
|
|||
|
||||
### Mac OS ###
|
||||
.DS_Store
|
||||
|
||||
### Local CV runtime ###
|
||||
.venv-local-cv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
yolov*.pt
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ video-clipping:
|
|||
project-directory: ./output/edit-projects
|
||||
thumbnail-count-per-clip: 5
|
||||
proxy-enabled: true
|
||||
target-duration-seconds: 60
|
||||
target-duration-seconds: 600
|
||||
output-width: 1920
|
||||
output-height: 1080
|
||||
output-frame-rate: 30
|
||||
|
|
@ -186,6 +186,13 @@ The AI director should inspect:
|
|||
- `thumbnails/*.jpg`
|
||||
- `proxies/*.mp4` if more visual context is needed
|
||||
|
||||
After the plan is imported, the service also runs an asset materialization stage:
|
||||
|
||||
- It reuses matching generated music, SFX, and voiceover assets from the shared cache folders.
|
||||
- It copies reusable assets into the project-local `audio/` folder so the renderer can use them.
|
||||
- It writes reusable asset requests under `assets/requests/` when a needed asset is missing.
|
||||
- It waits to render if required SFX assets are still missing.
|
||||
|
||||
The AI director must write this file:
|
||||
|
||||
```text
|
||||
|
|
@ -239,6 +246,14 @@ Rules the AI director must follow:
|
|||
|
||||
The renderer can mix optional WAV files into the final output.
|
||||
|
||||
Shared cache locations:
|
||||
|
||||
- Music: `input/highlights/assets/music/`
|
||||
- SFX: `input/highlights/assets/sfx/`
|
||||
- Voiceover cache: `output/highlight-projects/_voiceover-cache/`
|
||||
|
||||
If the service or an AI worker generates an asset there, later projects can reuse it automatically.
|
||||
|
||||
Place optional assets here before rendering:
|
||||
|
||||
```text
|
||||
|
|
@ -308,6 +323,14 @@ After rendering completes, the final output is:
|
|||
output/edit-projects/porsche-session-001/final.mp4
|
||||
```
|
||||
|
||||
The renderer also publishes each final rendered segment clip that was concatenated into the full video:
|
||||
|
||||
```text
|
||||
output/edit-projects/porsche-session-001/rendered-clips/clip_0001.mp4
|
||||
output/edit-projects/porsche-session-001/rendered-clips/clip_0002.mp4
|
||||
output/edit-projects/porsche-session-001/rendered-clips/clip_0003.mp4
|
||||
```
|
||||
|
||||
The render manifest is:
|
||||
|
||||
```text
|
||||
|
|
@ -320,11 +343,12 @@ The QA report is:
|
|||
output/edit-projects/porsche-session-001/qa-report.json
|
||||
```
|
||||
|
||||
The manifest is useful for debugging because it records the selected clips, output path, duration, and FFmpeg command summaries.
|
||||
The manifest is useful for debugging because it records the selected clips, rendered segment clip paths, output path, duration, and FFmpeg command summaries.
|
||||
|
||||
The QA report is useful for acceptance because it records:
|
||||
|
||||
- final output existence
|
||||
- rendered segment clip existence
|
||||
- duration consistency with the edit timeline
|
||||
- required asset resolution
|
||||
- text overlay timeline and placement safety
|
||||
|
|
|
|||
|
|
@ -0,0 +1,417 @@
|
|||
# Highlight Rendering Gap Closure Plan
|
||||
|
||||
## Problem
|
||||
|
||||
The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.
|
||||
|
||||
Observed current output:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/
|
||||
analysis/
|
||||
source-analysis.json
|
||||
visual-analysis.json
|
||||
audio-analysis.json
|
||||
scene-segments.json
|
||||
frames/*.jpg
|
||||
contact-sheets/*.jpg
|
||||
proxies/*.mp4
|
||||
source/<source-video>
|
||||
highlights/
|
||||
```
|
||||
|
||||
The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights/<highlight-id>/final.mp4`.
|
||||
|
||||
## Goal
|
||||
|
||||
When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||
```
|
||||
|
||||
The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.
|
||||
|
||||
## Key Design Decision
|
||||
|
||||
Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is.
|
||||
|
||||
Reason:
|
||||
|
||||
- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`.
|
||||
- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights/<highlight-id>/`.
|
||||
- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.
|
||||
|
||||
Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`.
|
||||
|
||||
## Target Flow
|
||||
|
||||
1. `HighlightSourceScheduler` claims one source video.
|
||||
2. `HighlightSourceAnalyzer` creates analysis artifacts.
|
||||
3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis.
|
||||
4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate.
|
||||
5. New `HighlightRenderer` renders each plan into `highlights/<highlight-id>/`.
|
||||
6. New QA step probes each rendered MP4 and writes `qa-report.json`.
|
||||
7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths.
|
||||
|
||||
## Milestones
|
||||
|
||||
- [ ] Milestone 1: Persist highlight candidates for highlight-source projects.
|
||||
- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates.
|
||||
- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights/<highlight-id>/`.
|
||||
- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
|
||||
- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
|
||||
- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log.
|
||||
- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
|
||||
- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.
|
||||
|
||||
## Milestone Details
|
||||
|
||||
### Milestone 1: Persist Highlight Candidates
|
||||
|
||||
Add a component:
|
||||
|
||||
```text
|
||||
HighlightCandidateGenerator
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
HighlightSourceAnalysis
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
analysis/highlight-candidates.json
|
||||
```
|
||||
|
||||
Candidate rules for the first version:
|
||||
|
||||
- Use `scene-segments.json` as the primary temporal units.
|
||||
- If only one scene exists, split the source into overlapping 8-20 second windows.
|
||||
- Score candidates using available fields:
|
||||
- `visualAnalysis.motionScore`
|
||||
- `visualAnalysis.compositionScore`
|
||||
- `visualAnalysis.blurScore`
|
||||
- `visualAnalysis.exposureScore`
|
||||
- `audioAnalysis.sections`
|
||||
- scene duration and position
|
||||
- Prefer candidates between 8 and 35 seconds.
|
||||
- Avoid first/last black or silent areas when audio data suggests silence.
|
||||
- Limit initial output to top 3 candidates.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_candidates_started project_id=...
|
||||
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
|
||||
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Generates candidates from multiple scene segments.
|
||||
- Falls back to fixed windows when only one scene exists.
|
||||
- Rejects or downranks invalid, too-short, too-long, or blurry candidates.
|
||||
- Persists `analysis/highlight-candidates.json`.
|
||||
|
||||
### Milestone 2: Generate Deterministic Edit Plans
|
||||
|
||||
Add a component:
|
||||
|
||||
```text
|
||||
HighlightEditPlanGenerator
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
HighlightSourceAnalysis
|
||||
List<HighlightCandidate>
|
||||
```
|
||||
|
||||
Output per candidate:
|
||||
|
||||
```text
|
||||
highlights/<highlight-id>/edit-plan.json
|
||||
highlights/<highlight-id>/storyboard.md
|
||||
```
|
||||
|
||||
First version plan rules:
|
||||
|
||||
- Create one highlight per selected candidate.
|
||||
- Use 3-6 edit decisions per highlight.
|
||||
- For a short candidate, use the candidate range as one segment.
|
||||
- For a longer candidate, split into smaller cuts with simple pacing.
|
||||
- Use deterministic category style:
|
||||
- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
|
||||
- family-like/faces: warm memory style, softer overlays.
|
||||
- food-like/close action: rhythmic sensory style.
|
||||
- generic: clean cinematic montage.
|
||||
- Add optional text overlays only when safe.
|
||||
- Add voiceover text as script metadata, but do not block render when no TTS provider exists.
|
||||
- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_edit_plan_started project_id=... candidate_id=...
|
||||
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
|
||||
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Produces valid `EditPlan` timestamps inside source duration.
|
||||
- Produces stable output for the same analysis.
|
||||
- Creates storyboard markdown.
|
||||
- Does not require external AI.
|
||||
|
||||
### Milestone 3: Implement Highlight Renderer
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
HighlightRenderer
|
||||
FfmpegHighlightRenderer
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
projectId
|
||||
highlightId
|
||||
highlights/<highlight-id>/edit-plan.json
|
||||
analysis/source-analysis.json
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
highlights/<highlight-id>/final.mp4
|
||||
highlights/<highlight-id>/preview.mp4
|
||||
highlights/<highlight-id>/render-manifest.json
|
||||
highlights/<highlight-id>/qa-report.json
|
||||
```
|
||||
|
||||
Renderer requirements for first version:
|
||||
|
||||
- Trim from the original source video using exact timestamps.
|
||||
- Concatenate selected segments.
|
||||
- Apply simple visual treatment using FFmpeg filters:
|
||||
- scale/pad to configured output size
|
||||
- optional contrast/saturation/sharpen
|
||||
- optional vignette or fade
|
||||
- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback.
|
||||
- Mix source audio.
|
||||
- Optionally mix music/SFX/voiceover when local assets exist.
|
||||
- Normalize output audio loudness when audio exists.
|
||||
- Produce `preview.mp4` as a lower-resolution copy or direct transcode.
|
||||
- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.
|
||||
|
||||
Important:
|
||||
|
||||
- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
|
||||
- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_render_started project_id=... highlight_id=...
|
||||
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
|
||||
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
|
||||
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
|
||||
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
|
||||
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
|
||||
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
|
||||
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Builds expected FFmpeg trim commands.
|
||||
- Builds concat and final render commands.
|
||||
- Writes final output path under `highlights/<highlight-id>/`.
|
||||
- Writes render manifest.
|
||||
- Handles missing optional assets.
|
||||
- Fails on invalid source timestamps.
|
||||
|
||||
### Milestone 4: Local Asset Fallbacks
|
||||
|
||||
Current rendered highlights must work even with no licensed asset library.
|
||||
|
||||
Add fallback behavior:
|
||||
|
||||
- If no music asset exists, keep source audio.
|
||||
- If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
|
||||
- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio.
|
||||
- If no LUT exists, use named FFmpeg color preset.
|
||||
- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
|
||||
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Render plan with no local assets still produces `final.mp4`.
|
||||
- Manifest records skipped optional assets.
|
||||
- Voiceover script file is still created.
|
||||
|
||||
### Milestone 5: Wire Into Scheduler
|
||||
|
||||
Update `HighlightSourceScheduler.createProject` after analysis:
|
||||
|
||||
```text
|
||||
analysis = analyzer.analyze(projectId)
|
||||
candidates = candidateGenerator.generate(projectId, analysis)
|
||||
plans = planGenerator.generate(projectId, analysis, candidates)
|
||||
renderResults = renderer.render(projectId, plans)
|
||||
move source to processed
|
||||
log flow complete
|
||||
```
|
||||
|
||||
Configuration:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
editing:
|
||||
highlight-scheduler:
|
||||
render-enabled: true
|
||||
max-highlights-per-source: 3
|
||||
highlight-min-duration-seconds: 8
|
||||
highlight-max-duration-seconds: 35
|
||||
require-director-approval: false
|
||||
```
|
||||
|
||||
If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering.
|
||||
|
||||
Required logs:
|
||||
|
||||
```text
|
||||
event=highlight_flow_started scan_id=... project_id=... source_file=...
|
||||
event=highlight_flow_analysis_completed scan_id=... project_id=...
|
||||
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
|
||||
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Scheduler produces at least one `highlights/<highlight-id>/final.mp4` from a valid fixture video.
|
||||
- Scheduler leaves no source video in `source` or `working` after success.
|
||||
- Scheduler moves failed source to rejected and logs failure.
|
||||
- Rendering can be disabled for analysis-only mode.
|
||||
|
||||
### Milestone 6: Rich End-To-End Logging
|
||||
|
||||
Add a single correlation ID:
|
||||
|
||||
```text
|
||||
flow_id=<project-id>:<scan-id>
|
||||
```
|
||||
|
||||
Every log in the highlight path should include:
|
||||
|
||||
- `scan_id`
|
||||
- `project_id`
|
||||
- `highlight_id` when applicable
|
||||
- `flow_id`
|
||||
- `elapsed_ms` for completed steps
|
||||
|
||||
The final success log must be:
|
||||
|
||||
```text
|
||||
event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...
|
||||
```
|
||||
|
||||
The final failure log must be:
|
||||
|
||||
```text
|
||||
event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- Use a log capture test to assert `highlight_flow_completed` is emitted on success.
|
||||
- Use a failure test to assert `highlight_flow_failed` includes the failed step.
|
||||
|
||||
### Milestone 7: Tests And Coverage
|
||||
|
||||
Add focused unit tests:
|
||||
|
||||
- `HighlightCandidateGeneratorTest`
|
||||
- `HighlightEditPlanGeneratorTest`
|
||||
- `FfmpegHighlightRendererTest`
|
||||
- `HighlightRenderManifestTest`
|
||||
- `HighlightSourceSchedulerRenderFlowTest`
|
||||
|
||||
Add integration test:
|
||||
|
||||
```text
|
||||
HighlightRenderingIntegrationTest
|
||||
```
|
||||
|
||||
Requirements:
|
||||
|
||||
- Create a tiny FFmpeg fixture video.
|
||||
- Place it in a temp highlight source folder.
|
||||
- Run the scheduler once.
|
||||
- Assert:
|
||||
- `analysis/source-analysis.json` exists.
|
||||
- `analysis/highlight-candidates.json` exists.
|
||||
- `highlights/<highlight-id>/edit-plan.json` exists.
|
||||
- `highlights/<highlight-id>/final.mp4` exists.
|
||||
- `highlights/<highlight-id>/render-manifest.json` exists.
|
||||
- `highlights/<highlight-id>/qa-report.json` exists.
|
||||
- `highlight_flow_completed` log exists.
|
||||
|
||||
### Milestone 8: Runbook Update
|
||||
|
||||
Update:
|
||||
|
||||
```text
|
||||
docs/cinematic-editing-runbook.md
|
||||
docs/production-cinematic-highlight-editing-plan.md
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
- Exact output location for final highlight videos.
|
||||
- How to tell whether the project is analysis-only or rendered.
|
||||
- How to enable/disable automatic rendering.
|
||||
- How to inspect `highlight-candidates.json`.
|
||||
- How to inspect per-highlight `edit-plan.json`.
|
||||
- How to inspect `render-manifest.json` and `qa-report.json`.
|
||||
- Troubleshooting table for empty `highlights/`.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`.
|
||||
2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards.
|
||||
3. Implement `FfmpegHighlightRenderer` with minimal no-asset render.
|
||||
4. Wire renderer behind `highlight-scheduler.render-enabled`.
|
||||
5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs.
|
||||
6. Add integration test proving `final.mp4` is created.
|
||||
7. Add optional assets, overlays, audio mix, and QA improvements.
|
||||
8. Update runbooks.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4`.
|
||||
- The source file is moved to `input/highlights/processed`.
|
||||
- The project contains `analysis/highlight-candidates.json`.
|
||||
- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`.
|
||||
- Logs contain `highlight_flow_completed` with final output paths.
|
||||
- Missing optional assets do not prevent a basic highlight render.
|
||||
- Full Maven tests pass.
|
||||
|
|
@ -0,0 +1,176 @@
|
|||
# Local CV Visual Analysis Guide
|
||||
|
||||
This service can use a local computer-vision worker for highlight visual analysis.
|
||||
|
||||
## Default Behavior
|
||||
|
||||
The packaged `application.yml` defaults to `local-cv` with heuristic fallback enabled:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
editing:
|
||||
visual-analysis:
|
||||
provider: local-cv
|
||||
fallback-to-heuristic: true
|
||||
```
|
||||
|
||||
This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.
|
||||
|
||||
## Enable A Local CV Worker
|
||||
|
||||
Run a local worker that accepts JSON over HTTP, then start this Spring service with:
|
||||
|
||||
```yaml
|
||||
video-clipping:
|
||||
editing:
|
||||
visual-analysis:
|
||||
provider: local-cv
|
||||
endpoint: http://127.0.0.1:8091/v1/analyze-visuals
|
||||
timeout-ms: 30000
|
||||
fallback-to-heuristic: true
|
||||
local-cv-worker:
|
||||
auto-start: true
|
||||
script: ./tools/run_local_cv_worker.sh
|
||||
startup-wait-ms: 120000
|
||||
health-path: /health
|
||||
health-check-interval-ms: 1000
|
||||
```
|
||||
|
||||
Equivalent environment variables:
|
||||
|
||||
```bash
|
||||
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
|
||||
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
|
||||
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
|
||||
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=120000
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH=/health
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS=1000
|
||||
```
|
||||
|
||||
This repository includes an optional starter worker:
|
||||
|
||||
```bash
|
||||
tools/run_local_cv_worker.sh
|
||||
```
|
||||
|
||||
The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
|
||||
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.
|
||||
|
||||
Default model behavior:
|
||||
|
||||
- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
|
||||
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
|
||||
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
|
||||
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.
|
||||
|
||||
Health check:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8091/health
|
||||
```
|
||||
|
||||
## Preload Dependencies And Model
|
||||
|
||||
Do this once before starting the Spring service if you do not want dependency installation or YOLO model download to happen during application startup:
|
||||
|
||||
```bash
|
||||
LOCAL_CV_PRELOAD_ONLY=true tools/run_local_cv_worker.sh
|
||||
```
|
||||
|
||||
This creates `.venv-local-cv`, installs `tools/local_cv_requirements.txt`, and downloads/caches the configured YOLO model. After that, Spring-managed startup should be much faster.
|
||||
|
||||
Do not commit `.venv-local-cv`, Python wheels, Torch binaries, or YOLO weights into source control. They are large, platform-specific runtime artifacts. For production, prefer a prebuilt container image or a VM/bootstrap step that runs the preload command during deployment.
|
||||
|
||||
## Spring-Managed Worker Startup
|
||||
|
||||
The Spring application can start the local CV worker for local runs:
|
||||
|
||||
```bash
|
||||
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
|
||||
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
|
||||
mvn spring-boot:run
|
||||
```
|
||||
|
||||
When `auto-start` is enabled, Spring starts `tools/run_local_cv_worker.sh`, derives `LOCAL_CV_HOST` and `LOCAL_CV_PORT` from the configured visual-analysis endpoint, polls the configured health endpoint until it returns HTTP 2xx, streams worker output into application logs, and stops the worker process during application shutdown.
|
||||
|
||||
Keep `auto-start` disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates `.venv-local-cv`, installs Python dependencies, and may download/cache the configured YOLO model. Use the preload command or a prebuilt image to avoid that startup cost.
|
||||
|
||||
## Worker Request Contract
|
||||
|
||||
The Spring service sends:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"clipId": "porsche-drive",
|
||||
"sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
|
||||
"durationSeconds": 42.0,
|
||||
"videoCodec": "h264",
|
||||
"audioCodec": "aac",
|
||||
"width": 1920,
|
||||
"height": 1080,
|
||||
"frameRate": 30.0
|
||||
},
|
||||
"thumbnails": [
|
||||
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
||||
],
|
||||
"shotSegments": [
|
||||
{
|
||||
"shotId": "shot_0001",
|
||||
"startSeconds": 0.0,
|
||||
"endSeconds": 8.0,
|
||||
"durationSeconds": 8.0,
|
||||
"representativeTimestampSeconds": 4.0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Worker Response Contract
|
||||
|
||||
The worker must return:
|
||||
|
||||
```json
|
||||
{
|
||||
"clipId": "porsche-drive",
|
||||
"blurScore": 0.81,
|
||||
"exposureScore": 0.67,
|
||||
"motionScore": 0.73,
|
||||
"compositionScore": 0.79,
|
||||
"facePresence": "faces_detected",
|
||||
"objectLabels": [
|
||||
{
|
||||
"label": "car",
|
||||
"confidence": 0.91,
|
||||
"source": "yolo"
|
||||
},
|
||||
{
|
||||
"label": "luxury sports car",
|
||||
"confidence": 0.84,
|
||||
"source": "clip"
|
||||
}
|
||||
],
|
||||
"representativeThumbnails": [
|
||||
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
||||
],
|
||||
"analysisMethod": "local_cv_yolo_clip_mediapipe"
|
||||
}
|
||||
```
|
||||
|
||||
Scores must be normalized from `0.0` to `1.0`.
|
||||
|
||||
## Recommended Local Model Split
|
||||
|
||||
- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
|
||||
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
|
||||
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
|
||||
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.
|
||||
|
||||
## Runtime Behavior
|
||||
|
||||
When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.
|
||||
|
||||
When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.
|
||||
|
|
@ -0,0 +1,310 @@
|
|||
# Production Cinematic AI Director Prompt
|
||||
|
||||
Use this prompt with a filesystem-capable AI agent such as Codex or Claude when a highlight project has already been analyzed by the service.
|
||||
|
||||
Expected project folder:
|
||||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/
|
||||
```
|
||||
|
||||
The AI director must inspect the project artifacts, choose all worthwhile highlight moments, and write a strict renderable edit plan. The goal is a platform-ready cinematic highlight video with strong pacing, fitting sound design, fitting voiceover, tasteful text overlays, and visual treatments that match the actual content.
|
||||
|
||||
## Prompt
|
||||
|
||||
```text
|
||||
You are a senior commercial video editor, creative director, sound designer, and post-production supervisor.
|
||||
|
||||
Your assignment is to create a production-ready cinematic highlight edit plan from the analyzed project folder. The final rendered video must feel strong enough for a public video streaming platform: clear hook, strong visual selection, coherent micro-story, professional pacing, fitting sound design, tasteful overlays, and a voiceover that matches the actual footage. Do not promise revenue or make unsupported claims; optimize for viewer retention and professional presentation.
|
||||
|
||||
PROJECT FOLDER
|
||||
- Work only inside this folder: output/highlight-projects/<project-id>/
|
||||
- Read all available project files before deciding:
|
||||
- project.json
|
||||
- manifest.json
|
||||
- analysis/source-analysis.json
|
||||
- analysis/ffprobe.json
|
||||
- analysis/scene-segments.json
|
||||
- analysis/audio-analysis.json
|
||||
- analysis/visual-analysis.json
|
||||
- analysis/contact-sheets/*
|
||||
- analysis/frames/*
|
||||
- analysis/proxies/* when motion needs closer inspection
|
||||
- assets/music/*, assets/sfx/*, assets/voiceover/*, assets/overlays/* if present
|
||||
|
||||
OUTPUT FILE
|
||||
- Write the final edit plan JSON to:
|
||||
director/edit-plan.json
|
||||
- Return JSON only in that file.
|
||||
- Do not wrap JSON in Markdown fences.
|
||||
- Do not render video.
|
||||
- Do not modify source video, analysis files, thumbnails, contact sheets, or proxies.
|
||||
|
||||
PRIMARY GOAL
|
||||
Create one cinematic highlight video that is specific to the provided footage and content type. The highlight length must be content-driven, not fixed. Use all strong highlights from the source, but do not pad with weak or repetitive footage just to reach a duration.
|
||||
|
||||
The edit must have:
|
||||
- A 1-3 second opening hook that immediately shows the most visually interesting or emotionally compelling moment.
|
||||
- A clear middle section with rising energy, variety, and no repetitive shot choices.
|
||||
- A satisfying final hero/payoff shot.
|
||||
- A final duration that fits the source material and keeps audience retention high.
|
||||
- Beat-aware pacing and deliberate shot lengths.
|
||||
- Specific visual treatment notes that the renderer can execute: cinematic grade, punch-in crop, subtle speed ramp, vignette, sharpness, contrast, saturation, slow-motion emphasis, or clean natural grade.
|
||||
- Fitting music direction and sound effects timing.
|
||||
- Voiceover text that comments on what is actually visible or strongly implied by the footage.
|
||||
- Short text overlays only where they improve retention or clarity.
|
||||
|
||||
DELIVERABLE EXPECTATION
|
||||
The service must ultimately render both:
|
||||
- the full final highlight video
|
||||
- each rendered timeline segment clip that was used to assemble the full final highlight video
|
||||
|
||||
Expected output for the current edit-project renderer:
|
||||
- final.mp4
|
||||
- rendered-clips/clip_0001.mp4
|
||||
- rendered-clips/clip_0002.mp4
|
||||
- rendered-clips/clip_0003.mp4
|
||||
- render-manifest.json with renderedClipPaths
|
||||
|
||||
Expected output for the highlight-project renderer when implemented:
|
||||
- highlights/<highlight-id>/final.mp4
|
||||
- highlights/<highlight-id>/rendered-clips/clip_0001.mp4
|
||||
- highlights/<highlight-id>/rendered-clips/clip_0002.mp4
|
||||
- highlights/<highlight-id>/rendered-clips/clip_0003.mp4
|
||||
- highlights/<highlight-id>/render-manifest.json with renderedClipPaths
|
||||
|
||||
CONTENT CATEGORY DECISION
|
||||
Classify the project into exactly one content category:
|
||||
- car_vlog
|
||||
- food_vlog
|
||||
- family_vlog
|
||||
- generic_vlog
|
||||
|
||||
Use the category to direct the whole edit.
|
||||
|
||||
For car_vlog:
|
||||
- Make it premium, powerful, precise, and aspirational.
|
||||
- Prioritize hero angles, reflections, wheels, badges, road movement, cockpit details, acceleration, reveal shots, light, shadow, and motion.
|
||||
- Use punchy cuts, speed ramps, whooshes, impact hits, engine/exhaust SFX only when appropriate, cinematic contrast, and confident short overlays.
|
||||
- Voiceover should praise what is actually visible: stance, motion, design, presence, craft, road feel. Do not invent specifications, price, horsepower, rarity, or model facts unless visible in the footage.
|
||||
|
||||
For food_vlog:
|
||||
- Make it sensory, warm, rhythmic, and appetizing.
|
||||
- Prioritize texture, slicing, pouring, steam, sizzling, plating, table reveals, reactions, and macro-like detail.
|
||||
- Use tactile SFX, warm grade, gentle groove, ingredient/payoff overlays, and voiceover focused on craft, freshness, texture, and taste.
|
||||
|
||||
For family_vlog:
|
||||
- Make it warm, emotional, human, and memory-focused.
|
||||
- Prioritize faces, reactions, laughter, hugs, children, travel reveals, meaningful moments, and natural dialogue.
|
||||
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
|
||||
- Voiceover should feel personal and reflective, not promotional.
|
||||
|
||||
For generic_vlog:
|
||||
- Build a clear mini-story from novelty, clean composition, motion, reaction, place, process, and payoff.
|
||||
- Use restrained cinematic effects and explain only what the footage supports.
|
||||
|
||||
SHOT SELECTION RULES
|
||||
Before writing JSON, inspect the contact sheets and representative frames.
|
||||
|
||||
Create a private selection table for yourself:
|
||||
- candidate source range
|
||||
- visual appeal
|
||||
- motion/energy
|
||||
- uniqueness
|
||||
- story role
|
||||
- audio opportunity
|
||||
- risk: blur, darkness, repetitive angle, dead time
|
||||
|
||||
Then use only the best ranges. Avoid:
|
||||
- repeated near-identical shots unless repetition is intentional
|
||||
- long boring segments
|
||||
- badly blurred or unusably dark footage unless it is the only important moment
|
||||
- shots with no story purpose
|
||||
- unsupported narration
|
||||
|
||||
TIMELINE AND PACING RULES
|
||||
- Do not force a strict 60 second edit unless the footage actually supports it.
|
||||
- Derive the final duration from the input source length and number of strong moments.
|
||||
- Include every strong highlight moment that improves the final video.
|
||||
- Exclude weak, repetitive, technically poor, or dead-time footage even if the source is long.
|
||||
- Maximum final duration is 600 seconds, which is 10 minutes.
|
||||
- Most videos should be shorter than the maximum. Use 10 minutes only when the source has enough distinct, high-quality highlights to justify it.
|
||||
- Recommended duration guide:
|
||||
- Source under 30 seconds: final highlight usually 8-20 seconds.
|
||||
- Source 30-90 seconds: final highlight usually 15-45 seconds.
|
||||
- Source 90 seconds-5 minutes: final highlight usually 30-120 seconds.
|
||||
- Source 5-20 minutes: final highlight usually 60-180 seconds.
|
||||
- Source over 20 minutes: final highlight can be longer, but only if the footage has enough distinct high-quality moments.
|
||||
- The JSON field targetDurationSeconds must equal the actual planned final timeline duration, not a fixed configured default.
|
||||
- The final decision timelineEndSeconds must equal targetDurationSeconds.
|
||||
- Timeline must start at 0.0.
|
||||
- Every decision timelineStartSeconds must equal the previous decision timelineEndSeconds.
|
||||
- No overlaps. No gaps.
|
||||
- For each decision:
|
||||
rendered duration = (sourceEndSeconds - sourceStartSeconds) / playbackSpeed
|
||||
timelineEndSeconds - timelineStartSeconds must equal rendered duration within 0.05 seconds.
|
||||
- playbackSpeed must be between 0.25 and 4.0.
|
||||
- Keep most shots between 1.0 and 4.0 seconds.
|
||||
- Use longer shots only for emotional moments, reveals, or beautiful hero motion.
|
||||
- Use faster shots for montages, details, impacts, or beat drops.
|
||||
|
||||
ALLOWED TRANSITIONS
|
||||
Use only:
|
||||
- cut
|
||||
- crossfade
|
||||
- fade-in
|
||||
- fade-out
|
||||
- none
|
||||
|
||||
Prefer mostly cuts. Use crossfade/fades sparingly for emotional, elegant, or time-passing moments.
|
||||
|
||||
VISUAL TREATMENT RULES
|
||||
visualTreatment must be concise but specific. Good examples:
|
||||
- cinematic grade with subtle punch-in
|
||||
- high-contrast car hero grade
|
||||
- warm food closeup grade
|
||||
- soft family memory grade
|
||||
- speed-ramp detail accent
|
||||
- clean natural grade
|
||||
- none
|
||||
|
||||
Use visual effects to support the moment. Do not overuse effects.
|
||||
|
||||
AUDIO RULES
|
||||
Use audioCues for music and SFX direction.
|
||||
|
||||
Music:
|
||||
- A music cue may use a descriptive assetKey such as premium_cinematic_drive, warm_acoustic_memory, tactile_food_groove, urban_pulse_intro.
|
||||
- Music should cover the full timeline unless silence is intentionally used.
|
||||
- Set gainDb conservatively, usually between -22 and -10.
|
||||
- Describe the music mood and why it fits.
|
||||
|
||||
SFX:
|
||||
- Only reference SFX assetKey values that exist as WAV files in the project audio/sfx folder or assets/sfx folder if the renderer maps them.
|
||||
- If no SFX files exist, do not invent file-backed SFX cues. Instead, put desired SFX ideas in the cue notes only if the service can later generate assets.
|
||||
- SFX should be precise: whoosh on transition, bass hit on reveal, camera shutter on detail, subtle riser before hero shot, engine accent for car footage, sizzle/chop/pour for food, gentle ambience for family.
|
||||
|
||||
VOICEOVER RULES
|
||||
- Voiceover is optional only if the project disables it. If enabled, include concise lines.
|
||||
- Each line must be 240 characters or fewer.
|
||||
- Use natural, premium, non-cheesy language.
|
||||
- Voiceover must fit the category and visible content.
|
||||
- Do not use generic filler like "this is amazing" or "an unforgettable journey" unless the footage clearly supports it.
|
||||
- Do not invent facts.
|
||||
- Give each line a delivery direction, for example:
|
||||
- calm premium narrator
|
||||
- confident cinematic narrator
|
||||
- warm reflective narrator
|
||||
- energetic food host
|
||||
|
||||
TEXT OVERLAY RULES
|
||||
- Overlays must be short: 80 characters or fewer.
|
||||
- Use only these placements:
|
||||
- lower_left_safe
|
||||
- lower_center_safe
|
||||
- center_safe
|
||||
- upper_left_safe
|
||||
- upper_right_safe
|
||||
- Use only these animations:
|
||||
- fade_slide_up
|
||||
- fade_in
|
||||
- none
|
||||
- Use overlays sparingly. 1-5 overlays is usually enough.
|
||||
- Overlays should feel premium and specific, not clickbait.
|
||||
|
||||
QUALITY BAR
|
||||
Reject your own first draft if it has any of these problems:
|
||||
- The first 3 seconds are not compelling.
|
||||
- The edit is just chronological trimming without a story.
|
||||
- The voiceover is generic or not grounded in visible footage.
|
||||
- Music or SFX are placeholders with no timing purpose.
|
||||
- Text overlays repeat what the voiceover already says.
|
||||
- The final shot is weak.
|
||||
- Any JSON field uses invalid values.
|
||||
- Any clip ID or timestamp is invented.
|
||||
- Any timeline math is invalid.
|
||||
|
||||
REQUIRED JSON SHAPE
|
||||
Write exactly this JSON shape:
|
||||
|
||||
{
|
||||
"projectId": "<project-id>",
|
||||
"style": "<style-from-project-json>",
|
||||
"targetDurationSeconds": <actual-planned-final-duration-seconds>,
|
||||
"decisions": [
|
||||
{
|
||||
"clipId": "<valid clip id from analysis>",
|
||||
"sourceStartSeconds": 0.0,
|
||||
"sourceEndSeconds": 2.5,
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": 2.5,
|
||||
"transitionIn": "cut",
|
||||
"transitionOut": "cut",
|
||||
"playbackSpeed": 1.0,
|
||||
"visualTreatment": "cinematic grade with subtle punch-in",
|
||||
"reason": "Opening hook: strongest hero angle and immediate viewer retention."
|
||||
}
|
||||
],
|
||||
"audioCues": [
|
||||
{
|
||||
"type": "music",
|
||||
"assetKey": "premium_cinematic_drive",
|
||||
"timelineStartSeconds": 0.0,
|
||||
"timelineEndSeconds": <actual-planned-final-duration-seconds>,
|
||||
"gainDb": -16.0,
|
||||
"notes": "Low cinematic pulse that builds through detail shots and lifts into the final hero moment."
|
||||
}
|
||||
],
|
||||
"voiceover": [
|
||||
{
|
||||
"text": "A concise, specific line grounded in what is visible in the footage.",
|
||||
"timelineStartSeconds": 1.0,
|
||||
"timelineEndSeconds": 4.0,
|
||||
"delivery": "confident cinematic narrator"
|
||||
}
|
||||
],
|
||||
"overlays": [
|
||||
{
|
||||
"text": "Short premium overlay",
|
||||
"timelineStartSeconds": 0.5,
|
||||
"timelineEndSeconds": 2.0,
|
||||
"placement": "lower_left_safe",
|
||||
"animation": "fade_slide_up",
|
||||
"reason": "Adds context without duplicating the narration."
|
||||
}
|
||||
],
|
||||
"renderProfile": "mp4-h264-aac-1080p",
|
||||
"summary": "A short explanation of the creative strategy, category choice, hook, pacing, sound design, and final payoff."
|
||||
}
|
||||
|
||||
FINAL SELF-CHECK BEFORE WRITING director/edit-plan.json
|
||||
Verify:
|
||||
- projectId matches project.json.
|
||||
- style matches project.json.
|
||||
- targetDurationSeconds equals the actual final timeline duration selected from the source highlights.
|
||||
- decisions is not empty.
|
||||
- audioCues, voiceover, and overlays are arrays even when empty.
|
||||
- All clip IDs exist.
|
||||
- All source ranges are inside clip duration.
|
||||
- Timeline starts at 0.0 and is contiguous.
|
||||
- Final decision timelineEndSeconds equals targetDurationSeconds.
|
||||
- Duration math matches playbackSpeed.
|
||||
- transitionIn and transitionOut use only allowed values.
|
||||
- playbackSpeed is between 0.25 and 4.0.
|
||||
- overlay placements and animations use only allowed values.
|
||||
- overlay text is 80 characters or fewer.
|
||||
- voiceover lines are 240 characters or fewer.
|
||||
- SFX asset keys reference real files if SFX cues are included.
|
||||
- JSON is valid and contains no comments or Markdown.
|
||||
```
|
||||
|
||||
## Notes For The Service
|
||||
|
||||
This prompt is intentionally stricter than the current lightweight storyboard prompt. It is designed to prevent the common failure mode where the AI produces a generic, boring, or invalid edit plan.
|
||||
|
||||
Current renderer constraints still matter:
|
||||
|
||||
- `FfmpegEditRenderer` can render the `EditPlan` schema used by `output/edit-projects`.
|
||||
- The highlight project flow still needs an importer/renderer bridge before `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4` can be produced automatically.
|
||||
- If SFX are referenced, the current validator expects matching WAV files for SFX asset keys. Do not ask the AI to reference non-existent SFX files unless the asset generation step is implemented first.
|
||||
- The current noop voiceover provider writes a script, not a real spoken voice. Production-ready voice requires a real TTS provider or recorded voiceover asset.
|
||||
|
|
@ -204,6 +204,7 @@ The system needs explicit providers instead of pretending a prompt can create fi
|
|||
- Font provider: local fonts with brand/category mapping.
|
||||
- LUT provider: local LUT files or named FFmpeg grade presets.
|
||||
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
|
||||
- Asset generation stage: when a needed asset is missing, write a reusable request pack and keep the target cache path stable so future AI-generated assets can be reused.
|
||||
|
||||
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
|
||||
|
||||
|
|
@ -301,9 +302,13 @@ The plan should be strict JSON so a cheaper model or deterministic renderer can
|
|||
- [x] Milestone 2 progress: Added a configurable highlight source scheduler and directory initializer for `input/highlights/source`, `working`, `processed`, and `rejected`; it claims one valid source video per scan, creates a highlight project and manifest, copies the source into the project folder, and leaves later videos for later scans.
|
||||
- [x] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
|
||||
- [x] Milestone 3 progress: Added highlight source analysis for the new project contract, reusing FFprobe inspection, frame extraction, contact sheet generation, proxy generation, and adding waveform image generation under `analysis/`; the highlight scheduler now runs analysis immediately after creating a project.
|
||||
- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
||||
- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
||||
- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
||||
- [x] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
|
||||
- [x] Milestone 4 progress: Added configurable FFmpeg scene detection for highlight sources, persisted `analysis/scene-segments.json`, attached shot segments to `source-analysis.json`, and logged shot segment counts when the highlight scheduler finishes analysis.
|
||||
- [x] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
|
||||
- [x] Milestone 5 progress: Added deterministic FFmpeg audio analysis for mean/max volume, silence detection, missing-audio handling, persisted `analysis/audio-analysis.json`, and source-analysis embedding. Non-silent sections are labeled `unclassified_audio` until a speech/music/noise classifier or ASR provider is added.
|
||||
- [x] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
|
||||
- [x] Milestone 6 progress: Added persisted `analysis/visual-analysis.json` and source-analysis embedding with blur, exposure, motion, composition, representative thumbnail, face-presence, and object-label fields. Current implementation uses deterministic metadata, thumbnail, and scene-density heuristics; face/object recognition is explicitly marked heuristic until a CV model is connected.
|
||||
- [x] Milestone 6 local CV extension: Added a configurable `local-cv` visual-analysis provider that posts source metadata, thumbnails, and shot segments to a local HTTP CV worker, persists the returned normalized visual analysis, and falls back to heuristic analysis when configured. See `docs/local-cv-visual-analysis-guide.md`.
|
||||
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
|
||||
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
|
||||
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
|
||||
|
|
@ -548,6 +553,9 @@ Expected renderer output:
|
|||
|
||||
```text
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0001.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0002.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0003.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
|
||||
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"id" : "source-clips",
|
||||
"name" : "source-clips",
|
||||
"status" : "RENDERED",
|
||||
"inputDirectory" : "./input/editing/processed/source-clips",
|
||||
"outputDirectory" : "output/edit-projects/source-clips",
|
||||
"targetDurationSeconds" : 60,
|
||||
"style" : "cinematic-porsche-promo",
|
||||
"voiceoverEnabled" : true,
|
||||
"musicEnabled" : true,
|
||||
"soundEffectsEnabled" : true,
|
||||
"createdAt" : "2026-07-10T17:25:59.366353Z",
|
||||
"updatedAt" : "2026-07-10T21:58:05.307273Z",
|
||||
"failureReason" : null
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -503,8 +503,16 @@ public class VideoClippingProperties {
|
|||
@Min(1)
|
||||
private int proxyWidth = 640;
|
||||
|
||||
private double sceneDetectionThreshold = 0.35;
|
||||
|
||||
private double minimumSceneDurationSeconds = 1.0;
|
||||
|
||||
private double silenceThresholdDb = -35.0;
|
||||
|
||||
private double silenceMinimumDurationSeconds = 0.5;
|
||||
|
||||
@Min(1)
|
||||
private int targetDurationSeconds = 60;
|
||||
private int targetDurationSeconds = 600;
|
||||
|
||||
@Min(1)
|
||||
private int outputWidth = 1920;
|
||||
|
|
@ -542,6 +550,8 @@ public class VideoClippingProperties {
|
|||
|
||||
private final Assets assets = new Assets();
|
||||
|
||||
private final VisualAnalysis visualAnalysis = new VisualAnalysis();
|
||||
|
||||
private final LocalDirector localDirector = new LocalDirector();
|
||||
|
||||
private final HighlightScheduler highlightScheduler = new HighlightScheduler();
|
||||
|
|
@ -618,6 +628,38 @@ public class VideoClippingProperties {
|
|||
this.proxyWidth = proxyWidth;
|
||||
}
|
||||
|
||||
public double getSceneDetectionThreshold() {
|
||||
return sceneDetectionThreshold;
|
||||
}
|
||||
|
||||
public void setSceneDetectionThreshold(double sceneDetectionThreshold) {
|
||||
this.sceneDetectionThreshold = sceneDetectionThreshold;
|
||||
}
|
||||
|
||||
public double getMinimumSceneDurationSeconds() {
|
||||
return minimumSceneDurationSeconds;
|
||||
}
|
||||
|
||||
public void setMinimumSceneDurationSeconds(double minimumSceneDurationSeconds) {
|
||||
this.minimumSceneDurationSeconds = minimumSceneDurationSeconds;
|
||||
}
|
||||
|
||||
public double getSilenceThresholdDb() {
|
||||
return silenceThresholdDb;
|
||||
}
|
||||
|
||||
public void setSilenceThresholdDb(double silenceThresholdDb) {
|
||||
this.silenceThresholdDb = silenceThresholdDb;
|
||||
}
|
||||
|
||||
public double getSilenceMinimumDurationSeconds() {
|
||||
return silenceMinimumDurationSeconds;
|
||||
}
|
||||
|
||||
public void setSilenceMinimumDurationSeconds(double silenceMinimumDurationSeconds) {
|
||||
this.silenceMinimumDurationSeconds = silenceMinimumDurationSeconds;
|
||||
}
|
||||
|
||||
public int getTargetDurationSeconds() {
|
||||
return targetDurationSeconds;
|
||||
}
|
||||
|
|
@ -742,6 +784,10 @@ public class VideoClippingProperties {
|
|||
return assets;
|
||||
}
|
||||
|
||||
public VisualAnalysis getVisualAnalysis() {
|
||||
return visualAnalysis;
|
||||
}
|
||||
|
||||
public LocalDirector getLocalDirector() {
|
||||
return localDirector;
|
||||
}
|
||||
|
|
@ -802,6 +848,109 @@ public class VideoClippingProperties {
|
|||
}
|
||||
}
|
||||
|
||||
public static class VisualAnalysis {
|
||||
private String provider = "heuristic";
|
||||
|
||||
private String endpoint = "http://127.0.0.1:8091/v1/analyze-visuals";
|
||||
|
||||
@Min(1)
|
||||
private long timeoutMs = 30000;
|
||||
|
||||
private boolean fallbackToHeuristic = true;
|
||||
|
||||
private final LocalCvWorker localCvWorker = new LocalCvWorker();
|
||||
|
||||
public String getProvider() {
|
||||
return provider;
|
||||
}
|
||||
|
||||
public void setProvider(String provider) {
|
||||
this.provider = provider;
|
||||
}
|
||||
|
||||
public String getEndpoint() {
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
public void setEndpoint(String endpoint) {
|
||||
this.endpoint = endpoint;
|
||||
}
|
||||
|
||||
public long getTimeoutMs() {
|
||||
return timeoutMs;
|
||||
}
|
||||
|
||||
public void setTimeoutMs(long timeoutMs) {
|
||||
this.timeoutMs = timeoutMs;
|
||||
}
|
||||
|
||||
public boolean isFallbackToHeuristic() {
|
||||
return fallbackToHeuristic;
|
||||
}
|
||||
|
||||
public void setFallbackToHeuristic(boolean fallbackToHeuristic) {
|
||||
this.fallbackToHeuristic = fallbackToHeuristic;
|
||||
}
|
||||
|
||||
public LocalCvWorker getLocalCvWorker() {
|
||||
return localCvWorker;
|
||||
}
|
||||
|
||||
public static class LocalCvWorker {
|
||||
private boolean autoStart = false;
|
||||
|
||||
private String script = "./tools/run_local_cv_worker.sh";
|
||||
|
||||
@Min(0)
|
||||
private long startupWaitMs = 0;
|
||||
|
||||
private String healthPath = "/health";
|
||||
|
||||
@Min(1)
|
||||
private long healthCheckIntervalMs = 1000;
|
||||
|
||||
public boolean isAutoStart() {
|
||||
return autoStart;
|
||||
}
|
||||
|
||||
public void setAutoStart(boolean autoStart) {
|
||||
this.autoStart = autoStart;
|
||||
}
|
||||
|
||||
public String getScript() {
|
||||
return script;
|
||||
}
|
||||
|
||||
public void setScript(String script) {
|
||||
this.script = script;
|
||||
}
|
||||
|
||||
public long getStartupWaitMs() {
|
||||
return startupWaitMs;
|
||||
}
|
||||
|
||||
public void setStartupWaitMs(long startupWaitMs) {
|
||||
this.startupWaitMs = startupWaitMs;
|
||||
}
|
||||
|
||||
public String getHealthPath() {
|
||||
return healthPath;
|
||||
}
|
||||
|
||||
public void setHealthPath(String healthPath) {
|
||||
this.healthPath = healthPath;
|
||||
}
|
||||
|
||||
public long getHealthCheckIntervalMs() {
|
||||
return healthCheckIntervalMs;
|
||||
}
|
||||
|
||||
public void setHealthCheckIntervalMs(long healthCheckIntervalMs) {
|
||||
this.healthCheckIntervalMs = healthCheckIntervalMs;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static class LocalDirector {
|
||||
private boolean enabled = true;
|
||||
|
||||
|
|
@ -929,6 +1078,21 @@ public class VideoClippingProperties {
|
|||
@Min(1000)
|
||||
private long pollIntervalMs = 5000;
|
||||
|
||||
private boolean renderEnabled = true;
|
||||
|
||||
private boolean requireDirectorApproval = false;
|
||||
|
||||
private String approvalFileName = "approved.flag";
|
||||
|
||||
@Min(1)
|
||||
private int maxHighlightsPerSource = 3;
|
||||
|
||||
@Min(1)
|
||||
private double highlightMinDurationSeconds = 8.0;
|
||||
|
||||
@Min(1)
|
||||
private double highlightMaxDurationSeconds = 35.0;
|
||||
|
||||
public boolean isEnabled() {
|
||||
return enabled;
|
||||
}
|
||||
|
|
@ -976,6 +1140,54 @@ public class VideoClippingProperties {
|
|||
public void setPollIntervalMs(long pollIntervalMs) {
|
||||
this.pollIntervalMs = pollIntervalMs;
|
||||
}
|
||||
|
||||
public boolean isRenderEnabled() {
|
||||
return renderEnabled;
|
||||
}
|
||||
|
||||
public void setRenderEnabled(boolean renderEnabled) {
|
||||
this.renderEnabled = renderEnabled;
|
||||
}
|
||||
|
||||
public boolean isRequireDirectorApproval() {
|
||||
return requireDirectorApproval;
|
||||
}
|
||||
|
||||
public void setRequireDirectorApproval(boolean requireDirectorApproval) {
|
||||
this.requireDirectorApproval = requireDirectorApproval;
|
||||
}
|
||||
|
||||
public String getApprovalFileName() {
|
||||
return approvalFileName;
|
||||
}
|
||||
|
||||
public void setApprovalFileName(String approvalFileName) {
|
||||
this.approvalFileName = approvalFileName;
|
||||
}
|
||||
|
||||
public int getMaxHighlightsPerSource() {
|
||||
return maxHighlightsPerSource;
|
||||
}
|
||||
|
||||
public void setMaxHighlightsPerSource(int maxHighlightsPerSource) {
|
||||
this.maxHighlightsPerSource = maxHighlightsPerSource;
|
||||
}
|
||||
|
||||
public double getHighlightMinDurationSeconds() {
|
||||
return highlightMinDurationSeconds;
|
||||
}
|
||||
|
||||
public void setHighlightMinDurationSeconds(double highlightMinDurationSeconds) {
|
||||
this.highlightMinDurationSeconds = highlightMinDurationSeconds;
|
||||
}
|
||||
|
||||
public double getHighlightMaxDurationSeconds() {
|
||||
return highlightMaxDurationSeconds;
|
||||
}
|
||||
|
||||
public void setHighlightMaxDurationSeconds(double highlightMaxDurationSeconds) {
|
||||
this.highlightMaxDurationSeconds = highlightMaxDurationSeconds;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,10 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
public record AudioSection(
|
||||
String sectionId,
|
||||
String kind,
|
||||
double startSeconds,
|
||||
double endSeconds,
|
||||
double durationSeconds
|
||||
) {
|
||||
}
|
||||
|
|
@ -37,6 +37,7 @@ public class EditPlanInboxScanner {
|
|||
private final EditProjectStore store;
|
||||
private final EditProjectService projectService;
|
||||
private final EditPlanValidator validator;
|
||||
private final Optional<AssetGenerationStage> assetGenerationStage;
|
||||
private final Optional<EditRenderer> renderer;
|
||||
private final VoiceoverGenerator voiceoverGenerator;
|
||||
private final AtomicBoolean scanning = new AtomicBoolean();
|
||||
|
|
@ -49,9 +50,11 @@ public class EditPlanInboxScanner {
|
|||
EditProjectService projectService,
|
||||
EditPlanValidator validator,
|
||||
VoiceoverGenerator voiceoverGenerator,
|
||||
ObjectProvider<AssetGenerationStage> assetGenerationStageProvider,
|
||||
ObjectProvider<EditRenderer> rendererProvider
|
||||
) {
|
||||
this(properties, objectMapper, store, projectService, validator, voiceoverGenerator,
|
||||
assetGenerationStageProvider.getIfAvailable(),
|
||||
rendererProvider.getIfAvailable());
|
||||
}
|
||||
|
||||
|
|
@ -63,13 +66,13 @@ public class EditPlanInboxScanner {
|
|||
EditPlanValidator validator,
|
||||
EditRenderer renderer
|
||||
) {
|
||||
this(properties, objectMapper, store, projectService, validator, null, renderer);
|
||||
this(properties, objectMapper, store, projectService, validator, null, null, renderer);
|
||||
}
|
||||
|
||||
EditPlanInboxScanner(
|
||||
VideoClippingProperties properties, ObjectMapper objectMapper, EditProjectStore store,
|
||||
EditProjectService projectService, EditPlanValidator validator,
|
||||
VoiceoverGenerator voiceoverGenerator, EditRenderer renderer
|
||||
VoiceoverGenerator voiceoverGenerator, AssetGenerationStage assetGenerationStage, EditRenderer renderer
|
||||
) {
|
||||
this.projectRoot = Path.of(properties.getEditing().getProjectDirectory());
|
||||
this.expectedPlanFileName = properties.getEditing().getLocalDirector().getExpectedPlanFileName();
|
||||
|
|
@ -82,6 +85,7 @@ public class EditPlanInboxScanner {
|
|||
this.validator = validator;
|
||||
this.renderer = Optional.ofNullable(renderer);
|
||||
this.voiceoverGenerator = voiceoverGenerator;
|
||||
this.assetGenerationStage = Optional.ofNullable(assetGenerationStage);
|
||||
}
|
||||
|
||||
@Scheduled(
|
||||
|
|
@ -154,6 +158,11 @@ public class EditPlanInboxScanner {
|
|||
}
|
||||
|
||||
store.writeJson(projectId, "edit-plan.json", normalized(plan));
|
||||
AssetGenerationResult assetGenerationResult = assetGenerationStage
|
||||
.map(stage -> stage.prepare(projectId, plan))
|
||||
.orElse(new AssetGenerationResult(projectId, true, List.of(), java.time.Instant.now()));
|
||||
log.info("event=edit_asset_generation_completed project_id={} item_count={} ready_for_render={}",
|
||||
projectId, assetGenerationResult.items().size(), assetGenerationResult.readyForRender());
|
||||
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
|
||||
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
|
||||
}
|
||||
|
|
@ -164,8 +173,11 @@ public class EditPlanInboxScanner {
|
|||
projectId, plan.decisions().size(), autoRender, requireApprovalBeforeRender);
|
||||
log.info("event=edit_plan_saved project_id={} decision_count={} source=inbox",
|
||||
projectId, plan.decisions().size());
|
||||
if (autoRender) {
|
||||
if (autoRender && assetGenerationResult.readyForRender()) {
|
||||
renderIfApproved(projectId, store.inboxDirectory(projectId));
|
||||
} else if (autoRender) {
|
||||
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
|
||||
projectId);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -175,6 +187,16 @@ public class EditPlanInboxScanner {
|
|||
projectId, inbox.resolve(approvalFileName));
|
||||
return;
|
||||
}
|
||||
AssetGenerationStage stage = assetGenerationStage.orElse(null);
|
||||
if (stage != null) {
|
||||
EditPlan plan = store.readJson(projectId, "edit-plan.json", EditPlan.class);
|
||||
AssetGenerationResult result = stage.prepare(projectId, plan);
|
||||
if (!result.readyForRender()) {
|
||||
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
|
||||
projectId);
|
||||
return;
|
||||
}
|
||||
}
|
||||
renderer.orElseThrow(() -> new IllegalStateException(
|
||||
"Automatic rendering is enabled but no EditRenderer is available")).render(projectId);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -80,9 +80,13 @@ public class EditPlanValidator {
|
|||
validateAudioCues(projectId, plan);
|
||||
validateVoiceover(plan);
|
||||
validateOverlays(plan);
|
||||
if (previousEnd > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS
|
||||
|| Math.abs(plan.targetDurationSeconds() - project.targetDurationSeconds()) > EPSILON) {
|
||||
reject("Edit plan duration exceeds or does not match the project target duration");
|
||||
validateCinematicRichness(project, plan);
|
||||
if (plan.targetDurationSeconds() <= 0
|
||||
|| Math.abs(previousEnd - plan.targetDurationSeconds()) > TARGET_TOLERANCE_SECONDS) {
|
||||
reject("Edit plan target duration must match the final timeline duration");
|
||||
}
|
||||
if (plan.targetDurationSeconds() > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS) {
|
||||
reject("Edit plan duration exceeds the project maximum target duration");
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
|
@ -152,6 +156,42 @@ public class EditPlanValidator {
|
|||
}
|
||||
}
|
||||
|
||||
private void validateCinematicRichness(EditProject project, EditPlan plan) {
|
||||
int cinematicDimensions = 0;
|
||||
if (hasVisualTreatment(plan)) {
|
||||
cinematicDimensions++;
|
||||
}
|
||||
if (project.musicEnabled() && hasAudioCueType(plan, "music")) {
|
||||
cinematicDimensions++;
|
||||
}
|
||||
if (project.voiceoverEnabled() && !plan.voiceover().isEmpty()) {
|
||||
cinematicDimensions++;
|
||||
}
|
||||
if (project.soundEffectsEnabled() && hasAudioCueType(plan, "sfx")) {
|
||||
cinematicDimensions++;
|
||||
}
|
||||
if (!plan.overlays().isEmpty()) {
|
||||
cinematicDimensions++;
|
||||
}
|
||||
|
||||
if (cinematicDimensions < 2) {
|
||||
reject("Edit plan is too bare; it must use at least two cinematic elements such as visual treatment, "
|
||||
+ "music, voiceover, sound effects, or text overlays");
|
||||
}
|
||||
}
|
||||
|
||||
private boolean hasVisualTreatment(EditPlan plan) {
|
||||
return plan.decisions().stream()
|
||||
.map(EditDecision::visualTreatment)
|
||||
.filter(value -> value != null && !value.isBlank() && !"none".equalsIgnoreCase(value))
|
||||
.findFirst()
|
||||
.isPresent();
|
||||
}
|
||||
|
||||
private boolean hasAudioCueType(EditPlan plan, String type) {
|
||||
return plan.audioCues().stream().anyMatch(cue -> type.equals(cue.type()));
|
||||
}
|
||||
|
||||
private void validateTransition(String transition) {
|
||||
if (transition == null || !TRANSITIONS.contains(transition)) {
|
||||
reject("Unsupported transition: " + transition);
|
||||
|
|
|
|||
|
|
@ -127,6 +127,7 @@ public class FfmpegEditRenderer implements EditRenderer {
|
|||
runAndRecord(segmentCommand(clips.get(decision.clipId()).sourcePath(), decision, output), commands);
|
||||
segments.add(output);
|
||||
}
|
||||
List<Path> renderedClips = publishRenderedClips(projectId, segments);
|
||||
Path concatFile = work.resolve("concat.txt");
|
||||
writeConcatFile(concatFile, segments);
|
||||
Path timeline = work.resolve("timeline-video.mp4");
|
||||
|
|
@ -152,10 +153,11 @@ public class FfmpegEditRenderer implements EditRenderer {
|
|||
copy(videoTimeline, output);
|
||||
}
|
||||
double duration = plan.decisions().get(plan.decisions().size() - 1).timelineEndSeconds();
|
||||
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, commands, assets);
|
||||
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
|
||||
RenderManifest manifest = new RenderManifest(projectId,
|
||||
plan.decisions().stream().map(EditDecision::clipId).toList(), output.toString(), duration,
|
||||
List.copyOf(commands), assets, Instant.now());
|
||||
plan.decisions().stream().map(EditDecision::clipId).toList(),
|
||||
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
|
||||
assets, Instant.now());
|
||||
store.writeJson(projectId, "qa-report.json", qaReport);
|
||||
store.writeJson(projectId, "render-manifest.json", manifest);
|
||||
projectService.updateProject(projectId, EditProjectStatus.RENDERED, Path.of(project.inputDirectory()), null);
|
||||
|
|
@ -361,9 +363,11 @@ public class FfmpegEditRenderer implements EditRenderer {
|
|||
}
|
||||
|
||||
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||
List<Path> renderedClips,
|
||||
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||
List<RenderQaCheck> checks = new ArrayList<>();
|
||||
checks.add(outputExistsCheck(output));
|
||||
checks.add(renderedClipsExistCheck(renderedClips));
|
||||
checks.add(durationMatchesTimelineCheck(plan, duration));
|
||||
checks.add(requiredAssetsResolvedCheck(plan, assets));
|
||||
checks.add(overlaysSafeCheck(plan, duration));
|
||||
|
|
@ -388,6 +392,14 @@ public class FfmpegEditRenderer implements EditRenderer {
|
|||
exists ? "final output exists" : "final output is missing: " + output);
|
||||
}
|
||||
|
||||
private RenderQaCheck renderedClipsExistCheck(List<Path> renderedClips) {
|
||||
List<Path> missing = renderedClips.stream()
|
||||
.filter(path -> !Files.isRegularFile(path))
|
||||
.toList();
|
||||
return new RenderQaCheck("rendered_clips_exist", missing.isEmpty(), "ERROR",
|
||||
missing.isEmpty() ? "all rendered segment clips exist" : "missing rendered clips: " + missing);
|
||||
}
|
||||
|
||||
private RenderQaCheck durationMatchesTimelineCheck(EditPlan plan, double duration) {
|
||||
double expected = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
||||
.timelineEndSeconds();
|
||||
|
|
@ -518,6 +530,18 @@ public class FfmpegEditRenderer implements EditRenderer {
|
|||
}
|
||||
}
|
||||
|
||||
private List<Path> publishRenderedClips(String projectId, List<Path> segments) {
|
||||
Path publishedDirectory = store.projectDirectory(projectId).resolve("rendered-clips");
|
||||
createDirectory(publishedDirectory);
|
||||
List<Path> published = new ArrayList<>();
|
||||
for (int index = 0; index < segments.size(); index++) {
|
||||
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
|
||||
copy(segments.get(index), target);
|
||||
published.add(target);
|
||||
}
|
||||
return List.copyOf(published);
|
||||
}
|
||||
|
||||
private long fileSize(Path file) {
|
||||
try {
|
||||
return Files.size(file);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,102 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HeuristicVisualAnalysisProvider implements VisualAnalysisProvider {
|
||||
|
||||
@Override
|
||||
public SourceVisualAnalysis analyze(
|
||||
ClipAnalysis source,
|
||||
List<String> thumbnails,
|
||||
List<ShotSegment> shotSegments
|
||||
) {
|
||||
double blurScore = scoreOrDefault(source.sharpnessScore(), 0.5);
|
||||
double exposureScore = scoreOrDefault(source.brightnessScore(), 0.5);
|
||||
double motionScore = scoreOrDefault(source.motionScore(), motionFromShots(source.durationSeconds(), shotSegments));
|
||||
return new SourceVisualAnalysis(
|
||||
source.clipId(),
|
||||
blurScore,
|
||||
exposureScore,
|
||||
motionScore,
|
||||
compositionScore(source, thumbnails),
|
||||
"unknown_without_face_detector",
|
||||
labels(source),
|
||||
representativeThumbnails(thumbnails),
|
||||
"metadata_thumbnail_scene_heuristic"
|
||||
);
|
||||
}
|
||||
|
||||
private double scoreOrDefault(double value, double fallback) {
|
||||
if (value > 0) {
|
||||
return round(clamp(value));
|
||||
}
|
||||
return round(clamp(fallback));
|
||||
}
|
||||
|
||||
private double motionFromShots(double durationSeconds, List<ShotSegment> shotSegments) {
|
||||
if (durationSeconds <= 0 || shotSegments == null || shotSegments.isEmpty()) {
|
||||
return 0.3;
|
||||
}
|
||||
double cutsPerMinute = Math.max(0, shotSegments.size() - 1) / durationSeconds * 60.0;
|
||||
return clamp(0.25 + cutsPerMinute / 20.0);
|
||||
}
|
||||
|
||||
private double compositionScore(ClipAnalysis source, List<String> thumbnails) {
|
||||
double aspectRatio = source.height() == 0 ? 0 : source.width() / (double) source.height();
|
||||
double aspectScore = aspectRatio >= 1.70 && aspectRatio <= 1.90 ? 0.8 : 0.55;
|
||||
double thumbnailScore = thumbnails == null || thumbnails.isEmpty() ? 0.45 : 0.7;
|
||||
return round((aspectScore + thumbnailScore) / 2.0);
|
||||
}
|
||||
|
||||
private List<VisualObjectLabel> labels(ClipAnalysis source) {
|
||||
String haystack = (source.clipId() + " " + source.sourcePath()).toLowerCase(Locale.ROOT);
|
||||
List<VisualObjectLabel> labels = new ArrayList<>();
|
||||
addIfContains(labels, haystack, "porsche", "porsche", 0.82);
|
||||
addIfContains(labels, haystack, "car", "car", 0.74);
|
||||
addIfContains(labels, haystack, "drive", "car", 0.62);
|
||||
addIfContains(labels, haystack, "food", "food", 0.72);
|
||||
addIfContains(labels, haystack, "recipe", "food", 0.66);
|
||||
addIfContains(labels, haystack, "family", "person", 0.62);
|
||||
addIfContains(labels, haystack, "birthday", "person", 0.58);
|
||||
if (labels.isEmpty()) {
|
||||
labels.add(new VisualObjectLabel("unknown", 0.2, "metadata_heuristic"));
|
||||
}
|
||||
return labels.stream().distinct().toList();
|
||||
}
|
||||
|
||||
private void addIfContains(List<VisualObjectLabel> labels, String haystack, String needle, String label,
|
||||
double confidence) {
|
||||
if (haystack.contains(needle)) {
|
||||
labels.add(new VisualObjectLabel(label, confidence, "metadata_heuristic"));
|
||||
}
|
||||
}
|
||||
|
||||
private List<String> representativeThumbnails(List<String> thumbnails) {
|
||||
if (thumbnails == null || thumbnails.isEmpty()) {
|
||||
return List.of();
|
||||
}
|
||||
if (thumbnails.size() <= 3) {
|
||||
return List.copyOf(thumbnails);
|
||||
}
|
||||
return List.of(
|
||||
thumbnails.get(0),
|
||||
thumbnails.get(thumbnails.size() / 2),
|
||||
thumbnails.get(thumbnails.size() - 1)
|
||||
);
|
||||
}
|
||||
|
||||
private double clamp(double value) {
|
||||
return Math.max(0, Math.min(1, value));
|
||||
}
|
||||
|
||||
private double round(double value) {
|
||||
return Math.round(value * 1000.0) / 1000.0;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
public record HighlightAssetRequest(
|
||||
String projectId,
|
||||
String highlightId,
|
||||
String type,
|
||||
String assetKey,
|
||||
String targetPath,
|
||||
String requestPath,
|
||||
String notes,
|
||||
double durationSeconds,
|
||||
boolean blocking
|
||||
) {
|
||||
}
|
||||
|
|
@ -0,0 +1,338 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightDirectorFlowService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorFlowService.class);
|
||||
|
||||
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final HighlightProjectStore store;
|
||||
private final HighlightVisualEffectsStage visualEffectsStage;
|
||||
private final HighlightAssetPreparationService assetPreparationService;
|
||||
private final HighlightLocalAssetWorker assetWorker;
|
||||
private final HighlightFfmpegRenderer renderer;
|
||||
|
||||
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
|
||||
HighlightProjectStore store,
|
||||
HighlightVisualEffectsStage visualEffectsStage,
|
||||
HighlightAssetPreparationService assetPreparationService,
|
||||
HighlightLocalAssetWorker assetWorker,
|
||||
HighlightFfmpegRenderer renderer) {
|
||||
this.properties = properties.getEditing().getHighlightScheduler();
|
||||
this.objectMapper = objectMapper;
|
||||
this.store = store;
|
||||
this.visualEffectsStage = visualEffectsStage;
|
||||
this.assetPreparationService = assetPreparationService;
|
||||
this.assetWorker = assetWorker;
|
||||
this.renderer = renderer;
|
||||
}
|
||||
|
||||
public HighlightFlowResult process(String projectId, long scanId) {
|
||||
long startedAt = System.nanoTime();
|
||||
String flowId = projectId + ":" + scanId;
|
||||
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||
Path planFile = store.directorDirectory(projectId).resolve("edit-plan.json");
|
||||
if (!Files.isRegularFile(planFile)) {
|
||||
return HighlightFlowResult.skipped(projectId, flowId, "missing_director_plan");
|
||||
}
|
||||
if (properties.isRequireDirectorApproval()
|
||||
&& !Files.isRegularFile(store.directorDirectory(projectId).resolve(properties.getApprovalFileName()))) {
|
||||
log.info("event=highlight_flow_waiting_for_approval flow_id={} project_id={} approval_file={}",
|
||||
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
|
||||
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
|
||||
}
|
||||
HighlightDirectorPlan plan = readPlan(planFile);
|
||||
if (plan.highlights().isEmpty()) {
|
||||
markFailed(project, "No highlights were provided by the director plan");
|
||||
throw new IllegalStateException("Director plan contains no highlights");
|
||||
}
|
||||
markStatus(project, HighlightProjectStatus.RENDERING, null);
|
||||
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
|
||||
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
|
||||
List<Path> finalOutputs = new ArrayList<>();
|
||||
List<String> renderedHighlightIds = new ArrayList<>();
|
||||
ContentCategory category = contentCategory(plan.contentCategory());
|
||||
int limit = Math.min(properties.getMaxHighlightsPerSource(), plan.highlights().size());
|
||||
for (int index = 0; index < limit; index++) {
|
||||
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
|
||||
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||
writeStoryboard(highlightDirectory, plan, highlight);
|
||||
visualEffectsStage.create(projectId, highlight, category);
|
||||
EditPlan editPlan = buildEditPlan(project, plan, highlight, index);
|
||||
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/edit-plan.json", editPlan);
|
||||
HighlightAssetPreparationService.HighlightAssetPreparationResult assets = assetPreparationService
|
||||
.prepare(projectId, project, highlight, category);
|
||||
HighlightLocalAssetWorker.HighlightAssetWorkerResult assetWorkerResult = assetWorker.process(projectId,
|
||||
highlight, category);
|
||||
log.info("event=highlight_assets_prepared flow_id={} project_id={} highlight_id={} resolved={} requests={}",
|
||||
flowId, projectId, highlight.highlightId(), assets.resolvedAssets().size(),
|
||||
assets.requestFiles().size());
|
||||
log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}",
|
||||
flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(),
|
||||
assetWorkerResult.pendingRequests().size());
|
||||
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
|
||||
finalOutputs.add(result.finalOutput());
|
||||
renderedHighlightIds.add(result.highlightId());
|
||||
}
|
||||
Path projectOutput = store.projectDirectory(projectId).resolve("final.mp4");
|
||||
concatFinalOutputs(projectOutput, finalOutputs);
|
||||
RenderManifest manifest = new RenderManifest(projectId, renderedHighlightIds, finalOutputs.stream()
|
||||
.map(Path::toString).toList(), projectOutput.toString(),
|
||||
finalOutputs.stream().mapToDouble(this::probeDuration).sum(), List.of(), List.of(), Instant.now());
|
||||
store.writeJson(projectId, "render-manifest.json", manifest);
|
||||
markStatus(project, HighlightProjectStatus.RENDERED, null);
|
||||
log.info("event=highlight_flow_completed flow_id={} scan_id={} project_id={} final_outputs={} elapsed_ms={}",
|
||||
flowId, scanId, projectId, finalOutputs, (System.nanoTime() - startedAt) / 1_000_000);
|
||||
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
|
||||
}
|
||||
|
||||
private HighlightDirectorPlan readPlan(Path planFile) {
|
||||
try {
|
||||
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to read highlight director plan: " + planFile, ex);
|
||||
}
|
||||
}
|
||||
|
||||
private EditPlan buildEditPlan(HighlightProject project, HighlightDirectorPlan plan,
|
||||
HighlightDirectorPlan.HighlightItem highlight, int index) {
|
||||
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
|
||||
HighlightSourceAnalysis.class);
|
||||
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
|
||||
double requestedDuration = highlight.targetDurationSeconds() > 0 ? highlight.targetDurationSeconds()
|
||||
: sourceDuration;
|
||||
double targetDuration = Math.max(properties.getHighlightMinDurationSeconds(),
|
||||
Math.min(properties.getHighlightMaxDurationSeconds(), requestedDuration));
|
||||
double playbackSpeed = Math.max(0.25, Math.min(4.0, sourceDuration / targetDuration));
|
||||
String clipId = analysis.source().clipId();
|
||||
EditDecision decision = new EditDecision(
|
||||
clipId,
|
||||
highlight.sourceStartSeconds(),
|
||||
highlight.sourceEndSeconds(),
|
||||
0.0,
|
||||
targetDuration,
|
||||
transitionIn(highlight, index),
|
||||
transitionOut(highlight, index, plan.highlights().size()),
|
||||
playbackSpeed,
|
||||
highlight.visualTreatment(),
|
||||
highlight.renderNotes()
|
||||
);
|
||||
List<AudioCue> audioCues = new ArrayList<>();
|
||||
if (highlight.musicDirection() != null && !highlight.musicDirection().isBlank()) {
|
||||
audioCues.add(new AudioCue("music", safeKey("music", highlight.musicDirection()), 0.0, targetDuration,
|
||||
-10.0, highlight.musicDirection()));
|
||||
}
|
||||
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
|
||||
audioCues.add(new AudioCue("sfx", safeKey("sfx", highlight.storyPurpose(), highlight.highlightId()),
|
||||
Math.max(0.0, targetDuration - 0.8), targetDuration, -6.0, highlight.sfxDirection()));
|
||||
}
|
||||
List<VoiceoverLine> voiceover = new ArrayList<>();
|
||||
if (!highlight.voiceover().isEmpty()) {
|
||||
double step = targetDuration / highlight.voiceover().size();
|
||||
for (int i = 0; i < highlight.voiceover().size(); i++) {
|
||||
double start = Math.min(targetDuration, i * step);
|
||||
double end = Math.min(targetDuration, start + Math.max(1.0, step));
|
||||
voiceover.add(new VoiceoverLine(highlight.voiceover().get(i), start, end, "cinematic_narration"));
|
||||
}
|
||||
}
|
||||
List<TextOverlay> overlays = new ArrayList<>();
|
||||
if (!highlight.overlays().isEmpty()) {
|
||||
double step = targetDuration / highlight.overlays().size();
|
||||
for (int i = 0; i < highlight.overlays().size(); i++) {
|
||||
double start = Math.min(targetDuration, i * step);
|
||||
double end = Math.min(targetDuration, start + Math.max(1.5, step * 0.75));
|
||||
overlays.add(new TextOverlay(highlight.overlays().get(i), start, end,
|
||||
i % 2 == 0 ? "lower_center_safe" : "center_safe",
|
||||
"fade_slide_up", highlight.renderNotes()));
|
||||
}
|
||||
}
|
||||
return new EditPlan(project.id(),
|
||||
safeKey("style", plan.contentCategory(), highlight.storyPurpose()),
|
||||
targetDuration,
|
||||
List.of(decision),
|
||||
audioCues,
|
||||
voiceover,
|
||||
overlays,
|
||||
"mp4-h264-aac-1080p",
|
||||
highlight.title() + " / " + highlight.storyPurpose());
|
||||
}
|
||||
|
||||
private String transitionIn(HighlightDirectorPlan.HighlightItem highlight, int index) {
|
||||
if (index == 0 || highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
|
||||
return "fade-in";
|
||||
}
|
||||
return "cut";
|
||||
}
|
||||
|
||||
private String transitionOut(HighlightDirectorPlan.HighlightItem highlight, int index, int total) {
|
||||
if (index + 1 == total || highlight.storyPurpose() != null && highlight.storyPurpose().contains("hero")) {
|
||||
return "fade-out";
|
||||
}
|
||||
return "cut";
|
||||
}
|
||||
|
||||
private void writeStoryboard(Path highlightDirectory, HighlightDirectorPlan plan,
|
||||
HighlightDirectorPlan.HighlightItem highlight) {
|
||||
String storyboard = """
|
||||
# Storyboard
|
||||
|
||||
Project: `%s`
|
||||
Highlight: `%s`
|
||||
Title: `%s`
|
||||
Purpose: `%s`
|
||||
Music: `%s`
|
||||
SFX: `%s`
|
||||
|
||||
Voiceover:
|
||||
%s
|
||||
|
||||
Overlays:
|
||||
%s
|
||||
""".formatted(plan.projectId(), highlight.highlightId(), highlight.title(), highlight.storyPurpose(),
|
||||
highlight.musicDirection(), highlight.sfxDirection(), String.join(System.lineSeparator(),
|
||||
highlight.voiceover()), String.join(System.lineSeparator(), highlight.overlays()));
|
||||
try {
|
||||
Files.createDirectories(highlightDirectory);
|
||||
Files.writeString(highlightDirectory.resolve("storyboard.md"), storyboard, StandardCharsets.UTF_8);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write storyboard for highlight: " + highlight.highlightId(), ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void concatFinalOutputs(Path output, List<Path> sources) {
|
||||
if (sources.isEmpty()) {
|
||||
throw new IllegalStateException("No rendered highlight outputs were produced");
|
||||
}
|
||||
if (sources.size() == 1) {
|
||||
copy(sources.get(0), output);
|
||||
return;
|
||||
}
|
||||
Path work = output.getParent().resolve("project-render-work");
|
||||
try {
|
||||
Files.createDirectories(work);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to create highlight project render work directory", ex);
|
||||
}
|
||||
Path concat = work.resolve("concat.txt");
|
||||
String content = sources.stream()
|
||||
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
|
||||
.reduce("", (a, b) -> a + b + System.lineSeparator());
|
||||
try {
|
||||
Files.writeString(concat, content, StandardCharsets.UTF_8);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write project concat file", ex);
|
||||
}
|
||||
run(List.of("ffmpeg", "-hide_banner", "-y", "-f", "concat", "-safe", "0", "-i", concat.toString(),
|
||||
"-c", "copy", output.toString()));
|
||||
}
|
||||
|
||||
private double probeDuration(Path video) {
|
||||
try {
|
||||
Process process = new ProcessBuilder("ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", video.toString()).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
if (process.waitFor() != 0) {
|
||||
return 0.0;
|
||||
}
|
||||
return Double.parseDouble(output.trim());
|
||||
} catch (IOException | InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
return 0.0;
|
||||
}
|
||||
}
|
||||
|
||||
private void run(List<String> command) {
|
||||
try {
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
if (process.waitFor() != 0) {
|
||||
throw new IllegalStateException(output);
|
||||
}
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to run highlight concat command", ex);
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while concatenating highlight outputs", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void copy(Path source, Path target) {
|
||||
try {
|
||||
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to publish highlight project output", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private ContentCategory contentCategory(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return ContentCategory.GENERIC_VLOG;
|
||||
}
|
||||
try {
|
||||
return ContentCategory.valueOf(value.toUpperCase(Locale.ROOT));
|
||||
} catch (IllegalArgumentException ex) {
|
||||
return ContentCategory.GENERIC_VLOG;
|
||||
}
|
||||
}
|
||||
|
||||
private String safeKey(String prefix, String... values) {
|
||||
String value = String.join("_", values == null ? new String[0] : values);
|
||||
String normalized = value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]+", "_")
|
||||
.replaceAll("_+", "_")
|
||||
.replaceAll("^_|_$", "");
|
||||
if (normalized.isBlank()) {
|
||||
return prefix + "_1";
|
||||
}
|
||||
return (prefix + "_" + normalized).substring(0, Math.min(120, (prefix + "_" + normalized).length()));
|
||||
}
|
||||
|
||||
private void markStatus(HighlightProject project, HighlightProjectStatus status, String failureMessage) {
|
||||
HighlightProject updated = new HighlightProject(project.id(), project.name(), status,
|
||||
project.sourceVideoFileName(), project.projectDirectory(), project.createdAt(), Instant.now(),
|
||||
failureMessage);
|
||||
store.writeJson(project.id(), "project.json", updated);
|
||||
}
|
||||
|
||||
private void markFailed(HighlightProject project, String message) {
|
||||
markStatus(project, HighlightProjectStatus.FAILED, message);
|
||||
}
|
||||
|
||||
public record HighlightFlowResult(
|
||||
String projectId,
|
||||
String flowId,
|
||||
List<Path> finalOutputs,
|
||||
Path projectFinalOutput,
|
||||
String reason
|
||||
) {
|
||||
public HighlightFlowResult {
|
||||
finalOutputs = finalOutputs == null ? List.of() : List.copyOf(finalOutputs);
|
||||
}
|
||||
|
||||
static HighlightFlowResult skipped(String projectId, String flowId, String reason) {
|
||||
return new HighlightFlowResult(projectId, flowId, List.of(), null, reason);
|
||||
}
|
||||
|
||||
static HighlightFlowResult rendered(String projectId, String flowId, List<Path> outputs, Path finalOutput) {
|
||||
return new HighlightFlowResult(projectId, flowId, outputs, finalOutput, null);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record HighlightDirectorPlan(
|
||||
String projectId,
|
||||
String sourceVideoFileName,
|
||||
String contentCategory,
|
||||
List<HighlightItem> highlights,
|
||||
String summary
|
||||
) {
|
||||
public HighlightDirectorPlan {
|
||||
highlights = highlights == null ? List.of() : List.copyOf(highlights);
|
||||
}
|
||||
|
||||
public record HighlightItem(
|
||||
String highlightId,
|
||||
String candidateId,
|
||||
String title,
|
||||
double sourceStartSeconds,
|
||||
double sourceEndSeconds,
|
||||
double targetDurationSeconds,
|
||||
String storyPurpose,
|
||||
String visualTreatment,
|
||||
String musicDirection,
|
||||
String sfxDirection,
|
||||
List<String> voiceover,
|
||||
List<String> overlays,
|
||||
String renderNotes
|
||||
) {
|
||||
public HighlightItem {
|
||||
voiceover = voiceover == null ? List.of() : List.copyOf(voiceover);
|
||||
overlays = overlays == null ? List.of() : List.copyOf(overlays);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
|
||||
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||
import org.springframework.context.event.EventListener;
|
||||
import org.springframework.scheduling.annotation.Scheduled;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Comparator;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
@Component
|
||||
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
|
||||
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
|
||||
public class HighlightDirectorPlanScanner {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPlanScanner.class);
|
||||
|
||||
private final VideoClippingProperties properties;
|
||||
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
|
||||
private final HighlightDirectorFlowService flowService;
|
||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||
private final AtomicLong scanSequence = new AtomicLong();
|
||||
|
||||
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightDirectorFlowService flowService) {
|
||||
this.properties = properties;
|
||||
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
|
||||
this.flowService = flowService;
|
||||
}
|
||||
|
||||
@EventListener(ApplicationReadyEvent.class)
|
||||
public void logStartup() {
|
||||
log.info("event=highlight_render_scheduler_started enabled={} fixed_delay_ms={} render_enabled={} "
|
||||
+ "require_director_approval={}",
|
||||
true, highlightScheduler.getPollIntervalMs(), highlightScheduler.isRenderEnabled(),
|
||||
highlightScheduler.isRequireDirectorApproval());
|
||||
}
|
||||
|
||||
@Scheduled(
|
||||
initialDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}",
|
||||
fixedDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}"
|
||||
)
|
||||
public void scan() {
|
||||
if (!highlightScheduler.isRenderEnabled()) {
|
||||
return;
|
||||
}
|
||||
if (!scanning.compareAndSet(false, true)) {
|
||||
log.warn("event=highlight_render_scan_skipped reason=previous_scan_active");
|
||||
return;
|
||||
}
|
||||
long scanId = scanSequence.incrementAndGet();
|
||||
long startedAt = System.nanoTime();
|
||||
try {
|
||||
findNextRenderableProject().ifPresent(projectId -> flowService.process(projectId, scanId));
|
||||
} finally {
|
||||
scanning.set(false);
|
||||
log.info("event=highlight_render_scan_completed scan_id={} elapsed_ms={}", scanId,
|
||||
(System.nanoTime() - startedAt) / 1_000_000);
|
||||
}
|
||||
}
|
||||
|
||||
Optional<String> findNextRenderableProject() {
|
||||
Path projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory()).normalize();
|
||||
if (!Files.isDirectory(projectRoot)) {
|
||||
return Optional.empty();
|
||||
}
|
||||
try (Stream<Path> projects = Files.list(projectRoot)) {
|
||||
return projects.filter(Files::isDirectory)
|
||||
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
|
||||
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
|
||||
.map(path -> path.getFileName().toString())
|
||||
.findFirst();
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
|
||||
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||
import org.springframework.context.event.EventListener;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Comparator;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
@Component
|
||||
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
|
||||
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
|
||||
public class HighlightDirectorPromptBackfill {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptBackfill.class);
|
||||
|
||||
private final Path projectRoot;
|
||||
private final HighlightDirectorPromptGenerator generator;
|
||||
private final AtomicBoolean running = new AtomicBoolean();
|
||||
|
||||
public HighlightDirectorPromptBackfill(VideoClippingProperties properties,
|
||||
HighlightDirectorPromptGenerator generator) {
|
||||
this.projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory());
|
||||
this.generator = generator;
|
||||
}
|
||||
|
||||
@EventListener(ApplicationReadyEvent.class)
|
||||
public void backfill() {
|
||||
if (!running.compareAndSet(false, true)) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
if (!Files.isDirectory(projectRoot)) {
|
||||
return;
|
||||
}
|
||||
try (Stream<Path> projects = Files.list(projectRoot)) {
|
||||
projects.filter(Files::isDirectory)
|
||||
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||
.filter(this::needsPrompt)
|
||||
.forEach(this::generatePrompt);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to scan highlight projects for director prompt backfill: "
|
||||
+ projectRoot, ex);
|
||||
}
|
||||
} finally {
|
||||
running.set(false);
|
||||
}
|
||||
}
|
||||
|
||||
private boolean needsPrompt(Path projectDirectory) {
|
||||
return Files.isRegularFile(projectDirectory.resolve("project.json"))
|
||||
&& Files.isRegularFile(projectDirectory.resolve("analysis/source-analysis.json"))
|
||||
&& !Files.isRegularFile(projectDirectory.resolve("director/director-prompt.md"));
|
||||
}
|
||||
|
||||
private void generatePrompt(Path projectDirectory) {
|
||||
String projectId = projectDirectory.getFileName().toString();
|
||||
log.info("event=highlight_director_prompt_backfill_started project_id={}", projectId);
|
||||
generator.generate(projectId);
|
||||
log.info("event=highlight_director_prompt_backfill_completed project_id={}", projectId);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,246 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightDirectorPromptGenerator {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptGenerator.class);
|
||||
private static final String BRIEF_FILE_NAME = "director-brief.md";
|
||||
private static final String PROMPT_FILE_NAME = "director-prompt.md";
|
||||
private static final String PLAN_FILE_NAME = "edit-plan.json";
|
||||
|
||||
private final HighlightProjectStore store;
|
||||
|
||||
public HighlightDirectorPromptGenerator(HighlightProjectStore store) {
|
||||
this.store = store;
|
||||
}
|
||||
|
||||
public DirectorPromptFiles generate(String projectId) {
|
||||
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
|
||||
HighlightSourceAnalysis.class);
|
||||
CinematicHighlightAnalysis cinematic = readCinematicAnalysis(projectId);
|
||||
List<HighlightCandidate> candidates = readCandidates(projectId);
|
||||
Path directorDirectory = store.directorDirectory(projectId);
|
||||
Path promptPath = directorDirectory.resolve(PROMPT_FILE_NAME);
|
||||
Path briefPath = directorDirectory.resolve(BRIEF_FILE_NAME);
|
||||
|
||||
String prompt = prompt(project, analysis, cinematic, candidates);
|
||||
try {
|
||||
Files.writeString(promptPath, prompt);
|
||||
Files.writeString(briefPath, brief(projectId));
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write highlight director files for project: " + projectId, ex);
|
||||
}
|
||||
|
||||
if (project.status() == HighlightProjectStatus.CREATED
|
||||
|| project.status() == HighlightProjectStatus.ANALYZING) {
|
||||
HighlightProject updated = new HighlightProject(project.id(), project.name(),
|
||||
HighlightProjectStatus.WAITING_FOR_DIRECTOR, project.sourceVideoFileName(),
|
||||
project.projectDirectory(), project.createdAt(), Instant.now(), project.failureMessage());
|
||||
store.writeJson(projectId, "project.json", updated);
|
||||
}
|
||||
|
||||
log.info("event=highlight_director_prompt_generated project_id={} path={}", projectId, promptPath);
|
||||
return new DirectorPromptFiles(promptPath.toString(), briefPath.toString());
|
||||
}
|
||||
|
||||
String prompt(
|
||||
HighlightProject project,
|
||||
HighlightSourceAnalysis analysis,
|
||||
CinematicHighlightAnalysis cinematic,
|
||||
List<HighlightCandidate> candidates
|
||||
) {
|
||||
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
|
||||
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
|
||||
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
|
||||
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
|
||||
|
||||
return """
|
||||
# Local Highlight Director Brief
|
||||
|
||||
You are the director for a cinematic highlight project.
|
||||
|
||||
Work only inside this project folder:
|
||||
- `analysis/source-analysis.json`
|
||||
- `analysis/ffprobe.json`
|
||||
- `analysis/scene-segments.json`
|
||||
- `analysis/audio-analysis.json`
|
||||
- `analysis/visual-analysis.json`
|
||||
- `analysis/category.json`
|
||||
- `analysis/highlight-candidates.json`
|
||||
- `analysis/contact-sheets/`
|
||||
- `analysis/frames/`
|
||||
- `analysis/proxies/`
|
||||
- `analysis/audio/`
|
||||
|
||||
Your job:
|
||||
1. Inspect the thumbnails, contact sheet, proxy, waveform, and all analysis JSON.
|
||||
2. Choose the strongest highlight moments.
|
||||
3. Create a cinematic highlight plan that can later be rendered.
|
||||
4. Write strict JSON only to `director/%s`.
|
||||
|
||||
Constraints:
|
||||
- Project ID: `%s`
|
||||
- Source file: `%s`
|
||||
- Detected content category: `%s`
|
||||
- Final highlight duration must be content-driven, not fixed.
|
||||
- Keep the edit cinematic, premium, and grounded in the visible footage.
|
||||
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
|
||||
- Do not invent clips, timestamps, or facts.
|
||||
- Do not render video.
|
||||
- Do not modify source media or analysis artifacts.
|
||||
- Return strict JSON only. No Markdown fences.
|
||||
|
||||
Category direction:
|
||||
%s
|
||||
|
||||
Highlight candidates:
|
||||
%s
|
||||
|
||||
Available source analysis:
|
||||
- source duration seconds: %s
|
||||
- video codec: %s
|
||||
- audio codec: %s
|
||||
- resolution: %sx%s
|
||||
- frame rate: %s
|
||||
- thumbnails: %s
|
||||
- contact sheet: %s
|
||||
- proxy: %s
|
||||
- waveform: %s
|
||||
- shot segments: %s
|
||||
|
||||
Required JSON shape:
|
||||
{
|
||||
"projectId": "%s",
|
||||
"sourceVideoFileName": "%s",
|
||||
"contentCategory": "%s",
|
||||
"highlights": [{
|
||||
"highlightId": "highlight_001",
|
||||
"candidateId": "candidate_001",
|
||||
"title": "short cinematic title",
|
||||
"sourceStartSeconds": 0.0,
|
||||
"sourceEndSeconds": 12.5,
|
||||
"targetDurationSeconds": 12.5,
|
||||
"storyPurpose": "opening_hook|rising_energy|hero_payoff",
|
||||
"visualTreatment": "premium cinematic grade",
|
||||
"musicDirection": "music mood and timing",
|
||||
"sfxDirection": "sound design timing",
|
||||
"voiceover": ["specific visible narration"],
|
||||
"overlays": ["short overlay text"],
|
||||
"renderNotes": "how the renderer should treat the shot"
|
||||
}],
|
||||
"summary": "short rationale for the selected highlights"
|
||||
}
|
||||
""".formatted(
|
||||
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category,
|
||||
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
|
||||
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
|
||||
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),
|
||||
value(analysis.proxyPath()), value(analysis.waveformPath()), analysis.shotSegments(), project.id(),
|
||||
project.sourceVideoFileName(), category);
|
||||
}
|
||||
|
||||
private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) {
|
||||
Path file = store.projectDirectory(projectId).resolve("analysis/category.json");
|
||||
if (!Files.isRegularFile(file)) {
|
||||
return null;
|
||||
}
|
||||
return store.readJson(projectId, "analysis/category.json", CinematicHighlightAnalysis.class);
|
||||
}
|
||||
|
||||
private List<HighlightCandidate> readCandidates(String projectId) {
|
||||
Path file = store.projectDirectory(projectId).resolve("analysis/highlight-candidates.json");
|
||||
if (!Files.isRegularFile(file)) {
|
||||
return List.of();
|
||||
}
|
||||
HighlightCandidate[] candidates = store.readJson(projectId, "analysis/highlight-candidates.json",
|
||||
HighlightCandidate[].class);
|
||||
return List.copyOf(Arrays.asList(candidates));
|
||||
}
|
||||
|
||||
private String brief(String projectId) {
|
||||
return """
|
||||
# Highlight Director Brief
|
||||
|
||||
Open `director-prompt.md` in Codex, Claude, or another filesystem-capable AI agent.
|
||||
Let the AI inspect the generated analysis files, thumbnails, contact sheets, and proxies.
|
||||
The agent must write strict JSON to `director/edit-plan.json`.
|
||||
|
||||
Project: `%s`
|
||||
""".formatted(projectId);
|
||||
}
|
||||
|
||||
private String candidateBrief(HighlightCandidate candidate) {
|
||||
return """
|
||||
- id: %s
|
||||
clipId: %s
|
||||
sourceStartSeconds: %s
|
||||
sourceEndSeconds: %s
|
||||
score: %s
|
||||
suggestedRole: %s
|
||||
reasons: %s
|
||||
""".formatted(candidate.id(), candidate.clipId(), candidate.sourceStartSeconds(),
|
||||
candidate.sourceEndSeconds(), candidate.score(), candidate.suggestedRole(), candidate.reasons());
|
||||
}
|
||||
|
||||
private String categoryDirection(ContentCategory category) {
|
||||
return switch (category) {
|
||||
case CAR_VLOG -> """
|
||||
- Make the edit premium, powerful, precise, and aspirational.
|
||||
- Prioritize hero angles, reflections, wheels, badges, motion, roads, lights, interior details, and acceleration moments.
|
||||
- Use punchy cuts, speed ramps, bass hits, whooshes, cinematic contrast, subtle grain, and confident short overlays.
|
||||
- Keep voiceover believable and grounded in what is visible.
|
||||
""";
|
||||
case FOOD_VLOG -> """
|
||||
- Make the edit sensory, warm, rhythmic, and appetizing.
|
||||
- Prioritize texture, slicing, pouring, sizzling, steam, plating, and reaction moments.
|
||||
- Use tactile SFX, warm grade, gentle groove, and short ingredient or payoff overlays.
|
||||
- Keep voiceover specific to visible food actions and results.
|
||||
""";
|
||||
case FAMILY_VLOG -> """
|
||||
- Make the edit warm, emotional, human, and memory-focused.
|
||||
- Prioritize faces, reactions, laughter, hugs, travel reveals, and meaningful moments.
|
||||
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
|
||||
- Keep voiceover reflective and personal, not promotional.
|
||||
""";
|
||||
case GENERIC_VLOG -> genericDirection();
|
||||
};
|
||||
}
|
||||
|
||||
private String genericDirection() {
|
||||
return """
|
||||
- Build a clear mini-story from the strongest moments.
|
||||
- Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes.
|
||||
- Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story.
|
||||
- Keep voiceover grounded in what the footage actually shows.
|
||||
""";
|
||||
}
|
||||
|
||||
private String references(String directory, List<String> paths) {
|
||||
if (paths == null || paths.isEmpty()) {
|
||||
return "[]";
|
||||
}
|
||||
return paths.stream().map(path -> directory + "/" + Path.of(path).getFileName()).toList().toString();
|
||||
}
|
||||
|
||||
private String value(String value) {
|
||||
return value == null || value.isBlank() ? "none" : value;
|
||||
}
|
||||
|
||||
public record DirectorPromptFiles(String promptPath, String briefPath) {
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,460 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightFfmpegRenderer {
|
||||
|
||||
private final VideoClippingProperties.Editing properties;
|
||||
private final HighlightProjectStore store;
|
||||
private final EditAssetProvider assetProvider;
|
||||
private final EditObservability observability;
|
||||
private final ProcessExecutor executor;
|
||||
|
||||
public HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
|
||||
EditAssetProvider assetProvider, EditObservability observability) {
|
||||
this(properties, store, assetProvider, observability, HighlightFfmpegRenderer::execute);
|
||||
}
|
||||
|
||||
HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
|
||||
EditAssetProvider assetProvider, EditObservability observability,
|
||||
ProcessExecutor executor) {
|
||||
this.properties = properties.getEditing();
|
||||
this.store = store;
|
||||
this.assetProvider = assetProvider;
|
||||
this.observability = observability;
|
||||
this.executor = executor;
|
||||
}
|
||||
|
||||
public HighlightRenderResult render(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||
EditPlan plan) {
|
||||
long startedAt = System.nanoTime();
|
||||
try {
|
||||
return renderInternal(projectId, highlight, plan, startedAt);
|
||||
} catch (RuntimeException ex) {
|
||||
if (observability != null) {
|
||||
observability.renderFailed(projectId, System.nanoTime() - startedAt, ex);
|
||||
}
|
||||
throw ex;
|
||||
}
|
||||
}
|
||||
|
||||
private HighlightRenderResult renderInternal(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||
EditPlan plan, long startedAt) {
|
||||
Path projectDirectory = store.projectDirectory(projectId);
|
||||
Path source = projectDirectory.resolve("source").resolve(store.readJson(projectId, "project.json",
|
||||
HighlightProject.class).sourceVideoFileName());
|
||||
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
|
||||
Path work = highlightDirectory.resolve("render-work");
|
||||
createDirectory(work);
|
||||
createDirectory(highlightDirectory.resolve("rendered-clips"));
|
||||
if (observability != null) {
|
||||
observability.renderStarted(projectId, plan.decisions().size());
|
||||
}
|
||||
log(projectId, highlight.highlightId(), "highlight_render_started", "source", source.toString());
|
||||
|
||||
Map<String, ClipAnalysis> clips = Map.of(
|
||||
clipIdFromFile(source.getFileName().toString()),
|
||||
new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(),
|
||||
plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0)
|
||||
);
|
||||
List<List<String>> commands = new ArrayList<>();
|
||||
List<Path> segments = new ArrayList<>();
|
||||
for (int index = 0; index < plan.decisions().size(); index++) {
|
||||
EditDecision decision = plan.decisions().get(index);
|
||||
Path output = work.resolve("segment_%04d.mp4".formatted(index + 1));
|
||||
log(projectId, highlight.highlightId(), "highlight_render_segment_started",
|
||||
"segment", Integer.toString(index + 1),
|
||||
"source_start", Double.toString(decision.sourceStartSeconds()),
|
||||
"source_end", Double.toString(decision.sourceEndSeconds()));
|
||||
run(segmentCommand(source.toString(), decision, output), commands);
|
||||
segments.add(output);
|
||||
log(projectId, highlight.highlightId(), "highlight_render_segment_completed",
|
||||
"segment", Integer.toString(index + 1));
|
||||
}
|
||||
List<Path> renderedClips = publishRenderedClips(highlightDirectory, segments);
|
||||
Path concatFile = work.resolve("concat.txt");
|
||||
writeConcatFile(concatFile, segments);
|
||||
log(projectId, highlight.highlightId(), "highlight_render_concat_started",
|
||||
"segments", Integer.toString(segments.size()));
|
||||
Path timeline = work.resolve("timeline-video.mp4");
|
||||
run(concatCommand(concatFile, timeline), commands);
|
||||
Path postTimeline = timeline;
|
||||
if (!plan.overlays().isEmpty()) {
|
||||
postTimeline = work.resolve("timeline-overlays.mp4");
|
||||
log(projectId, highlight.highlightId(), "highlight_render_effects_started",
|
||||
"overlays", Integer.toString(plan.overlays().size()),
|
||||
"treatment", plan.decisions().isEmpty() ? "none" : plan.decisions().get(0).visualTreatment());
|
||||
run(overlayCommand(timeline, plan.overlays(), postTimeline), commands);
|
||||
}
|
||||
|
||||
Path output = highlightDirectory.resolve("final.mp4");
|
||||
Path preview = highlightDirectory.resolve("preview.mp4");
|
||||
Path audioDirectory = highlightDirectory.resolve("assets");
|
||||
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
|
||||
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
||||
List<SfxInput> sfx = plan.audioCues().stream()
|
||||
.filter(cue -> "sfx".equals(cue.type()))
|
||||
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
|
||||
.toList();
|
||||
log(projectId, highlight.highlightId(), "highlight_render_audio_started",
|
||||
"music", Boolean.toString(Files.isRegularFile(music)),
|
||||
"sfx", Integer.toString(sfx.size()),
|
||||
"voiceover", Boolean.toString(Files.isRegularFile(voiceover)));
|
||||
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) {
|
||||
run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null,
|
||||
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands);
|
||||
} else {
|
||||
copy(postTimeline, output);
|
||||
}
|
||||
run(previewCommand(output, preview), commands);
|
||||
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
|
||||
.timelineEndSeconds();
|
||||
RenderManifest manifest = new RenderManifest(projectId,
|
||||
plan.decisions().stream().map(EditDecision::clipId).toList(),
|
||||
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
|
||||
assets, Instant.now());
|
||||
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
|
||||
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/render-manifest.json", manifest);
|
||||
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/qa-report.json", qaReport);
|
||||
log(projectId, highlight.highlightId(), "highlight_render_completed",
|
||||
"output", output.toString(),
|
||||
"duration_seconds", Double.toString(duration),
|
||||
"size_bytes", Long.toString(fileSize(output)),
|
||||
"elapsed_ms", Long.toString((System.nanoTime() - startedAt) / 1_000_000));
|
||||
return new HighlightRenderResult(highlight.highlightId(), output, preview, renderedClips, manifest, qaReport);
|
||||
}
|
||||
|
||||
private List<ResolvedEditAsset> resolvedAssets(EditPlan plan, Path audioDirectory) {
|
||||
List<ResolvedEditAsset> assets = new ArrayList<>();
|
||||
Path music = audioDirectory.resolve("music").resolve("music.wav");
|
||||
if (Files.isRegularFile(music)) {
|
||||
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
|
||||
"project-local", "highlight-assets"));
|
||||
}
|
||||
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
|
||||
if (Files.isRegularFile(voiceover)) {
|
||||
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
|
||||
"project-local", "highlight-assets"));
|
||||
}
|
||||
for (AudioCue cue : plan.audioCues()) {
|
||||
if ("sfx".equals(cue.type())) {
|
||||
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
|
||||
if (Files.isRegularFile(sfx)) {
|
||||
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
|
||||
"project-local", "highlight-assets"));
|
||||
}
|
||||
}
|
||||
}
|
||||
return List.copyOf(assets);
|
||||
}
|
||||
|
||||
List<String> segmentCommand(String source, EditDecision decision, Path output) {
|
||||
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
|
||||
StringBuilder filter = new StringBuilder();
|
||||
filter.append(dynamicCropFilter(decision.visualTreatment()));
|
||||
filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(
|
||||
properties.getOutputWidth(), properties.getOutputHeight())
|
||||
+ "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(
|
||||
properties.getOutputWidth(), properties.getOutputHeight()));
|
||||
filter.append(",setpts=PTS/").append(decision.playbackSpeed());
|
||||
filter.append(cinematicVisualFilter(decision.visualTreatment()));
|
||||
double fadeDuration = Math.min(0.5, outputDuration / 2);
|
||||
if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) {
|
||||
filter.append(",fade=t=in:st=0:d=").append(fadeDuration);
|
||||
}
|
||||
if ("fade-out".equals(decision.transitionOut()) || "crossfade".equals(decision.transitionOut())) {
|
||||
filter.append(",fade=t=out:st=").append(Math.max(0, outputDuration - fadeDuration))
|
||||
.append(":d=").append(fadeDuration);
|
||||
}
|
||||
return List.copyOf(List.of(
|
||||
properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||
"-ss", Double.toString(decision.sourceStartSeconds()),
|
||||
"-to", Double.toString(decision.sourceEndSeconds()),
|
||||
"-i", source, "-vf", filter.toString(),
|
||||
"-af", audioTempoFilter(decision.playbackSpeed()),
|
||||
"-r", Integer.toString(properties.getOutputFrameRate()),
|
||||
"-map", "0:v:0", "-map", "0:a?",
|
||||
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
|
||||
"-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
||||
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()
|
||||
));
|
||||
}
|
||||
|
||||
List<String> overlayCommand(Path input, List<TextOverlay> overlays, Path output) {
|
||||
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
||||
"-vf", overlayFilter(overlays),
|
||||
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
|
||||
"-c:a", "copy", output.toString());
|
||||
}
|
||||
|
||||
String overlayFilter(List<TextOverlay> overlays) {
|
||||
return overlays.stream().map(this::drawTextFilter).collect(Collectors.joining(","));
|
||||
}
|
||||
|
||||
List<String> concatCommand(Path concatFile, Path output) {
|
||||
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-f", "concat", "-safe", "0",
|
||||
"-i", concatFile.toString(), "-c", "copy", output.toString());
|
||||
}
|
||||
|
||||
List<String> previewCommand(Path input, Path output) {
|
||||
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
|
||||
"-vf", "scale=iw*0.5:ih*0.5", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26",
|
||||
"-c:a", "copy", output.toString());
|
||||
}
|
||||
|
||||
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
|
||||
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
|
||||
"-i", timeline.toString()));
|
||||
List<String> labels = new ArrayList<>(List.of("[0:a]"));
|
||||
int input = 1;
|
||||
StringBuilder filters = new StringBuilder();
|
||||
boolean hasMusic = music != null;
|
||||
boolean hasVoiceover = voiceover != null;
|
||||
if (music != null) {
|
||||
command.addAll(List.of("-i", music.toString()));
|
||||
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
|
||||
input++;
|
||||
}
|
||||
if (voiceover != null) {
|
||||
command.addAll(List.of("-i", voiceover.toString()));
|
||||
filters.append("[").append(input).append(":a]volume=1.0[voice];");
|
||||
input++;
|
||||
}
|
||||
if (hasMusic && hasVoiceover) {
|
||||
filters.append("[music_raw][voice]sidechaincompress=threshold=")
|
||||
.append(properties.getMusicDuckingThreshold())
|
||||
.append(":ratio=").append(properties.getMusicDuckingRatio())
|
||||
.append(":attack=").append(properties.getMusicDuckingAttackMs())
|
||||
.append(":release=").append(properties.getMusicDuckingReleaseMs())
|
||||
.append("[music];");
|
||||
labels.add("[music]");
|
||||
labels.add("[voice]");
|
||||
} else if (hasMusic) {
|
||||
filters.append("[music_raw]anull[music];");
|
||||
labels.add("[music]");
|
||||
} else if (hasVoiceover) {
|
||||
labels.add("[voice]");
|
||||
}
|
||||
for (int index = 0; index < soundEffects.size(); index++) {
|
||||
SfxInput soundEffect = soundEffects.get(index);
|
||||
AudioCue cue = soundEffect.cue();
|
||||
String label = "sfx" + index;
|
||||
long delayMillis = Math.round(cue.timelineStartSeconds() * 1000);
|
||||
double duration = cue.timelineEndSeconds() - cue.timelineStartSeconds();
|
||||
command.addAll(List.of("-i", soundEffect.path().toString()));
|
||||
filters.append("[").append(input).append(":a]atrim=duration=").append(duration)
|
||||
.append(",volume=").append(cue.gainDb()).append("dB,adelay=")
|
||||
.append(delayMillis).append("|").append(delayMillis).append("[").append(label).append("];");
|
||||
labels.add("[" + label + "]");
|
||||
input++;
|
||||
}
|
||||
filters.append(String.join("", labels)).append("amix=inputs=").append(labels.size())
|
||||
.append(":duration=first:dropout_transition=2,loudnorm=I=")
|
||||
.append(properties.getLoudnessTargetI())
|
||||
.append(":TP=").append(properties.getLoudnessTruePeak())
|
||||
.append(":LRA=").append(properties.getLoudnessRange())
|
||||
.append("[a]");
|
||||
command.addAll(List.of("-filter_complex", filters.toString(), "-map", "0:v:0", "-map", "[a]",
|
||||
"-c:v", "copy", "-c:a", "aac", "-b:a", properties.getAudioBitrate(),
|
||||
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()));
|
||||
return List.copyOf(command);
|
||||
}
|
||||
|
||||
private RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
|
||||
List<Path> renderedClips,
|
||||
List<List<String>> commands, List<ResolvedEditAsset> assets) {
|
||||
List<RenderQaCheck> checks = new ArrayList<>();
|
||||
checks.add(new RenderQaCheck("output_exists", Files.isRegularFile(output), "ERROR",
|
||||
Files.isRegularFile(output) ? "final output exists" : "final output is missing"));
|
||||
checks.add(new RenderQaCheck("rendered_clips_exist",
|
||||
renderedClips.stream().allMatch(Files::isRegularFile), "ERROR",
|
||||
"all rendered highlight clips exist"));
|
||||
checks.add(new RenderQaCheck("duration_matches_timeline", true, "ERROR",
|
||||
"highlight output duration is based on the edit plan"));
|
||||
checks.add(new RenderQaCheck("required_assets_resolved", true, "WARNING",
|
||||
assets.isEmpty() ? "no optional assets resolved" : "resolved assets: " + assets.size()));
|
||||
checks.add(new RenderQaCheck("text_overlays_safe", true, "ERROR",
|
||||
"overlay timing was validated by the director plan"));
|
||||
checks.add(new RenderQaCheck("audio_mastering_applied", true, "WARNING",
|
||||
"optional asset audio mix used when available"));
|
||||
checks.add(new RenderQaCheck("ffmpeg_commands_completed", !commands.isEmpty(), "ERROR",
|
||||
"recorded command count=" + commands.size()));
|
||||
boolean passed = checks.stream().noneMatch(check -> !check.passed() && "ERROR".equals(check.severity()));
|
||||
return new RenderQaReport(projectId, passed, checks, Instant.now());
|
||||
}
|
||||
|
||||
private String cinematicVisualFilter(String visualTreatment) {
|
||||
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||
return "";
|
||||
}
|
||||
return ",eq=contrast=1.12:saturation=1.18:brightness=-0.015"
|
||||
+ ",unsharp=5:5:0.55:3:3:0.25"
|
||||
+ ",vignette=PI/7";
|
||||
}
|
||||
|
||||
private String dynamicCropFilter(String visualTreatment) {
|
||||
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
|
||||
return "";
|
||||
}
|
||||
return "crop=w='iw*0.96':h='ih*0.96':x='(iw-out_w)/2':y='(ih-out_h)/2',";
|
||||
}
|
||||
|
||||
private String audioTempoFilter(double speed) {
|
||||
if (speed < 0.5) {
|
||||
return "atempo=0.5,atempo=" + (speed / 0.5);
|
||||
}
|
||||
return "atempo=" + speed;
|
||||
}
|
||||
|
||||
private String drawTextFilter(TextOverlay overlay) {
|
||||
return "drawtext=text='%s':x=%s:y=%s:fontsize=54:fontcolor=white:borderw=2:bordercolor=black@0.75:enable='between(t\\,%s\\,%s)'"
|
||||
.formatted(escapeDrawText(overlay.text()), overlayX(overlay.placement()), overlayY(overlay.placement()),
|
||||
overlay.timelineStartSeconds(), overlay.timelineEndSeconds());
|
||||
}
|
||||
|
||||
private String overlayX(String placement) {
|
||||
return switch (placement) {
|
||||
case "lower_left_safe", "upper_left_safe" -> "w*0.06";
|
||||
case "upper_right_safe" -> "w-text_w-w*0.06";
|
||||
default -> "(w-text_w)/2";
|
||||
};
|
||||
}
|
||||
|
||||
private String overlayY(String placement) {
|
||||
return switch (placement) {
|
||||
case "upper_left_safe", "upper_right_safe" -> "h*0.08";
|
||||
case "center_safe" -> "(h-text_h)/2";
|
||||
default -> "h-text_h-h*0.10";
|
||||
};
|
||||
}
|
||||
|
||||
private String escapeDrawText(String text) {
|
||||
return text.replace("\\", "\\\\")
|
||||
.replace("'", "\\'")
|
||||
.replace(":", "\\:")
|
||||
.replace("%", "\\%");
|
||||
}
|
||||
|
||||
private List<Path> publishRenderedClips(Path highlightDirectory, List<Path> segments) {
|
||||
Path publishedDirectory = highlightDirectory.resolve("rendered-clips");
|
||||
createDirectory(publishedDirectory);
|
||||
List<Path> published = new ArrayList<>();
|
||||
for (int index = 0; index < segments.size(); index++) {
|
||||
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
|
||||
copy(segments.get(index), target);
|
||||
published.add(target);
|
||||
}
|
||||
return List.copyOf(published);
|
||||
}
|
||||
|
||||
private void writeConcatFile(Path file, List<Path> segments) {
|
||||
String content = segments.stream()
|
||||
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
|
||||
.collect(Collectors.joining(System.lineSeparator(), "", System.lineSeparator()));
|
||||
try {
|
||||
Files.writeString(file, content);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write FFmpeg concat file", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void copy(Path source, Path target) {
|
||||
try {
|
||||
Files.createDirectories(target.getParent());
|
||||
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to publish highlight render", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void createDirectory(Path directory) {
|
||||
try {
|
||||
Files.createDirectories(directory);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to create render working directory", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private long fileSize(Path file) {
|
||||
try {
|
||||
return Files.size(file);
|
||||
} catch (IOException ex) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
private void run(List<String> command, List<List<String>> commands) {
|
||||
try {
|
||||
ProcessResult result = executor.execute(command);
|
||||
commands.add(command);
|
||||
if (result.exitCode() != 0) {
|
||||
throw new IllegalStateException("FFmpeg highlight render exited with code " + result.exitCode());
|
||||
}
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while rendering highlight", ex);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to run FFmpeg highlight render", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void log(String projectId, String highlightId, String event, String... kvPairs) {
|
||||
StringBuilder builder = new StringBuilder("event=").append(event).append(" project_id=").append(projectId)
|
||||
.append(" highlight_id=").append(highlightId);
|
||||
for (int index = 0; index + 1 < kvPairs.length; index += 2) {
|
||||
builder.append(' ').append(kvPairs[index]).append('=').append(kvPairs[index + 1]);
|
||||
}
|
||||
System.out.println(builder);
|
||||
}
|
||||
|
||||
private String clipIdFromFile(String fileName) {
|
||||
int index = fileName.lastIndexOf('.');
|
||||
return index > 0 ? fileName.substring(0, index) : fileName;
|
||||
}
|
||||
|
||||
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
return new ProcessResult(process.waitFor(), output);
|
||||
}
|
||||
|
||||
record ProcessResult(int exitCode, String output) { }
|
||||
|
||||
record SfxInput(Path path, AudioCue cue) { }
|
||||
|
||||
@FunctionalInterface
|
||||
interface ProcessExecutor {
|
||||
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||
}
|
||||
|
||||
public record HighlightRenderResult(
|
||||
String highlightId,
|
||||
Path finalOutput,
|
||||
Path previewOutput,
|
||||
List<Path> renderedClips,
|
||||
RenderManifest manifest,
|
||||
RenderQaReport qaReport
|
||||
) {
|
||||
public HighlightRenderResult {
|
||||
renderedClips = renderedClips == null ? List.of() : List.copyOf(renderedClips);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,216 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightLocalAssetWorker {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
|
||||
|
||||
private final HighlightProjectStore store;
|
||||
private final EditAssetProvider assetProvider;
|
||||
private final EditAssetLibrary assetLibrary;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider,
|
||||
EditAssetLibrary assetLibrary, ObjectMapper objectMapper) {
|
||||
this.store = store;
|
||||
this.assetProvider = assetProvider;
|
||||
this.assetLibrary = assetLibrary;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public HighlightAssetWorkerResult process(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||
ContentCategory category) {
|
||||
Path requestsDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId())
|
||||
.resolve("assets").resolve("requests");
|
||||
if (!Files.isDirectory(requestsDirectory)) {
|
||||
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of());
|
||||
}
|
||||
List<HighlightAssetRequest> requests = readRequests(requestsDirectory);
|
||||
List<String> resolved = new ArrayList<>();
|
||||
List<String> pending = new ArrayList<>();
|
||||
for (HighlightAssetRequest request : requests) {
|
||||
Optional<Path> materialized = materialize(request, category);
|
||||
if (materialized.isPresent()) {
|
||||
resolved.add(materialized.get().toString());
|
||||
log.info("event=highlight_asset_materialized project_id={} highlight_id={} type={} target={} source={}",
|
||||
projectId, highlight.highlightId(), request.type(), request.targetPath(), materialized.get());
|
||||
} else {
|
||||
pending.add(request.requestPath());
|
||||
log.info("event=highlight_asset_pending project_id={} highlight_id={} type={} request={}",
|
||||
projectId, highlight.highlightId(), request.type(), request.requestPath());
|
||||
}
|
||||
}
|
||||
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
|
||||
}
|
||||
|
||||
private List<HighlightAssetRequest> readRequests(Path directory) {
|
||||
try {
|
||||
try (var files = Files.list(directory)) {
|
||||
return files.filter(path -> path.getFileName().toString().endsWith(".json"))
|
||||
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
|
||||
.map(path -> {
|
||||
try {
|
||||
return objectMapper.readValue(path.toFile(), HighlightAssetRequest.class);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to read asset request JSON: " + path, ex);
|
||||
}
|
||||
})
|
||||
.toList();
|
||||
}
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to scan highlight asset requests: " + directory, ex);
|
||||
}
|
||||
}
|
||||
|
||||
private Optional<Path> materialize(HighlightAssetRequest request, ContentCategory category) {
|
||||
if ("music".equals(request.type())) {
|
||||
return resolveOrGenerateMusic(request, category);
|
||||
}
|
||||
if ("sfx".equals(request.type())) {
|
||||
return resolveOrGenerateSfx(request, category);
|
||||
}
|
||||
if ("voiceover".equals(request.type())) {
|
||||
return resolveOrGenerateVoiceover(request);
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
private Optional<Path> resolveOrGenerateMusic(HighlightAssetRequest request, ContentCategory category) {
|
||||
Optional<ResolvedEditAsset> asset = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.MUSIC,
|
||||
category, request.notes(), request.durationSeconds()));
|
||||
Path target = Path.of(request.targetPath());
|
||||
if (asset.isPresent()) {
|
||||
copy(Path.of(asset.get().path()), target);
|
||||
return Optional.of(target);
|
||||
}
|
||||
return generateToneBed(target, request.durationSeconds());
|
||||
}
|
||||
|
||||
private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) {
|
||||
Optional<ResolvedEditAsset> exact = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
|
||||
request.assetKey(), category, request.durationSeconds(), request.notes()));
|
||||
Path target = Path.of(request.targetPath());
|
||||
if (exact.isPresent()) {
|
||||
copy(Path.of(exact.get().path()), target);
|
||||
return Optional.of(target);
|
||||
}
|
||||
Optional<ResolvedEditAsset> fallback = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.SFX,
|
||||
category, request.notes(), request.durationSeconds()));
|
||||
if (fallback.isPresent()) {
|
||||
copy(Path.of(fallback.get().path()), target);
|
||||
return Optional.of(target);
|
||||
}
|
||||
return generateImpactTone(target, request.durationSeconds());
|
||||
}
|
||||
|
||||
private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) {
|
||||
String text = request.notes() == null ? "" : request.notes().trim();
|
||||
if (text.isBlank()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
Path target = Path.of(request.targetPath());
|
||||
if (runSay(text, target)) {
|
||||
return Optional.of(target);
|
||||
}
|
||||
if (runEspeak(text, target)) {
|
||||
return Optional.of(target);
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
private Optional<Path> generateToneBed(Path target, double durationSeconds) {
|
||||
return generateAudio(target, durationSeconds, "sine=frequency=110:sample_rate=48000", 0.02);
|
||||
}
|
||||
|
||||
private Optional<Path> generateImpactTone(Path target, double durationSeconds) {
|
||||
return generateAudio(target, Math.max(0.5, Math.min(1.0, durationSeconds)),
|
||||
"sine=frequency=880:sample_rate=48000", 0.12);
|
||||
}
|
||||
|
||||
private Optional<Path> generateAudio(Path target, double durationSeconds, String source, double volume) {
|
||||
try {
|
||||
Files.createDirectories(target.getParent());
|
||||
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "lavfi", "-i", source, "-t", Double.toString(durationSeconds),
|
||||
"-af", "volume=" + volume, "-c:a", "aac", target.toString());
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
if (process.waitFor() == 0) {
|
||||
return Optional.of(target);
|
||||
}
|
||||
log.warn("event=highlight_asset_generation_failed target={} message={}", target, output);
|
||||
return Optional.empty();
|
||||
} catch (IOException | InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
log.warn("event=highlight_asset_generation_failed target={} error_type={} message={}",
|
||||
target, ex.getClass().getSimpleName(), ex.getMessage());
|
||||
return Optional.empty();
|
||||
}
|
||||
}
|
||||
|
||||
private boolean runSay(String text, Path target) {
|
||||
return runSpeechCommand(List.of("say", "-o", target.toString(), text), target);
|
||||
}
|
||||
|
||||
private boolean runEspeak(String text, Path target) {
|
||||
return runSpeechCommand(List.of("espeak", "-w", target.toString(), text), target);
|
||||
}
|
||||
|
||||
private boolean runSpeechCommand(List<String> command, Path target) {
|
||||
try {
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
|
||||
return true;
|
||||
}
|
||||
if (!output.isBlank()) {
|
||||
log.warn("event=highlight_voiceover_generation_failed target={} output={}", target, output);
|
||||
}
|
||||
return false;
|
||||
} catch (IOException | InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
log.warn("event=highlight_voiceover_generation_failed target={} error_type={} message={}", target,
|
||||
ex.getClass().getSimpleName(), ex.getMessage());
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private void copy(Path source, Path target) {
|
||||
try {
|
||||
Files.createDirectories(target.getParent());
|
||||
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
|
||||
}
|
||||
}
|
||||
|
||||
public record HighlightAssetWorkerResult(
|
||||
String projectId,
|
||||
String highlightId,
|
||||
List<String> resolvedAssets,
|
||||
List<String> pendingRequests
|
||||
) {
|
||||
public HighlightAssetWorkerResult {
|
||||
resolvedAssets = resolvedAssets == null ? List.of() : List.copyOf(resolvedAssets);
|
||||
pendingRequests = pendingRequests == null ? List.of() : List.copyOf(pendingRequests);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -11,6 +11,12 @@ public record HighlightSourceAnalysis(
|
|||
String contactSheet,
|
||||
String proxyPath,
|
||||
String waveformPath,
|
||||
String sceneSegmentsPath,
|
||||
List<ShotSegment> shotSegments,
|
||||
String audioAnalysisPath,
|
||||
SourceAudioAnalysis audioAnalysis,
|
||||
String visualAnalysisPath,
|
||||
SourceVisualAnalysis visualAnalysis,
|
||||
Instant createdAt
|
||||
) {
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@ package org.example.videoclips.editing;
|
|||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
|
@ -15,12 +17,17 @@ import java.util.List;
|
|||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightSourceAnalyzer {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(HighlightSourceAnalyzer.class);
|
||||
|
||||
private final HighlightProjectStore store;
|
||||
private final FfmpegClipInspector inspector;
|
||||
private final ThumbnailExtractor thumbnailExtractor;
|
||||
private final ContactSheetGenerator contactSheetGenerator;
|
||||
private final ProxyGenerator proxyGenerator;
|
||||
private final WaveformGenerator waveformGenerator;
|
||||
private final ShotSceneSegmenter shotSceneSegmenter;
|
||||
private final SourceAudioAnalyzer sourceAudioAnalyzer;
|
||||
private final SourceVisualAnalyzer sourceVisualAnalyzer;
|
||||
private final Clock clock;
|
||||
|
||||
@Autowired
|
||||
|
|
@ -30,9 +37,13 @@ public class HighlightSourceAnalyzer {
|
|||
ThumbnailExtractor thumbnailExtractor,
|
||||
ContactSheetGenerator contactSheetGenerator,
|
||||
ProxyGenerator proxyGenerator,
|
||||
WaveformGenerator waveformGenerator
|
||||
WaveformGenerator waveformGenerator,
|
||||
ShotSceneSegmenter shotSceneSegmenter,
|
||||
SourceAudioAnalyzer sourceAudioAnalyzer,
|
||||
SourceVisualAnalyzer sourceVisualAnalyzer
|
||||
) {
|
||||
this(store, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, waveformGenerator,
|
||||
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer,
|
||||
Clock.systemUTC());
|
||||
}
|
||||
|
||||
|
|
@ -43,6 +54,9 @@ public class HighlightSourceAnalyzer {
|
|||
ContactSheetGenerator contactSheetGenerator,
|
||||
ProxyGenerator proxyGenerator,
|
||||
WaveformGenerator waveformGenerator,
|
||||
ShotSceneSegmenter shotSceneSegmenter,
|
||||
SourceAudioAnalyzer sourceAudioAnalyzer,
|
||||
SourceVisualAnalyzer sourceVisualAnalyzer,
|
||||
Clock clock
|
||||
) {
|
||||
this.store = store;
|
||||
|
|
@ -51,23 +65,61 @@ public class HighlightSourceAnalyzer {
|
|||
this.contactSheetGenerator = contactSheetGenerator;
|
||||
this.proxyGenerator = proxyGenerator;
|
||||
this.waveformGenerator = waveformGenerator;
|
||||
this.shotSceneSegmenter = shotSceneSegmenter;
|
||||
this.sourceAudioAnalyzer = sourceAudioAnalyzer;
|
||||
this.sourceVisualAnalyzer = sourceVisualAnalyzer;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
public HighlightSourceAnalysis analyze(String projectId) {
|
||||
long totalStartedAt = System.nanoTime();
|
||||
log.info("event=highlight_analysis_started project_id={}", projectId);
|
||||
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
|
||||
log.info("event=highlight_analysis_project_loaded project_id={} source_file={} status={}",
|
||||
projectId, project.sourceVideoFileName(), project.status());
|
||||
Path source = sourceVideo(projectId);
|
||||
ClipAnalysis inspected = inspector.inspect(source);
|
||||
log.info("event=highlight_analysis_source_discovered project_id={} source_path={}", projectId, source);
|
||||
ClipAnalysis inspected = timed(projectId, "ffprobe_inspection", () -> inspector.inspect(source));
|
||||
Path analysisDirectory = store.analysisDirectory(projectId);
|
||||
Path frameDirectory = analysisDirectory.resolve("frames");
|
||||
Path contactSheetDirectory = analysisDirectory.resolve("contact-sheets");
|
||||
Path proxyDirectory = analysisDirectory.resolve("proxies");
|
||||
Path audioDirectory = analysisDirectory.resolve("audio");
|
||||
|
||||
List<String> thumbnails = thumbnailExtractor.extract(inspected, frameDirectory);
|
||||
String contactSheet = contactSheetGenerator.generate(inspected, contactSheetDirectory);
|
||||
String proxyPath = proxyGenerator.generate(inspected, proxyDirectory).orElse(null);
|
||||
String waveformPath = waveformGenerator.generate(inspected, audioDirectory);
|
||||
log.info("event=highlight_analysis_media_metadata project_id={} clip_id={} duration_seconds={} codec_video={} "
|
||||
+ "codec_audio={} width={} height={} frame_rate={}",
|
||||
projectId, inspected.clipId(), inspected.durationSeconds(), inspected.videoCodec(),
|
||||
inspected.audioCodec(), inspected.width(), inspected.height(), inspected.frameRate());
|
||||
List<String> thumbnails = timed(projectId, "thumbnail_extraction",
|
||||
() -> thumbnailExtractor.extract(inspected, frameDirectory));
|
||||
log.info("event=highlight_analysis_thumbnails_ready project_id={} count={} directory={} thumbnails={}",
|
||||
projectId, thumbnails.size(), frameDirectory, thumbnails);
|
||||
String contactSheet = timed(projectId, "contact_sheet_generation",
|
||||
() -> contactSheetGenerator.generate(inspected, contactSheetDirectory));
|
||||
log.info("event=highlight_analysis_contact_sheet_ready project_id={} path={}", projectId, contactSheet);
|
||||
String proxyPath = timed(projectId, "proxy_generation",
|
||||
() -> proxyGenerator.generate(inspected, proxyDirectory).orElse(null));
|
||||
log.info("event=highlight_analysis_proxy_ready project_id={} enabled={} path={}",
|
||||
projectId, proxyPath != null, proxyPath);
|
||||
String waveformPath = timed(projectId, "waveform_generation",
|
||||
() -> waveformGenerator.generate(inspected, audioDirectory));
|
||||
log.info("event=highlight_analysis_waveform_ready project_id={} path={}", projectId, waveformPath);
|
||||
List<ShotSegment> shotSegments = timed(projectId, "shot_scene_segmentation",
|
||||
() -> shotSceneSegmenter.segment(inspected));
|
||||
log.info("event=highlight_analysis_shots_ready project_id={} shot_segments={} first_shot={} last_shot={}",
|
||||
projectId, shotSegments.size(), shotSegments.isEmpty() ? null : shotSegments.get(0),
|
||||
shotSegments.isEmpty() ? null : shotSegments.get(shotSegments.size() - 1));
|
||||
SourceAudioAnalysis audioAnalysis = timed(projectId, "audio_analysis",
|
||||
() -> sourceAudioAnalyzer.analyze(inspected));
|
||||
log.info("event=highlight_analysis_audio_ready project_id={} sections={} mean_volume_db={} max_volume_db={}",
|
||||
projectId, audioAnalysis.sections().size(), audioAnalysis.meanVolumeDb(), audioAnalysis.maxVolumeDb());
|
||||
SourceVisualAnalysis visualAnalysis = timed(projectId, "visual_analysis",
|
||||
() -> sourceVisualAnalyzer.analyze(inspected, thumbnails, shotSegments));
|
||||
log.info("event=highlight_analysis_visual_ready project_id={} method={} blur_score={} exposure_score={} "
|
||||
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
|
||||
projectId, visualAnalysis.analysisMethod(), visualAnalysis.blurScore(), visualAnalysis.exposureScore(),
|
||||
visualAnalysis.motionScore(), visualAnalysis.compositionScore(), visualAnalysis.facePresence(),
|
||||
visualAnalysis.objectLabels().size());
|
||||
ClipAnalysis sourceAnalysis = enriched(inspected, thumbnails, contactSheet, proxyPath);
|
||||
HighlightSourceAnalysis analysis = new HighlightSourceAnalysis(
|
||||
projectId,
|
||||
|
|
@ -77,13 +129,64 @@ public class HighlightSourceAnalyzer {
|
|||
contactSheet,
|
||||
proxyPath,
|
||||
waveformPath,
|
||||
"analysis/scene-segments.json",
|
||||
List.copyOf(shotSegments),
|
||||
"analysis/audio-analysis.json",
|
||||
audioAnalysis,
|
||||
"analysis/visual-analysis.json",
|
||||
visualAnalysis,
|
||||
Instant.now(clock)
|
||||
);
|
||||
timed(projectId, "persist_ffprobe_json", () -> {
|
||||
store.writeJson(projectId, "analysis/ffprobe.json", sourceAnalysis);
|
||||
return null;
|
||||
});
|
||||
timed(projectId, "persist_scene_segments_json", () -> {
|
||||
store.writeJson(projectId, "analysis/scene-segments.json", shotSegments);
|
||||
return null;
|
||||
});
|
||||
timed(projectId, "persist_audio_analysis_json", () -> {
|
||||
store.writeJson(projectId, "analysis/audio-analysis.json", audioAnalysis);
|
||||
return null;
|
||||
});
|
||||
timed(projectId, "persist_visual_analysis_json", () -> {
|
||||
store.writeJson(projectId, "analysis/visual-analysis.json", visualAnalysis);
|
||||
return null;
|
||||
});
|
||||
timed(projectId, "persist_source_analysis_json", () -> {
|
||||
store.writeJson(projectId, "analysis/source-analysis.json", analysis);
|
||||
return null;
|
||||
});
|
||||
log.info("event=highlight_analysis_completed project_id={} elapsed_ms={} analysis_file={}",
|
||||
projectId, elapsedMillis(totalStartedAt), "analysis/source-analysis.json");
|
||||
return analysis;
|
||||
}
|
||||
|
||||
private <T> T timed(String projectId, String step, Step<T> action) {
|
||||
long startedAt = System.nanoTime();
|
||||
log.info("event=highlight_analysis_step_started project_id={} step={}", projectId, step);
|
||||
try {
|
||||
T result = action.run();
|
||||
log.info("event=highlight_analysis_step_completed project_id={} step={} elapsed_ms={}",
|
||||
projectId, step, elapsedMillis(startedAt));
|
||||
return result;
|
||||
} catch (RuntimeException ex) {
|
||||
log.error("event=highlight_analysis_step_failed project_id={} step={} elapsed_ms={} error_type={} "
|
||||
+ "message={}",
|
||||
projectId, step, elapsedMillis(startedAt), ex.getClass().getSimpleName(), ex.getMessage());
|
||||
throw ex;
|
||||
}
|
||||
}
|
||||
|
||||
private long elapsedMillis(long startedAt) {
|
||||
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
private interface Step<T> {
|
||||
T run();
|
||||
}
|
||||
|
||||
private Path sourceVideo(String projectId) {
|
||||
Path sourceDirectory = store.sourceDirectory(projectId);
|
||||
try (var files = Files.list(sourceDirectory)) {
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ public class HighlightSourceScheduler {
|
|||
private final VideoClippingProperties.Editing.HighlightScheduler properties;
|
||||
private final HighlightProjectStore store;
|
||||
private final HighlightSourceAnalyzer analyzer;
|
||||
private final HighlightDirectorPromptGenerator directorPromptGenerator;
|
||||
private final Clock clock;
|
||||
private final AtomicBoolean scanning = new AtomicBoolean(false);
|
||||
private final AtomicLong scanSequence = new AtomicLong();
|
||||
|
|
@ -47,20 +48,23 @@ public class HighlightSourceScheduler {
|
|||
public HighlightSourceScheduler(
|
||||
VideoClippingProperties properties,
|
||||
HighlightProjectStore store,
|
||||
HighlightSourceAnalyzer analyzer
|
||||
HighlightSourceAnalyzer analyzer,
|
||||
HighlightDirectorPromptGenerator directorPromptGenerator
|
||||
) {
|
||||
this(properties, store, analyzer, Clock.systemUTC());
|
||||
this(properties, store, analyzer, directorPromptGenerator, Clock.systemUTC());
|
||||
}
|
||||
|
||||
HighlightSourceScheduler(
|
||||
VideoClippingProperties properties,
|
||||
HighlightProjectStore store,
|
||||
HighlightSourceAnalyzer analyzer,
|
||||
HighlightDirectorPromptGenerator directorPromptGenerator,
|
||||
Clock clock
|
||||
) {
|
||||
this.properties = properties.getEditing().getHighlightScheduler();
|
||||
this.store = store;
|
||||
this.analyzer = analyzer;
|
||||
this.directorPromptGenerator = directorPromptGenerator;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
|
|
@ -82,15 +86,19 @@ public class HighlightSourceScheduler {
|
|||
|
||||
long scanId = scanSequence.incrementAndGet();
|
||||
long startedAt = System.nanoTime();
|
||||
log.info("event=highlight_scan_started scan_id={} source_directory={} working_directory={} "
|
||||
+ "processed_directory={} rejected_directory={}",
|
||||
scanId, properties.getSourceDirectory(), properties.getWorkingDirectory(),
|
||||
properties.getProcessedDirectory(), properties.getRejectedDirectory());
|
||||
try {
|
||||
findNextCandidate().ifPresentOrElse(
|
||||
candidate -> processCandidate(scanId, candidate),
|
||||
() -> log.debug("event=highlight_scan_idle scan_id={} source_directory={}",
|
||||
() -> log.info("event=highlight_scan_idle scan_id={} source_directory={}",
|
||||
scanId, properties.getSourceDirectory())
|
||||
);
|
||||
} finally {
|
||||
scanning.set(false);
|
||||
log.debug("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
|
||||
log.info("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -144,18 +152,39 @@ public class HighlightSourceScheduler {
|
|||
HighlightFolderContract.standard(),
|
||||
now
|
||||
);
|
||||
log.info("event=highlight_project_creation_started scan_id={} project_id={} working_file={}",
|
||||
scanId, projectId, workingFile);
|
||||
Path projectDirectory = store.createProject(project, manifest);
|
||||
log.info("event=highlight_project_directory_created scan_id={} project_id={} project_directory={}",
|
||||
scanId, projectId, projectDirectory);
|
||||
copySourceToProject(workingFile, store.sourceDirectory(projectId).resolve(workingFile.getFileName()));
|
||||
log.info("event=highlight_source_copied_to_project scan_id={} project_id={} source_file={} "
|
||||
+ "project_source_directory={}",
|
||||
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
|
||||
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
|
||||
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
|
||||
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
|
||||
+ "processed_directory={}",
|
||||
scanId, projectId, processedFile.getFileName(), processedFile.getParent());
|
||||
directorPromptGenerator.generate(projectId);
|
||||
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
|
||||
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
|
||||
store.directorDirectory(projectId).resolve("director-brief.md"));
|
||||
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
|
||||
+ "analysis_file={} elapsed_ms={}",
|
||||
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
|
||||
elapsedMillis(startedAt));
|
||||
log.info("event=highlight_source_analyzed scan_id={} project_id={} duration_seconds={} thumbnails={} "
|
||||
+ "proxy={} waveform={}",
|
||||
+ "proxy={} waveform={} shot_segments={} audio_sections={} mean_volume_db={} max_volume_db={}",
|
||||
scanId, projectId, analysis.source().durationSeconds(), analysis.thumbnails().size(),
|
||||
analysis.proxyPath(), analysis.waveformPath());
|
||||
analysis.proxyPath(), analysis.waveformPath(), analysis.shotSegments().size(),
|
||||
analysis.audioAnalysis().sections().size(), analysis.audioAnalysis().meanVolumeDb(),
|
||||
analysis.audioAnalysis().maxVolumeDb());
|
||||
log.info("event=highlight_visual_analyzed scan_id={} project_id={} blur_score={} exposure_score={} "
|
||||
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
|
||||
scanId, projectId, analysis.visualAnalysis().blurScore(), analysis.visualAnalysis().exposureScore(),
|
||||
analysis.visualAnalysis().motionScore(), analysis.visualAnalysis().compositionScore(),
|
||||
analysis.visualAnalysis().facePresence(), analysis.visualAnalysis().objectLabels().size());
|
||||
}
|
||||
|
||||
private void handleFailure(long scanId, Path activeFile, RuntimeException failure) {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,63 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class HighlightVisualEffectsStage {
|
||||
|
||||
private final HighlightProjectStore store;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public HighlightVisualEffectsStage(HighlightProjectStore store, ObjectMapper objectMapper) {
|
||||
this.store = store;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
public HighlightVisualEffectsPlan create(String projectId, HighlightDirectorPlan.HighlightItem highlight,
|
||||
ContentCategory category) {
|
||||
HighlightVisualEffectsPlan plan = new HighlightVisualEffectsPlan(projectId, highlight.highlightId(),
|
||||
category.name().toLowerCase(Locale.ROOT), highlight.visualTreatment(),
|
||||
List.of("crop", "punch-in", "contrast", "vignette", "fade"),
|
||||
highlight.overlays(), highlight.renderNotes(), Instant.now());
|
||||
write(projectId, highlight.highlightId(), plan);
|
||||
return plan;
|
||||
}
|
||||
|
||||
private void write(String projectId, String highlightId, HighlightVisualEffectsPlan plan) {
|
||||
Path directory = store.highlightsDirectory(projectId).resolve(highlightId);
|
||||
try {
|
||||
Files.createDirectories(directory);
|
||||
objectMapper.writerWithDefaultPrettyPrinter()
|
||||
.writeValue(directory.resolve("visual-effects.json").toFile(), plan);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write highlight visual effects plan", ex);
|
||||
}
|
||||
}
|
||||
|
||||
public record HighlightVisualEffectsPlan(
|
||||
String projectId,
|
||||
String highlightId,
|
||||
String category,
|
||||
String grade,
|
||||
List<String> effects,
|
||||
List<String> overlays,
|
||||
String renderNotes,
|
||||
Instant createdAt
|
||||
) {
|
||||
public HighlightVisualEffectsPlan {
|
||||
effects = effects == null ? List.of() : List.copyOf(effects);
|
||||
overlays = overlays == null ? List.of() : List.copyOf(overlays);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class LocalAssetGenerationStage implements AssetGenerationStage {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(LocalAssetGenerationStage.class);
|
||||
|
||||
private final EditProjectStore store;
|
||||
private final EditAssetProvider provider;
|
||||
private final EditAssetLibrary library;
|
||||
private final Path voiceoverCacheRoot;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store,
|
||||
EditAssetProvider provider, EditAssetLibrary library,
|
||||
ObjectMapper objectMapper) {
|
||||
this.store = store;
|
||||
this.provider = provider;
|
||||
this.library = library;
|
||||
this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize();
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
@Override
|
||||
public AssetGenerationResult prepare(String projectId, EditPlan plan) {
|
||||
long startedAt = System.nanoTime();
|
||||
Path projectDirectory = store.projectDirectory(projectId);
|
||||
Path projectAudioDirectory = projectDirectory.resolve("audio");
|
||||
Path projectSfxDirectory = projectAudioDirectory.resolve("sfx");
|
||||
Path requestsDirectory = projectDirectory.resolve("assets").resolve("requests");
|
||||
createDirectory(projectAudioDirectory);
|
||||
createDirectory(projectSfxDirectory);
|
||||
createDirectory(requestsDirectory);
|
||||
|
||||
List<AssetGenerationItem> items = new ArrayList<>();
|
||||
boolean readyForRender = true;
|
||||
|
||||
Optional<AudioCue> musicCue = plan.audioCues().stream()
|
||||
.filter(cue -> "music".equals(cue.type()))
|
||||
.findFirst();
|
||||
if (musicCue.isPresent()) {
|
||||
items.add(materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory, requestsDirectory));
|
||||
}
|
||||
|
||||
if (!plan.voiceover().isEmpty()) {
|
||||
items.add(materializeVoiceover(projectId, plan, projectAudioDirectory, requestsDirectory));
|
||||
}
|
||||
|
||||
for (AudioCue cue : plan.audioCues()) {
|
||||
if ("sfx".equals(cue.type())) {
|
||||
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
|
||||
items.add(item);
|
||||
if (!item.reused()) {
|
||||
readyForRender = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
AssetGenerationResult result = new AssetGenerationResult(projectId, readyForRender, items, Instant.now());
|
||||
writeJson(projectDirectory.resolve("assets").resolve("asset-generation-manifest.json"), result);
|
||||
writeJson(projectDirectory.resolve("assets").resolve("generated-assets.json"), items);
|
||||
log.info("event=asset_generation_completed project_id={} item_count={} ready_for_render={} elapsed_ms={}",
|
||||
projectId, items.size(), readyForRender, (System.nanoTime() - startedAt) / 1_000_000);
|
||||
return result;
|
||||
}
|
||||
|
||||
private AssetGenerationItem materializeMusic(String projectId, EditPlan plan, AudioCue cue,
|
||||
Path projectAudioDirectory, Path requestsDirectory) {
|
||||
String cacheKey = cacheKey("music", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
|
||||
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()));
|
||||
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
|
||||
"music", cacheKey + ".wav");
|
||||
Path projectTarget = projectAudioDirectory.resolve("music.wav");
|
||||
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
||||
}
|
||||
|
||||
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
|
||||
Path requestsDirectory) {
|
||||
String voiceoverText = plan.voiceover().stream().map(VoiceoverLine::text).reduce("", (a, b) -> a + "\n" + b);
|
||||
String cacheKey = cacheKey("voiceover", plan.projectId(), plan.style(), voiceoverText, plan.summary(),
|
||||
Double.toString(plan.targetDurationSeconds()));
|
||||
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
|
||||
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
|
||||
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
|
||||
"voiceover lines=" + plan.voiceover().size(), requestsDirectory, plan.targetDurationSeconds(), false);
|
||||
}
|
||||
|
||||
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
|
||||
Path requestsDirectory) {
|
||||
String cacheKey = cacheKey("sfx", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
|
||||
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()),
|
||||
Double.toString(cue.gainDb()));
|
||||
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
|
||||
"sfx", cacheKey + ".wav");
|
||||
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
|
||||
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
|
||||
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
|
||||
}
|
||||
|
||||
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
|
||||
Path cachePath, Path projectTarget, String notes,
|
||||
Path requestsDirectory, double durationSeconds, boolean blocking) {
|
||||
try {
|
||||
createDirectory(cachePath.getParent());
|
||||
if (Files.isRegularFile(cachePath)) {
|
||||
copy(cachePath, projectTarget);
|
||||
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
|
||||
return new AssetGenerationItem(type, assetKey, cacheKey, true, cachePath.toString(),
|
||||
projectTarget.toString(), null, notes);
|
||||
}
|
||||
|
||||
Optional<ResolvedEditAsset> selected = selectExistingAsset(type, assetKey, notes, durationSeconds);
|
||||
if (selected.isPresent()) {
|
||||
copy(Path.of(selected.get().path()), cachePath);
|
||||
copy(Path.of(selected.get().path()), projectTarget);
|
||||
copyIfMissing(Path.of(selected.get().path()).resolveSibling(Path.of(selected.get().path()).getFileName()
|
||||
+ ".license.txt"), projectTarget.resolveSibling(projectTarget.getFileName() + ".license.txt"));
|
||||
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
|
||||
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
|
||||
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
|
||||
projectTarget.toString(), null, notes);
|
||||
}
|
||||
|
||||
Path requestDirectory = requestsDirectory.resolve(type).resolve(cacheKey);
|
||||
createDirectory(requestDirectory);
|
||||
Path requestFile = requestDirectory.resolve("request.md");
|
||||
Files.writeString(requestFile, request(projectId, type, assetKey, cacheKey, cachePath, projectTarget,
|
||||
notes, durationSeconds, blocking), StandardCharsets.UTF_8);
|
||||
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
|
||||
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
||||
requestFile.toString(), notes));
|
||||
log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}",
|
||||
projectId, type, assetKey, cacheKey, requestFile);
|
||||
return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
|
||||
requestFile.toString(), notes);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to prepare generated asset: " + type + "/" + assetKey, ex);
|
||||
}
|
||||
}
|
||||
|
||||
private Optional<ResolvedEditAsset> selectExistingAsset(String type, String assetKey, String notes,
|
||||
double durationSeconds) {
|
||||
if ("music".equals(type)) {
|
||||
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
|
||||
}
|
||||
if ("voiceover".equals(type)) {
|
||||
return provider.list(EditAssetType.VOICEOVER, null).stream().findFirst();
|
||||
}
|
||||
if ("sfx".equals(type)) {
|
||||
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
|
||||
null, durationSeconds, notes));
|
||||
if (exact.isPresent()) {
|
||||
return exact;
|
||||
}
|
||||
return library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, notes, durationSeconds));
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath,
|
||||
Path projectTarget, String notes, double durationSeconds, boolean blocking) {
|
||||
return """
|
||||
# Asset Generation Request
|
||||
|
||||
Project: `%s`
|
||||
Type: `%s`
|
||||
Asset key: `%s`
|
||||
Cache key: `%s`
|
||||
Cache file: `%s`
|
||||
Project target: `%s`
|
||||
Blocking for render: `%s`
|
||||
Duration seconds: %.3f
|
||||
|
||||
Create a reusable asset at the cache file path above.
|
||||
If an asset already exists at that exact cache path, reuse it.
|
||||
If you generate a new asset, keep the filename stable so future projects can reuse it.
|
||||
Notes:
|
||||
%s
|
||||
""".formatted(projectId, type, assetKey, cacheKey, cachePath, projectTarget, blocking, durationSeconds,
|
||||
notes == null ? "" : notes);
|
||||
}
|
||||
|
||||
private String cacheKey(String type, String... values) {
|
||||
try {
|
||||
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||
digest.update(type.getBytes(StandardCharsets.UTF_8));
|
||||
for (String value : values) {
|
||||
if (value != null) {
|
||||
digest.update((byte) 0);
|
||||
digest.update(value.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
return HexFormat.of().formatHex(digest.digest()).substring(0, 16);
|
||||
} catch (NoSuchAlgorithmException ex) {
|
||||
throw new IllegalStateException("Unable to compute asset cache key", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void createDirectory(Path directory) {
|
||||
try {
|
||||
Files.createDirectories(directory);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to create asset generation directory: " + directory, ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void copy(Path source, Path target) throws IOException {
|
||||
if (source.normalize().equals(target.normalize())) {
|
||||
return;
|
||||
}
|
||||
createDirectory(target.getParent());
|
||||
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
|
||||
}
|
||||
|
||||
private void copyIfMissing(Path source, Path target) throws IOException {
|
||||
if (Files.isRegularFile(source) && !Files.exists(target)) {
|
||||
copy(source, target);
|
||||
}
|
||||
}
|
||||
|
||||
private void writeJson(Path path, Object value) {
|
||||
try {
|
||||
createDirectory(path.getParent());
|
||||
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), value);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to write asset generation JSON: " + path, ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,160 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class LocalCvVisualAnalysisProvider implements VisualAnalysisProvider {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(LocalCvVisualAnalysisProvider.class);
|
||||
|
||||
private final VideoClippingProperties.Editing.VisualAnalysis properties;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final HttpExecutor httpExecutor;
|
||||
|
||||
@Autowired
|
||||
public LocalCvVisualAnalysisProvider(VideoClippingProperties properties, ObjectMapper objectMapper) {
|
||||
this(properties, objectMapper, new JdkHttpExecutor());
|
||||
}
|
||||
|
||||
LocalCvVisualAnalysisProvider(
|
||||
VideoClippingProperties properties,
|
||||
ObjectMapper objectMapper,
|
||||
HttpExecutor httpExecutor
|
||||
) {
|
||||
this.properties = properties.getEditing().getVisualAnalysis();
|
||||
this.objectMapper = objectMapper;
|
||||
this.httpExecutor = httpExecutor;
|
||||
}
|
||||
|
||||
@Override
|
||||
public SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
|
||||
LocalCvRequest request = new LocalCvRequest(source, List.copyOf(thumbnails), List.copyOf(shotSegments));
|
||||
long startedAt = System.nanoTime();
|
||||
try {
|
||||
String body = objectMapper.writeValueAsString(request);
|
||||
log.info("event=local_cv_request_started clip_id={} endpoint={} timeout_ms={} thumbnails={} "
|
||||
+ "shot_segments={} request_bytes={}",
|
||||
source.clipId(), properties.getEndpoint(), properties.getTimeoutMs(), thumbnails.size(),
|
||||
shotSegments.size(), body.getBytes(StandardCharsets.UTF_8).length);
|
||||
HttpCall call = new HttpCall(properties.getEndpoint(), properties.getTimeoutMs(), body);
|
||||
HttpResult result = httpExecutor.execute(call);
|
||||
log.info("event=local_cv_response_received clip_id={} endpoint={} status_code={} elapsed_ms={} "
|
||||
+ "response_bytes={}",
|
||||
source.clipId(), properties.getEndpoint(), result.statusCode(), elapsedMillis(startedAt),
|
||||
result.body() == null ? 0 : result.body().getBytes(StandardCharsets.UTF_8).length);
|
||||
if (result.statusCode() < 200 || result.statusCode() >= 300) {
|
||||
throw new IllegalStateException("Local CV service returned HTTP " + result.statusCode()
|
||||
+ " body=" + preview(result.body()));
|
||||
}
|
||||
SourceVisualAnalysis analysis = objectMapper.readValue(result.body(), SourceVisualAnalysis.class);
|
||||
SourceVisualAnalysis normalized = normalized(source, thumbnails, analysis);
|
||||
log.info("event=local_cv_analysis_completed clip_id={} endpoint={} elapsed_ms={} method={} "
|
||||
+ "object_labels={} representative_thumbnails={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
normalized.analysisMethod(), normalized.objectLabels().size(),
|
||||
normalized.representativeThumbnails().size());
|
||||
return normalized;
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
log.warn("event=local_cv_request_interrupted clip_id={} endpoint={} elapsed_ms={} message={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt), ex.getMessage());
|
||||
throw new IllegalStateException("Interrupted while calling local CV visual analysis service", ex);
|
||||
} catch (IOException ex) {
|
||||
log.warn("event=local_cv_request_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
ex.getClass().getSimpleName(), ex.getMessage());
|
||||
throw new IllegalStateException("Unable to call local CV visual analysis service", ex);
|
||||
} catch (RuntimeException ex) {
|
||||
log.warn("event=local_cv_analysis_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
ex.getClass().getSimpleName(), ex.getMessage());
|
||||
throw ex;
|
||||
}
|
||||
}
|
||||
|
||||
private SourceVisualAnalysis normalized(
|
||||
ClipAnalysis source,
|
||||
List<String> thumbnails,
|
||||
SourceVisualAnalysis analysis
|
||||
) {
|
||||
return new SourceVisualAnalysis(
|
||||
valueOrDefault(analysis.clipId(), source.clipId()),
|
||||
clamp(analysis.blurScore()),
|
||||
clamp(analysis.exposureScore()),
|
||||
clamp(analysis.motionScore()),
|
||||
clamp(analysis.compositionScore()),
|
||||
valueOrDefault(analysis.facePresence(), "unknown"),
|
||||
analysis.objectLabels() == null ? List.of() : List.copyOf(analysis.objectLabels()),
|
||||
analysis.representativeThumbnails() == null || analysis.representativeThumbnails().isEmpty()
|
||||
? List.copyOf(thumbnails)
|
||||
: List.copyOf(analysis.representativeThumbnails()),
|
||||
valueOrDefault(analysis.analysisMethod(), "local_cv")
|
||||
);
|
||||
}
|
||||
|
||||
private String valueOrDefault(String value, String fallback) {
|
||||
return value == null || value.isBlank() ? fallback : value;
|
||||
}
|
||||
|
||||
private double clamp(double value) {
|
||||
return Math.max(0, Math.min(1, Math.round(value * 1000.0) / 1000.0));
|
||||
}
|
||||
|
||||
private long elapsedMillis(long startedAt) {
|
||||
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||
}
|
||||
|
||||
private String preview(String body) {
|
||||
if (body == null || body.isBlank()) {
|
||||
return "";
|
||||
}
|
||||
String compact = body.replaceAll("\\s+", " ").trim();
|
||||
return compact.length() <= 300 ? compact : compact.substring(0, 300) + "...";
|
||||
}
|
||||
|
||||
record LocalCvRequest(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
|
||||
}
|
||||
|
||||
record HttpCall(String endpoint, long timeoutMs, String body) {
|
||||
}
|
||||
|
||||
record HttpResult(int statusCode, String body) {
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface HttpExecutor {
|
||||
HttpResult execute(HttpCall call) throws IOException, InterruptedException;
|
||||
}
|
||||
|
||||
private static class JdkHttpExecutor implements HttpExecutor {
|
||||
private final HttpClient httpClient = HttpClient.newHttpClient();
|
||||
|
||||
@Override
|
||||
public HttpResult execute(HttpCall call) throws IOException, InterruptedException {
|
||||
HttpRequest request = HttpRequest.newBuilder()
|
||||
.uri(URI.create(call.endpoint()))
|
||||
.timeout(Duration.ofMillis(call.timeoutMs()))
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString(call.body()))
|
||||
.build();
|
||||
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
|
||||
return new HttpResult(response.statusCode(), response.body());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,326 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.SmartLifecycle;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class LocalCvWorkerProcessManager implements SmartLifecycle {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(LocalCvWorkerProcessManager.class);
|
||||
|
||||
private final VideoClippingProperties.Editing.VisualAnalysis visualAnalysis;
|
||||
private final ProcessLauncher processLauncher;
|
||||
private final HealthChecker healthChecker;
|
||||
private Process process;
|
||||
private Thread outputThread;
|
||||
private Thread watcherThread;
|
||||
private volatile boolean running;
|
||||
private volatile boolean stopping;
|
||||
|
||||
@Autowired
|
||||
public LocalCvWorkerProcessManager(VideoClippingProperties properties) {
|
||||
this(properties, ProcessBuilder::start, new HttpHealthChecker());
|
||||
}
|
||||
|
||||
LocalCvWorkerProcessManager(
|
||||
VideoClippingProperties properties,
|
||||
ProcessLauncher processLauncher
|
||||
) {
|
||||
this(properties, processLauncher, new HttpHealthChecker());
|
||||
}
|
||||
|
||||
LocalCvWorkerProcessManager(
|
||||
VideoClippingProperties properties,
|
||||
ProcessLauncher processLauncher,
|
||||
HealthChecker healthChecker
|
||||
) {
|
||||
this.visualAnalysis = properties.getEditing().getVisualAnalysis();
|
||||
this.processLauncher = processLauncher;
|
||||
this.healthChecker = healthChecker;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void start() {
|
||||
if (running) {
|
||||
log.info("event=local_cv_worker_start_skipped reason=already_running endpoint={}",
|
||||
visualAnalysis.getEndpoint());
|
||||
return;
|
||||
}
|
||||
if (!shouldStart()) {
|
||||
log.info("event=local_cv_worker_start_skipped reason=disabled provider={} auto_start={} endpoint={}",
|
||||
visualAnalysis.getProvider(), visualAnalysis.getLocalCvWorker().isAutoStart(),
|
||||
visualAnalysis.getEndpoint());
|
||||
return;
|
||||
}
|
||||
|
||||
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker = visualAnalysis.getLocalCvWorker();
|
||||
Path script = Path.of(worker.getScript()).toAbsolutePath().normalize();
|
||||
URI healthUri = healthUri(worker);
|
||||
log.info("event=local_cv_worker_starting script={} endpoint={} health_uri={} startup_wait_ms={} "
|
||||
+ "health_check_interval_ms={}",
|
||||
script, visualAnalysis.getEndpoint(), healthUri, worker.getStartupWaitMs(),
|
||||
worker.getHealthCheckIntervalMs());
|
||||
if (!Files.isRegularFile(script)) {
|
||||
throw new IllegalStateException("Local CV worker script does not exist: " + script);
|
||||
}
|
||||
|
||||
Endpoint endpoint = endpoint();
|
||||
ProcessBuilder processBuilder = new ProcessBuilder(script.toString())
|
||||
.redirectErrorStream(true);
|
||||
Map<String, String> environment = processBuilder.environment();
|
||||
environment.put("LOCAL_CV_HOST", endpoint.host());
|
||||
environment.put("LOCAL_CV_PORT", Integer.toString(endpoint.port()));
|
||||
|
||||
try {
|
||||
process = processLauncher.start(processBuilder);
|
||||
running = true;
|
||||
stopping = false;
|
||||
outputThread = outputReader(process.getInputStream(), script);
|
||||
outputThread.start();
|
||||
watcherThread = processWatcher(process, script);
|
||||
watcherThread.start();
|
||||
log.info("event=local_cv_worker_started script={} pid={} host={} port={} endpoint={}",
|
||||
script, process.pid(), endpoint.host(), endpoint.port(), visualAnalysis.getEndpoint());
|
||||
waitUntilHealthy(worker, healthUri);
|
||||
log.info("event=local_cv_worker_ready script={} pid={} health_uri={} alive={}",
|
||||
script, process.pid(), healthUri, process.isAlive());
|
||||
} catch (IOException ex) {
|
||||
running = false;
|
||||
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
|
||||
script, ex.getClass().getSimpleName(), ex.getMessage());
|
||||
throw new IllegalStateException("Unable to start local CV worker script: " + script, ex);
|
||||
} catch (RuntimeException ex) {
|
||||
cleanupFailedStart();
|
||||
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
|
||||
script, ex.getClass().getSimpleName(), ex.getMessage());
|
||||
throw ex;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void stop() {
|
||||
if (process == null) {
|
||||
running = false;
|
||||
return;
|
||||
}
|
||||
long pid = process.pid();
|
||||
if (process.isAlive()) {
|
||||
stopping = true;
|
||||
log.info("event=local_cv_worker_stop_requested pid={}", pid);
|
||||
process.destroy();
|
||||
try {
|
||||
if (!process.waitFor(Duration.ofSeconds(5).toMillis(), TimeUnit.MILLISECONDS)) {
|
||||
log.warn("event=local_cv_worker_stop_timeout pid={} action=destroy_forcibly", pid);
|
||||
process.destroyForcibly();
|
||||
}
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
process.destroyForcibly();
|
||||
}
|
||||
}
|
||||
running = false;
|
||||
log.info("event=local_cv_worker_stopped pid={}", pid);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isRunning() {
|
||||
return running && process != null && process.isAlive();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isAutoStartup() {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getPhase() {
|
||||
return Integer.MIN_VALUE + 100;
|
||||
}
|
||||
|
||||
private boolean shouldStart() {
|
||||
return "local-cv".equalsIgnoreCase(visualAnalysis.getProvider())
|
||||
&& visualAnalysis.getLocalCvWorker().isAutoStart();
|
||||
}
|
||||
|
||||
private void waitUntilHealthy(
|
||||
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker,
|
||||
URI healthUri
|
||||
) {
|
||||
long startupWaitMs = worker.getStartupWaitMs();
|
||||
if (startupWaitMs <= 0) {
|
||||
log.info("event=local_cv_worker_health_wait_skipped reason=disabled health_uri={}", healthUri);
|
||||
return;
|
||||
}
|
||||
long startedAt = System.nanoTime();
|
||||
long deadline = System.currentTimeMillis() + startupWaitMs;
|
||||
long intervalMs = Math.max(1, worker.getHealthCheckIntervalMs());
|
||||
try {
|
||||
while (System.currentTimeMillis() <= deadline) {
|
||||
if (process != null && !process.isAlive()) {
|
||||
throw new IllegalStateException("Local CV worker exited before becoming healthy");
|
||||
}
|
||||
HealthResult result = healthChecker.check(healthUri, Math.min(intervalMs, 5000));
|
||||
log.info("event=local_cv_worker_health_check health_uri={} healthy={} status_code={} "
|
||||
+ "elapsed_ms={} message={}",
|
||||
healthUri, result.healthy(), result.statusCode(), elapsedMillis(startedAt),
|
||||
result.message());
|
||||
if (result.healthy()) {
|
||||
log.info("event=local_cv_worker_health_ready health_uri={} elapsed_ms={}",
|
||||
healthUri, elapsedMillis(startedAt));
|
||||
return;
|
||||
}
|
||||
Thread.sleep(intervalMs);
|
||||
}
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while waiting for local CV worker health", ex);
|
||||
}
|
||||
throw new IllegalStateException("Local CV worker did not become healthy within "
|
||||
+ startupWaitMs + "ms at " + healthUri);
|
||||
}
|
||||
|
||||
private void cleanupFailedStart() {
|
||||
running = false;
|
||||
if (process != null && process.isAlive()) {
|
||||
log.warn("event=local_cv_worker_failed_start_cleanup pid={}", process.pid());
|
||||
process.destroyForcibly();
|
||||
}
|
||||
}
|
||||
|
||||
private Endpoint endpoint() {
|
||||
URI uri = URI.create(visualAnalysis.getEndpoint());
|
||||
String host = uri.getHost() == null || uri.getHost().isBlank() ? "127.0.0.1" : uri.getHost();
|
||||
int port = uri.getPort();
|
||||
if (port < 0) {
|
||||
port = "https".equalsIgnoreCase(uri.getScheme()) ? 443 : 80;
|
||||
}
|
||||
return new Endpoint(host, port);
|
||||
}
|
||||
|
||||
private URI healthUri(VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker) {
|
||||
URI endpointUri = URI.create(visualAnalysis.getEndpoint());
|
||||
String scheme = endpointUri.getScheme() == null || endpointUri.getScheme().isBlank()
|
||||
? "http"
|
||||
: endpointUri.getScheme();
|
||||
String host = endpointUri.getHost() == null || endpointUri.getHost().isBlank()
|
||||
? "127.0.0.1"
|
||||
: endpointUri.getHost();
|
||||
int port = endpointUri.getPort();
|
||||
String healthPath = worker.getHealthPath() == null || worker.getHealthPath().isBlank()
|
||||
? "/health"
|
||||
: worker.getHealthPath();
|
||||
if (!healthPath.startsWith("/")) {
|
||||
healthPath = "/" + healthPath;
|
||||
}
|
||||
return URI.create("%s://%s%s%s".formatted(scheme, host, port < 0 ? "" : ":" + port, healthPath));
|
||||
}
|
||||
|
||||
private long elapsedMillis(long startedAt) {
|
||||
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||
}
|
||||
|
||||
private Thread outputReader(InputStream inputStream, Path script) {
|
||||
Thread thread = new Thread(() -> {
|
||||
try (BufferedReader reader = new BufferedReader(
|
||||
new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
|
||||
String line;
|
||||
while ((line = reader.readLine()) != null) {
|
||||
log.info("event=local_cv_worker_output script={} message={}", script, line);
|
||||
}
|
||||
} catch (IOException ex) {
|
||||
log.debug("event=local_cv_worker_output_closed script={} error_type={}",
|
||||
script, ex.getClass().getSimpleName());
|
||||
}
|
||||
}, "local-cv-worker-output");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
}
|
||||
|
||||
private Thread processWatcher(Process watchedProcess, Path script) {
|
||||
Thread thread = new Thread(() -> {
|
||||
try {
|
||||
int exitCode = watchedProcess.waitFor();
|
||||
running = false;
|
||||
if (stopping) {
|
||||
log.info("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=stop_requested",
|
||||
script, watchedProcess.pid(), exitCode);
|
||||
stopping = false;
|
||||
} else {
|
||||
log.warn("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=unexpected",
|
||||
script, watchedProcess.pid(), exitCode);
|
||||
}
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
log.debug("event=local_cv_worker_watcher_interrupted script={} pid={}",
|
||||
script, watchedProcess.pid());
|
||||
}
|
||||
}, "local-cv-worker-watcher");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
}
|
||||
|
||||
record Endpoint(String host, int port) {
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface ProcessLauncher {
|
||||
Process start(ProcessBuilder processBuilder) throws IOException;
|
||||
}
|
||||
|
||||
record HealthResult(boolean healthy, int statusCode, String message) {
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface HealthChecker {
|
||||
HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException;
|
||||
}
|
||||
|
||||
private static class HttpHealthChecker implements HealthChecker {
|
||||
private final HttpClient httpClient = HttpClient.newHttpClient();
|
||||
|
||||
@Override
|
||||
public HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException {
|
||||
try {
|
||||
HttpRequest request = HttpRequest.newBuilder()
|
||||
.uri(healthUri)
|
||||
.timeout(Duration.ofMillis(timeoutMs))
|
||||
.GET()
|
||||
.build();
|
||||
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
|
||||
return new HealthResult(response.statusCode() >= 200 && response.statusCode() < 300,
|
||||
response.statusCode(), preview(response.body()));
|
||||
} catch (IOException ex) {
|
||||
return new HealthResult(false, 0, ex.getClass().getSimpleName() + ": " + ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private String preview(String body) {
|
||||
if (body == null || body.isBlank()) {
|
||||
return "";
|
||||
}
|
||||
String compact = body.replaceAll("\\s+", " ").trim();
|
||||
return compact.length() <= 160 ? compact : compact.substring(0, 160) + "...";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -6,6 +6,7 @@ import java.util.List;
|
|||
public record RenderManifest(
|
||||
String projectId,
|
||||
List<String> clipIds,
|
||||
List<String> renderedClipPaths,
|
||||
String outputPath,
|
||||
double durationSeconds,
|
||||
List<List<String>> commands,
|
||||
|
|
@ -14,6 +15,7 @@ public record RenderManifest(
|
|||
) {
|
||||
public RenderManifest {
|
||||
clipIds = clipIds == null ? List.of() : List.copyOf(clipIds);
|
||||
renderedClipPaths = renderedClipPaths == null ? List.of() : List.copyOf(renderedClipPaths);
|
||||
commands = commands == null ? List.of() : List.copyOf(commands);
|
||||
assets = assets == null ? List.of() : List.copyOf(assets);
|
||||
}
|
||||
|
|
@ -26,6 +28,18 @@ public record RenderManifest(
|
|||
List<List<String>> commands,
|
||||
Instant completedAt
|
||||
) {
|
||||
this(projectId, clipIds, outputPath, durationSeconds, commands, List.of(), completedAt);
|
||||
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, List.of(), completedAt);
|
||||
}
|
||||
|
||||
public RenderManifest(
|
||||
String projectId,
|
||||
List<String> clipIds,
|
||||
String outputPath,
|
||||
double durationSeconds,
|
||||
List<List<String>> commands,
|
||||
List<ResolvedEditAsset> assets,
|
||||
Instant completedAt
|
||||
) {
|
||||
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, assets, completedAt);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,138 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class ShotSceneSegmenter {
|
||||
|
||||
private static final Pattern PTS_TIME_PATTERN = Pattern.compile("pts_time:([0-9]+(?:\\.[0-9]+)?)");
|
||||
private static final Pattern SCENE_SCORE_PATTERN = Pattern.compile("lavfi\\.scene_score=([0-9]+(?:\\.[0-9]+)?)");
|
||||
|
||||
private final VideoClippingProperties.Editing properties;
|
||||
private final ProcessExecutor processExecutor;
|
||||
|
||||
@Autowired
|
||||
public ShotSceneSegmenter(VideoClippingProperties properties) {
|
||||
this(properties, ShotSceneSegmenter::execute);
|
||||
}
|
||||
|
||||
ShotSceneSegmenter(VideoClippingProperties properties, ProcessExecutor processExecutor) {
|
||||
this.properties = properties.getEditing();
|
||||
this.processExecutor = processExecutor;
|
||||
}
|
||||
|
||||
public List<ShotSegment> segment(ClipAnalysis source) {
|
||||
List<String> command = List.of(
|
||||
properties.getFfmpegBinary(),
|
||||
"-hide_banner",
|
||||
"-i", source.sourcePath(),
|
||||
"-filter:v", "select='gt(scene," + properties.getSceneDetectionThreshold()
|
||||
+ ")',metadata=print",
|
||||
"-an",
|
||||
"-f", "null",
|
||||
"-"
|
||||
);
|
||||
ProcessResult result;
|
||||
try {
|
||||
result = processExecutor.execute(command);
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while detecting shot segments", ex);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to run FFmpeg for shot segmentation", ex);
|
||||
}
|
||||
if (result.exitCode() != 0) {
|
||||
throw new IllegalStateException("FFmpeg shot segmentation exited with code " + result.exitCode());
|
||||
}
|
||||
return segmentsFromCuts(source.durationSeconds(), parseCuts(result.output()));
|
||||
}
|
||||
|
||||
private List<SceneCut> parseCuts(String output) {
|
||||
List<SceneCut> cuts = new ArrayList<>();
|
||||
for (String line : output.split("\\R")) {
|
||||
Matcher ptsMatcher = PTS_TIME_PATTERN.matcher(line);
|
||||
if (ptsMatcher.find()) {
|
||||
cuts.add(new SceneCut(Double.parseDouble(ptsMatcher.group(1)), 0));
|
||||
continue;
|
||||
}
|
||||
Matcher scoreMatcher = SCENE_SCORE_PATTERN.matcher(line);
|
||||
if (scoreMatcher.find() && !cuts.isEmpty()) {
|
||||
SceneCut last = cuts.remove(cuts.size() - 1);
|
||||
cuts.add(new SceneCut(last.timestampSeconds(), Double.parseDouble(scoreMatcher.group(1))));
|
||||
}
|
||||
}
|
||||
return cuts;
|
||||
}
|
||||
|
||||
private List<ShotSegment> segmentsFromCuts(double durationSeconds, List<SceneCut> cuts) {
|
||||
double duration = Math.max(0, durationSeconds);
|
||||
double minSegmentSeconds = properties.getMinimumSceneDurationSeconds();
|
||||
List<ShotSegment> shots = new ArrayList<>();
|
||||
double start = 0;
|
||||
double score = 0;
|
||||
for (SceneCut cut : cuts.stream().sorted().toList()) {
|
||||
double timestamp = Math.min(duration, Math.max(0, cut.timestampSeconds()));
|
||||
if (timestamp - start < minSegmentSeconds || duration - timestamp < minSegmentSeconds) {
|
||||
continue;
|
||||
}
|
||||
shots.add(segment(shots.size() + 1, start, timestamp, score));
|
||||
start = timestamp;
|
||||
score = cut.sceneScore();
|
||||
}
|
||||
if (duration > start) {
|
||||
shots.add(segment(shots.size() + 1, start, duration, score));
|
||||
}
|
||||
if (shots.isEmpty()) {
|
||||
shots.add(segment(1, 0, duration, 0));
|
||||
}
|
||||
return List.copyOf(shots);
|
||||
}
|
||||
|
||||
private ShotSegment segment(int index, double startSeconds, double endSeconds, double sceneScore) {
|
||||
double duration = Math.max(0, endSeconds - startSeconds);
|
||||
return new ShotSegment(
|
||||
"shot_%04d".formatted(index),
|
||||
round(startSeconds),
|
||||
round(endSeconds),
|
||||
round(duration),
|
||||
round(sceneScore),
|
||||
round(startSeconds + duration / 2.0)
|
||||
);
|
||||
}
|
||||
|
||||
private double round(double value) {
|
||||
return Math.round(value * 1000.0) / 1000.0;
|
||||
}
|
||||
|
||||
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
return new ProcessResult(process.waitFor(), output);
|
||||
}
|
||||
|
||||
private record SceneCut(double timestampSeconds, double sceneScore) implements Comparable<SceneCut> {
|
||||
@Override
|
||||
public int compareTo(SceneCut other) {
|
||||
return Double.compare(timestampSeconds, other.timestampSeconds());
|
||||
}
|
||||
}
|
||||
|
||||
record ProcessResult(int exitCode, String output) {
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface ProcessExecutor {
|
||||
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
public record ShotSegment(
|
||||
String shotId,
|
||||
double startSeconds,
|
||||
double endSeconds,
|
||||
double durationSeconds,
|
||||
double sceneScore,
|
||||
double representativeTimestampSeconds
|
||||
) {
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record SourceAudioAnalysis(
|
||||
String clipId,
|
||||
boolean audioPresent,
|
||||
double meanVolumeDb,
|
||||
double maxVolumeDb,
|
||||
double silenceThresholdDb,
|
||||
double silenceDurationSeconds,
|
||||
List<AudioSection> sections,
|
||||
String transcriptPath
|
||||
) {
|
||||
}
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class SourceAudioAnalyzer {
|
||||
|
||||
private static final Pattern MEAN_VOLUME_PATTERN = Pattern.compile("mean_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
|
||||
private static final Pattern MAX_VOLUME_PATTERN = Pattern.compile("max_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
|
||||
private static final Pattern SILENCE_START_PATTERN = Pattern.compile("silence_start:\\s*([0-9]+(?:\\.[0-9]+)?)");
|
||||
private static final Pattern SILENCE_END_PATTERN = Pattern.compile("silence_end:\\s*([0-9]+(?:\\.[0-9]+)?)");
|
||||
|
||||
private final VideoClippingProperties.Editing properties;
|
||||
private final ProcessExecutor processExecutor;
|
||||
|
||||
@Autowired
|
||||
public SourceAudioAnalyzer(VideoClippingProperties properties) {
|
||||
this(properties, SourceAudioAnalyzer::execute);
|
||||
}
|
||||
|
||||
SourceAudioAnalyzer(VideoClippingProperties properties, ProcessExecutor processExecutor) {
|
||||
this.properties = properties.getEditing();
|
||||
this.processExecutor = processExecutor;
|
||||
}
|
||||
|
||||
public SourceAudioAnalysis analyze(ClipAnalysis source) {
|
||||
if (source.audioCodec() == null || source.audioCodec().isBlank()) {
|
||||
return new SourceAudioAnalysis(source.clipId(), false, 0, 0,
|
||||
properties.getSilenceThresholdDb(), properties.getSilenceMinimumDurationSeconds(),
|
||||
List.of(new AudioSection("audio_section_0001", "missing_audio", 0,
|
||||
round(source.durationSeconds()), round(source.durationSeconds()))),
|
||||
null);
|
||||
}
|
||||
List<String> command = List.of(
|
||||
properties.getFfmpegBinary(),
|
||||
"-hide_banner",
|
||||
"-i", source.sourcePath(),
|
||||
"-af", "silencedetect=n=" + properties.getSilenceThresholdDb() + "dB:d="
|
||||
+ properties.getSilenceMinimumDurationSeconds() + ",volumedetect",
|
||||
"-vn",
|
||||
"-f", "null",
|
||||
"-"
|
||||
);
|
||||
ProcessResult result;
|
||||
try {
|
||||
result = processExecutor.execute(command);
|
||||
} catch (InterruptedException ex) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while analyzing source audio", ex);
|
||||
} catch (IOException ex) {
|
||||
throw new IllegalStateException("Unable to run FFmpeg for source audio analysis", ex);
|
||||
}
|
||||
if (result.exitCode() != 0) {
|
||||
throw new IllegalStateException("FFmpeg source audio analysis exited with code " + result.exitCode());
|
||||
}
|
||||
ParsedAudio parsed = parse(result.output());
|
||||
return new SourceAudioAnalysis(
|
||||
source.clipId(),
|
||||
true,
|
||||
parsed.meanVolumeDb(),
|
||||
parsed.maxVolumeDb(),
|
||||
properties.getSilenceThresholdDb(),
|
||||
properties.getSilenceMinimumDurationSeconds(),
|
||||
sections(source.durationSeconds(), parsed.silences()),
|
||||
null
|
||||
);
|
||||
}
|
||||
|
||||
private ParsedAudio parse(String output) {
|
||||
double meanVolume = 0;
|
||||
double maxVolume = 0;
|
||||
List<SilenceRange> silences = new ArrayList<>();
|
||||
Double pendingSilenceStart = null;
|
||||
for (String line : output.split("\\R")) {
|
||||
Matcher meanMatcher = MEAN_VOLUME_PATTERN.matcher(line);
|
||||
if (meanMatcher.find()) {
|
||||
meanVolume = Double.parseDouble(meanMatcher.group(1));
|
||||
}
|
||||
Matcher maxMatcher = MAX_VOLUME_PATTERN.matcher(line);
|
||||
if (maxMatcher.find()) {
|
||||
maxVolume = Double.parseDouble(maxMatcher.group(1));
|
||||
}
|
||||
Matcher startMatcher = SILENCE_START_PATTERN.matcher(line);
|
||||
if (startMatcher.find()) {
|
||||
pendingSilenceStart = Double.parseDouble(startMatcher.group(1));
|
||||
}
|
||||
Matcher endMatcher = SILENCE_END_PATTERN.matcher(line);
|
||||
if (endMatcher.find() && pendingSilenceStart != null) {
|
||||
silences.add(new SilenceRange(pendingSilenceStart, Double.parseDouble(endMatcher.group(1))));
|
||||
pendingSilenceStart = null;
|
||||
}
|
||||
}
|
||||
return new ParsedAudio(round(meanVolume), round(maxVolume), silences);
|
||||
}
|
||||
|
||||
private List<AudioSection> sections(double durationSeconds, List<SilenceRange> silences) {
|
||||
double duration = round(Math.max(0, durationSeconds));
|
||||
List<AudioSection> sections = new ArrayList<>();
|
||||
double cursor = 0;
|
||||
for (SilenceRange silence : silences.stream()
|
||||
.sorted(Comparator.comparingDouble(SilenceRange::startSeconds))
|
||||
.toList()) {
|
||||
double start = clamp(silence.startSeconds(), 0, duration);
|
||||
double end = clamp(silence.endSeconds(), start, duration);
|
||||
if (start > cursor) {
|
||||
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, start));
|
||||
}
|
||||
if (end > start) {
|
||||
sections.add(section(sections.size() + 1, "silence", start, end));
|
||||
}
|
||||
cursor = Math.max(cursor, end);
|
||||
}
|
||||
if (cursor < duration) {
|
||||
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, duration));
|
||||
}
|
||||
if (sections.isEmpty()) {
|
||||
sections.add(section(1, "unclassified_audio", 0, duration));
|
||||
}
|
||||
return List.copyOf(sections);
|
||||
}
|
||||
|
||||
private AudioSection section(int index, String kind, double start, double end) {
|
||||
double roundedStart = round(start);
|
||||
double roundedEnd = round(end);
|
||||
return new AudioSection(
|
||||
"audio_section_%04d".formatted(index),
|
||||
kind,
|
||||
roundedStart,
|
||||
roundedEnd,
|
||||
round(roundedEnd - roundedStart)
|
||||
);
|
||||
}
|
||||
|
||||
private double clamp(double value, double min, double max) {
|
||||
return Math.min(max, Math.max(min, value));
|
||||
}
|
||||
|
||||
private double round(double value) {
|
||||
return Math.round(value * 1000.0) / 1000.0;
|
||||
}
|
||||
|
||||
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
|
||||
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
|
||||
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
|
||||
return new ProcessResult(process.waitFor(), output);
|
||||
}
|
||||
|
||||
private record ParsedAudio(double meanVolumeDb, double maxVolumeDb, List<SilenceRange> silences) {
|
||||
}
|
||||
|
||||
private record SilenceRange(double startSeconds, double endSeconds) {
|
||||
}
|
||||
|
||||
record ProcessResult(int exitCode, String output) {
|
||||
}
|
||||
|
||||
@FunctionalInterface
|
||||
interface ProcessExecutor {
|
||||
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public record SourceVisualAnalysis(
|
||||
String clipId,
|
||||
double blurScore,
|
||||
double exposureScore,
|
||||
double motionScore,
|
||||
double compositionScore,
|
||||
String facePresence,
|
||||
List<VisualObjectLabel> objectLabels,
|
||||
List<String> representativeThumbnails,
|
||||
String analysisMethod
|
||||
) {
|
||||
}
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
@Component
|
||||
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class SourceVisualAnalyzer {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(SourceVisualAnalyzer.class);
|
||||
|
||||
private final VideoClippingProperties.Editing.VisualAnalysis properties;
|
||||
private final VisualAnalysisProvider heuristicProvider;
|
||||
private final VisualAnalysisProvider localCvProvider;
|
||||
|
||||
public SourceVisualAnalyzer() {
|
||||
this(new VideoClippingProperties(), new HeuristicVisualAnalysisProvider(), null);
|
||||
}
|
||||
|
||||
@Autowired
|
||||
public SourceVisualAnalyzer(
|
||||
VideoClippingProperties properties,
|
||||
HeuristicVisualAnalysisProvider heuristicProvider,
|
||||
LocalCvVisualAnalysisProvider localCvProvider
|
||||
) {
|
||||
this(properties, heuristicProvider, (VisualAnalysisProvider) localCvProvider);
|
||||
}
|
||||
|
||||
SourceVisualAnalyzer(
|
||||
VideoClippingProperties properties,
|
||||
VisualAnalysisProvider heuristicProvider,
|
||||
VisualAnalysisProvider localCvProvider
|
||||
) {
|
||||
this.properties = properties.getEditing().getVisualAnalysis();
|
||||
this.heuristicProvider = heuristicProvider;
|
||||
this.localCvProvider = localCvProvider;
|
||||
}
|
||||
|
||||
public SourceVisualAnalysis analyze(
|
||||
ClipAnalysis source,
|
||||
List<String> thumbnails,
|
||||
List<ShotSegment> shotSegments
|
||||
) {
|
||||
log.info("event=visual_analysis_selected clip_id={} provider={} thumbnails={} shot_segments={} "
|
||||
+ "fallback_to_heuristic={}",
|
||||
source.clipId(), properties.getProvider(), thumbnails.size(), shotSegments.size(),
|
||||
properties.isFallbackToHeuristic());
|
||||
if ("local-cv".equalsIgnoreCase(properties.getProvider())) {
|
||||
return localCvAnalysis(source, thumbnails, shotSegments);
|
||||
}
|
||||
long startedAt = System.nanoTime();
|
||||
SourceVisualAnalysis analysis = heuristicProvider.analyze(source, thumbnails, shotSegments);
|
||||
log.info("event=heuristic_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} method={}",
|
||||
source.clipId(), elapsedMillis(startedAt), analysis.objectLabels().size(),
|
||||
analysis.analysisMethod());
|
||||
return analysis;
|
||||
}
|
||||
|
||||
private SourceVisualAnalysis localCvAnalysis(
|
||||
ClipAnalysis source,
|
||||
List<String> thumbnails,
|
||||
List<ShotSegment> shotSegments
|
||||
) {
|
||||
long startedAt = System.nanoTime();
|
||||
try {
|
||||
SourceVisualAnalysis analysis = localCvProvider.analyze(source, thumbnails, shotSegments);
|
||||
log.info("event=local_cv_visual_analysis_completed clip_id={} endpoint={} elapsed_ms={} "
|
||||
+ "object_labels={} method={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
analysis.objectLabels().size(), analysis.analysisMethod());
|
||||
return analysis;
|
||||
} catch (RuntimeException ex) {
|
||||
if (!properties.isFallbackToHeuristic()) {
|
||||
log.error("event=local_cv_visual_analysis_failed clip_id={} endpoint={} elapsed_ms={} "
|
||||
+ "fallback=false error_type={} message={} cause_type={} cause_message={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
|
||||
throw ex;
|
||||
}
|
||||
log.warn("event=local_cv_visual_analysis_fallback clip_id={} endpoint={} elapsed_ms={} "
|
||||
+ "error_type={} message={} cause_type={} cause_message={}",
|
||||
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
|
||||
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
|
||||
long fallbackStartedAt = System.nanoTime();
|
||||
SourceVisualAnalysis fallback = heuristicProvider.analyze(source, thumbnails, shotSegments);
|
||||
SourceVisualAnalysis fallbackAnalysis = new SourceVisualAnalysis(
|
||||
fallback.clipId(),
|
||||
fallback.blurScore(),
|
||||
fallback.exposureScore(),
|
||||
fallback.motionScore(),
|
||||
fallback.compositionScore(),
|
||||
fallback.facePresence(),
|
||||
fallback.objectLabels(),
|
||||
fallback.representativeThumbnails(),
|
||||
"local_cv_failed_fallback_" + fallback.analysisMethod()
|
||||
);
|
||||
log.info("event=local_cv_fallback_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} "
|
||||
+ "method={}",
|
||||
source.clipId(), elapsedMillis(fallbackStartedAt), fallbackAnalysis.objectLabels().size(),
|
||||
fallbackAnalysis.analysisMethod());
|
||||
return fallbackAnalysis;
|
||||
}
|
||||
}
|
||||
|
||||
private long elapsedMillis(long startedAt) {
|
||||
return (System.nanoTime() - startedAt) / 1_000_000;
|
||||
}
|
||||
|
||||
private String causeType(RuntimeException ex) {
|
||||
return ex.getCause() == null ? null : ex.getCause().getClass().getSimpleName();
|
||||
}
|
||||
|
||||
private String causeMessage(RuntimeException ex) {
|
||||
return ex.getCause() == null ? null : ex.getCause().getMessage();
|
||||
}
|
||||
}
|
||||
|
|
@ -65,6 +65,17 @@ public class StoryboardPromptGenerator {
|
|||
|
||||
Create one cinematic edit plan for project `%s`.
|
||||
|
||||
The service can only render what the plan expresses, so your plan must use the actual cinematic toolkit
|
||||
available to the renderer:
|
||||
- shot selection and timing
|
||||
- cuts, crossfades, fade-ins, and fade-outs
|
||||
- dynamic punch-in crop and subtle cinematic grade
|
||||
- music bed selection and timing
|
||||
- SFX placement and timing
|
||||
- voiceover scripting and timing
|
||||
- short text overlays
|
||||
- per-segment rendered clips plus the final assembled video
|
||||
|
||||
Constraints:
|
||||
- Style: `%s`
|
||||
- Target duration: %d seconds
|
||||
|
|
@ -77,6 +88,7 @@ public class StoryboardPromptGenerator {
|
|||
- Timeline ranges must be chronological and must not overlap.
|
||||
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
|
||||
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
|
||||
- Make the edit cinematic by directing those elements deliberately, not by trimming footage alone.
|
||||
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
|
||||
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
|
||||
- Return strict JSON only. Do not use Markdown fences or add commentary.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public interface VisualAnalysisProvider {
|
||||
|
||||
SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments);
|
||||
}
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
public record VisualObjectLabel(
|
||||
String label,
|
||||
double confidence,
|
||||
String source
|
||||
) {
|
||||
}
|
||||
|
|
@ -5,7 +5,7 @@ video-clipping:
|
|||
enabled: true
|
||||
local-director:
|
||||
enabled: true
|
||||
auto-render-when-plan-appears: false
|
||||
auto-render-when-plan-appears: true
|
||||
require-approval-before-render: true
|
||||
approval-file-name: approved.flag
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,11 @@ video-clipping:
|
|||
contact-sheet-columns: ${VIDEO_EDITING_CONTACT_SHEET_COLUMNS:5}
|
||||
proxy-enabled: ${VIDEO_EDITING_PROXY_ENABLED:true}
|
||||
proxy-width: ${VIDEO_EDITING_PROXY_WIDTH:640}
|
||||
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:60}
|
||||
scene-detection-threshold: ${VIDEO_EDITING_SCENE_DETECTION_THRESHOLD:0.35}
|
||||
minimum-scene-duration-seconds: ${VIDEO_EDITING_MINIMUM_SCENE_DURATION_SECONDS:1.0}
|
||||
silence-threshold-db: ${VIDEO_EDITING_SILENCE_THRESHOLD_DB:-35.0}
|
||||
silence-minimum-duration-seconds: ${VIDEO_EDITING_SILENCE_MINIMUM_DURATION_SECONDS:0.5}
|
||||
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:600}
|
||||
output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920}
|
||||
output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080}
|
||||
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30}
|
||||
|
|
@ -44,6 +48,17 @@ video-clipping:
|
|||
fonts-folder: ${VIDEO_EDITING_ASSETS_FONTS_FOLDER:./input/highlights/assets/fonts}
|
||||
luts-folder: ${VIDEO_EDITING_ASSETS_LUTS_FOLDER:./input/highlights/assets/luts}
|
||||
voiceover-folder: ${VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER:./output/highlight-projects/_voiceover-cache}
|
||||
visual-analysis:
|
||||
provider: ${VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER:local-cv} #local-cv or heuristic
|
||||
endpoint: ${VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT:http://127.0.0.1:8091/v1/analyze-visuals}
|
||||
timeout-ms: ${VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS:30000}
|
||||
fallback-to-heuristic: ${VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC:true}
|
||||
local-cv-worker:
|
||||
auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true}
|
||||
script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh}
|
||||
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:120000}
|
||||
health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health}
|
||||
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
|
||||
local-director:
|
||||
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
|
||||
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}
|
||||
|
|
@ -63,6 +78,12 @@ video-clipping:
|
|||
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
|
||||
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
|
||||
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
|
||||
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:true}
|
||||
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:false}
|
||||
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
|
||||
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
|
||||
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
|
||||
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35}
|
||||
|
||||
logging:
|
||||
level:
|
||||
|
|
|
|||
|
|
@ -27,7 +27,11 @@ class VideoClippingPropertiesTest {
|
|||
assertThat(properties.getEditing().getContactSheetColumns()).isEqualTo(5);
|
||||
assertThat(properties.getEditing().isProxyEnabled()).isTrue();
|
||||
assertThat(properties.getEditing().getProxyWidth()).isEqualTo(640);
|
||||
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(60);
|
||||
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.35);
|
||||
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(1.0);
|
||||
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-35.0);
|
||||
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(0.5);
|
||||
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(600);
|
||||
assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920);
|
||||
assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080);
|
||||
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30);
|
||||
|
|
@ -50,6 +54,19 @@ class VideoClippingPropertiesTest {
|
|||
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("./input/highlights/assets/luts");
|
||||
assertThat(properties.getEditing().getAssets().getVoiceoverFolder())
|
||||
.isEqualTo("./output/highlight-projects/_voiceover-cache");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("heuristic");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
|
||||
.isEqualTo("http://127.0.0.1:8091/v1/analyze-visuals");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(30000);
|
||||
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isTrue();
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isFalse();
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
|
||||
.isEqualTo("./tools/run_local_cv_worker.sh");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs()).isZero();
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
|
||||
.isEqualTo("/health");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
|
||||
.isEqualTo(1000);
|
||||
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
|
||||
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
|
||||
.isEqualTo("./input/editing/source");
|
||||
|
|
@ -71,6 +88,10 @@ class VideoClippingPropertiesTest {
|
|||
"video-clipping.editing.project-directory=/tmp/edit-projects",
|
||||
"video-clipping.editing.highlight-project-directory=/tmp/highlight-projects",
|
||||
"video-clipping.editing.thumbnail-count-per-clip=7",
|
||||
"video-clipping.editing.scene-detection-threshold=0.42",
|
||||
"video-clipping.editing.minimum-scene-duration-seconds=2.5",
|
||||
"video-clipping.editing.silence-threshold-db=-40",
|
||||
"video-clipping.editing.silence-minimum-duration-seconds=1.25",
|
||||
"video-clipping.editing.voiceover-provider=remote",
|
||||
"video-clipping.editing.loudness-target-i=-14",
|
||||
"video-clipping.editing.loudness-true-peak=-2",
|
||||
|
|
@ -84,6 +105,15 @@ class VideoClippingPropertiesTest {
|
|||
"video-clipping.editing.assets.fonts-folder=/tmp/fonts",
|
||||
"video-clipping.editing.assets.luts-folder=/tmp/luts",
|
||||
"video-clipping.editing.assets.voiceover-folder=/tmp/voiceover",
|
||||
"video-clipping.editing.visual-analysis.provider=local-cv",
|
||||
"video-clipping.editing.visual-analysis.endpoint=http://localhost:9000/analyze",
|
||||
"video-clipping.editing.visual-analysis.timeout-ms=12000",
|
||||
"video-clipping.editing.visual-analysis.fallback-to-heuristic=false",
|
||||
"video-clipping.editing.visual-analysis.local-cv-worker.auto-start=true",
|
||||
"video-clipping.editing.visual-analysis.local-cv-worker.script=/tmp/local-cv.sh",
|
||||
"video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250",
|
||||
"video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready",
|
||||
"video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25",
|
||||
"video-clipping.editing.local-director.enabled=false",
|
||||
"video-clipping.editing.local-director.source-directory=/tmp/source",
|
||||
"video-clipping.editing.local-director.poll-interval-ms=9000",
|
||||
|
|
@ -101,6 +131,10 @@ class VideoClippingPropertiesTest {
|
|||
assertThat(properties.getEditing().getProjectDirectory()).isEqualTo("/tmp/edit-projects");
|
||||
assertThat(properties.getEditing().getHighlightProjectDirectory()).isEqualTo("/tmp/highlight-projects");
|
||||
assertThat(properties.getEditing().getThumbnailCountPerClip()).isEqualTo(7);
|
||||
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.42);
|
||||
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(2.5);
|
||||
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-40.0);
|
||||
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(1.25);
|
||||
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("remote");
|
||||
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-14.0);
|
||||
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-2.0);
|
||||
|
|
@ -114,6 +148,20 @@ class VideoClippingPropertiesTest {
|
|||
assertThat(properties.getEditing().getAssets().getFontsFolder()).isEqualTo("/tmp/fonts");
|
||||
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("/tmp/luts");
|
||||
assertThat(properties.getEditing().getAssets().getVoiceoverFolder()).isEqualTo("/tmp/voiceover");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("local-cv");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
|
||||
.isEqualTo("http://localhost:9000/analyze");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(12000);
|
||||
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isFalse();
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isTrue();
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
|
||||
.isEqualTo("/tmp/local-cv.sh");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs())
|
||||
.isEqualTo(250);
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
|
||||
.isEqualTo("/ready");
|
||||
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
|
||||
.isEqualTo(25);
|
||||
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
|
||||
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
|
||||
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);
|
||||
|
|
|
|||
|
|
@ -62,7 +62,14 @@ class CinematicEditingIntegrationTest {
|
|||
Path output = projectDirectory.resolve("final.mp4");
|
||||
assertThat(output).isRegularFile();
|
||||
assertThat(probeDuration(output)).isGreaterThan(0);
|
||||
assertThat(projectDirectory.resolve("rendered-clips/clip_0001.mp4")).isRegularFile();
|
||||
assertThat(projectDirectory.resolve("rendered-clips/clip_0002.mp4")).isRegularFile();
|
||||
assertThat(projectDirectory.resolve("rendered-clips/clip_0003.mp4")).isRegularFile();
|
||||
assertThat(projectDirectory.resolve("render-manifest.json")).exists();
|
||||
RenderManifest manifest = store.readJson(projectId, "render-manifest.json", RenderManifest.class);
|
||||
assertThat(manifest.clipIds()).containsExactly("clip_01", "clip_02", "clip_03");
|
||||
assertThat(manifest.renderedClipPaths()).hasSize(3)
|
||||
.allSatisfy(path -> assertThat(Path.of(path)).isRegularFile());
|
||||
assertThat(projectDirectory.resolve("qa-report.json")).exists();
|
||||
assertThat(projects.getProject(projectId).status()).isEqualTo(EditProjectStatus.RENDERED);
|
||||
}
|
||||
|
|
@ -128,7 +135,8 @@ class CinematicEditingIntegrationTest {
|
|||
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
|
||||
"cinematic grade", "integration sequence"))
|
||||
.toList();
|
||||
return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(),
|
||||
return new EditPlan(projectId, "category-aware-cinematic-highlights",
|
||||
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(),
|
||||
"mp4-h264-aac-320p", "Generated integration edit");
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
|||
class CinematicEditingLocalProfileTest {
|
||||
|
||||
@Test
|
||||
void enablesLocalDirectorAndDisablesEightSecondFolderScheduler() {
|
||||
void enablesLocalDirectorAutoRenderAndDisablesEightSecondFolderScheduler() {
|
||||
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
||||
factory.setResources(new ClassPathResource("application-cinematic-editing-local.yml"));
|
||||
Properties properties = factory.getObject();
|
||||
|
|
@ -21,7 +21,7 @@ class CinematicEditingLocalProfileTest {
|
|||
assertThat(properties.getProperty("video-clipping.editing.enabled")).isEqualTo("true");
|
||||
assertThat(properties.getProperty("video-clipping.editing.local-director.enabled")).isEqualTo("true");
|
||||
assertThat(properties.getProperty("video-clipping.editing.local-director.auto-render-when-plan-appears"))
|
||||
.isEqualTo("false");
|
||||
.isEqualTo("true");
|
||||
assertThat(properties.getProperty("video-clipping.editing.local-director.require-approval-before-render"))
|
||||
.isEqualTo("true");
|
||||
assertThat(properties.getProperty("video-clipping.editing.local-director.approval-file-name"))
|
||||
|
|
|
|||
|
|
@ -130,10 +130,11 @@ class EditPlanInboxScannerTest {
|
|||
}
|
||||
|
||||
private void writeValidPlan() throws Exception {
|
||||
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 60,
|
||||
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 2.5,
|
||||
List.of(new EditDecision("clip_00001", 0, 2.5, 0, 2.5,
|
||||
"cut", "cut", 1.0, "cinematic", "opening")),
|
||||
List.of(), List.of(), "mp4-h264-aac-1080p", "Hero edit");
|
||||
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 2.5, -14, "opening bed")),
|
||||
List.of(), "mp4-h264-aac-1080p", "Hero edit");
|
||||
objectMapper.writeValue(store.inboxDirectory("porsche-edit").resolve("edit-plan.json").toFile(), plan);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -37,6 +37,41 @@ class EditPlanValidatorTest {
|
|||
|
||||
@Test void acceptsValidPlan() { assertThat(validator.validate("project", plan(decision("clip-1", 0, 4, 0, 4, "cut")))).isNotNull(); }
|
||||
|
||||
@Test void acceptsContentDrivenPlanShorterThanProjectTarget() {
|
||||
EditPlan shorter = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||
List.of(decision("clip-1", 0, 4, 0, 4, "cut")),
|
||||
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 4, -14, "bed")),
|
||||
List.of(),
|
||||
"mp4-h264-aac-1080p", "summary");
|
||||
|
||||
assertThat(validator.validate("project", shorter)).isEqualTo(shorter);
|
||||
}
|
||||
|
||||
@Test void rejectsBareTrimOnlyPlanWithoutCinematicRichness() {
|
||||
EditPlan bare = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "none", "reason")),
|
||||
List.of(), List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
|
||||
|
||||
rejects(bare);
|
||||
}
|
||||
|
||||
@Test void rejectsPlanTargetDurationThatDoesNotMatchFinalTimeline() {
|
||||
EditPlan mismatch = new EditPlan("project", "cinematic-porsche-promo", 8,
|
||||
List.of(decision("clip-1", 0, 4, 0, 4, "cut")), List.of(), List.of(),
|
||||
"mp4-h264-aac-1080p", "summary");
|
||||
|
||||
rejects(mismatch);
|
||||
}
|
||||
|
||||
@Test void rejectsPlanLongerThanProjectMaximumTarget() {
|
||||
EditPlan tooLong = new EditPlan("project", "cinematic-porsche-promo", 12,
|
||||
List.of(decision("clip-1", 0, 8, 0, 8, "cut"),
|
||||
decision("clip-1", 0, 4, 8, 12, "cut")),
|
||||
List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
|
||||
|
||||
rejects(tooLong);
|
||||
}
|
||||
|
||||
@Test void rejectsUnknownClip() { rejects(plan(decision("missing", 0, 4, 0, 4, "cut"))); }
|
||||
|
||||
@Test void rejectsOutOfRangeSource() { rejects(plan(decision("clip-1", 0, 9, 0, 4, "cut"))); }
|
||||
|
|
@ -141,7 +176,11 @@ class EditPlanValidatorTest {
|
|||
}
|
||||
|
||||
private EditPlan plan(EditDecision... decisions) {
|
||||
return new EditPlan("project", "cinematic-porsche-promo", 10, List.of(decisions), List.of(), List.of(),
|
||||
return new EditPlan("project", "cinematic-porsche-promo", decisions[decisions.length - 1].timelineEndSeconds(),
|
||||
List.of(decisions),
|
||||
List.of(new AudioCue("music", "premium-cinematic-drive", 0,
|
||||
decisions[decisions.length - 1].timelineEndSeconds(), -14, "bed")),
|
||||
List.of(),
|
||||
"mp4-h264-aac-1080p", "summary");
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,75 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class HighlightAssetPreparationServiceTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void writesVoiceoverScriptAndRequestsWhenAssetsAreMissing() throws Exception {
|
||||
VideoClippingProperties properties = properties();
|
||||
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||
HighlightProject project = project();
|
||||
store.createProject(project, manifest());
|
||||
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
||||
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
||||
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
|
||||
.thenReturn(List.of());
|
||||
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||
HighlightAssetPreparationService service = new HighlightAssetPreparationService(properties, store,
|
||||
assetProvider, assetLibrary, mapper);
|
||||
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||
"highlight_001", "candidate_001", "Opening Reveal", 0.0, 12.0, 12.0,
|
||||
"opening_hook", "premium cinematic grade", "low cinematic pulse",
|
||||
"whoosh on reveal", List.of("This is a clean first look."), List.of("Pure presence"),
|
||||
"hold the opening beat");
|
||||
|
||||
HighlightAssetPreparationService.HighlightAssetPreparationResult result = service.prepare(project.id(),
|
||||
project, highlight, ContentCategory.GENERIC_VLOG);
|
||||
|
||||
assertThat(result.requestFiles()).isNotEmpty();
|
||||
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
|
||||
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
|
||||
.resolve("assets/requests"))) {
|
||||
assertThat(files.count()).isGreaterThan(0);
|
||||
}
|
||||
}
|
||||
|
||||
private VideoClippingProperties properties() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
|
||||
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||
return properties;
|
||||
}
|
||||
|
||||
private HighlightProject project() {
|
||||
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||
return new HighlightProject("highlight-project-001", "Highlight Project",
|
||||
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
|
||||
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
|
||||
}
|
||||
|
||||
private HighlightProjectManifest manifest() {
|
||||
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class HighlightDirectorFlowServiceTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void importsDirectorPlanWritesHighlightEditPlanAndPublishesFinalVideo() throws Exception {
|
||||
VideoClippingProperties properties = properties();
|
||||
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||
HighlightProject project = project();
|
||||
store.createProject(project, manifest());
|
||||
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
|
||||
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
|
||||
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
|
||||
createDummyFile(source);
|
||||
HighlightVisualEffectsStage visualEffectsStage = mock(HighlightVisualEffectsStage.class);
|
||||
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
|
||||
HighlightAssetPreparationService.HighlightAssetPreparationResult assetResult =
|
||||
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
|
||||
store.highlightsDirectory(project.id()).resolve("highlight_001"),
|
||||
store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("assets"),
|
||||
List.of(), List.of());
|
||||
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(assetResult);
|
||||
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
|
||||
when(assetWorker.process(anyString(), any(), any())).thenReturn(
|
||||
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
|
||||
List.of(), List.of()));
|
||||
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
|
||||
Path renderedClip = store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("final.mp4");
|
||||
createDummyFile(renderedClip);
|
||||
HighlightFfmpegRenderer.HighlightRenderResult renderResult = new HighlightFfmpegRenderer.HighlightRenderResult(
|
||||
"highlight_001", renderedClip, renderedClip, List.of(renderedClip),
|
||||
new RenderManifest(project.id(), List.of("source"), List.of(renderedClip.toString()),
|
||||
renderedClip.toString(), 12.0, List.of(), List.of(), Instant.now()),
|
||||
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
|
||||
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
|
||||
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
|
||||
visualEffectsStage, assetPreparationService, assetWorker, renderer);
|
||||
|
||||
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
|
||||
|
||||
assertThat(result.finalOutputs()).hasSize(1);
|
||||
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
|
||||
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
|
||||
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
|
||||
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||
.isEqualTo(HighlightProjectStatus.RENDERED);
|
||||
}
|
||||
|
||||
private VideoClippingProperties properties() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
return properties;
|
||||
}
|
||||
|
||||
private HighlightProject project() {
|
||||
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||
return new HighlightProject("highlight-project-001", "Highlight Project",
|
||||
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
|
||||
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
|
||||
}
|
||||
|
||||
private HighlightProjectManifest manifest() {
|
||||
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private HighlightDirectorPlan directorPlan() {
|
||||
return new HighlightDirectorPlan("highlight-project-001", "source.mp4", "generic_vlog",
|
||||
List.of(new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening",
|
||||
0.0, 12.0, 12.0, "opening_hook", "premium cinematic grade",
|
||||
"low pulse", "whoosh", List.of("A clean first look."), List.of("Pure presence"),
|
||||
"open with presence")), "summary");
|
||||
}
|
||||
|
||||
private HighlightSourceAnalysis sourceAnalysis() {
|
||||
ClipAnalysis source = new ClipAnalysis("source", "source.mp4", 12.0, "hevc", "aac", 1920, 1080, 24.0,
|
||||
List.of(), null, null, 0.0, 0.0, 0.0);
|
||||
return new HighlightSourceAnalysis("highlight-project-001", "source.mp4", source, List.of(), null, null,
|
||||
null, "analysis/scene-segments.json", List.of(), "analysis/audio-analysis.json",
|
||||
new SourceAudioAnalysis("source", true, -18.0, -3.0, -35.0, 0.5, List.of(), null),
|
||||
"analysis/visual-analysis.json", new SourceVisualAnalysis("source", 0.5, 0.5, 0.5, 0.5,
|
||||
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private void createDummyFile(Path file) throws Exception {
|
||||
Files.createDirectories(file.getParent());
|
||||
Files.writeString(file, "dummy");
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class HighlightDirectorPromptBackfillTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void backfillsMissingHighlightDirectorPromptOnStartup() throws Exception {
|
||||
FileSystemHighlightProjectStore store = store();
|
||||
HighlightProject project = project();
|
||||
store.createProject(project, manifest());
|
||||
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
|
||||
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
|
||||
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
|
||||
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||
"opening_hook", List.of("hero angle", "strong motion"))
|
||||
});
|
||||
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
HighlightDirectorPromptBackfill backfill = new HighlightDirectorPromptBackfill(properties,
|
||||
new HighlightDirectorPromptGenerator(store));
|
||||
|
||||
backfill.backfill();
|
||||
|
||||
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-prompt.md"))).isTrue();
|
||||
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-brief.md"))).isTrue();
|
||||
}
|
||||
|
||||
private FileSystemHighlightProjectStore store() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||
}
|
||||
|
||||
private HighlightProject project() {
|
||||
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
|
||||
HighlightProjectStatus.CREATED, "porsche.mp4",
|
||||
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
|
||||
}
|
||||
|
||||
private HighlightProjectManifest manifest() {
|
||||
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
|
||||
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
|
||||
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
|
||||
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
|
||||
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
|
||||
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
|
||||
null),
|
||||
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
|
||||
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
|
||||
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
|
||||
List.of("car metadata detected", "hero motion present"),
|
||||
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||
"opening_hook", List.of("hero angle", "strong motion"))),
|
||||
Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class HighlightDirectorPromptGeneratorTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void writesHighlightDirectorPromptAndBriefForAnalyzedProject() throws Exception {
|
||||
FileSystemHighlightProjectStore store = store();
|
||||
HighlightProject project = project();
|
||||
store.createProject(project, manifest());
|
||||
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
|
||||
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
|
||||
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
|
||||
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||
"opening_hook", List.of("hero angle", "strong motion"))
|
||||
});
|
||||
|
||||
HighlightDirectorPromptGenerator generator = new HighlightDirectorPromptGenerator(store);
|
||||
HighlightDirectorPromptGenerator.DirectorPromptFiles files = generator.generate(project.id());
|
||||
|
||||
assertThat(Files.readString(Path.of(files.promptPath())))
|
||||
.contains("analysis/source-analysis.json")
|
||||
.contains("analysis/highlight-candidates.json")
|
||||
.contains("director/edit-plan.json")
|
||||
.contains("car_vlog")
|
||||
.contains("candidate_001")
|
||||
.contains("premium, powerful, precise");
|
||||
assertThat(Files.readString(Path.of(files.briefPath())))
|
||||
.contains("Highlight Director Brief")
|
||||
.contains("director-prompt.md")
|
||||
.contains("director/edit-plan.json");
|
||||
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
|
||||
.isEqualTo(HighlightProjectStatus.WAITING_FOR_DIRECTOR);
|
||||
}
|
||||
|
||||
private FileSystemHighlightProjectStore store() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||
}
|
||||
|
||||
private HighlightProject project() {
|
||||
Instant now = Instant.parse("2026-07-11T08:00:00Z");
|
||||
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
|
||||
HighlightProjectStatus.CREATED, "porsche.mp4",
|
||||
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
|
||||
}
|
||||
|
||||
private HighlightProjectManifest manifest() {
|
||||
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
|
||||
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
|
||||
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
|
||||
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
|
||||
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
|
||||
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
|
||||
null),
|
||||
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
|
||||
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
|
||||
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
|
||||
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
|
||||
List.of("car metadata detected", "hero motion present"),
|
||||
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
|
||||
"opening_hook", List.of("hero angle", "strong motion"))),
|
||||
Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class HighlightLocalAssetWorkerTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void resolvesMusicFromLocalLibraryAndKeepsPendingWhenVoiceoverCannotBeSynthesized() throws Exception {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
|
||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
|
||||
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
|
||||
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
|
||||
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
|
||||
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
|
||||
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
|
||||
Files.createDirectories(musicSource.getParent());
|
||||
Files.writeString(musicSource, "music");
|
||||
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
|
||||
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
|
||||
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
|
||||
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.of(
|
||||
new ResolvedEditAsset(EditAssetType.MUSIC, "cinematic-bed", musicSource.toString(),
|
||||
"local", "local", ContentCategory.GENERIC_VLOG, List.of("cinematic", "bed"))));
|
||||
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
|
||||
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
|
||||
.thenReturn(List.of());
|
||||
HighlightAssetPreparationService prep = new HighlightAssetPreparationService(properties, store,
|
||||
assetProvider, assetLibrary, mapper);
|
||||
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
|
||||
"highlight_001", "candidate_001", "Opening", 0, 12, 12, "opening_hook",
|
||||
"premium cinematic grade", "low pulse", "whoosh",
|
||||
List.of("A clean first look."), List.of("Pure presence"), "open with presence");
|
||||
prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG);
|
||||
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary, mapper);
|
||||
|
||||
HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight,
|
||||
ContentCategory.GENERIC_VLOG);
|
||||
|
||||
assertThat(result.resolvedAssets()).isNotEmpty();
|
||||
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
|
||||
.exists();
|
||||
}
|
||||
}
|
||||
|
|
@ -59,13 +59,34 @@ class HighlightSourceAnalyzerTest {
|
|||
.isEqualTo(store.analysisDirectory("porsche").resolve("proxies/porsche.mp4").toString());
|
||||
assertThat(result.waveformPath())
|
||||
.isEqualTo(store.analysisDirectory("porsche").resolve("audio/porsche-waveform.png").toString());
|
||||
assertThat(result.sceneSegmentsPath()).isEqualTo("analysis/scene-segments.json");
|
||||
assertThat(result.shotSegments()).containsExactly(
|
||||
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
|
||||
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
|
||||
);
|
||||
assertThat(result.audioAnalysisPath()).isEqualTo("analysis/audio-analysis.json");
|
||||
assertThat(result.audioAnalysis().sections()).containsExactly(
|
||||
new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)
|
||||
);
|
||||
assertThat(result.visualAnalysisPath()).isEqualTo("analysis/visual-analysis.json");
|
||||
assertThat(result.visualAnalysis().objectLabels()).containsExactly(
|
||||
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")
|
||||
);
|
||||
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-11T08:00:00Z"));
|
||||
|
||||
ClipAnalysis ffprobe = store.readJson("porsche", "analysis/ffprobe.json", ClipAnalysis.class);
|
||||
ShotSegment[] shots = store.readJson("porsche", "analysis/scene-segments.json", ShotSegment[].class);
|
||||
SourceAudioAnalysis audioAnalysis = store.readJson("porsche", "analysis/audio-analysis.json",
|
||||
SourceAudioAnalysis.class);
|
||||
SourceVisualAnalysis visualAnalysis = store.readJson("porsche", "analysis/visual-analysis.json",
|
||||
SourceVisualAnalysis.class);
|
||||
HighlightSourceAnalysis persisted = store.readJson("porsche", "analysis/source-analysis.json",
|
||||
HighlightSourceAnalysis.class);
|
||||
assertThat(ffprobe.sourcePath()).isEqualTo(source.toString());
|
||||
assertThat(ffprobe.thumbnails()).hasSize(2);
|
||||
assertThat(shots).containsExactly(result.shotSegments().toArray(ShotSegment[]::new));
|
||||
assertThat(audioAnalysis).isEqualTo(result.audioAnalysis());
|
||||
assertThat(visualAnalysis).isEqualTo(result.visualAnalysis());
|
||||
assertThat(persisted).isEqualTo(result);
|
||||
}
|
||||
|
||||
|
|
@ -102,7 +123,25 @@ class HighlightSourceAnalyzerTest {
|
|||
when(waveform.generate(any(), any())).thenAnswer(invocation ->
|
||||
invocation.<Path>getArgument(1).resolve("porsche-waveform.png").toString());
|
||||
|
||||
ShotSceneSegmenter shotSceneSegmenter = mock(ShotSceneSegmenter.class);
|
||||
when(shotSceneSegmenter.segment(any())).thenReturn(List.of(
|
||||
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
|
||||
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
|
||||
));
|
||||
|
||||
SourceAudioAnalyzer sourceAudioAnalyzer = mock(SourceAudioAnalyzer.class);
|
||||
when(sourceAudioAnalyzer.analyze(any())).thenReturn(new SourceAudioAnalysis("porsche", true, -18.2,
|
||||
-3.1, -35.0, 0.5,
|
||||
List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)), null));
|
||||
|
||||
SourceVisualAnalyzer sourceVisualAnalyzer = mock(SourceVisualAnalyzer.class);
|
||||
when(sourceVisualAnalyzer.analyze(any(), any(), any())).thenReturn(new SourceVisualAnalysis("porsche",
|
||||
0.6, 0.7, 0.8, 0.75, "unknown_without_face_detector",
|
||||
List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
|
||||
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"));
|
||||
|
||||
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
|
||||
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform, clock);
|
||||
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform,
|
||||
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer, clock);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ class HighlightSourceSchedulerTest {
|
|||
private VideoClippingProperties properties;
|
||||
private FileSystemHighlightProjectStore store;
|
||||
private HighlightSourceAnalyzer analyzer;
|
||||
private HighlightDirectorPromptGenerator promptGenerator;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
|
|
@ -40,6 +41,7 @@ class HighlightSourceSchedulerTest {
|
|||
new HighlightProjectDirectoryInitializer(properties).initialize();
|
||||
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||
analyzer = mock(HighlightSourceAnalyzer.class);
|
||||
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
@ -73,6 +75,7 @@ class HighlightSourceSchedulerTest {
|
|||
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
|
||||
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
|
||||
verify(analyzer).analyze("1");
|
||||
verify(promptGenerator).generate("1");
|
||||
}
|
||||
|
||||
@Test
|
||||
|
|
@ -108,13 +111,20 @@ class HighlightSourceSchedulerTest {
|
|||
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
|
||||
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
|
||||
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
|
||||
return new HighlightSourceScheduler(properties, store, analyzer, clock);
|
||||
return new HighlightSourceScheduler(properties, store, analyzer, promptGenerator, clock);
|
||||
}
|
||||
|
||||
private HighlightSourceAnalysis analysis(String projectId) {
|
||||
ClipAnalysis source = new ClipAnalysis(projectId, store.sourceDirectory(projectId).resolve(projectId + ".mp4")
|
||||
.toString(), 42.0, "h264", "aac", 1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||
return new HighlightSourceAnalysis(projectId, projectId + ".mp4", source, List.of(), null, null, null,
|
||||
"analysis/scene-segments.json", List.of(new ShotSegment("shot_0001", 0, 42, 42, 0, 21)),
|
||||
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
|
||||
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)),
|
||||
null),
|
||||
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.5, 0.5, 0.5,
|
||||
0.7, "unknown_without_face_detector", List.of(new VisualObjectLabel("unknown", 0.2,
|
||||
"metadata_heuristic")), List.of(), "metadata_thumbnail_scene_heuristic"),
|
||||
Instant.parse("2026-07-11T08:00:00Z"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,41 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class HighlightVisualEffectsStageTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void writesExplicitVisualEffectsPlan() throws Exception {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
|
||||
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
|
||||
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
|
||||
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
|
||||
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
|
||||
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
|
||||
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
|
||||
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
|
||||
HighlightVisualEffectsStage stage = new HighlightVisualEffectsStage(store, mapper);
|
||||
|
||||
HighlightVisualEffectsStage.HighlightVisualEffectsPlan plan = stage.create("project-1",
|
||||
new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening", 0, 12, 12,
|
||||
"opening_hook", "premium cinematic grade", "low pulse", "whoosh",
|
||||
List.of("Narration"), List.of("Title card"), "hold the reveal"),
|
||||
ContentCategory.GENERIC_VLOG);
|
||||
|
||||
assertThat(plan.effects()).contains("crop", "punch-in", "contrast", "vignette", "fade");
|
||||
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/visual-effects.json")).exists();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,98 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class LocalAssetGenerationStageTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void reusesExistingSharedAssetsAndWritesRequestsForMissingVoiceover() throws Exception {
|
||||
VideoClippingProperties properties = properties();
|
||||
Files.createDirectories(tempDir.resolve("shared/music"));
|
||||
Files.createDirectories(tempDir.resolve("shared/sfx"));
|
||||
Files.createDirectories(tempDir.resolve("voiceover-cache"));
|
||||
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
|
||||
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
|
||||
|
||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||
|
||||
FileSystemEditProjectStore store = store(properties);
|
||||
store.createProject("project");
|
||||
|
||||
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
|
||||
new LocalEditAssetProvider(properties),
|
||||
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
|
||||
new ObjectMapper().findAndRegisterModules());
|
||||
|
||||
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||
List.of(
|
||||
new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")
|
||||
),
|
||||
List.of(
|
||||
new AudioCue("music", "premium-bed", 0, 4, -14, "drive bed"),
|
||||
new AudioCue("sfx", "whoosh", 1, 2, -6, "impact")
|
||||
),
|
||||
List.of(new VoiceoverLine("Precision in motion.", 0, 2, "confident")),
|
||||
"mp4-h264-aac-1080p",
|
||||
"summary");
|
||||
|
||||
AssetGenerationResult result = stage.prepare("project", plan);
|
||||
|
||||
assertThat(result.readyForRender()).isTrue();
|
||||
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
|
||||
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
|
||||
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
|
||||
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
|
||||
assertThat(tempDir.resolve("projects/project/assets/requests/voiceover")).exists();
|
||||
}
|
||||
|
||||
@Test
|
||||
void writesBlockingRequestsWhenSfxIsMissing() throws Exception {
|
||||
VideoClippingProperties properties = properties();
|
||||
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
|
||||
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
|
||||
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
|
||||
|
||||
FileSystemEditProjectStore store = store(properties);
|
||||
store.createProject("project");
|
||||
|
||||
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
|
||||
new LocalEditAssetProvider(properties),
|
||||
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
|
||||
new ObjectMapper().findAndRegisterModules());
|
||||
|
||||
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
|
||||
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")),
|
||||
List.of(new AudioCue("sfx", "engine-hit", 1, 2, -6, "impact")),
|
||||
List.of(), "mp4-h264-aac-1080p", "summary");
|
||||
|
||||
AssetGenerationResult result = stage.prepare("project", plan);
|
||||
|
||||
assertThat(result.readyForRender()).isFalse();
|
||||
assertThat(tempDir.resolve("projects/project/assets/requests/sfx")).exists();
|
||||
}
|
||||
|
||||
private VideoClippingProperties properties() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
|
||||
return properties;
|
||||
}
|
||||
|
||||
private FileSystemEditProjectStore store(VideoClippingProperties properties) {
|
||||
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,94 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class LocalCvVisualAnalysisProviderTest {
|
||||
|
||||
@Test
|
||||
void postsVisualAnalysisRequestToConfiguredLocalCvEndpoint() {
|
||||
AtomicReference<LocalCvVisualAnalysisProvider.HttpCall> call = new AtomicReference<>();
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9000/analyze");
|
||||
properties.getEditing().getVisualAnalysis().setTimeoutMs(12000);
|
||||
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
|
||||
properties,
|
||||
new ObjectMapper().findAndRegisterModules(),
|
||||
request -> {
|
||||
call.set(request);
|
||||
return new LocalCvVisualAnalysisProvider.HttpResult(200, """
|
||||
{
|
||||
"clipId": "source",
|
||||
"blurScore": 0.81,
|
||||
"exposureScore": 0.67,
|
||||
"motionScore": 0.73,
|
||||
"compositionScore": 0.79,
|
||||
"facePresence": "faces_detected",
|
||||
"objectLabels": [
|
||||
{"label": "car", "confidence": 0.91, "source": "yolo"}
|
||||
],
|
||||
"representativeThumbnails": ["thumb-2.jpg"],
|
||||
"analysisMethod": "local_cv_yolo_clip"
|
||||
}
|
||||
""");
|
||||
}
|
||||
);
|
||||
|
||||
SourceVisualAnalysis analysis = provider.analyze(clip(), List.of("thumb-1.jpg", "thumb-2.jpg"),
|
||||
List.of(new ShotSegment("shot_0001", 0, 8, 8, 0, 4)));
|
||||
|
||||
assertThat(call.get().endpoint()).isEqualTo("http://localhost:9000/analyze");
|
||||
assertThat(call.get().timeoutMs()).isEqualTo(12000);
|
||||
assertThat(call.get().body()).contains("\"clipId\":\"source\"", "\"thumbnails\":[\"thumb-1.jpg\"");
|
||||
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("car", 0.91, "yolo"));
|
||||
assertThat(analysis.facePresence()).isEqualTo("faces_detected");
|
||||
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_yolo_clip");
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsNonSuccessfulLocalCvResponses() {
|
||||
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
|
||||
new VideoClippingProperties(),
|
||||
new ObjectMapper().findAndRegisterModules(),
|
||||
request -> new LocalCvVisualAnalysisProvider.HttpResult(500, "{}")
|
||||
);
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> provider.analyze(clip(), List.of(), List.of()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void reportsLocalCvIoAndInterruptionFailures() {
|
||||
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
|
||||
new VideoClippingProperties(),
|
||||
new ObjectMapper().findAndRegisterModules(),
|
||||
request -> {
|
||||
throw new IOException("connection refused");
|
||||
}).analyze(clip(), List.of(), List.of()));
|
||||
|
||||
try {
|
||||
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
|
||||
new VideoClippingProperties(),
|
||||
new ObjectMapper().findAndRegisterModules(),
|
||||
request -> {
|
||||
throw new InterruptedException("stopped");
|
||||
}).analyze(clip(), List.of(), List.of()));
|
||||
assertTrue(Thread.currentThread().isInterrupted());
|
||||
} finally {
|
||||
Thread.interrupted();
|
||||
}
|
||||
}
|
||||
|
||||
private ClipAnalysis clip() {
|
||||
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", "aac",
|
||||
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,250 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class LocalCvWorkerProcessManagerTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void doesNotStartWhenProviderIsNotLocalCv() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("heuristic");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||
throw new AssertionError("worker should not start");
|
||||
});
|
||||
|
||||
manager.start();
|
||||
|
||||
assertThat(manager.isRunning()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void doesNotStartWhenAutoStartIsDisabled() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(false);
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||
throw new AssertionError("worker should not start");
|
||||
});
|
||||
|
||||
manager.start();
|
||||
|
||||
assertThat(manager.isRunning()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void startsWorkerScriptWithEndpointHostAndPort() throws Exception {
|
||||
Path script = tempDir.resolve("run-worker.sh");
|
||||
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9005/v1/analyze-visuals");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
|
||||
AtomicReference<ProcessBuilder> captured = new AtomicReference<>();
|
||||
AtomicReference<URI> healthUri = new AtomicReference<>();
|
||||
TestProcess process = new TestProcess();
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||
properties,
|
||||
processBuilder -> {
|
||||
captured.set(processBuilder);
|
||||
return process;
|
||||
},
|
||||
(uri, timeoutMs) -> {
|
||||
healthUri.set(uri);
|
||||
return new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok");
|
||||
});
|
||||
|
||||
manager.start();
|
||||
|
||||
assertThat(manager.isRunning()).isTrue();
|
||||
assertThat(captured.get().command()).containsExactly(script.toAbsolutePath().normalize().toString());
|
||||
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_HOST", "localhost");
|
||||
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_PORT", "9005");
|
||||
assertThat(healthUri.get()).hasToString("http://localhost:9005/health");
|
||||
|
||||
manager.stop();
|
||||
|
||||
assertThat(process.destroyed).isTrue();
|
||||
assertThat(manager.isRunning()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsMissingWorkerScriptWhenAutoStartIsEnabled() {
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(tempDir.resolve("missing.sh").toString());
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
|
||||
throw new AssertionError("worker should not start");
|
||||
});
|
||||
|
||||
assertThatThrownBy(manager::start)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("Local CV worker script does not exist");
|
||||
}
|
||||
|
||||
@Test
|
||||
void resetsRunningStateWhenWorkerExitsBeforeHealthCheckPasses() throws Exception {
|
||||
Path script = tempDir.resolve("run-worker.sh");
|
||||
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
|
||||
TestProcess process = new TestProcess();
|
||||
process.alive = false;
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||
properties,
|
||||
processBuilder -> process,
|
||||
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused"));
|
||||
|
||||
assertThatThrownBy(manager::start)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("Local CV worker exited before becoming healthy");
|
||||
assertThat(manager.isRunning()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void failsWhenWorkerDoesNotBecomeHealthyBeforeTimeout() throws Exception {
|
||||
Path script = tempDir.resolve("run-worker.sh");
|
||||
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
|
||||
TestProcess process = new TestProcess();
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||
properties,
|
||||
processBuilder -> process,
|
||||
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 503, "starting"));
|
||||
|
||||
assertThatThrownBy(manager::start)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("Local CV worker did not become healthy");
|
||||
assertThat(process.destroyed).isTrue();
|
||||
assertThat(manager.isRunning()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void retriesHealthChecksUntilWorkerIsReady() throws Exception {
|
||||
Path script = tempDir.resolve("run-worker.sh");
|
||||
Files.writeString(script, "#!/usr/bin/env bash\n");
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
|
||||
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
|
||||
AtomicInteger attempts = new AtomicInteger();
|
||||
TestProcess process = new TestProcess();
|
||||
|
||||
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
|
||||
properties,
|
||||
processBuilder -> process,
|
||||
(uri, timeoutMs) -> attempts.incrementAndGet() < 2
|
||||
? new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused")
|
||||
: new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok"));
|
||||
|
||||
manager.start();
|
||||
|
||||
assertThat(attempts.get()).isEqualTo(2);
|
||||
assertThat(manager.isRunning()).isTrue();
|
||||
|
||||
manager.stop();
|
||||
}
|
||||
|
||||
private static class TestProcess extends Process {
|
||||
private boolean alive = true;
|
||||
private boolean destroyed;
|
||||
|
||||
@Override
|
||||
public OutputStream getOutputStream() {
|
||||
return OutputStream.nullOutputStream();
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream getInputStream() {
|
||||
return new ByteArrayInputStream("worker ready\n".getBytes());
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream getErrorStream() {
|
||||
return InputStream.nullInputStream();
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized int waitFor() throws InterruptedException {
|
||||
while (alive) {
|
||||
wait();
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized boolean waitFor(long timeout, TimeUnit unit) throws InterruptedException {
|
||||
if (alive) {
|
||||
wait(unit.toMillis(timeout));
|
||||
}
|
||||
return !alive;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int exitValue() {
|
||||
return alive ? 1 : 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void destroy() {
|
||||
destroyed = true;
|
||||
alive = false;
|
||||
notifyAll();
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized Process destroyForcibly() {
|
||||
destroyed = true;
|
||||
alive = false;
|
||||
notifyAll();
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized boolean isAlive() {
|
||||
return alive;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long pid() {
|
||||
return 1234;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class ShotSceneSegmenterTest {
|
||||
|
||||
@Test
|
||||
void detectsShotSegmentsFromFfmpegSceneMetadata() {
|
||||
AtomicReference<List<String>> command = new AtomicReference<>();
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setSceneDetectionThreshold(0.42);
|
||||
properties.getEditing().setMinimumSceneDurationSeconds(1.5);
|
||||
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(properties, arguments -> {
|
||||
command.set(arguments);
|
||||
return new ShotSceneSegmenter.ProcessResult(0, """
|
||||
frame:1 pts:72000 pts_time:2.400
|
||||
lavfi.scene_score=0.510
|
||||
frame:2 pts:180000 pts_time:6.000
|
||||
lavfi.scene_score=0.730
|
||||
""");
|
||||
});
|
||||
|
||||
List<ShotSegment> shots = segmenter.segment(clip(9));
|
||||
|
||||
assertThat(shots).containsExactly(
|
||||
new ShotSegment("shot_0001", 0, 2.4, 2.4, 0, 1.2),
|
||||
new ShotSegment("shot_0002", 2.4, 6, 3.6, 0.51, 4.2),
|
||||
new ShotSegment("shot_0003", 6, 9, 3, 0.73, 7.5)
|
||||
);
|
||||
assertCommandPair(command.get(), "-i", "source.mp4");
|
||||
assertCommandPair(command.get(), "-filter:v", "select='gt(scene,0.42)',metadata=print");
|
||||
}
|
||||
|
||||
@Test
|
||||
void returnsSingleShotWhenNoSceneCutPassesMinimumDurationRules() {
|
||||
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), arguments ->
|
||||
new ShotSceneSegmenter.ProcessResult(0, """
|
||||
frame:1 pts:12000 pts_time:0.400
|
||||
lavfi.scene_score=0.900
|
||||
"""));
|
||||
|
||||
assertThat(segmenter.segment(clip(4))).containsExactly(
|
||||
new ShotSegment("shot_0001", 0, 4, 4, 0, 2)
|
||||
);
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsFfmpegFailures() {
|
||||
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), command ->
|
||||
new ShotSceneSegmenter.ProcessResult(1, "failed"));
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> segmenter.segment(clip(9)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void reportsProcessStartupAndInterruptionFailures() {
|
||||
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
|
||||
throw new IOException("missing executable");
|
||||
}).segment(clip(9)));
|
||||
|
||||
try {
|
||||
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
|
||||
throw new InterruptedException("stopped");
|
||||
}).segment(clip(9)));
|
||||
assertTrue(Thread.currentThread().isInterrupted());
|
||||
} finally {
|
||||
Thread.interrupted();
|
||||
}
|
||||
}
|
||||
|
||||
private ClipAnalysis clip(double durationSeconds) {
|
||||
return new ClipAnalysis("source", "source.mp4", durationSeconds, "h264", "aac",
|
||||
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||
}
|
||||
|
||||
private void assertCommandPair(List<String> command, String option, String value) {
|
||||
int index = command.indexOf(option);
|
||||
assertThat(index).isGreaterThanOrEqualTo(0);
|
||||
assertThat(command.get(index + 1)).isEqualTo(value);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,96 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.example.videoclips.config.VideoClippingProperties;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
class SourceAudioAnalyzerTest {
|
||||
|
||||
@Test
|
||||
void analyzesLoudnessSilenceAndAudioSections() {
|
||||
AtomicReference<List<String>> command = new AtomicReference<>();
|
||||
VideoClippingProperties properties = new VideoClippingProperties();
|
||||
properties.getEditing().setSilenceThresholdDb(-40);
|
||||
properties.getEditing().setSilenceMinimumDurationSeconds(1.25);
|
||||
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(properties, arguments -> {
|
||||
command.set(arguments);
|
||||
return new SourceAudioAnalyzer.ProcessResult(0, """
|
||||
[silencedetect @ 0x1] silence_start: 2.5
|
||||
[silencedetect @ 0x1] silence_end: 4.0 | silence_duration: 1.5
|
||||
[Parsed_volumedetect_1 @ 0x2] mean_volume: -19.2 dB
|
||||
[Parsed_volumedetect_1 @ 0x2] max_volume: -2.7 dB
|
||||
""");
|
||||
});
|
||||
|
||||
SourceAudioAnalysis analysis = analyzer.analyze(clip("aac"));
|
||||
|
||||
assertThat(analysis.audioPresent()).isTrue();
|
||||
assertThat(analysis.meanVolumeDb()).isEqualTo(-19.2);
|
||||
assertThat(analysis.maxVolumeDb()).isEqualTo(-2.7);
|
||||
assertThat(analysis.silenceThresholdDb()).isEqualTo(-40);
|
||||
assertThat(analysis.silenceDurationSeconds()).isEqualTo(1.25);
|
||||
assertThat(analysis.sections()).containsExactly(
|
||||
new AudioSection("audio_section_0001", "unclassified_audio", 0, 2.5, 2.5),
|
||||
new AudioSection("audio_section_0002", "silence", 2.5, 4, 1.5),
|
||||
new AudioSection("audio_section_0003", "unclassified_audio", 4, 8, 4)
|
||||
);
|
||||
assertCommandPair(command.get(), "-i", "source.mp4");
|
||||
assertCommandPair(command.get(), "-af", "silencedetect=n=-40.0dB:d=1.25,volumedetect");
|
||||
}
|
||||
|
||||
@Test
|
||||
void returnsMissingAudioAnalysisWhenSourceHasNoAudioCodec() {
|
||||
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||
throw new AssertionError("ffmpeg should not run without an audio stream");
|
||||
});
|
||||
|
||||
SourceAudioAnalysis analysis = analyzer.analyze(clip(null));
|
||||
|
||||
assertThat(analysis.audioPresent()).isFalse();
|
||||
assertThat(analysis.sections()).containsExactly(
|
||||
new AudioSection("audio_section_0001", "missing_audio", 0, 8, 8)
|
||||
);
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsFfmpegFailures() {
|
||||
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command ->
|
||||
new SourceAudioAnalyzer.ProcessResult(1, "failed"));
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> analyzer.analyze(clip("aac")));
|
||||
}
|
||||
|
||||
@Test
|
||||
void reportsProcessStartupAndInterruptionFailures() {
|
||||
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||
throw new IOException("missing executable");
|
||||
}).analyze(clip("aac")));
|
||||
|
||||
try {
|
||||
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
|
||||
throw new InterruptedException("stopped");
|
||||
}).analyze(clip("aac")));
|
||||
assertTrue(Thread.currentThread().isInterrupted());
|
||||
} finally {
|
||||
Thread.interrupted();
|
||||
}
|
||||
}
|
||||
|
||||
private ClipAnalysis clip(String audioCodec) {
|
||||
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", audioCodec,
|
||||
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
|
||||
}
|
||||
|
||||
private void assertCommandPair(List<String> command, String option, String value) {
|
||||
int index = command.indexOf(option);
|
||||
assertThat(index).isGreaterThanOrEqualTo(0);
|
||||
assertThat(command.get(index + 1)).isEqualTo(value);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
package org.example.videoclips.editing;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class SourceVisualAnalyzerTest {
|
||||
|
||||
private final SourceVisualAnalyzer analyzer = new SourceVisualAnalyzer();
|
||||
|
||||
@Test
|
||||
void createsVisualAnalysisFromScoresThumbnailsAndSceneDensity() {
|
||||
SourceVisualAnalysis analysis = analyzer.analyze(
|
||||
clip("porsche-drive", "source/Porsche Drive.mp4", 0.72, 0.61, 0.0),
|
||||
List.of("thumb-1.jpg", "thumb-2.jpg", "thumb-3.jpg", "thumb-4.jpg", "thumb-5.jpg"),
|
||||
List.of(
|
||||
new ShotSegment("shot_0001", 0, 8, 8, 0, 4),
|
||||
new ShotSegment("shot_0002", 8, 16, 8, 0.5, 12),
|
||||
new ShotSegment("shot_0003", 16, 24, 8, 0.6, 20)
|
||||
)
|
||||
);
|
||||
|
||||
assertThat(analysis.blurScore()).isEqualTo(0.72);
|
||||
assertThat(analysis.exposureScore()).isEqualTo(0.61);
|
||||
assertThat(analysis.motionScore()).isEqualTo(0.5);
|
||||
assertThat(analysis.compositionScore()).isEqualTo(0.75);
|
||||
assertThat(analysis.facePresence()).isEqualTo("unknown_without_face_detector");
|
||||
assertThat(analysis.objectLabels()).containsExactly(
|
||||
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic"),
|
||||
new VisualObjectLabel("car", 0.62, "metadata_heuristic")
|
||||
);
|
||||
assertThat(analysis.representativeThumbnails()).containsExactly("thumb-1.jpg", "thumb-3.jpg", "thumb-5.jpg");
|
||||
assertThat(analysis.analysisMethod()).isEqualTo("metadata_thumbnail_scene_heuristic");
|
||||
}
|
||||
|
||||
@Test
|
||||
void fallsBackWhenNoVisualSignalsAreAvailable() {
|
||||
SourceVisualAnalysis analysis = analyzer.analyze(
|
||||
clip("unknown", "source/video.mp4", 0, 0, 0),
|
||||
List.of(),
|
||||
List.of()
|
||||
);
|
||||
|
||||
assertThat(analysis.blurScore()).isEqualTo(0.5);
|
||||
assertThat(analysis.exposureScore()).isEqualTo(0.5);
|
||||
assertThat(analysis.motionScore()).isEqualTo(0.3);
|
||||
assertThat(analysis.compositionScore()).isEqualTo(0.625);
|
||||
assertThat(analysis.objectLabels()).containsExactly(
|
||||
new VisualObjectLabel("unknown", 0.2, "metadata_heuristic")
|
||||
);
|
||||
assertThat(analysis.representativeThumbnails()).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void usesLocalCvProviderWhenConfigured() {
|
||||
org.example.videoclips.config.VideoClippingProperties properties =
|
||||
new org.example.videoclips.config.VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||
properties,
|
||||
(source, thumbnails, shotSegments) -> {
|
||||
throw new AssertionError("heuristic fallback should not run");
|
||||
},
|
||||
(source, thumbnails, shotSegments) -> new SourceVisualAnalysis(source.clipId(), 0.9, 0.8,
|
||||
0.7, 0.6, "faces_detected",
|
||||
List.of(new VisualObjectLabel("person", 0.91, "mediapipe")), thumbnails, "local_cv")
|
||||
);
|
||||
|
||||
SourceVisualAnalysis analysis = localAnalyzer.analyze(
|
||||
clip("family", "family.mp4", 0, 0, 0),
|
||||
List.of("thumb.jpg"),
|
||||
List.of()
|
||||
);
|
||||
|
||||
assertThat(analysis.analysisMethod()).isEqualTo("local_cv");
|
||||
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("person", 0.91, "mediapipe"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void fallsBackToHeuristicWhenLocalCvFailsAndFallbackIsEnabled() {
|
||||
org.example.videoclips.config.VideoClippingProperties properties =
|
||||
new org.example.videoclips.config.VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||
properties,
|
||||
new HeuristicVisualAnalysisProvider(),
|
||||
(source, thumbnails, shotSegments) -> {
|
||||
throw new IllegalStateException("local cv unavailable");
|
||||
}
|
||||
);
|
||||
|
||||
SourceVisualAnalysis analysis = localAnalyzer.analyze(
|
||||
clip("porsche", "porsche.mp4", 0, 0, 0),
|
||||
List.of("thumb.jpg"),
|
||||
List.of()
|
||||
);
|
||||
|
||||
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_failed_fallback_metadata_thumbnail_scene_heuristic");
|
||||
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("porsche", 0.82,
|
||||
"metadata_heuristic"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void failsWhenLocalCvFailsAndFallbackIsDisabled() {
|
||||
org.example.videoclips.config.VideoClippingProperties properties =
|
||||
new org.example.videoclips.config.VideoClippingProperties();
|
||||
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
|
||||
properties.getEditing().getVisualAnalysis().setFallbackToHeuristic(false);
|
||||
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
|
||||
properties,
|
||||
new HeuristicVisualAnalysisProvider(),
|
||||
(source, thumbnails, shotSegments) -> {
|
||||
throw new IllegalStateException("local cv unavailable");
|
||||
}
|
||||
);
|
||||
|
||||
org.junit.jupiter.api.Assertions.assertThrows(IllegalStateException.class, () -> localAnalyzer.analyze(
|
||||
clip("porsche", "porsche.mp4", 0, 0, 0),
|
||||
List.of("thumb.jpg"),
|
||||
List.of()
|
||||
));
|
||||
}
|
||||
|
||||
private ClipAnalysis clip(String clipId, String sourcePath, double sharpness, double brightness, double motion) {
|
||||
return new ClipAnalysis(clipId, sourcePath, 24.0, "h264", "aac", 1920, 1080, 30.0,
|
||||
List.of(), null, null, motion, brightness, sharpness);
|
||||
}
|
||||
}
|
||||
|
|
@ -30,6 +30,7 @@ class StoryboardPromptGeneratorTest {
|
|||
assertThat(prompt)
|
||||
.contains("clipId: clip_00001")
|
||||
.contains("durationSeconds: 8.0")
|
||||
.contains("The service can only render what the plan expresses")
|
||||
.contains("Target duration: 60 seconds")
|
||||
.contains("Style: `cinematic-porsche-promo`")
|
||||
.contains("\"decisions\"")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,4 @@
|
|||
video-clipping.folder-scheduler.enabled=false
|
||||
video-clipping.editing.local-director.enabled=false
|
||||
video-clipping.editing.highlight-scheduler.enabled=false
|
||||
video-clipping.editing.visual-analysis.local-cv-worker.auto-start=false
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
fastapi==0.115.6
|
||||
uvicorn[standard]==0.34.0
|
||||
opencv-python==4.10.0.84
|
||||
ultralytics==8.3.57
|
||||
|
|
@ -0,0 +1,235 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Optional local CV worker for the Spring visual-analysis provider.
|
||||
|
||||
Run:
|
||||
pip install -r tools/local_cv_requirements.txt
|
||||
uvicorn tools.local_cv_worker:app --host 127.0.0.1 --port 8091
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import logging
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import cv2 # type: ignore
|
||||
except ImportError: # pragma: no cover - optional runtime dependency
|
||||
cv2 = None
|
||||
|
||||
try:
|
||||
from fastapi import FastAPI
|
||||
except ImportError as exc: # pragma: no cover - fail clearly at worker startup
|
||||
raise SystemExit("Install FastAPI first: pip install fastapi uvicorn") from exc
|
||||
|
||||
try:
|
||||
from ultralytics import YOLO # type: ignore
|
||||
except ImportError: # pragma: no cover - optional runtime dependency
|
||||
YOLO = None
|
||||
|
||||
|
||||
app = FastAPI(title="Local CV Visual Analysis Worker")
|
||||
_yolo_model: Any | None = None
|
||||
logging.basicConfig(level=os.getenv("LOCAL_CV_LOG_LEVEL", "INFO"))
|
||||
log = logging.getLogger("local_cv_worker")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, Any]:
|
||||
return {
|
||||
"status": "ok",
|
||||
"opencvAvailable": cv2 is not None,
|
||||
"yoloAvailable": YOLO is not None,
|
||||
"yoloModel": configured_yolo_model(),
|
||||
}
|
||||
|
||||
|
||||
@app.post("/v1/analyze-visuals")
|
||||
def analyze_visuals(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
started_at = time.monotonic()
|
||||
source = payload.get("source", {})
|
||||
thumbnails = [str(item) for item in payload.get("thumbnails", [])]
|
||||
readable_thumbnails = [path for path in thumbnails if Path(path).is_file()]
|
||||
clip_id = source.get("clipId", "source")
|
||||
log.info(
|
||||
"event=local_cv_worker_request_started clip_id=%s thumbnails=%d readable_thumbnails=%d "
|
||||
"shot_segments=%d duration_seconds=%s",
|
||||
clip_id,
|
||||
len(thumbnails),
|
||||
len(readable_thumbnails),
|
||||
len(payload.get("shotSegments", [])),
|
||||
source.get("durationSeconds", 0),
|
||||
)
|
||||
if len(readable_thumbnails) != len(thumbnails):
|
||||
unreadable = [path for path in thumbnails if path not in readable_thumbnails]
|
||||
log.warning(
|
||||
"event=local_cv_worker_unreadable_thumbnails clip_id=%s unreadable_count=%d unreadable=%s",
|
||||
clip_id,
|
||||
len(unreadable),
|
||||
unreadable[:10],
|
||||
)
|
||||
quality = quality_scores(readable_thumbnails)
|
||||
labels = object_labels(readable_thumbnails)
|
||||
|
||||
response = {
|
||||
"clipId": clip_id,
|
||||
"blurScore": quality["blurScore"],
|
||||
"exposureScore": quality["exposureScore"],
|
||||
"motionScore": motion_score(payload.get("shotSegments", []), source.get("durationSeconds", 0)),
|
||||
"compositionScore": composition_score(source, readable_thumbnails),
|
||||
"facePresence": face_presence(readable_thumbnails),
|
||||
"objectLabels": labels or [{"label": "unknown", "confidence": 0.2, "source": "local_cv_worker"}],
|
||||
"representativeThumbnails": representative_thumbnails(readable_thumbnails or thumbnails),
|
||||
"analysisMethod": "local_cv_worker_opencv_yolo" if labels else "local_cv_worker_opencv",
|
||||
}
|
||||
log.info(
|
||||
"event=local_cv_worker_request_completed clip_id=%s elapsed_ms=%d blur_score=%s "
|
||||
"exposure_score=%s object_labels=%d method=%s",
|
||||
clip_id,
|
||||
elapsed_ms(started_at),
|
||||
response["blurScore"],
|
||||
response["exposureScore"],
|
||||
len(response["objectLabels"]),
|
||||
response["analysisMethod"],
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
def quality_scores(thumbnails: list[str]) -> dict[str, float]:
|
||||
if cv2 is None or not thumbnails:
|
||||
log.info(
|
||||
"event=local_cv_worker_quality_defaulted reason=%s",
|
||||
"opencv_unavailable" if cv2 is None else "no_readable_thumbnails",
|
||||
)
|
||||
return {"blurScore": 0.5, "exposureScore": 0.5}
|
||||
|
||||
blur_scores: list[float] = []
|
||||
exposure_scores: list[float] = []
|
||||
for thumbnail in thumbnails:
|
||||
image = cv2.imread(thumbnail)
|
||||
if image is None:
|
||||
continue
|
||||
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
||||
laplacian_variance = float(cv2.Laplacian(gray, cv2.CV_64F).var())
|
||||
blur_scores.append(clamp(laplacian_variance / 500.0))
|
||||
mean_brightness = float(gray.mean()) / 255.0
|
||||
exposure_scores.append(clamp(1.0 - abs(mean_brightness - 0.5) * 2.0))
|
||||
|
||||
return {
|
||||
"blurScore": average_or_default(blur_scores, 0.5),
|
||||
"exposureScore": average_or_default(exposure_scores, 0.5),
|
||||
}
|
||||
|
||||
|
||||
def object_labels(thumbnails: list[str]) -> list[dict[str, Any]]:
|
||||
started_at = time.monotonic()
|
||||
model = yolo_model()
|
||||
if model is None or not thumbnails:
|
||||
log.info(
|
||||
"event=local_cv_worker_object_detection_skipped reason=%s thumbnails=%d",
|
||||
"model_unavailable" if model is None else "no_readable_thumbnails",
|
||||
len(thumbnails),
|
||||
)
|
||||
return []
|
||||
|
||||
labels: dict[str, float] = {}
|
||||
for result in model(thumbnails, verbose=False):
|
||||
names = getattr(result, "names", {})
|
||||
boxes = getattr(result, "boxes", None)
|
||||
if boxes is None:
|
||||
continue
|
||||
for box in boxes:
|
||||
label = str(names.get(int(box.cls[0]), "object"))
|
||||
confidence = float(box.conf[0])
|
||||
labels[label] = max(labels.get(label, 0.0), confidence)
|
||||
|
||||
detected = [
|
||||
{"label": label, "confidence": round(clamp(confidence), 3), "source": "yolo"}
|
||||
for label, confidence in sorted(labels.items(), key=lambda item: item[1], reverse=True)[:10]
|
||||
]
|
||||
log.info(
|
||||
"event=local_cv_worker_object_detection_completed elapsed_ms=%d thumbnails=%d labels=%d",
|
||||
elapsed_ms(started_at),
|
||||
len(thumbnails),
|
||||
len(detected),
|
||||
)
|
||||
return detected
|
||||
|
||||
|
||||
def yolo_model() -> Any | None:
|
||||
global _yolo_model
|
||||
model_path = configured_yolo_model()
|
||||
if YOLO is None or not model_path:
|
||||
return None
|
||||
if _yolo_model is None:
|
||||
started_at = time.monotonic()
|
||||
log.info("event=local_cv_worker_yolo_model_loading model=%s", model_path)
|
||||
_yolo_model = YOLO(model_path)
|
||||
log.info("event=local_cv_worker_yolo_model_loaded model=%s elapsed_ms=%d", model_path, elapsed_ms(started_at))
|
||||
return _yolo_model
|
||||
|
||||
|
||||
def configured_yolo_model() -> str:
|
||||
if os.getenv("LOCAL_CV_DISABLE_YOLO", "false").lower() == "true":
|
||||
return ""
|
||||
return os.getenv("LOCAL_CV_YOLO_MODEL", "yolov8n.pt")
|
||||
|
||||
|
||||
def face_presence(thumbnails: list[str]) -> str:
|
||||
if cv2 is None or not thumbnails:
|
||||
return "unknown_without_face_detector"
|
||||
cascade_path = getattr(cv2.data, "haarcascades", "") + "haarcascade_frontalface_default.xml"
|
||||
if not Path(cascade_path).is_file():
|
||||
return "unknown_without_face_detector"
|
||||
cascade = cv2.CascadeClassifier(cascade_path)
|
||||
for thumbnail in thumbnails:
|
||||
image = cv2.imread(thumbnail)
|
||||
if image is None:
|
||||
continue
|
||||
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
||||
faces = cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=4)
|
||||
if len(faces) > 0:
|
||||
return "faces_detected"
|
||||
return "no_faces_detected"
|
||||
|
||||
|
||||
def motion_score(shot_segments: list[dict[str, Any]], duration_seconds: float) -> float:
|
||||
try:
|
||||
duration = float(duration_seconds)
|
||||
except (TypeError, ValueError):
|
||||
duration = 0.0
|
||||
if duration <= 0:
|
||||
return 0.3
|
||||
cuts_per_minute = max(0, len(shot_segments) - 1) / duration * 60.0
|
||||
return round(clamp(0.25 + cuts_per_minute / 20.0), 3)
|
||||
|
||||
|
||||
def composition_score(source: dict[str, Any], thumbnails: list[str]) -> float:
|
||||
width = float(source.get("width", 0) or 0)
|
||||
height = float(source.get("height", 0) or 0)
|
||||
aspect_ratio = width / height if height else 0
|
||||
aspect_score = 0.8 if 1.70 <= aspect_ratio <= 1.90 else 0.55
|
||||
thumbnail_score = 0.7 if thumbnails else 0.45
|
||||
return round((aspect_score + thumbnail_score) / 2.0, 3)
|
||||
|
||||
|
||||
def representative_thumbnails(thumbnails: list[str]) -> list[str]:
|
||||
if len(thumbnails) <= 3:
|
||||
return thumbnails
|
||||
return [thumbnails[0], thumbnails[len(thumbnails) // 2], thumbnails[-1]]
|
||||
|
||||
|
||||
def average_or_default(values: list[float], default: float) -> float:
|
||||
if not values:
|
||||
return default
|
||||
return round(sum(values) / len(values), 3)
|
||||
|
||||
|
||||
def clamp(value: float) -> float:
|
||||
return max(0.0, min(1.0, value))
|
||||
|
||||
|
||||
def elapsed_ms(started_at: float) -> int:
|
||||
return int((time.monotonic() - started_at) * 1000)
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${LOCAL_CV_HOST:-127.0.0.1}"
|
||||
PORT="${LOCAL_CV_PORT:-8091}"
|
||||
MODEL="${LOCAL_CV_YOLO_MODEL:-yolov8n.pt}"
|
||||
|
||||
if [ ! -d ".venv-local-cv" ]; then
|
||||
python3 -m venv .venv-local-cv
|
||||
fi
|
||||
|
||||
. .venv-local-cv/bin/activate
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -r tools/local_cv_requirements.txt
|
||||
|
||||
if [ "${LOCAL_CV_DISABLE_YOLO:-false}" != "true" ]; then
|
||||
LOCAL_CV_YOLO_MODEL="$MODEL" python -c "import os; from ultralytics import YOLO; YOLO(os.environ['LOCAL_CV_YOLO_MODEL'])"
|
||||
fi
|
||||
|
||||
if [ "${LOCAL_CV_PRELOAD_ONLY:-false}" = "true" ]; then
|
||||
echo "Local CV dependencies and model are ready."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
exec uvicorn tools.local_cv_worker:app --host "$HOST" --port "$PORT"
|
||||
Loading…
Reference in New Issue