Compare commits

..

10 Commits

Author SHA1 Message Date
JSLMPR 618ba9af49 - Implement end-to-end cinematic highlight rendering pipeline
- Wire visual effects and local asset generation into highlight flow
  - Add director plan rendering with local asset worker
2026-07-12 01:20:18 +02:00
JSLMPR adc979eea8 Add local CV worker readiness checks 2026-07-11 17:46:40 +02:00
JSLMPR 80eca92d56 Add highlight rendering gap closure plan 2026-07-11 17:26:36 +02:00
JSLMPR 8195b58552 Add rich video editing diagnostics 2026-07-11 17:11:35 +02:00
JSLMPR e2f49e2e5a Add managed local CV worker startup 2026-07-11 16:50:54 +02:00
JSLMPR 9dc720b0cc Add runnable local CV model worker 2026-07-11 16:40:59 +02:00
JSLMPR 82e557905e Add local CV visual analysis provider 2026-07-11 10:49:11 +02:00
JSLMPR eea619a2d3 Add highlight visual analysis 2026-07-11 10:35:07 +02:00
JSLMPR fc17578549 Add highlight audio analysis 2026-07-11 10:23:07 +02:00
JSLMPR 5d5f8022f2 Add highlight scene segmentation 2026-07-11 10:16:07 +02:00
65 changed files with 5993 additions and 42 deletions

8
.gitignore vendored
View File

@ -35,4 +35,10 @@ build/
.vscode/
### Mac OS ###
.DS_Store
.DS_Store
### Local CV runtime ###
.venv-local-cv/
__pycache__/
*.pyc
yolov*.pt

View File

@ -50,7 +50,7 @@ video-clipping:
project-directory: ./output/edit-projects
thumbnail-count-per-clip: 5
proxy-enabled: true
target-duration-seconds: 60
target-duration-seconds: 600
output-width: 1920
output-height: 1080
output-frame-rate: 30
@ -186,6 +186,13 @@ The AI director should inspect:
- `thumbnails/*.jpg`
- `proxies/*.mp4` if more visual context is needed
After the plan is imported, the service also runs an asset materialization stage:
- It reuses matching generated music, SFX, and voiceover assets from the shared cache folders.
- It copies reusable assets into the project-local `audio/` folder so the renderer can use them.
- It writes reusable asset requests under `assets/requests/` when a needed asset is missing.
- It waits to render if required SFX assets are still missing.
The AI director must write this file:
```text
@ -239,6 +246,14 @@ Rules the AI director must follow:
The renderer can mix optional WAV files into the final output.
Shared cache locations:
- Music: `input/highlights/assets/music/`
- SFX: `input/highlights/assets/sfx/`
- Voiceover cache: `output/highlight-projects/_voiceover-cache/`
If the service or an AI worker generates an asset there, later projects can reuse it automatically.
Place optional assets here before rendering:
```text
@ -308,6 +323,14 @@ After rendering completes, the final output is:
output/edit-projects/porsche-session-001/final.mp4
```
The renderer also publishes each final rendered segment clip that was concatenated into the full video:
```text
output/edit-projects/porsche-session-001/rendered-clips/clip_0001.mp4
output/edit-projects/porsche-session-001/rendered-clips/clip_0002.mp4
output/edit-projects/porsche-session-001/rendered-clips/clip_0003.mp4
```
The render manifest is:
```text
@ -320,11 +343,12 @@ The QA report is:
output/edit-projects/porsche-session-001/qa-report.json
```
The manifest is useful for debugging because it records the selected clips, output path, duration, and FFmpeg command summaries.
The manifest is useful for debugging because it records the selected clips, rendered segment clip paths, output path, duration, and FFmpeg command summaries.
The QA report is useful for acceptance because it records:
- final output existence
- rendered segment clip existence
- duration consistency with the edit timeline
- required asset resolution
- text overlay timeline and placement safety

View File

@ -0,0 +1,417 @@
# Highlight Rendering Gap Closure Plan
## Problem
The highlight source scheduler currently creates a highlight project and analysis artifacts, but it does not render final highlight videos.
Observed current output:
```text
output/highlight-projects/<project-id>/
analysis/
source-analysis.json
visual-analysis.json
audio-analysis.json
scene-segments.json
frames/*.jpg
contact-sheets/*.jpg
proxies/*.mp4
source/<source-video>
highlights/
```
The `highlights/` directory is empty because there is no connected pipeline that converts `analysis/source-analysis.json` into renderable highlight plans and then renders `highlights/<highlight-id>/final.mp4`.
## Goal
When a user drops one source video into `input/highlights/source`, the service must eventually produce one or more rendered highlight videos:
```text
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/edit-plan.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json
```
The first implementation should be deterministic and local-first. AI-director integration can improve the plan later, but rendering should not depend on manually running an AI instance.
## Key Design Decision
Do not reuse `FfmpegEditRenderer` directly for highlight projects as-is.
Reason:
- `FfmpegEditRenderer` expects the older edit-project contract: `analysis.json`, `edit-plan.json`, `final.mp4` at project root, and `EditProjectService`.
- Highlight projects use a different contract: `analysis/source-analysis.json`, `director/`, and `highlights/<highlight-id>/`.
- Forcing highlight projects into the edit-project renderer would create path hacks and status-model confusion.
Instead, add a dedicated highlight rendering pipeline that can reuse lower-level concepts such as `EditPlan`, `EditDecision`, `AudioCue`, `VoiceoverLine`, `TextOverlay`, `RenderManifest`, and `RenderQaReport`.
## Target Flow
1. `HighlightSourceScheduler` claims one source video.
2. `HighlightSourceAnalyzer` creates analysis artifacts.
3. New `HighlightCandidateGenerator` creates ranked highlight candidates from scene, audio, and visual analysis.
4. New `HighlightEditPlanGenerator` creates one deterministic `EditPlan` per selected candidate.
5. New `HighlightRenderer` renders each plan into `highlights/<highlight-id>/`.
6. New QA step probes each rendered MP4 and writes `qa-report.json`.
7. Scheduler logs `event=highlight_flow_completed` with final output count and output paths.
## Milestones
- [ ] Milestone 1: Persist highlight candidates for highlight-source projects.
- [ ] Milestone 2: Generate deterministic highlight edit plans from candidates.
- [ ] Milestone 3: Implement a highlight-project renderer that writes into `highlights/<highlight-id>/`.
- [ ] Milestone 4: Add local asset fallback behavior so highlights render even without music, SFX, or TTS assets.
- [ ] Milestone 5: Wire candidate generation, plan generation, and rendering into the scheduler flow.
- [ ] Milestone 6: Add full rich logs for every highlight rendering step and a final `highlight_flow_completed` log.
- [ ] Milestone 7: Add tests for candidate generation, plan generation, renderer commands, scheduler integration, and end-to-end output.
- [ ] Milestone 8: Update the runbook so users can find rendered highlights and troubleshoot missing outputs.
## Milestone Details
### Milestone 1: Persist Highlight Candidates
Add a component:
```text
HighlightCandidateGenerator
```
Input:
```text
HighlightSourceAnalysis
```
Output:
```text
analysis/highlight-candidates.json
```
Candidate rules for the first version:
- Use `scene-segments.json` as the primary temporal units.
- If only one scene exists, split the source into overlapping 8-20 second windows.
- Score candidates using available fields:
- `visualAnalysis.motionScore`
- `visualAnalysis.compositionScore`
- `visualAnalysis.blurScore`
- `visualAnalysis.exposureScore`
- `audioAnalysis.sections`
- scene duration and position
- Prefer candidates between 8 and 35 seconds.
- Avoid first/last black or silent areas when audio data suggests silence.
- Limit initial output to top 3 candidates.
Required logs:
```text
event=highlight_candidates_started project_id=...
event=highlight_candidate_scored project_id=... candidate_id=... start=... end=... score=... reasons=...
event=highlight_candidates_completed project_id=... count=... elapsed_ms=...
```
Tests:
- Generates candidates from multiple scene segments.
- Falls back to fixed windows when only one scene exists.
- Rejects or downranks invalid, too-short, too-long, or blurry candidates.
- Persists `analysis/highlight-candidates.json`.
### Milestone 2: Generate Deterministic Edit Plans
Add a component:
```text
HighlightEditPlanGenerator
```
Input:
```text
HighlightSourceAnalysis
List<HighlightCandidate>
```
Output per candidate:
```text
highlights/<highlight-id>/edit-plan.json
highlights/<highlight-id>/storyboard.md
```
First version plan rules:
- Create one highlight per selected candidate.
- Use 3-6 edit decisions per highlight.
- For a short candidate, use the candidate range as one segment.
- For a longer candidate, split into smaller cuts with simple pacing.
- Use deterministic category style:
- car-like/object-motion: premium cinematic, punch-in, warm contrast, bass-hit cuts if assets exist.
- family-like/faces: warm memory style, softer overlays.
- food-like/close action: rhythmic sensory style.
- generic: clean cinematic montage.
- Add optional text overlays only when safe.
- Add voiceover text as script metadata, but do not block render when no TTS provider exists.
- Add music/SFX cues only when local assets exist or when renderer can skip missing optional assets.
Required logs:
```text
event=highlight_edit_plan_started project_id=... candidate_id=...
event=highlight_edit_plan_decision_created project_id=... highlight_id=... clip_id=... source_start=... source_end=...
event=highlight_edit_plan_completed project_id=... highlight_id=... decisions=... overlays=... audio_cues=...
```
Tests:
- Produces valid `EditPlan` timestamps inside source duration.
- Produces stable output for the same analysis.
- Creates storyboard markdown.
- Does not require external AI.
### Milestone 3: Implement Highlight Renderer
Add:
```text
HighlightRenderer
FfmpegHighlightRenderer
```
Input:
```text
projectId
highlightId
highlights/<highlight-id>/edit-plan.json
analysis/source-analysis.json
```
Output:
```text
highlights/<highlight-id>/final.mp4
highlights/<highlight-id>/preview.mp4
highlights/<highlight-id>/render-manifest.json
highlights/<highlight-id>/qa-report.json
```
Renderer requirements for first version:
- Trim from the original source video using exact timestamps.
- Concatenate selected segments.
- Apply simple visual treatment using FFmpeg filters:
- scale/pad to configured output size
- optional contrast/saturation/sharpen
- optional vignette or fade
- render overlays with `drawtext` when fonts are available or use default FFmpeg font fallback.
- Mix source audio.
- Optionally mix music/SFX/voiceover when local assets exist.
- Normalize output audio loudness when audio exists.
- Produce `preview.mp4` as a lower-resolution copy or direct transcode.
- Write render manifest with commands, decisions, output paths, assets, duration, and exit codes.
Important:
- Missing music, SFX, voiceover, LUT, or font assets must not block a basic render.
- Missing source video, invalid timestamps, or FFmpeg failure must block render and mark the highlight failed.
Required logs:
```text
event=highlight_render_started project_id=... highlight_id=...
event=highlight_render_segment_started project_id=... highlight_id=... segment=... source_start=... source_end=...
event=highlight_render_segment_completed project_id=... highlight_id=... segment=... elapsed_ms=...
event=highlight_render_concat_started project_id=... highlight_id=... segments=...
event=highlight_render_effects_started project_id=... highlight_id=... overlays=... treatment=...
event=highlight_render_audio_started project_id=... highlight_id=... music=... sfx=... voiceover=...
event=highlight_render_completed project_id=... highlight_id=... output=... duration_seconds=... size_bytes=... elapsed_ms=...
event=highlight_render_failed project_id=... highlight_id=... error_type=... message=...
```
Tests:
- Builds expected FFmpeg trim commands.
- Builds concat and final render commands.
- Writes final output path under `highlights/<highlight-id>/`.
- Writes render manifest.
- Handles missing optional assets.
- Fails on invalid source timestamps.
### Milestone 4: Local Asset Fallbacks
Current rendered highlights must work even with no licensed asset library.
Add fallback behavior:
- If no music asset exists, keep source audio.
- If no SFX asset exists, skip SFX cues and record skipped assets in manifest.
- If no TTS provider exists, write voiceover script to `assets/voiceover/voiceover-script.txt` and skip voiceover audio.
- If no LUT exists, use named FFmpeg color preset.
- If no font exists, use FFmpeg default drawtext behavior or skip overlays if drawtext cannot resolve a font.
Required logs:
```text
event=highlight_asset_resolved project_id=... highlight_id=... type=... asset=...
event=highlight_asset_skipped project_id=... highlight_id=... type=... reason=missing_optional_asset
```
Tests:
- Render plan with no local assets still produces `final.mp4`.
- Manifest records skipped optional assets.
- Voiceover script file is still created.
### Milestone 5: Wire Into Scheduler
Update `HighlightSourceScheduler.createProject` after analysis:
```text
analysis = analyzer.analyze(projectId)
candidates = candidateGenerator.generate(projectId, analysis)
plans = planGenerator.generate(projectId, analysis, candidates)
renderResults = renderer.render(projectId, plans)
move source to processed
log flow complete
```
Configuration:
```yaml
video-clipping:
editing:
highlight-scheduler:
render-enabled: true
max-highlights-per-source: 3
highlight-min-duration-seconds: 8
highlight-max-duration-seconds: 35
require-director-approval: false
```
If `require-director-approval=true`, the scheduler should stop after writing plans and wait for an approval flag before rendering.
Required logs:
```text
event=highlight_flow_started scan_id=... project_id=... source_file=...
event=highlight_flow_analysis_completed scan_id=... project_id=...
event=highlight_flow_candidates_completed scan_id=... project_id=... count=...
event=highlight_flow_plans_completed scan_id=... project_id=... count=...
event=highlight_flow_renders_completed scan_id=... project_id=... count=...
event=highlight_flow_completed scan_id=... project_id=... final_outputs=... elapsed_ms=...
```
Tests:
- Scheduler produces at least one `highlights/<highlight-id>/final.mp4` from a valid fixture video.
- Scheduler leaves no source video in `source` or `working` after success.
- Scheduler moves failed source to rejected and logs failure.
- Rendering can be disabled for analysis-only mode.
### Milestone 6: Rich End-To-End Logging
Add a single correlation ID:
```text
flow_id=<project-id>:<scan-id>
```
Every log in the highlight path should include:
- `scan_id`
- `project_id`
- `highlight_id` when applicable
- `flow_id`
- `elapsed_ms` for completed steps
The final success log must be:
```text
event=highlight_flow_completed flow_id=... scan_id=... project_id=... final_outputs=[...] elapsed_ms=...
```
The final failure log must be:
```text
event=highlight_flow_failed flow_id=... scan_id=... project_id=... step=... error_type=... message=... elapsed_ms=...
```
Tests:
- Use a log capture test to assert `highlight_flow_completed` is emitted on success.
- Use a failure test to assert `highlight_flow_failed` includes the failed step.
### Milestone 7: Tests And Coverage
Add focused unit tests:
- `HighlightCandidateGeneratorTest`
- `HighlightEditPlanGeneratorTest`
- `FfmpegHighlightRendererTest`
- `HighlightRenderManifestTest`
- `HighlightSourceSchedulerRenderFlowTest`
Add integration test:
```text
HighlightRenderingIntegrationTest
```
Requirements:
- Create a tiny FFmpeg fixture video.
- Place it in a temp highlight source folder.
- Run the scheduler once.
- Assert:
- `analysis/source-analysis.json` exists.
- `analysis/highlight-candidates.json` exists.
- `highlights/<highlight-id>/edit-plan.json` exists.
- `highlights/<highlight-id>/final.mp4` exists.
- `highlights/<highlight-id>/render-manifest.json` exists.
- `highlights/<highlight-id>/qa-report.json` exists.
- `highlight_flow_completed` log exists.
### Milestone 8: Runbook Update
Update:
```text
docs/cinematic-editing-runbook.md
docs/production-cinematic-highlight-editing-plan.md
```
Add:
- Exact output location for final highlight videos.
- How to tell whether the project is analysis-only or rendered.
- How to enable/disable automatic rendering.
- How to inspect `highlight-candidates.json`.
- How to inspect per-highlight `edit-plan.json`.
- How to inspect `render-manifest.json` and `qa-report.json`.
- Troubleshooting table for empty `highlights/`.
## Implementation Order
1. Implement `HighlightCandidateGenerator` and persist `analysis/highlight-candidates.json`.
2. Implement `HighlightEditPlanGenerator` and write per-highlight plans/storyboards.
3. Implement `FfmpegHighlightRenderer` with minimal no-asset render.
4. Wire renderer behind `highlight-scheduler.render-enabled`.
5. Add final `highlight_flow_completed` and `highlight_flow_failed` logs.
6. Add integration test proving `final.mp4` is created.
7. Add optional assets, overlays, audio mix, and QA improvements.
8. Update runbooks.
## Definition Of Done
- Dropping one valid source video into `input/highlights/source` produces at least one rendered MP4 under `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4`.
- The source file is moved to `input/highlights/processed`.
- The project contains `analysis/highlight-candidates.json`.
- Each rendered highlight contains `edit-plan.json`, `storyboard.md`, `render-manifest.json`, and `qa-report.json`.
- Logs contain `highlight_flow_completed` with final output paths.
- Missing optional assets do not prevent a basic highlight render.
- Full Maven tests pass.

View File

@ -0,0 +1,176 @@
# Local CV Visual Analysis Guide
This service can use a local computer-vision worker for highlight visual analysis.
## Default Behavior
The packaged `application.yml` defaults to `local-cv` with heuristic fallback enabled:
```yaml
video-clipping:
editing:
visual-analysis:
provider: local-cv
fallback-to-heuristic: true
```
This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.
## Enable A Local CV Worker
Run a local worker that accepts JSON over HTTP, then start this Spring service with:
```yaml
video-clipping:
editing:
visual-analysis:
provider: local-cv
endpoint: http://127.0.0.1:8091/v1/analyze-visuals
timeout-ms: 30000
fallback-to-heuristic: true
local-cv-worker:
auto-start: true
script: ./tools/run_local_cv_worker.sh
startup-wait-ms: 120000
health-path: /health
health-check-interval-ms: 1000
```
Equivalent environment variables:
```bash
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=120000
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH=/health
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS=1000
```
This repository includes an optional starter worker:
```bash
tools/run_local_cv_worker.sh
```
The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.
Default model behavior:
- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.
Health check:
```bash
curl http://127.0.0.1:8091/health
```
## Preload Dependencies And Model
Do this once before starting the Spring service if you do not want dependency installation or YOLO model download to happen during application startup:
```bash
LOCAL_CV_PRELOAD_ONLY=true tools/run_local_cv_worker.sh
```
This creates `.venv-local-cv`, installs `tools/local_cv_requirements.txt`, and downloads/caches the configured YOLO model. After that, Spring-managed startup should be much faster.
Do not commit `.venv-local-cv`, Python wheels, Torch binaries, or YOLO weights into source control. They are large, platform-specific runtime artifacts. For production, prefer a prebuilt container image or a VM/bootstrap step that runs the preload command during deployment.
## Spring-Managed Worker Startup
The Spring application can start the local CV worker for local runs:
```bash
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
mvn spring-boot:run
```
When `auto-start` is enabled, Spring starts `tools/run_local_cv_worker.sh`, derives `LOCAL_CV_HOST` and `LOCAL_CV_PORT` from the configured visual-analysis endpoint, polls the configured health endpoint until it returns HTTP 2xx, streams worker output into application logs, and stops the worker process during application shutdown.
Keep `auto-start` disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates `.venv-local-cv`, installs Python dependencies, and may download/cache the configured YOLO model. Use the preload command or a prebuilt image to avoid that startup cost.
## Worker Request Contract
The Spring service sends:
```json
{
"source": {
"clipId": "porsche-drive",
"sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
"durationSeconds": 42.0,
"videoCodec": "h264",
"audioCodec": "aac",
"width": 1920,
"height": 1080,
"frameRate": 30.0
},
"thumbnails": [
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
],
"shotSegments": [
{
"shotId": "shot_0001",
"startSeconds": 0.0,
"endSeconds": 8.0,
"durationSeconds": 8.0,
"representativeTimestampSeconds": 4.0
}
]
}
```
## Worker Response Contract
The worker must return:
```json
{
"clipId": "porsche-drive",
"blurScore": 0.81,
"exposureScore": 0.67,
"motionScore": 0.73,
"compositionScore": 0.79,
"facePresence": "faces_detected",
"objectLabels": [
{
"label": "car",
"confidence": 0.91,
"source": "yolo"
},
{
"label": "luxury sports car",
"confidence": 0.84,
"source": "clip"
}
],
"representativeThumbnails": [
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
],
"analysisMethod": "local_cv_yolo_clip_mediapipe"
}
```
Scores must be normalized from `0.0` to `1.0`.
## Recommended Local Model Split
- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.
## Runtime Behavior
When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.
When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.

View File

@ -0,0 +1,310 @@
# Production Cinematic AI Director Prompt
Use this prompt with a filesystem-capable AI agent such as Codex or Claude when a highlight project has already been analyzed by the service.
Expected project folder:
```text
output/highlight-projects/<project-id>/
```
The AI director must inspect the project artifacts, choose all worthwhile highlight moments, and write a strict renderable edit plan. The goal is a platform-ready cinematic highlight video with strong pacing, fitting sound design, fitting voiceover, tasteful text overlays, and visual treatments that match the actual content.
## Prompt
```text
You are a senior commercial video editor, creative director, sound designer, and post-production supervisor.
Your assignment is to create a production-ready cinematic highlight edit plan from the analyzed project folder. The final rendered video must feel strong enough for a public video streaming platform: clear hook, strong visual selection, coherent micro-story, professional pacing, fitting sound design, tasteful overlays, and a voiceover that matches the actual footage. Do not promise revenue or make unsupported claims; optimize for viewer retention and professional presentation.
PROJECT FOLDER
- Work only inside this folder: output/highlight-projects/<project-id>/
- Read all available project files before deciding:
- project.json
- manifest.json
- analysis/source-analysis.json
- analysis/ffprobe.json
- analysis/scene-segments.json
- analysis/audio-analysis.json
- analysis/visual-analysis.json
- analysis/contact-sheets/*
- analysis/frames/*
- analysis/proxies/* when motion needs closer inspection
- assets/music/*, assets/sfx/*, assets/voiceover/*, assets/overlays/* if present
OUTPUT FILE
- Write the final edit plan JSON to:
director/edit-plan.json
- Return JSON only in that file.
- Do not wrap JSON in Markdown fences.
- Do not render video.
- Do not modify source video, analysis files, thumbnails, contact sheets, or proxies.
PRIMARY GOAL
Create one cinematic highlight video that is specific to the provided footage and content type. The highlight length must be content-driven, not fixed. Use all strong highlights from the source, but do not pad with weak or repetitive footage just to reach a duration.
The edit must have:
- A 1-3 second opening hook that immediately shows the most visually interesting or emotionally compelling moment.
- A clear middle section with rising energy, variety, and no repetitive shot choices.
- A satisfying final hero/payoff shot.
- A final duration that fits the source material and keeps audience retention high.
- Beat-aware pacing and deliberate shot lengths.
- Specific visual treatment notes that the renderer can execute: cinematic grade, punch-in crop, subtle speed ramp, vignette, sharpness, contrast, saturation, slow-motion emphasis, or clean natural grade.
- Fitting music direction and sound effects timing.
- Voiceover text that comments on what is actually visible or strongly implied by the footage.
- Short text overlays only where they improve retention or clarity.
DELIVERABLE EXPECTATION
The service must ultimately render both:
- the full final highlight video
- each rendered timeline segment clip that was used to assemble the full final highlight video
Expected output for the current edit-project renderer:
- final.mp4
- rendered-clips/clip_0001.mp4
- rendered-clips/clip_0002.mp4
- rendered-clips/clip_0003.mp4
- render-manifest.json with renderedClipPaths
Expected output for the highlight-project renderer when implemented:
- highlights/<highlight-id>/final.mp4
- highlights/<highlight-id>/rendered-clips/clip_0001.mp4
- highlights/<highlight-id>/rendered-clips/clip_0002.mp4
- highlights/<highlight-id>/rendered-clips/clip_0003.mp4
- highlights/<highlight-id>/render-manifest.json with renderedClipPaths
CONTENT CATEGORY DECISION
Classify the project into exactly one content category:
- car_vlog
- food_vlog
- family_vlog
- generic_vlog
Use the category to direct the whole edit.
For car_vlog:
- Make it premium, powerful, precise, and aspirational.
- Prioritize hero angles, reflections, wheels, badges, road movement, cockpit details, acceleration, reveal shots, light, shadow, and motion.
- Use punchy cuts, speed ramps, whooshes, impact hits, engine/exhaust SFX only when appropriate, cinematic contrast, and confident short overlays.
- Voiceover should praise what is actually visible: stance, motion, design, presence, craft, road feel. Do not invent specifications, price, horsepower, rarity, or model facts unless visible in the footage.
For food_vlog:
- Make it sensory, warm, rhythmic, and appetizing.
- Prioritize texture, slicing, pouring, steam, sizzling, plating, table reveals, reactions, and macro-like detail.
- Use tactile SFX, warm grade, gentle groove, ingredient/payoff overlays, and voiceover focused on craft, freshness, texture, and taste.
For family_vlog:
- Make it warm, emotional, human, and memory-focused.
- Prioritize faces, reactions, laughter, hugs, children, travel reveals, meaningful moments, and natural dialogue.
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
- Voiceover should feel personal and reflective, not promotional.
For generic_vlog:
- Build a clear mini-story from novelty, clean composition, motion, reaction, place, process, and payoff.
- Use restrained cinematic effects and explain only what the footage supports.
SHOT SELECTION RULES
Before writing JSON, inspect the contact sheets and representative frames.
Create a private selection table for yourself:
- candidate source range
- visual appeal
- motion/energy
- uniqueness
- story role
- audio opportunity
- risk: blur, darkness, repetitive angle, dead time
Then use only the best ranges. Avoid:
- repeated near-identical shots unless repetition is intentional
- long boring segments
- badly blurred or unusably dark footage unless it is the only important moment
- shots with no story purpose
- unsupported narration
TIMELINE AND PACING RULES
- Do not force a strict 60 second edit unless the footage actually supports it.
- Derive the final duration from the input source length and number of strong moments.
- Include every strong highlight moment that improves the final video.
- Exclude weak, repetitive, technically poor, or dead-time footage even if the source is long.
- Maximum final duration is 600 seconds, which is 10 minutes.
- Most videos should be shorter than the maximum. Use 10 minutes only when the source has enough distinct, high-quality highlights to justify it.
- Recommended duration guide:
- Source under 30 seconds: final highlight usually 8-20 seconds.
- Source 30-90 seconds: final highlight usually 15-45 seconds.
- Source 90 seconds-5 minutes: final highlight usually 30-120 seconds.
- Source 5-20 minutes: final highlight usually 60-180 seconds.
- Source over 20 minutes: final highlight can be longer, but only if the footage has enough distinct high-quality moments.
- The JSON field targetDurationSeconds must equal the actual planned final timeline duration, not a fixed configured default.
- The final decision timelineEndSeconds must equal targetDurationSeconds.
- Timeline must start at 0.0.
- Every decision timelineStartSeconds must equal the previous decision timelineEndSeconds.
- No overlaps. No gaps.
- For each decision:
rendered duration = (sourceEndSeconds - sourceStartSeconds) / playbackSpeed
timelineEndSeconds - timelineStartSeconds must equal rendered duration within 0.05 seconds.
- playbackSpeed must be between 0.25 and 4.0.
- Keep most shots between 1.0 and 4.0 seconds.
- Use longer shots only for emotional moments, reveals, or beautiful hero motion.
- Use faster shots for montages, details, impacts, or beat drops.
ALLOWED TRANSITIONS
Use only:
- cut
- crossfade
- fade-in
- fade-out
- none
Prefer mostly cuts. Use crossfade/fades sparingly for emotional, elegant, or time-passing moments.
VISUAL TREATMENT RULES
visualTreatment must be concise but specific. Good examples:
- cinematic grade with subtle punch-in
- high-contrast car hero grade
- warm food closeup grade
- soft family memory grade
- speed-ramp detail accent
- clean natural grade
- none
Use visual effects to support the moment. Do not overuse effects.
AUDIO RULES
Use audioCues for music and SFX direction.
Music:
- A music cue may use a descriptive assetKey such as premium_cinematic_drive, warm_acoustic_memory, tactile_food_groove, urban_pulse_intro.
- Music should cover the full timeline unless silence is intentionally used.
- Set gainDb conservatively, usually between -22 and -10.
- Describe the music mood and why it fits.
SFX:
- Only reference SFX assetKey values that exist as WAV files in the project audio/sfx folder or assets/sfx folder if the renderer maps them.
- If no SFX files exist, do not invent file-backed SFX cues. Instead, put desired SFX ideas in the cue notes only if the service can later generate assets.
- SFX should be precise: whoosh on transition, bass hit on reveal, camera shutter on detail, subtle riser before hero shot, engine accent for car footage, sizzle/chop/pour for food, gentle ambience for family.
VOICEOVER RULES
- Voiceover is optional only if the project disables it. If enabled, include concise lines.
- Each line must be 240 characters or fewer.
- Use natural, premium, non-cheesy language.
- Voiceover must fit the category and visible content.
- Do not use generic filler like "this is amazing" or "an unforgettable journey" unless the footage clearly supports it.
- Do not invent facts.
- Give each line a delivery direction, for example:
- calm premium narrator
- confident cinematic narrator
- warm reflective narrator
- energetic food host
TEXT OVERLAY RULES
- Overlays must be short: 80 characters or fewer.
- Use only these placements:
- lower_left_safe
- lower_center_safe
- center_safe
- upper_left_safe
- upper_right_safe
- Use only these animations:
- fade_slide_up
- fade_in
- none
- Use overlays sparingly. 1-5 overlays is usually enough.
- Overlays should feel premium and specific, not clickbait.
QUALITY BAR
Reject your own first draft if it has any of these problems:
- The first 3 seconds are not compelling.
- The edit is just chronological trimming without a story.
- The voiceover is generic or not grounded in visible footage.
- Music or SFX are placeholders with no timing purpose.
- Text overlays repeat what the voiceover already says.
- The final shot is weak.
- Any JSON field uses invalid values.
- Any clip ID or timestamp is invented.
- Any timeline math is invalid.
REQUIRED JSON SHAPE
Write exactly this JSON shape:
{
"projectId": "<project-id>",
"style": "<style-from-project-json>",
"targetDurationSeconds": <actual-planned-final-duration-seconds>,
"decisions": [
{
"clipId": "<valid clip id from analysis>",
"sourceStartSeconds": 0.0,
"sourceEndSeconds": 2.5,
"timelineStartSeconds": 0.0,
"timelineEndSeconds": 2.5,
"transitionIn": "cut",
"transitionOut": "cut",
"playbackSpeed": 1.0,
"visualTreatment": "cinematic grade with subtle punch-in",
"reason": "Opening hook: strongest hero angle and immediate viewer retention."
}
],
"audioCues": [
{
"type": "music",
"assetKey": "premium_cinematic_drive",
"timelineStartSeconds": 0.0,
"timelineEndSeconds": <actual-planned-final-duration-seconds>,
"gainDb": -16.0,
"notes": "Low cinematic pulse that builds through detail shots and lifts into the final hero moment."
}
],
"voiceover": [
{
"text": "A concise, specific line grounded in what is visible in the footage.",
"timelineStartSeconds": 1.0,
"timelineEndSeconds": 4.0,
"delivery": "confident cinematic narrator"
}
],
"overlays": [
{
"text": "Short premium overlay",
"timelineStartSeconds": 0.5,
"timelineEndSeconds": 2.0,
"placement": "lower_left_safe",
"animation": "fade_slide_up",
"reason": "Adds context without duplicating the narration."
}
],
"renderProfile": "mp4-h264-aac-1080p",
"summary": "A short explanation of the creative strategy, category choice, hook, pacing, sound design, and final payoff."
}
FINAL SELF-CHECK BEFORE WRITING director/edit-plan.json
Verify:
- projectId matches project.json.
- style matches project.json.
- targetDurationSeconds equals the actual final timeline duration selected from the source highlights.
- decisions is not empty.
- audioCues, voiceover, and overlays are arrays even when empty.
- All clip IDs exist.
- All source ranges are inside clip duration.
- Timeline starts at 0.0 and is contiguous.
- Final decision timelineEndSeconds equals targetDurationSeconds.
- Duration math matches playbackSpeed.
- transitionIn and transitionOut use only allowed values.
- playbackSpeed is between 0.25 and 4.0.
- overlay placements and animations use only allowed values.
- overlay text is 80 characters or fewer.
- voiceover lines are 240 characters or fewer.
- SFX asset keys reference real files if SFX cues are included.
- JSON is valid and contains no comments or Markdown.
```
## Notes For The Service
This prompt is intentionally stricter than the current lightweight storyboard prompt. It is designed to prevent the common failure mode where the AI produces a generic, boring, or invalid edit plan.
Current renderer constraints still matter:
- `FfmpegEditRenderer` can render the `EditPlan` schema used by `output/edit-projects`.
- The highlight project flow still needs an importer/renderer bridge before `output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4` can be produced automatically.
- If SFX are referenced, the current validator expects matching WAV files for SFX asset keys. Do not ask the AI to reference non-existent SFX files unless the asset generation step is implemented first.
- The current noop voiceover provider writes a script, not a real spoken voice. Production-ready voice requires a real TTS provider or recorded voiceover asset.

View File

@ -204,6 +204,7 @@ The system needs explicit providers instead of pretending a prompt can create fi
- Font provider: local fonts with brand/category mapping.
- LUT provider: local LUT files or named FFmpeg grade presets.
- Asset cache: deterministic reuse by prompt, category, duration, and provider settings.
- Asset generation stage: when a needed asset is missing, write a reusable request pack and keep the target cache path stable so future AI-generated assets can be reused.
Licensing must be tracked in `render-manifest.json` for any music, SFX, voice, font, or LUT asset used in a final export.
@ -301,9 +302,13 @@ The plan should be strict JSON so a cheaper model or deterministic renderer can
- [x] Milestone 2 progress: Added a configurable highlight source scheduler and directory initializer for `input/highlights/source`, `working`, `processed`, and `rejected`; it claims one valid source video per scan, creates a highlight project and manifest, copies the source into the project folder, and leaves later videos for later scans.
- [x] Milestone 3: Implement source probing, proxy generation, frame extraction, waveform extraction, and contact sheet creation.
- [x] Milestone 3 progress: Added highlight source analysis for the new project contract, reusing FFprobe inspection, frame extraction, contact sheet generation, proxy generation, and adding waveform image generation under `analysis/`; the highlight scheduler now runs analysis immediately after creating a project.
- [ ] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
- [ ] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
- [ ] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
- [x] Milestone 4: Implement shot and scene segmentation using FFmpeg scene detection or PySceneDetect.
- [x] Milestone 4 progress: Added configurable FFmpeg scene detection for highlight sources, persisted `analysis/scene-segments.json`, attached shot segments to `source-analysis.json`, and logged shot segment counts when the highlight scheduler finishes analysis.
- [x] Milestone 5: Implement audio analysis for loudness, silence, peaks, speech/music/noise sections, and optional ASR transcript.
- [x] Milestone 5 progress: Added deterministic FFmpeg audio analysis for mean/max volume, silence detection, missing-audio handling, persisted `analysis/audio-analysis.json`, and source-analysis embedding. Non-silent sections are labeled `unclassified_audio` until a speech/music/noise classifier or ASR provider is added.
- [x] Milestone 6: Implement visual analysis for blur, exposure, motion, faces, object labels, composition quality, and representative thumbnails.
- [x] Milestone 6 progress: Added persisted `analysis/visual-analysis.json` and source-analysis embedding with blur, exposure, motion, composition, representative thumbnail, face-presence, and object-label fields. Current implementation uses deterministic metadata, thumbnail, and scene-density heuristics; face/object recognition is explicitly marked heuristic until a CV model is connected.
- [x] Milestone 6 local CV extension: Added a configurable `local-cv` visual-analysis provider that posts source metadata, thumbnails, and shot segments to a local HTTP CV worker, persists the returned normalized visual analysis, and falls back to heuristic analysis when configured. See `docs/local-cv-visual-analysis-guide.md`.
- [x] Milestone 7: Implement content category classification for family vlog, food vlog, car vlog, and generic fallback.
- [x] Milestone 8: Implement category-specific highlight candidate scoring.
- [x] Milestone 9: Implement strict AI director prompts and JSON schemas for category-aware highlight edit plans.
@ -548,6 +553,9 @@ Expected renderer output:
```text
output/highlight-projects/<project-id>/highlights/<highlight-id>/final.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0001.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0002.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/rendered-clips/clip_0003.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/preview.mp4
output/highlight-projects/<project-id>/highlights/<highlight-id>/render-manifest.json
output/highlight-projects/<project-id>/highlights/<highlight-id>/qa-report.json

View File

@ -0,0 +1,15 @@
{
"id" : "source-clips",
"name" : "source-clips",
"status" : "RENDERED",
"inputDirectory" : "./input/editing/processed/source-clips",
"outputDirectory" : "output/edit-projects/source-clips",
"targetDurationSeconds" : 60,
"style" : "cinematic-porsche-promo",
"voiceoverEnabled" : true,
"musicEnabled" : true,
"soundEffectsEnabled" : true,
"createdAt" : "2026-07-10T17:25:59.366353Z",
"updatedAt" : "2026-07-10T21:58:05.307273Z",
"failureReason" : null
}

File diff suppressed because one or more lines are too long

View File

@ -503,8 +503,16 @@ public class VideoClippingProperties {
@Min(1)
private int proxyWidth = 640;
private double sceneDetectionThreshold = 0.35;
private double minimumSceneDurationSeconds = 1.0;
private double silenceThresholdDb = -35.0;
private double silenceMinimumDurationSeconds = 0.5;
@Min(1)
private int targetDurationSeconds = 60;
private int targetDurationSeconds = 600;
@Min(1)
private int outputWidth = 1920;
@ -542,6 +550,8 @@ public class VideoClippingProperties {
private final Assets assets = new Assets();
private final VisualAnalysis visualAnalysis = new VisualAnalysis();
private final LocalDirector localDirector = new LocalDirector();
private final HighlightScheduler highlightScheduler = new HighlightScheduler();
@ -618,6 +628,38 @@ public class VideoClippingProperties {
this.proxyWidth = proxyWidth;
}
public double getSceneDetectionThreshold() {
return sceneDetectionThreshold;
}
public void setSceneDetectionThreshold(double sceneDetectionThreshold) {
this.sceneDetectionThreshold = sceneDetectionThreshold;
}
public double getMinimumSceneDurationSeconds() {
return minimumSceneDurationSeconds;
}
public void setMinimumSceneDurationSeconds(double minimumSceneDurationSeconds) {
this.minimumSceneDurationSeconds = minimumSceneDurationSeconds;
}
public double getSilenceThresholdDb() {
return silenceThresholdDb;
}
public void setSilenceThresholdDb(double silenceThresholdDb) {
this.silenceThresholdDb = silenceThresholdDb;
}
public double getSilenceMinimumDurationSeconds() {
return silenceMinimumDurationSeconds;
}
public void setSilenceMinimumDurationSeconds(double silenceMinimumDurationSeconds) {
this.silenceMinimumDurationSeconds = silenceMinimumDurationSeconds;
}
public int getTargetDurationSeconds() {
return targetDurationSeconds;
}
@ -742,6 +784,10 @@ public class VideoClippingProperties {
return assets;
}
public VisualAnalysis getVisualAnalysis() {
return visualAnalysis;
}
public LocalDirector getLocalDirector() {
return localDirector;
}
@ -802,6 +848,109 @@ public class VideoClippingProperties {
}
}
public static class VisualAnalysis {
private String provider = "heuristic";
private String endpoint = "http://127.0.0.1:8091/v1/analyze-visuals";
@Min(1)
private long timeoutMs = 30000;
private boolean fallbackToHeuristic = true;
private final LocalCvWorker localCvWorker = new LocalCvWorker();
public String getProvider() {
return provider;
}
public void setProvider(String provider) {
this.provider = provider;
}
public String getEndpoint() {
return endpoint;
}
public void setEndpoint(String endpoint) {
this.endpoint = endpoint;
}
public long getTimeoutMs() {
return timeoutMs;
}
public void setTimeoutMs(long timeoutMs) {
this.timeoutMs = timeoutMs;
}
public boolean isFallbackToHeuristic() {
return fallbackToHeuristic;
}
public void setFallbackToHeuristic(boolean fallbackToHeuristic) {
this.fallbackToHeuristic = fallbackToHeuristic;
}
public LocalCvWorker getLocalCvWorker() {
return localCvWorker;
}
public static class LocalCvWorker {
private boolean autoStart = false;
private String script = "./tools/run_local_cv_worker.sh";
@Min(0)
private long startupWaitMs = 0;
private String healthPath = "/health";
@Min(1)
private long healthCheckIntervalMs = 1000;
public boolean isAutoStart() {
return autoStart;
}
public void setAutoStart(boolean autoStart) {
this.autoStart = autoStart;
}
public String getScript() {
return script;
}
public void setScript(String script) {
this.script = script;
}
public long getStartupWaitMs() {
return startupWaitMs;
}
public void setStartupWaitMs(long startupWaitMs) {
this.startupWaitMs = startupWaitMs;
}
public String getHealthPath() {
return healthPath;
}
public void setHealthPath(String healthPath) {
this.healthPath = healthPath;
}
public long getHealthCheckIntervalMs() {
return healthCheckIntervalMs;
}
public void setHealthCheckIntervalMs(long healthCheckIntervalMs) {
this.healthCheckIntervalMs = healthCheckIntervalMs;
}
}
}
public static class LocalDirector {
private boolean enabled = true;
@ -929,6 +1078,21 @@ public class VideoClippingProperties {
@Min(1000)
private long pollIntervalMs = 5000;
private boolean renderEnabled = true;
private boolean requireDirectorApproval = false;
private String approvalFileName = "approved.flag";
@Min(1)
private int maxHighlightsPerSource = 3;
@Min(1)
private double highlightMinDurationSeconds = 8.0;
@Min(1)
private double highlightMaxDurationSeconds = 35.0;
public boolean isEnabled() {
return enabled;
}
@ -976,6 +1140,54 @@ public class VideoClippingProperties {
public void setPollIntervalMs(long pollIntervalMs) {
this.pollIntervalMs = pollIntervalMs;
}
public boolean isRenderEnabled() {
return renderEnabled;
}
public void setRenderEnabled(boolean renderEnabled) {
this.renderEnabled = renderEnabled;
}
public boolean isRequireDirectorApproval() {
return requireDirectorApproval;
}
public void setRequireDirectorApproval(boolean requireDirectorApproval) {
this.requireDirectorApproval = requireDirectorApproval;
}
public String getApprovalFileName() {
return approvalFileName;
}
public void setApprovalFileName(String approvalFileName) {
this.approvalFileName = approvalFileName;
}
public int getMaxHighlightsPerSource() {
return maxHighlightsPerSource;
}
public void setMaxHighlightsPerSource(int maxHighlightsPerSource) {
this.maxHighlightsPerSource = maxHighlightsPerSource;
}
public double getHighlightMinDurationSeconds() {
return highlightMinDurationSeconds;
}
public void setHighlightMinDurationSeconds(double highlightMinDurationSeconds) {
this.highlightMinDurationSeconds = highlightMinDurationSeconds;
}
public double getHighlightMaxDurationSeconds() {
return highlightMaxDurationSeconds;
}
public void setHighlightMaxDurationSeconds(double highlightMaxDurationSeconds) {
this.highlightMaxDurationSeconds = highlightMaxDurationSeconds;
}
}
}

View File

@ -0,0 +1,10 @@
package org.example.videoclips.editing;
public record AudioSection(
String sectionId,
String kind,
double startSeconds,
double endSeconds,
double durationSeconds
) {
}

View File

@ -37,6 +37,7 @@ public class EditPlanInboxScanner {
private final EditProjectStore store;
private final EditProjectService projectService;
private final EditPlanValidator validator;
private final Optional<AssetGenerationStage> assetGenerationStage;
private final Optional<EditRenderer> renderer;
private final VoiceoverGenerator voiceoverGenerator;
private final AtomicBoolean scanning = new AtomicBoolean();
@ -49,9 +50,11 @@ public class EditPlanInboxScanner {
EditProjectService projectService,
EditPlanValidator validator,
VoiceoverGenerator voiceoverGenerator,
ObjectProvider<AssetGenerationStage> assetGenerationStageProvider,
ObjectProvider<EditRenderer> rendererProvider
) {
this(properties, objectMapper, store, projectService, validator, voiceoverGenerator,
assetGenerationStageProvider.getIfAvailable(),
rendererProvider.getIfAvailable());
}
@ -63,13 +66,13 @@ public class EditPlanInboxScanner {
EditPlanValidator validator,
EditRenderer renderer
) {
this(properties, objectMapper, store, projectService, validator, null, renderer);
this(properties, objectMapper, store, projectService, validator, null, null, renderer);
}
EditPlanInboxScanner(
VideoClippingProperties properties, ObjectMapper objectMapper, EditProjectStore store,
EditProjectService projectService, EditPlanValidator validator,
VoiceoverGenerator voiceoverGenerator, EditRenderer renderer
VoiceoverGenerator voiceoverGenerator, AssetGenerationStage assetGenerationStage, EditRenderer renderer
) {
this.projectRoot = Path.of(properties.getEditing().getProjectDirectory());
this.expectedPlanFileName = properties.getEditing().getLocalDirector().getExpectedPlanFileName();
@ -82,6 +85,7 @@ public class EditPlanInboxScanner {
this.validator = validator;
this.renderer = Optional.ofNullable(renderer);
this.voiceoverGenerator = voiceoverGenerator;
this.assetGenerationStage = Optional.ofNullable(assetGenerationStage);
}
@Scheduled(
@ -154,6 +158,11 @@ public class EditPlanInboxScanner {
}
store.writeJson(projectId, "edit-plan.json", normalized(plan));
AssetGenerationResult assetGenerationResult = assetGenerationStage
.map(stage -> stage.prepare(projectId, plan))
.orElse(new AssetGenerationResult(projectId, true, List.of(), java.time.Instant.now()));
log.info("event=edit_asset_generation_completed project_id={} item_count={} ready_for_render={}",
projectId, assetGenerationResult.items().size(), assetGenerationResult.readyForRender());
if (voiceoverGenerator != null && !plan.voiceover().isEmpty()) {
voiceoverGenerator.generateVoiceover(projectId, plan.voiceover());
}
@ -164,8 +173,11 @@ public class EditPlanInboxScanner {
projectId, plan.decisions().size(), autoRender, requireApprovalBeforeRender);
log.info("event=edit_plan_saved project_id={} decision_count={} source=inbox",
projectId, plan.decisions().size());
if (autoRender) {
if (autoRender && assetGenerationResult.readyForRender()) {
renderIfApproved(projectId, store.inboxDirectory(projectId));
} else if (autoRender) {
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
projectId);
}
}
@ -175,6 +187,16 @@ public class EditPlanInboxScanner {
projectId, inbox.resolve(approvalFileName));
return;
}
AssetGenerationStage stage = assetGenerationStage.orElse(null);
if (stage != null) {
EditPlan plan = store.readJson(projectId, "edit-plan.json", EditPlan.class);
AssetGenerationResult result = stage.prepare(projectId, plan);
if (!result.readyForRender()) {
log.info("event=edit_render_waiting_for_assets project_id={} blocking_assets_pending=true",
projectId);
return;
}
}
renderer.orElseThrow(() -> new IllegalStateException(
"Automatic rendering is enabled but no EditRenderer is available")).render(projectId);
}

View File

@ -80,9 +80,13 @@ public class EditPlanValidator {
validateAudioCues(projectId, plan);
validateVoiceover(plan);
validateOverlays(plan);
if (previousEnd > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS
|| Math.abs(plan.targetDurationSeconds() - project.targetDurationSeconds()) > EPSILON) {
reject("Edit plan duration exceeds or does not match the project target duration");
validateCinematicRichness(project, plan);
if (plan.targetDurationSeconds() <= 0
|| Math.abs(previousEnd - plan.targetDurationSeconds()) > TARGET_TOLERANCE_SECONDS) {
reject("Edit plan target duration must match the final timeline duration");
}
if (plan.targetDurationSeconds() > project.targetDurationSeconds() + TARGET_TOLERANCE_SECONDS) {
reject("Edit plan duration exceeds the project maximum target duration");
}
return plan;
}
@ -152,6 +156,42 @@ public class EditPlanValidator {
}
}
private void validateCinematicRichness(EditProject project, EditPlan plan) {
int cinematicDimensions = 0;
if (hasVisualTreatment(plan)) {
cinematicDimensions++;
}
if (project.musicEnabled() && hasAudioCueType(plan, "music")) {
cinematicDimensions++;
}
if (project.voiceoverEnabled() && !plan.voiceover().isEmpty()) {
cinematicDimensions++;
}
if (project.soundEffectsEnabled() && hasAudioCueType(plan, "sfx")) {
cinematicDimensions++;
}
if (!plan.overlays().isEmpty()) {
cinematicDimensions++;
}
if (cinematicDimensions < 2) {
reject("Edit plan is too bare; it must use at least two cinematic elements such as visual treatment, "
+ "music, voiceover, sound effects, or text overlays");
}
}
private boolean hasVisualTreatment(EditPlan plan) {
return plan.decisions().stream()
.map(EditDecision::visualTreatment)
.filter(value -> value != null && !value.isBlank() && !"none".equalsIgnoreCase(value))
.findFirst()
.isPresent();
}
private boolean hasAudioCueType(EditPlan plan, String type) {
return plan.audioCues().stream().anyMatch(cue -> type.equals(cue.type()));
}
private void validateTransition(String transition) {
if (transition == null || !TRANSITIONS.contains(transition)) {
reject("Unsupported transition: " + transition);

View File

@ -127,6 +127,7 @@ public class FfmpegEditRenderer implements EditRenderer {
runAndRecord(segmentCommand(clips.get(decision.clipId()).sourcePath(), decision, output), commands);
segments.add(output);
}
List<Path> renderedClips = publishRenderedClips(projectId, segments);
Path concatFile = work.resolve("concat.txt");
writeConcatFile(concatFile, segments);
Path timeline = work.resolve("timeline-video.mp4");
@ -152,10 +153,11 @@ public class FfmpegEditRenderer implements EditRenderer {
copy(videoTimeline, output);
}
double duration = plan.decisions().get(plan.decisions().size() - 1).timelineEndSeconds();
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, commands, assets);
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
RenderManifest manifest = new RenderManifest(projectId,
plan.decisions().stream().map(EditDecision::clipId).toList(), output.toString(), duration,
List.copyOf(commands), assets, Instant.now());
plan.decisions().stream().map(EditDecision::clipId).toList(),
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
assets, Instant.now());
store.writeJson(projectId, "qa-report.json", qaReport);
store.writeJson(projectId, "render-manifest.json", manifest);
projectService.updateProject(projectId, EditProjectStatus.RENDERED, Path.of(project.inputDirectory()), null);
@ -361,9 +363,11 @@ public class FfmpegEditRenderer implements EditRenderer {
}
RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
List<Path> renderedClips,
List<List<String>> commands, List<ResolvedEditAsset> assets) {
List<RenderQaCheck> checks = new ArrayList<>();
checks.add(outputExistsCheck(output));
checks.add(renderedClipsExistCheck(renderedClips));
checks.add(durationMatchesTimelineCheck(plan, duration));
checks.add(requiredAssetsResolvedCheck(plan, assets));
checks.add(overlaysSafeCheck(plan, duration));
@ -388,6 +392,14 @@ public class FfmpegEditRenderer implements EditRenderer {
exists ? "final output exists" : "final output is missing: " + output);
}
private RenderQaCheck renderedClipsExistCheck(List<Path> renderedClips) {
List<Path> missing = renderedClips.stream()
.filter(path -> !Files.isRegularFile(path))
.toList();
return new RenderQaCheck("rendered_clips_exist", missing.isEmpty(), "ERROR",
missing.isEmpty() ? "all rendered segment clips exist" : "missing rendered clips: " + missing);
}
private RenderQaCheck durationMatchesTimelineCheck(EditPlan plan, double duration) {
double expected = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
.timelineEndSeconds();
@ -518,6 +530,18 @@ public class FfmpegEditRenderer implements EditRenderer {
}
}
private List<Path> publishRenderedClips(String projectId, List<Path> segments) {
Path publishedDirectory = store.projectDirectory(projectId).resolve("rendered-clips");
createDirectory(publishedDirectory);
List<Path> published = new ArrayList<>();
for (int index = 0; index < segments.size(); index++) {
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
copy(segments.get(index), target);
published.add(target);
}
return List.copyOf(published);
}
private long fileSize(Path file) {
try {
return Files.size(file);

View File

@ -0,0 +1,102 @@
package org.example.videoclips.editing;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HeuristicVisualAnalysisProvider implements VisualAnalysisProvider {
@Override
public SourceVisualAnalysis analyze(
ClipAnalysis source,
List<String> thumbnails,
List<ShotSegment> shotSegments
) {
double blurScore = scoreOrDefault(source.sharpnessScore(), 0.5);
double exposureScore = scoreOrDefault(source.brightnessScore(), 0.5);
double motionScore = scoreOrDefault(source.motionScore(), motionFromShots(source.durationSeconds(), shotSegments));
return new SourceVisualAnalysis(
source.clipId(),
blurScore,
exposureScore,
motionScore,
compositionScore(source, thumbnails),
"unknown_without_face_detector",
labels(source),
representativeThumbnails(thumbnails),
"metadata_thumbnail_scene_heuristic"
);
}
private double scoreOrDefault(double value, double fallback) {
if (value > 0) {
return round(clamp(value));
}
return round(clamp(fallback));
}
private double motionFromShots(double durationSeconds, List<ShotSegment> shotSegments) {
if (durationSeconds <= 0 || shotSegments == null || shotSegments.isEmpty()) {
return 0.3;
}
double cutsPerMinute = Math.max(0, shotSegments.size() - 1) / durationSeconds * 60.0;
return clamp(0.25 + cutsPerMinute / 20.0);
}
private double compositionScore(ClipAnalysis source, List<String> thumbnails) {
double aspectRatio = source.height() == 0 ? 0 : source.width() / (double) source.height();
double aspectScore = aspectRatio >= 1.70 && aspectRatio <= 1.90 ? 0.8 : 0.55;
double thumbnailScore = thumbnails == null || thumbnails.isEmpty() ? 0.45 : 0.7;
return round((aspectScore + thumbnailScore) / 2.0);
}
private List<VisualObjectLabel> labels(ClipAnalysis source) {
String haystack = (source.clipId() + " " + source.sourcePath()).toLowerCase(Locale.ROOT);
List<VisualObjectLabel> labels = new ArrayList<>();
addIfContains(labels, haystack, "porsche", "porsche", 0.82);
addIfContains(labels, haystack, "car", "car", 0.74);
addIfContains(labels, haystack, "drive", "car", 0.62);
addIfContains(labels, haystack, "food", "food", 0.72);
addIfContains(labels, haystack, "recipe", "food", 0.66);
addIfContains(labels, haystack, "family", "person", 0.62);
addIfContains(labels, haystack, "birthday", "person", 0.58);
if (labels.isEmpty()) {
labels.add(new VisualObjectLabel("unknown", 0.2, "metadata_heuristic"));
}
return labels.stream().distinct().toList();
}
private void addIfContains(List<VisualObjectLabel> labels, String haystack, String needle, String label,
double confidence) {
if (haystack.contains(needle)) {
labels.add(new VisualObjectLabel(label, confidence, "metadata_heuristic"));
}
}
private List<String> representativeThumbnails(List<String> thumbnails) {
if (thumbnails == null || thumbnails.isEmpty()) {
return List.of();
}
if (thumbnails.size() <= 3) {
return List.copyOf(thumbnails);
}
return List.of(
thumbnails.get(0),
thumbnails.get(thumbnails.size() / 2),
thumbnails.get(thumbnails.size() - 1)
);
}
private double clamp(double value) {
return Math.max(0, Math.min(1, value));
}
private double round(double value) {
return Math.round(value * 1000.0) / 1000.0;
}
}

View File

@ -0,0 +1,14 @@
package org.example.videoclips.editing;
public record HighlightAssetRequest(
String projectId,
String highlightId,
String type,
String assetKey,
String targetPath,
String requestPath,
String notes,
double durationSeconds,
boolean blocking
) {
}

View File

@ -0,0 +1,338 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightDirectorFlowService {
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorFlowService.class);
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final ObjectMapper objectMapper;
private final HighlightProjectStore store;
private final HighlightVisualEffectsStage visualEffectsStage;
private final HighlightAssetPreparationService assetPreparationService;
private final HighlightLocalAssetWorker assetWorker;
private final HighlightFfmpegRenderer renderer;
public HighlightDirectorFlowService(VideoClippingProperties properties, ObjectMapper objectMapper,
HighlightProjectStore store,
HighlightVisualEffectsStage visualEffectsStage,
HighlightAssetPreparationService assetPreparationService,
HighlightLocalAssetWorker assetWorker,
HighlightFfmpegRenderer renderer) {
this.properties = properties.getEditing().getHighlightScheduler();
this.objectMapper = objectMapper;
this.store = store;
this.visualEffectsStage = visualEffectsStage;
this.assetPreparationService = assetPreparationService;
this.assetWorker = assetWorker;
this.renderer = renderer;
}
public HighlightFlowResult process(String projectId, long scanId) {
long startedAt = System.nanoTime();
String flowId = projectId + ":" + scanId;
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
Path planFile = store.directorDirectory(projectId).resolve("edit-plan.json");
if (!Files.isRegularFile(planFile)) {
return HighlightFlowResult.skipped(projectId, flowId, "missing_director_plan");
}
if (properties.isRequireDirectorApproval()
&& !Files.isRegularFile(store.directorDirectory(projectId).resolve(properties.getApprovalFileName()))) {
log.info("event=highlight_flow_waiting_for_approval flow_id={} project_id={} approval_file={}",
flowId, projectId, store.directorDirectory(projectId).resolve(properties.getApprovalFileName()));
return HighlightFlowResult.skipped(projectId, flowId, "approval_missing");
}
HighlightDirectorPlan plan = readPlan(planFile);
if (plan.highlights().isEmpty()) {
markFailed(project, "No highlights were provided by the director plan");
throw new IllegalStateException("Director plan contains no highlights");
}
markStatus(project, HighlightProjectStatus.RENDERING, null);
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
List<Path> finalOutputs = new ArrayList<>();
List<String> renderedHighlightIds = new ArrayList<>();
ContentCategory category = contentCategory(plan.contentCategory());
int limit = Math.min(properties.getMaxHighlightsPerSource(), plan.highlights().size());
for (int index = 0; index < limit; index++) {
HighlightDirectorPlan.HighlightItem highlight = plan.highlights().get(index);
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
writeStoryboard(highlightDirectory, plan, highlight);
visualEffectsStage.create(projectId, highlight, category);
EditPlan editPlan = buildEditPlan(project, plan, highlight, index);
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/edit-plan.json", editPlan);
HighlightAssetPreparationService.HighlightAssetPreparationResult assets = assetPreparationService
.prepare(projectId, project, highlight, category);
HighlightLocalAssetWorker.HighlightAssetWorkerResult assetWorkerResult = assetWorker.process(projectId,
highlight, category);
log.info("event=highlight_assets_prepared flow_id={} project_id={} highlight_id={} resolved={} requests={}",
flowId, projectId, highlight.highlightId(), assets.resolvedAssets().size(),
assets.requestFiles().size());
log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}",
flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(),
assetWorkerResult.pendingRequests().size());
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
finalOutputs.add(result.finalOutput());
renderedHighlightIds.add(result.highlightId());
}
Path projectOutput = store.projectDirectory(projectId).resolve("final.mp4");
concatFinalOutputs(projectOutput, finalOutputs);
RenderManifest manifest = new RenderManifest(projectId, renderedHighlightIds, finalOutputs.stream()
.map(Path::toString).toList(), projectOutput.toString(),
finalOutputs.stream().mapToDouble(this::probeDuration).sum(), List.of(), List.of(), Instant.now());
store.writeJson(projectId, "render-manifest.json", manifest);
markStatus(project, HighlightProjectStatus.RENDERED, null);
log.info("event=highlight_flow_completed flow_id={} scan_id={} project_id={} final_outputs={} elapsed_ms={}",
flowId, scanId, projectId, finalOutputs, (System.nanoTime() - startedAt) / 1_000_000);
return HighlightFlowResult.rendered(projectId, flowId, finalOutputs, projectOutput);
}
private HighlightDirectorPlan readPlan(Path planFile) {
try {
return objectMapper.readValue(planFile.toFile(), HighlightDirectorPlan.class);
} catch (IOException ex) {
throw new IllegalStateException("Unable to read highlight director plan: " + planFile, ex);
}
}
private EditPlan buildEditPlan(HighlightProject project, HighlightDirectorPlan plan,
HighlightDirectorPlan.HighlightItem highlight, int index) {
HighlightSourceAnalysis analysis = store.readJson(project.id(), "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
double sourceDuration = Math.max(0.01, highlight.sourceEndSeconds() - highlight.sourceStartSeconds());
double requestedDuration = highlight.targetDurationSeconds() > 0 ? highlight.targetDurationSeconds()
: sourceDuration;
double targetDuration = Math.max(properties.getHighlightMinDurationSeconds(),
Math.min(properties.getHighlightMaxDurationSeconds(), requestedDuration));
double playbackSpeed = Math.max(0.25, Math.min(4.0, sourceDuration / targetDuration));
String clipId = analysis.source().clipId();
EditDecision decision = new EditDecision(
clipId,
highlight.sourceStartSeconds(),
highlight.sourceEndSeconds(),
0.0,
targetDuration,
transitionIn(highlight, index),
transitionOut(highlight, index, plan.highlights().size()),
playbackSpeed,
highlight.visualTreatment(),
highlight.renderNotes()
);
List<AudioCue> audioCues = new ArrayList<>();
if (highlight.musicDirection() != null && !highlight.musicDirection().isBlank()) {
audioCues.add(new AudioCue("music", safeKey("music", highlight.musicDirection()), 0.0, targetDuration,
-10.0, highlight.musicDirection()));
}
if (highlight.sfxDirection() != null && !highlight.sfxDirection().isBlank()) {
audioCues.add(new AudioCue("sfx", safeKey("sfx", highlight.storyPurpose(), highlight.highlightId()),
Math.max(0.0, targetDuration - 0.8), targetDuration, -6.0, highlight.sfxDirection()));
}
List<VoiceoverLine> voiceover = new ArrayList<>();
if (!highlight.voiceover().isEmpty()) {
double step = targetDuration / highlight.voiceover().size();
for (int i = 0; i < highlight.voiceover().size(); i++) {
double start = Math.min(targetDuration, i * step);
double end = Math.min(targetDuration, start + Math.max(1.0, step));
voiceover.add(new VoiceoverLine(highlight.voiceover().get(i), start, end, "cinematic_narration"));
}
}
List<TextOverlay> overlays = new ArrayList<>();
if (!highlight.overlays().isEmpty()) {
double step = targetDuration / highlight.overlays().size();
for (int i = 0; i < highlight.overlays().size(); i++) {
double start = Math.min(targetDuration, i * step);
double end = Math.min(targetDuration, start + Math.max(1.5, step * 0.75));
overlays.add(new TextOverlay(highlight.overlays().get(i), start, end,
i % 2 == 0 ? "lower_center_safe" : "center_safe",
"fade_slide_up", highlight.renderNotes()));
}
}
return new EditPlan(project.id(),
safeKey("style", plan.contentCategory(), highlight.storyPurpose()),
targetDuration,
List.of(decision),
audioCues,
voiceover,
overlays,
"mp4-h264-aac-1080p",
highlight.title() + " / " + highlight.storyPurpose());
}
private String transitionIn(HighlightDirectorPlan.HighlightItem highlight, int index) {
if (index == 0 || highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
return "fade-in";
}
return "cut";
}
private String transitionOut(HighlightDirectorPlan.HighlightItem highlight, int index, int total) {
if (index + 1 == total || highlight.storyPurpose() != null && highlight.storyPurpose().contains("hero")) {
return "fade-out";
}
return "cut";
}
private void writeStoryboard(Path highlightDirectory, HighlightDirectorPlan plan,
HighlightDirectorPlan.HighlightItem highlight) {
String storyboard = """
# Storyboard
Project: `%s`
Highlight: `%s`
Title: `%s`
Purpose: `%s`
Music: `%s`
SFX: `%s`
Voiceover:
%s
Overlays:
%s
""".formatted(plan.projectId(), highlight.highlightId(), highlight.title(), highlight.storyPurpose(),
highlight.musicDirection(), highlight.sfxDirection(), String.join(System.lineSeparator(),
highlight.voiceover()), String.join(System.lineSeparator(), highlight.overlays()));
try {
Files.createDirectories(highlightDirectory);
Files.writeString(highlightDirectory.resolve("storyboard.md"), storyboard, StandardCharsets.UTF_8);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write storyboard for highlight: " + highlight.highlightId(), ex);
}
}
private void concatFinalOutputs(Path output, List<Path> sources) {
if (sources.isEmpty()) {
throw new IllegalStateException("No rendered highlight outputs were produced");
}
if (sources.size() == 1) {
copy(sources.get(0), output);
return;
}
Path work = output.getParent().resolve("project-render-work");
try {
Files.createDirectories(work);
} catch (IOException ex) {
throw new IllegalStateException("Unable to create highlight project render work directory", ex);
}
Path concat = work.resolve("concat.txt");
String content = sources.stream()
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
.reduce("", (a, b) -> a + b + System.lineSeparator());
try {
Files.writeString(concat, content, StandardCharsets.UTF_8);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write project concat file", ex);
}
run(List.of("ffmpeg", "-hide_banner", "-y", "-f", "concat", "-safe", "0", "-i", concat.toString(),
"-c", "copy", output.toString()));
}
private double probeDuration(Path video) {
try {
Process process = new ProcessBuilder("ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1", video.toString()).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() != 0) {
return 0.0;
}
return Double.parseDouble(output.trim());
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
return 0.0;
}
}
private void run(List<String> command) {
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() != 0) {
throw new IllegalStateException(output);
}
} catch (IOException ex) {
throw new IllegalStateException("Unable to run highlight concat command", ex);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while concatenating highlight outputs", ex);
}
}
private void copy(Path source, Path target) {
try {
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException ex) {
throw new IllegalStateException("Unable to publish highlight project output", ex);
}
}
private ContentCategory contentCategory(String value) {
if (value == null || value.isBlank()) {
return ContentCategory.GENERIC_VLOG;
}
try {
return ContentCategory.valueOf(value.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException ex) {
return ContentCategory.GENERIC_VLOG;
}
}
private String safeKey(String prefix, String... values) {
String value = String.join("_", values == null ? new String[0] : values);
String normalized = value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]+", "_")
.replaceAll("_+", "_")
.replaceAll("^_|_$", "");
if (normalized.isBlank()) {
return prefix + "_1";
}
return (prefix + "_" + normalized).substring(0, Math.min(120, (prefix + "_" + normalized).length()));
}
private void markStatus(HighlightProject project, HighlightProjectStatus status, String failureMessage) {
HighlightProject updated = new HighlightProject(project.id(), project.name(), status,
project.sourceVideoFileName(), project.projectDirectory(), project.createdAt(), Instant.now(),
failureMessage);
store.writeJson(project.id(), "project.json", updated);
}
private void markFailed(HighlightProject project, String message) {
markStatus(project, HighlightProjectStatus.FAILED, message);
}
public record HighlightFlowResult(
String projectId,
String flowId,
List<Path> finalOutputs,
Path projectFinalOutput,
String reason
) {
public HighlightFlowResult {
finalOutputs = finalOutputs == null ? List.of() : List.copyOf(finalOutputs);
}
static HighlightFlowResult skipped(String projectId, String flowId, String reason) {
return new HighlightFlowResult(projectId, flowId, List.of(), null, reason);
}
static HighlightFlowResult rendered(String projectId, String flowId, List<Path> outputs, Path finalOutput) {
return new HighlightFlowResult(projectId, flowId, outputs, finalOutput, null);
}
}
}

View File

@ -0,0 +1,36 @@
package org.example.videoclips.editing;
import java.util.List;
public record HighlightDirectorPlan(
String projectId,
String sourceVideoFileName,
String contentCategory,
List<HighlightItem> highlights,
String summary
) {
public HighlightDirectorPlan {
highlights = highlights == null ? List.of() : List.copyOf(highlights);
}
public record HighlightItem(
String highlightId,
String candidateId,
String title,
double sourceStartSeconds,
double sourceEndSeconds,
double targetDurationSeconds,
String storyPurpose,
String visualTreatment,
String musicDirection,
String sfxDirection,
List<String> voiceover,
List<String> overlays,
String renderNotes
) {
public HighlightItem {
voiceover = voiceover == null ? List.of() : List.copyOf(voiceover);
overlays = overlays == null ? List.of() : List.copyOf(overlays);
}
}
}

View File

@ -0,0 +1,87 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Comparator;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Stream;
@Component
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
public class HighlightDirectorPlanScanner {
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPlanScanner.class);
private final VideoClippingProperties properties;
private final VideoClippingProperties.Editing.HighlightScheduler highlightScheduler;
private final HighlightDirectorFlowService flowService;
private final AtomicBoolean scanning = new AtomicBoolean(false);
private final AtomicLong scanSequence = new AtomicLong();
public HighlightDirectorPlanScanner(VideoClippingProperties properties, HighlightDirectorFlowService flowService) {
this.properties = properties;
this.highlightScheduler = properties.getEditing().getHighlightScheduler();
this.flowService = flowService;
}
@EventListener(ApplicationReadyEvent.class)
public void logStartup() {
log.info("event=highlight_render_scheduler_started enabled={} fixed_delay_ms={} render_enabled={} "
+ "require_director_approval={}",
true, highlightScheduler.getPollIntervalMs(), highlightScheduler.isRenderEnabled(),
highlightScheduler.isRequireDirectorApproval());
}
@Scheduled(
initialDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}",
fixedDelayString = "${video-clipping.editing.highlight-scheduler.poll-interval-ms:5000}"
)
public void scan() {
if (!highlightScheduler.isRenderEnabled()) {
return;
}
if (!scanning.compareAndSet(false, true)) {
log.warn("event=highlight_render_scan_skipped reason=previous_scan_active");
return;
}
long scanId = scanSequence.incrementAndGet();
long startedAt = System.nanoTime();
try {
findNextRenderableProject().ifPresent(projectId -> flowService.process(projectId, scanId));
} finally {
scanning.set(false);
log.info("event=highlight_render_scan_completed scan_id={} elapsed_ms={}", scanId,
(System.nanoTime() - startedAt) / 1_000_000);
}
}
Optional<String> findNextRenderableProject() {
Path projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory()).normalize();
if (!Files.isDirectory(projectRoot)) {
return Optional.empty();
}
try (Stream<Path> projects = Files.list(projectRoot)) {
return projects.filter(Files::isDirectory)
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
.filter(path -> Files.isRegularFile(path.resolve("director").resolve("edit-plan.json")))
.filter(path -> !Files.isRegularFile(path.resolve("final.mp4")))
.map(path -> path.getFileName().toString())
.findFirst();
} catch (IOException ex) {
throw new IllegalStateException("Unable to scan highlight project directories: " + projectRoot, ex);
}
}
}

View File

@ -0,0 +1,70 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Comparator;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.stream.Stream;
@Component
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
+ "${video-clipping.editing.highlight-scheduler.enabled:true}")
public class HighlightDirectorPromptBackfill {
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptBackfill.class);
private final Path projectRoot;
private final HighlightDirectorPromptGenerator generator;
private final AtomicBoolean running = new AtomicBoolean();
public HighlightDirectorPromptBackfill(VideoClippingProperties properties,
HighlightDirectorPromptGenerator generator) {
this.projectRoot = Path.of(properties.getEditing().getHighlightProjectDirectory());
this.generator = generator;
}
@EventListener(ApplicationReadyEvent.class)
public void backfill() {
if (!running.compareAndSet(false, true)) {
return;
}
try {
if (!Files.isDirectory(projectRoot)) {
return;
}
try (Stream<Path> projects = Files.list(projectRoot)) {
projects.filter(Files::isDirectory)
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
.filter(this::needsPrompt)
.forEach(this::generatePrompt);
} catch (IOException ex) {
throw new IllegalStateException("Unable to scan highlight projects for director prompt backfill: "
+ projectRoot, ex);
}
} finally {
running.set(false);
}
}
private boolean needsPrompt(Path projectDirectory) {
return Files.isRegularFile(projectDirectory.resolve("project.json"))
&& Files.isRegularFile(projectDirectory.resolve("analysis/source-analysis.json"))
&& !Files.isRegularFile(projectDirectory.resolve("director/director-prompt.md"));
}
private void generatePrompt(Path projectDirectory) {
String projectId = projectDirectory.getFileName().toString();
log.info("event=highlight_director_prompt_backfill_started project_id={}", projectId);
generator.generate(projectId);
log.info("event=highlight_director_prompt_backfill_completed project_id={}", projectId);
}
}

View File

@ -0,0 +1,246 @@
package org.example.videoclips.editing;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightDirectorPromptGenerator {
private static final Logger log = LoggerFactory.getLogger(HighlightDirectorPromptGenerator.class);
private static final String BRIEF_FILE_NAME = "director-brief.md";
private static final String PROMPT_FILE_NAME = "director-prompt.md";
private static final String PLAN_FILE_NAME = "edit-plan.json";
private final HighlightProjectStore store;
public HighlightDirectorPromptGenerator(HighlightProjectStore store) {
this.store = store;
}
public DirectorPromptFiles generate(String projectId) {
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
HighlightSourceAnalysis analysis = store.readJson(projectId, "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
CinematicHighlightAnalysis cinematic = readCinematicAnalysis(projectId);
List<HighlightCandidate> candidates = readCandidates(projectId);
Path directorDirectory = store.directorDirectory(projectId);
Path promptPath = directorDirectory.resolve(PROMPT_FILE_NAME);
Path briefPath = directorDirectory.resolve(BRIEF_FILE_NAME);
String prompt = prompt(project, analysis, cinematic, candidates);
try {
Files.writeString(promptPath, prompt);
Files.writeString(briefPath, brief(projectId));
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight director files for project: " + projectId, ex);
}
if (project.status() == HighlightProjectStatus.CREATED
|| project.status() == HighlightProjectStatus.ANALYZING) {
HighlightProject updated = new HighlightProject(project.id(), project.name(),
HighlightProjectStatus.WAITING_FOR_DIRECTOR, project.sourceVideoFileName(),
project.projectDirectory(), project.createdAt(), Instant.now(), project.failureMessage());
store.writeJson(projectId, "project.json", updated);
}
log.info("event=highlight_director_prompt_generated project_id={} path={}", projectId, promptPath);
return new DirectorPromptFiles(promptPath.toString(), briefPath.toString());
}
String prompt(
HighlightProject project,
HighlightSourceAnalysis analysis,
CinematicHighlightAnalysis cinematic,
List<HighlightCandidate> candidates
) {
String category = cinematic == null ? "generic_vlog" : cinematic.category().name().toLowerCase();
String categoryDirection = cinematic == null ? genericDirection() : categoryDirection(cinematic.category());
String candidateBrief = candidates.isEmpty() ? "No highlight candidates were found."
: candidates.stream().map(this::candidateBrief).reduce((a, b) -> a + "\n" + b).orElse("");
return """
# Local Highlight Director Brief
You are the director for a cinematic highlight project.
Work only inside this project folder:
- `analysis/source-analysis.json`
- `analysis/ffprobe.json`
- `analysis/scene-segments.json`
- `analysis/audio-analysis.json`
- `analysis/visual-analysis.json`
- `analysis/category.json`
- `analysis/highlight-candidates.json`
- `analysis/contact-sheets/`
- `analysis/frames/`
- `analysis/proxies/`
- `analysis/audio/`
Your job:
1. Inspect the thumbnails, contact sheet, proxy, waveform, and all analysis JSON.
2. Choose the strongest highlight moments.
3. Create a cinematic highlight plan that can later be rendered.
4. Write strict JSON only to `director/%s`.
Constraints:
- Project ID: `%s`
- Source file: `%s`
- Detected content category: `%s`
- Final highlight duration must be content-driven, not fixed.
- Keep the edit cinematic, premium, and grounded in the visible footage.
- Use music, SFX, voiceover, overlays, and visual treatment only when they fit the footage.
- Do not invent clips, timestamps, or facts.
- Do not render video.
- Do not modify source media or analysis artifacts.
- Return strict JSON only. No Markdown fences.
Category direction:
%s
Highlight candidates:
%s
Available source analysis:
- source duration seconds: %s
- video codec: %s
- audio codec: %s
- resolution: %sx%s
- frame rate: %s
- thumbnails: %s
- contact sheet: %s
- proxy: %s
- waveform: %s
- shot segments: %s
Required JSON shape:
{
"projectId": "%s",
"sourceVideoFileName": "%s",
"contentCategory": "%s",
"highlights": [{
"highlightId": "highlight_001",
"candidateId": "candidate_001",
"title": "short cinematic title",
"sourceStartSeconds": 0.0,
"sourceEndSeconds": 12.5,
"targetDurationSeconds": 12.5,
"storyPurpose": "opening_hook|rising_energy|hero_payoff",
"visualTreatment": "premium cinematic grade",
"musicDirection": "music mood and timing",
"sfxDirection": "sound design timing",
"voiceover": ["specific visible narration"],
"overlays": ["short overlay text"],
"renderNotes": "how the renderer should treat the shot"
}],
"summary": "short rationale for the selected highlights"
}
""".formatted(
PLAN_FILE_NAME, project.id(), project.sourceVideoFileName(), category,
categoryDirection, candidateBrief, analysis.source().durationSeconds(), analysis.source().videoCodec(),
value(analysis.source().audioCodec()), analysis.source().width(), analysis.source().height(),
analysis.source().frameRate(), references("thumbnails", analysis.thumbnails()), analysis.contactSheet(),
value(analysis.proxyPath()), value(analysis.waveformPath()), analysis.shotSegments(), project.id(),
project.sourceVideoFileName(), category);
}
private CinematicHighlightAnalysis readCinematicAnalysis(String projectId) {
Path file = store.projectDirectory(projectId).resolve("analysis/category.json");
if (!Files.isRegularFile(file)) {
return null;
}
return store.readJson(projectId, "analysis/category.json", CinematicHighlightAnalysis.class);
}
private List<HighlightCandidate> readCandidates(String projectId) {
Path file = store.projectDirectory(projectId).resolve("analysis/highlight-candidates.json");
if (!Files.isRegularFile(file)) {
return List.of();
}
HighlightCandidate[] candidates = store.readJson(projectId, "analysis/highlight-candidates.json",
HighlightCandidate[].class);
return List.copyOf(Arrays.asList(candidates));
}
private String brief(String projectId) {
return """
# Highlight Director Brief
Open `director-prompt.md` in Codex, Claude, or another filesystem-capable AI agent.
Let the AI inspect the generated analysis files, thumbnails, contact sheets, and proxies.
The agent must write strict JSON to `director/edit-plan.json`.
Project: `%s`
""".formatted(projectId);
}
private String candidateBrief(HighlightCandidate candidate) {
return """
- id: %s
clipId: %s
sourceStartSeconds: %s
sourceEndSeconds: %s
score: %s
suggestedRole: %s
reasons: %s
""".formatted(candidate.id(), candidate.clipId(), candidate.sourceStartSeconds(),
candidate.sourceEndSeconds(), candidate.score(), candidate.suggestedRole(), candidate.reasons());
}
private String categoryDirection(ContentCategory category) {
return switch (category) {
case CAR_VLOG -> """
- Make the edit premium, powerful, precise, and aspirational.
- Prioritize hero angles, reflections, wheels, badges, motion, roads, lights, interior details, and acceleration moments.
- Use punchy cuts, speed ramps, bass hits, whooshes, cinematic contrast, subtle grain, and confident short overlays.
- Keep voiceover believable and grounded in what is visible.
""";
case FOOD_VLOG -> """
- Make the edit sensory, warm, rhythmic, and appetizing.
- Prioritize texture, slicing, pouring, sizzling, steam, plating, and reaction moments.
- Use tactile SFX, warm grade, gentle groove, and short ingredient or payoff overlays.
- Keep voiceover specific to visible food actions and results.
""";
case FAMILY_VLOG -> """
- Make the edit warm, emotional, human, and memory-focused.
- Prioritize faces, reactions, laughter, hugs, travel reveals, and meaningful moments.
- Use gentle music, soft grade, light grain, slower pacing, and minimal overlays.
- Keep voiceover reflective and personal, not promotional.
""";
case GENERIC_VLOG -> genericDirection();
};
}
private String genericDirection() {
return """
- Build a clear mini-story from the strongest moments.
- Prioritize visual novelty, clean composition, usable audio, expressive reactions, and strong scene changes.
- Use tasteful music, limited SFX, cinematic grade, and overlays only when they clarify the story.
- Keep voiceover grounded in what the footage actually shows.
""";
}
private String references(String directory, List<String> paths) {
if (paths == null || paths.isEmpty()) {
return "[]";
}
return paths.stream().map(path -> directory + "/" + Path.of(path).getFileName()).toList().toString();
}
private String value(String value) {
return value == null || value.isBlank() ? "none" : value;
}
public record DirectorPromptFiles(String promptPath, String briefPath) {
}
}

View File

@ -0,0 +1,460 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightFfmpegRenderer {
private final VideoClippingProperties.Editing properties;
private final HighlightProjectStore store;
private final EditAssetProvider assetProvider;
private final EditObservability observability;
private final ProcessExecutor executor;
public HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
EditAssetProvider assetProvider, EditObservability observability) {
this(properties, store, assetProvider, observability, HighlightFfmpegRenderer::execute);
}
HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
EditAssetProvider assetProvider, EditObservability observability,
ProcessExecutor executor) {
this.properties = properties.getEditing();
this.store = store;
this.assetProvider = assetProvider;
this.observability = observability;
this.executor = executor;
}
public HighlightRenderResult render(String projectId, HighlightDirectorPlan.HighlightItem highlight,
EditPlan plan) {
long startedAt = System.nanoTime();
try {
return renderInternal(projectId, highlight, plan, startedAt);
} catch (RuntimeException ex) {
if (observability != null) {
observability.renderFailed(projectId, System.nanoTime() - startedAt, ex);
}
throw ex;
}
}
private HighlightRenderResult renderInternal(String projectId, HighlightDirectorPlan.HighlightItem highlight,
EditPlan plan, long startedAt) {
Path projectDirectory = store.projectDirectory(projectId);
Path source = projectDirectory.resolve("source").resolve(store.readJson(projectId, "project.json",
HighlightProject.class).sourceVideoFileName());
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
Path work = highlightDirectory.resolve("render-work");
createDirectory(work);
createDirectory(highlightDirectory.resolve("rendered-clips"));
if (observability != null) {
observability.renderStarted(projectId, plan.decisions().size());
}
log(projectId, highlight.highlightId(), "highlight_render_started", "source", source.toString());
Map<String, ClipAnalysis> clips = Map.of(
clipIdFromFile(source.getFileName().toString()),
new ClipAnalysis(clipIdFromFile(source.getFileName().toString()), source.toString(),
plan.targetDurationSeconds(), "hevc", "aac", 0, 0, 0, List.of(), null, null, 0, 0, 0)
);
List<List<String>> commands = new ArrayList<>();
List<Path> segments = new ArrayList<>();
for (int index = 0; index < plan.decisions().size(); index++) {
EditDecision decision = plan.decisions().get(index);
Path output = work.resolve("segment_%04d.mp4".formatted(index + 1));
log(projectId, highlight.highlightId(), "highlight_render_segment_started",
"segment", Integer.toString(index + 1),
"source_start", Double.toString(decision.sourceStartSeconds()),
"source_end", Double.toString(decision.sourceEndSeconds()));
run(segmentCommand(source.toString(), decision, output), commands);
segments.add(output);
log(projectId, highlight.highlightId(), "highlight_render_segment_completed",
"segment", Integer.toString(index + 1));
}
List<Path> renderedClips = publishRenderedClips(highlightDirectory, segments);
Path concatFile = work.resolve("concat.txt");
writeConcatFile(concatFile, segments);
log(projectId, highlight.highlightId(), "highlight_render_concat_started",
"segments", Integer.toString(segments.size()));
Path timeline = work.resolve("timeline-video.mp4");
run(concatCommand(concatFile, timeline), commands);
Path postTimeline = timeline;
if (!plan.overlays().isEmpty()) {
postTimeline = work.resolve("timeline-overlays.mp4");
log(projectId, highlight.highlightId(), "highlight_render_effects_started",
"overlays", Integer.toString(plan.overlays().size()),
"treatment", plan.decisions().isEmpty() ? "none" : plan.decisions().get(0).visualTreatment());
run(overlayCommand(timeline, plan.overlays(), postTimeline), commands);
}
Path output = highlightDirectory.resolve("final.mp4");
Path preview = highlightDirectory.resolve("preview.mp4");
Path audioDirectory = highlightDirectory.resolve("assets");
List<ResolvedEditAsset> assets = resolvedAssets(plan, audioDirectory);
Path music = audioDirectory.resolve("music").resolve("music.wav");
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
List<SfxInput> sfx = plan.audioCues().stream()
.filter(cue -> "sfx".equals(cue.type()))
.map(cue -> new SfxInput(audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav"), cue))
.toList();
log(projectId, highlight.highlightId(), "highlight_render_audio_started",
"music", Boolean.toString(Files.isRegularFile(music)),
"sfx", Integer.toString(sfx.size()),
"voiceover", Boolean.toString(Files.isRegularFile(voiceover)));
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) {
run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null,
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands);
} else {
copy(postTimeline, output);
}
run(previewCommand(output, preview), commands);
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
.timelineEndSeconds();
RenderManifest manifest = new RenderManifest(projectId,
plan.decisions().stream().map(EditDecision::clipId).toList(),
renderedClips.stream().map(Path::toString).toList(), output.toString(), duration, List.copyOf(commands),
assets, Instant.now());
RenderQaReport qaReport = buildQaReport(projectId, plan, output, duration, renderedClips, commands, assets);
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/render-manifest.json", manifest);
store.writeJson(projectId, "highlights/" + highlight.highlightId() + "/qa-report.json", qaReport);
log(projectId, highlight.highlightId(), "highlight_render_completed",
"output", output.toString(),
"duration_seconds", Double.toString(duration),
"size_bytes", Long.toString(fileSize(output)),
"elapsed_ms", Long.toString((System.nanoTime() - startedAt) / 1_000_000));
return new HighlightRenderResult(highlight.highlightId(), output, preview, renderedClips, manifest, qaReport);
}
private List<ResolvedEditAsset> resolvedAssets(EditPlan plan, Path audioDirectory) {
List<ResolvedEditAsset> assets = new ArrayList<>();
Path music = audioDirectory.resolve("music").resolve("music.wav");
if (Files.isRegularFile(music)) {
assets.add(new ResolvedEditAsset(EditAssetType.MUSIC, "music", music.toString(),
"project-local", "highlight-assets"));
}
Path voiceover = audioDirectory.resolve("voiceover").resolve("voiceover.wav");
if (Files.isRegularFile(voiceover)) {
assets.add(new ResolvedEditAsset(EditAssetType.VOICEOVER, "voiceover", voiceover.toString(),
"project-local", "highlight-assets"));
}
for (AudioCue cue : plan.audioCues()) {
if ("sfx".equals(cue.type())) {
Path sfx = audioDirectory.resolve("sfx").resolve(cue.assetKey() + ".wav");
if (Files.isRegularFile(sfx)) {
assets.add(new ResolvedEditAsset(EditAssetType.SFX, cue.assetKey(), sfx.toString(),
"project-local", "highlight-assets"));
}
}
}
return List.copyOf(assets);
}
List<String> segmentCommand(String source, EditDecision decision, Path output) {
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
StringBuilder filter = new StringBuilder();
filter.append(dynamicCropFilter(decision.visualTreatment()));
filter.append("scale=%d:%d:force_original_aspect_ratio=decrease,".formatted(
properties.getOutputWidth(), properties.getOutputHeight())
+ "pad=%d:%d:(ow-iw)/2:(oh-ih)/2,format=yuv420p".formatted(
properties.getOutputWidth(), properties.getOutputHeight()));
filter.append(",setpts=PTS/").append(decision.playbackSpeed());
filter.append(cinematicVisualFilter(decision.visualTreatment()));
double fadeDuration = Math.min(0.5, outputDuration / 2);
if ("fade-in".equals(decision.transitionIn()) || "crossfade".equals(decision.transitionIn())) {
filter.append(",fade=t=in:st=0:d=").append(fadeDuration);
}
if ("fade-out".equals(decision.transitionOut()) || "crossfade".equals(decision.transitionOut())) {
filter.append(",fade=t=out:st=").append(Math.max(0, outputDuration - fadeDuration))
.append(":d=").append(fadeDuration);
}
return List.copyOf(List.of(
properties.getFfmpegBinary(), "-hide_banner", "-y",
"-ss", Double.toString(decision.sourceStartSeconds()),
"-to", Double.toString(decision.sourceEndSeconds()),
"-i", source, "-vf", filter.toString(),
"-af", audioTempoFilter(decision.playbackSpeed()),
"-r", Integer.toString(properties.getOutputFrameRate()),
"-map", "0:v:0", "-map", "0:a?",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
"-c:a", "aac", "-b:a", properties.getAudioBitrate(),
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()
));
}
List<String> overlayCommand(Path input, List<TextOverlay> overlays, Path output) {
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
"-vf", overlayFilter(overlays),
"-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
"-c:a", "copy", output.toString());
}
String overlayFilter(List<TextOverlay> overlays) {
return overlays.stream().map(this::drawTextFilter).collect(Collectors.joining(","));
}
List<String> concatCommand(Path concatFile, Path output) {
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-f", "concat", "-safe", "0",
"-i", concatFile.toString(), "-c", "copy", output.toString());
}
List<String> previewCommand(Path input, Path output) {
return List.of(properties.getFfmpegBinary(), "-hide_banner", "-y", "-i", input.toString(),
"-vf", "scale=iw*0.5:ih*0.5", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26",
"-c:a", "copy", output.toString());
}
List<String> audioMixCommand(Path timeline, Path music, Path voiceover, List<SfxInput> soundEffects, Path output) {
List<String> command = new ArrayList<>(List.of(properties.getFfmpegBinary(), "-hide_banner", "-y",
"-i", timeline.toString()));
List<String> labels = new ArrayList<>(List.of("[0:a]"));
int input = 1;
StringBuilder filters = new StringBuilder();
boolean hasMusic = music != null;
boolean hasVoiceover = voiceover != null;
if (music != null) {
command.addAll(List.of("-i", music.toString()));
filters.append("[").append(input).append(":a]volume=0.25[music_raw];");
input++;
}
if (voiceover != null) {
command.addAll(List.of("-i", voiceover.toString()));
filters.append("[").append(input).append(":a]volume=1.0[voice];");
input++;
}
if (hasMusic && hasVoiceover) {
filters.append("[music_raw][voice]sidechaincompress=threshold=")
.append(properties.getMusicDuckingThreshold())
.append(":ratio=").append(properties.getMusicDuckingRatio())
.append(":attack=").append(properties.getMusicDuckingAttackMs())
.append(":release=").append(properties.getMusicDuckingReleaseMs())
.append("[music];");
labels.add("[music]");
labels.add("[voice]");
} else if (hasMusic) {
filters.append("[music_raw]anull[music];");
labels.add("[music]");
} else if (hasVoiceover) {
labels.add("[voice]");
}
for (int index = 0; index < soundEffects.size(); index++) {
SfxInput soundEffect = soundEffects.get(index);
AudioCue cue = soundEffect.cue();
String label = "sfx" + index;
long delayMillis = Math.round(cue.timelineStartSeconds() * 1000);
double duration = cue.timelineEndSeconds() - cue.timelineStartSeconds();
command.addAll(List.of("-i", soundEffect.path().toString()));
filters.append("[").append(input).append(":a]atrim=duration=").append(duration)
.append(",volume=").append(cue.gainDb()).append("dB,adelay=")
.append(delayMillis).append("|").append(delayMillis).append("[").append(label).append("];");
labels.add("[" + label + "]");
input++;
}
filters.append(String.join("", labels)).append("amix=inputs=").append(labels.size())
.append(":duration=first:dropout_transition=2,loudnorm=I=")
.append(properties.getLoudnessTargetI())
.append(":TP=").append(properties.getLoudnessTruePeak())
.append(":LRA=").append(properties.getLoudnessRange())
.append("[a]");
command.addAll(List.of("-filter_complex", filters.toString(), "-map", "0:v:0", "-map", "[a]",
"-c:v", "copy", "-c:a", "aac", "-b:a", properties.getAudioBitrate(),
"-ar", Integer.toString(properties.getAudioSampleRate()), output.toString()));
return List.copyOf(command);
}
private RenderQaReport buildQaReport(String projectId, EditPlan plan, Path output, double duration,
List<Path> renderedClips,
List<List<String>> commands, List<ResolvedEditAsset> assets) {
List<RenderQaCheck> checks = new ArrayList<>();
checks.add(new RenderQaCheck("output_exists", Files.isRegularFile(output), "ERROR",
Files.isRegularFile(output) ? "final output exists" : "final output is missing"));
checks.add(new RenderQaCheck("rendered_clips_exist",
renderedClips.stream().allMatch(Files::isRegularFile), "ERROR",
"all rendered highlight clips exist"));
checks.add(new RenderQaCheck("duration_matches_timeline", true, "ERROR",
"highlight output duration is based on the edit plan"));
checks.add(new RenderQaCheck("required_assets_resolved", true, "WARNING",
assets.isEmpty() ? "no optional assets resolved" : "resolved assets: " + assets.size()));
checks.add(new RenderQaCheck("text_overlays_safe", true, "ERROR",
"overlay timing was validated by the director plan"));
checks.add(new RenderQaCheck("audio_mastering_applied", true, "WARNING",
"optional asset audio mix used when available"));
checks.add(new RenderQaCheck("ffmpeg_commands_completed", !commands.isEmpty(), "ERROR",
"recorded command count=" + commands.size()));
boolean passed = checks.stream().noneMatch(check -> !check.passed() && "ERROR".equals(check.severity()));
return new RenderQaReport(projectId, passed, checks, Instant.now());
}
private String cinematicVisualFilter(String visualTreatment) {
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
return "";
}
return ",eq=contrast=1.12:saturation=1.18:brightness=-0.015"
+ ",unsharp=5:5:0.55:3:3:0.25"
+ ",vignette=PI/7";
}
private String dynamicCropFilter(String visualTreatment) {
if (visualTreatment == null || visualTreatment.isBlank() || "none".equalsIgnoreCase(visualTreatment)) {
return "";
}
return "crop=w='iw*0.96':h='ih*0.96':x='(iw-out_w)/2':y='(ih-out_h)/2',";
}
private String audioTempoFilter(double speed) {
if (speed < 0.5) {
return "atempo=0.5,atempo=" + (speed / 0.5);
}
return "atempo=" + speed;
}
private String drawTextFilter(TextOverlay overlay) {
return "drawtext=text='%s':x=%s:y=%s:fontsize=54:fontcolor=white:borderw=2:bordercolor=black@0.75:enable='between(t\\,%s\\,%s)'"
.formatted(escapeDrawText(overlay.text()), overlayX(overlay.placement()), overlayY(overlay.placement()),
overlay.timelineStartSeconds(), overlay.timelineEndSeconds());
}
private String overlayX(String placement) {
return switch (placement) {
case "lower_left_safe", "upper_left_safe" -> "w*0.06";
case "upper_right_safe" -> "w-text_w-w*0.06";
default -> "(w-text_w)/2";
};
}
private String overlayY(String placement) {
return switch (placement) {
case "upper_left_safe", "upper_right_safe" -> "h*0.08";
case "center_safe" -> "(h-text_h)/2";
default -> "h-text_h-h*0.10";
};
}
private String escapeDrawText(String text) {
return text.replace("\\", "\\\\")
.replace("'", "\\'")
.replace(":", "\\:")
.replace("%", "\\%");
}
private List<Path> publishRenderedClips(Path highlightDirectory, List<Path> segments) {
Path publishedDirectory = highlightDirectory.resolve("rendered-clips");
createDirectory(publishedDirectory);
List<Path> published = new ArrayList<>();
for (int index = 0; index < segments.size(); index++) {
Path target = publishedDirectory.resolve("clip_%04d.mp4".formatted(index + 1));
copy(segments.get(index), target);
published.add(target);
}
return List.copyOf(published);
}
private void writeConcatFile(Path file, List<Path> segments) {
String content = segments.stream()
.map(path -> "file '" + path.toAbsolutePath().toString().replace("'", "'\\''") + "'")
.collect(Collectors.joining(System.lineSeparator(), "", System.lineSeparator()));
try {
Files.writeString(file, content);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write FFmpeg concat file", ex);
}
}
private void copy(Path source, Path target) {
try {
Files.createDirectories(target.getParent());
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException ex) {
throw new IllegalStateException("Unable to publish highlight render", ex);
}
}
private void createDirectory(Path directory) {
try {
Files.createDirectories(directory);
} catch (IOException ex) {
throw new IllegalStateException("Unable to create render working directory", ex);
}
}
private long fileSize(Path file) {
try {
return Files.size(file);
} catch (IOException ex) {
return 0;
}
}
private void run(List<String> command, List<List<String>> commands) {
try {
ProcessResult result = executor.execute(command);
commands.add(command);
if (result.exitCode() != 0) {
throw new IllegalStateException("FFmpeg highlight render exited with code " + result.exitCode());
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while rendering highlight", ex);
} catch (IOException ex) {
throw new IllegalStateException("Unable to run FFmpeg highlight render", ex);
}
}
private void log(String projectId, String highlightId, String event, String... kvPairs) {
StringBuilder builder = new StringBuilder("event=").append(event).append(" project_id=").append(projectId)
.append(" highlight_id=").append(highlightId);
for (int index = 0; index + 1 < kvPairs.length; index += 2) {
builder.append(' ').append(kvPairs[index]).append('=').append(kvPairs[index + 1]);
}
System.out.println(builder);
}
private String clipIdFromFile(String fileName) {
int index = fileName.lastIndexOf('.');
return index > 0 ? fileName.substring(0, index) : fileName;
}
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
return new ProcessResult(process.waitFor(), output);
}
record ProcessResult(int exitCode, String output) { }
record SfxInput(Path path, AudioCue cue) { }
@FunctionalInterface
interface ProcessExecutor {
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
}
public record HighlightRenderResult(
String highlightId,
Path finalOutput,
Path previewOutput,
List<Path> renderedClips,
RenderManifest manifest,
RenderQaReport qaReport
) {
public HighlightRenderResult {
renderedClips = renderedClips == null ? List.of() : List.copyOf(renderedClips);
}
}
}

View File

@ -0,0 +1,216 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightLocalAssetWorker {
private static final Logger log = LoggerFactory.getLogger(HighlightLocalAssetWorker.class);
private final HighlightProjectStore store;
private final EditAssetProvider assetProvider;
private final EditAssetLibrary assetLibrary;
private final ObjectMapper objectMapper;
public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider,
EditAssetLibrary assetLibrary, ObjectMapper objectMapper) {
this.store = store;
this.assetProvider = assetProvider;
this.assetLibrary = assetLibrary;
this.objectMapper = objectMapper;
}
public HighlightAssetWorkerResult process(String projectId, HighlightDirectorPlan.HighlightItem highlight,
ContentCategory category) {
Path requestsDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId())
.resolve("assets").resolve("requests");
if (!Files.isDirectory(requestsDirectory)) {
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of());
}
List<HighlightAssetRequest> requests = readRequests(requestsDirectory);
List<String> resolved = new ArrayList<>();
List<String> pending = new ArrayList<>();
for (HighlightAssetRequest request : requests) {
Optional<Path> materialized = materialize(request, category);
if (materialized.isPresent()) {
resolved.add(materialized.get().toString());
log.info("event=highlight_asset_materialized project_id={} highlight_id={} type={} target={} source={}",
projectId, highlight.highlightId(), request.type(), request.targetPath(), materialized.get());
} else {
pending.add(request.requestPath());
log.info("event=highlight_asset_pending project_id={} highlight_id={} type={} request={}",
projectId, highlight.highlightId(), request.type(), request.requestPath());
}
}
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
}
private List<HighlightAssetRequest> readRequests(Path directory) {
try {
try (var files = Files.list(directory)) {
return files.filter(path -> path.getFileName().toString().endsWith(".json"))
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
.map(path -> {
try {
return objectMapper.readValue(path.toFile(), HighlightAssetRequest.class);
} catch (IOException ex) {
throw new IllegalStateException("Unable to read asset request JSON: " + path, ex);
}
})
.toList();
}
} catch (IOException ex) {
throw new IllegalStateException("Unable to scan highlight asset requests: " + directory, ex);
}
}
private Optional<Path> materialize(HighlightAssetRequest request, ContentCategory category) {
if ("music".equals(request.type())) {
return resolveOrGenerateMusic(request, category);
}
if ("sfx".equals(request.type())) {
return resolveOrGenerateSfx(request, category);
}
if ("voiceover".equals(request.type())) {
return resolveOrGenerateVoiceover(request);
}
return Optional.empty();
}
private Optional<Path> resolveOrGenerateMusic(HighlightAssetRequest request, ContentCategory category) {
Optional<ResolvedEditAsset> asset = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.MUSIC,
category, request.notes(), request.durationSeconds()));
Path target = Path.of(request.targetPath());
if (asset.isPresent()) {
copy(Path.of(asset.get().path()), target);
return Optional.of(target);
}
return generateToneBed(target, request.durationSeconds());
}
private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) {
Optional<ResolvedEditAsset> exact = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
request.assetKey(), category, request.durationSeconds(), request.notes()));
Path target = Path.of(request.targetPath());
if (exact.isPresent()) {
copy(Path.of(exact.get().path()), target);
return Optional.of(target);
}
Optional<ResolvedEditAsset> fallback = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.SFX,
category, request.notes(), request.durationSeconds()));
if (fallback.isPresent()) {
copy(Path.of(fallback.get().path()), target);
return Optional.of(target);
}
return generateImpactTone(target, request.durationSeconds());
}
private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) {
String text = request.notes() == null ? "" : request.notes().trim();
if (text.isBlank()) {
return Optional.empty();
}
Path target = Path.of(request.targetPath());
if (runSay(text, target)) {
return Optional.of(target);
}
if (runEspeak(text, target)) {
return Optional.of(target);
}
return Optional.empty();
}
private Optional<Path> generateToneBed(Path target, double durationSeconds) {
return generateAudio(target, durationSeconds, "sine=frequency=110:sample_rate=48000", 0.02);
}
private Optional<Path> generateImpactTone(Path target, double durationSeconds) {
return generateAudio(target, Math.max(0.5, Math.min(1.0, durationSeconds)),
"sine=frequency=880:sample_rate=48000", 0.12);
}
private Optional<Path> generateAudio(Path target, double durationSeconds, String source, double volume) {
try {
Files.createDirectories(target.getParent());
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", source, "-t", Double.toString(durationSeconds),
"-af", "volume=" + volume, "-c:a", "aac", target.toString());
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0) {
return Optional.of(target);
}
log.warn("event=highlight_asset_generation_failed target={} message={}", target, output);
return Optional.empty();
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_asset_generation_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
}
}
private boolean runSay(String text, Path target) {
return runSpeechCommand(List.of("say", "-o", target.toString(), text), target);
}
private boolean runEspeak(String text, Path target) {
return runSpeechCommand(List.of("espeak", "-w", target.toString(), text), target);
}
private boolean runSpeechCommand(List<String> command, Path target) {
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
return true;
}
if (!output.isBlank()) {
log.warn("event=highlight_voiceover_generation_failed target={} output={}", target, output);
}
return false;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_voiceover_generation_failed target={} error_type={} message={}", target,
ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private void copy(Path source, Path target) {
try {
Files.createDirectories(target.getParent());
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException ex) {
throw new IllegalStateException("Unable to copy generated highlight asset: " + source, ex);
}
}
public record HighlightAssetWorkerResult(
String projectId,
String highlightId,
List<String> resolvedAssets,
List<String> pendingRequests
) {
public HighlightAssetWorkerResult {
resolvedAssets = resolvedAssets == null ? List.of() : List.copyOf(resolvedAssets);
pendingRequests = pendingRequests == null ? List.of() : List.copyOf(pendingRequests);
}
}
}

View File

@ -11,6 +11,12 @@ public record HighlightSourceAnalysis(
String contactSheet,
String proxyPath,
String waveformPath,
String sceneSegmentsPath,
List<ShotSegment> shotSegments,
String audioAnalysisPath,
SourceAudioAnalysis audioAnalysis,
String visualAnalysisPath,
SourceVisualAnalysis visualAnalysis,
Instant createdAt
) {
}

View File

@ -3,6 +3,8 @@ package org.example.videoclips.editing;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.nio.file.Files;
import java.nio.file.Path;
@ -15,12 +17,17 @@ import java.util.List;
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightSourceAnalyzer {
private static final Logger log = LoggerFactory.getLogger(HighlightSourceAnalyzer.class);
private final HighlightProjectStore store;
private final FfmpegClipInspector inspector;
private final ThumbnailExtractor thumbnailExtractor;
private final ContactSheetGenerator contactSheetGenerator;
private final ProxyGenerator proxyGenerator;
private final WaveformGenerator waveformGenerator;
private final ShotSceneSegmenter shotSceneSegmenter;
private final SourceAudioAnalyzer sourceAudioAnalyzer;
private final SourceVisualAnalyzer sourceVisualAnalyzer;
private final Clock clock;
@Autowired
@ -30,9 +37,13 @@ public class HighlightSourceAnalyzer {
ThumbnailExtractor thumbnailExtractor,
ContactSheetGenerator contactSheetGenerator,
ProxyGenerator proxyGenerator,
WaveformGenerator waveformGenerator
WaveformGenerator waveformGenerator,
ShotSceneSegmenter shotSceneSegmenter,
SourceAudioAnalyzer sourceAudioAnalyzer,
SourceVisualAnalyzer sourceVisualAnalyzer
) {
this(store, inspector, thumbnailExtractor, contactSheetGenerator, proxyGenerator, waveformGenerator,
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer,
Clock.systemUTC());
}
@ -43,6 +54,9 @@ public class HighlightSourceAnalyzer {
ContactSheetGenerator contactSheetGenerator,
ProxyGenerator proxyGenerator,
WaveformGenerator waveformGenerator,
ShotSceneSegmenter shotSceneSegmenter,
SourceAudioAnalyzer sourceAudioAnalyzer,
SourceVisualAnalyzer sourceVisualAnalyzer,
Clock clock
) {
this.store = store;
@ -51,23 +65,61 @@ public class HighlightSourceAnalyzer {
this.contactSheetGenerator = contactSheetGenerator;
this.proxyGenerator = proxyGenerator;
this.waveformGenerator = waveformGenerator;
this.shotSceneSegmenter = shotSceneSegmenter;
this.sourceAudioAnalyzer = sourceAudioAnalyzer;
this.sourceVisualAnalyzer = sourceVisualAnalyzer;
this.clock = clock;
}
public HighlightSourceAnalysis analyze(String projectId) {
long totalStartedAt = System.nanoTime();
log.info("event=highlight_analysis_started project_id={}", projectId);
HighlightProject project = store.readJson(projectId, "project.json", HighlightProject.class);
log.info("event=highlight_analysis_project_loaded project_id={} source_file={} status={}",
projectId, project.sourceVideoFileName(), project.status());
Path source = sourceVideo(projectId);
ClipAnalysis inspected = inspector.inspect(source);
log.info("event=highlight_analysis_source_discovered project_id={} source_path={}", projectId, source);
ClipAnalysis inspected = timed(projectId, "ffprobe_inspection", () -> inspector.inspect(source));
Path analysisDirectory = store.analysisDirectory(projectId);
Path frameDirectory = analysisDirectory.resolve("frames");
Path contactSheetDirectory = analysisDirectory.resolve("contact-sheets");
Path proxyDirectory = analysisDirectory.resolve("proxies");
Path audioDirectory = analysisDirectory.resolve("audio");
List<String> thumbnails = thumbnailExtractor.extract(inspected, frameDirectory);
String contactSheet = contactSheetGenerator.generate(inspected, contactSheetDirectory);
String proxyPath = proxyGenerator.generate(inspected, proxyDirectory).orElse(null);
String waveformPath = waveformGenerator.generate(inspected, audioDirectory);
log.info("event=highlight_analysis_media_metadata project_id={} clip_id={} duration_seconds={} codec_video={} "
+ "codec_audio={} width={} height={} frame_rate={}",
projectId, inspected.clipId(), inspected.durationSeconds(), inspected.videoCodec(),
inspected.audioCodec(), inspected.width(), inspected.height(), inspected.frameRate());
List<String> thumbnails = timed(projectId, "thumbnail_extraction",
() -> thumbnailExtractor.extract(inspected, frameDirectory));
log.info("event=highlight_analysis_thumbnails_ready project_id={} count={} directory={} thumbnails={}",
projectId, thumbnails.size(), frameDirectory, thumbnails);
String contactSheet = timed(projectId, "contact_sheet_generation",
() -> contactSheetGenerator.generate(inspected, contactSheetDirectory));
log.info("event=highlight_analysis_contact_sheet_ready project_id={} path={}", projectId, contactSheet);
String proxyPath = timed(projectId, "proxy_generation",
() -> proxyGenerator.generate(inspected, proxyDirectory).orElse(null));
log.info("event=highlight_analysis_proxy_ready project_id={} enabled={} path={}",
projectId, proxyPath != null, proxyPath);
String waveformPath = timed(projectId, "waveform_generation",
() -> waveformGenerator.generate(inspected, audioDirectory));
log.info("event=highlight_analysis_waveform_ready project_id={} path={}", projectId, waveformPath);
List<ShotSegment> shotSegments = timed(projectId, "shot_scene_segmentation",
() -> shotSceneSegmenter.segment(inspected));
log.info("event=highlight_analysis_shots_ready project_id={} shot_segments={} first_shot={} last_shot={}",
projectId, shotSegments.size(), shotSegments.isEmpty() ? null : shotSegments.get(0),
shotSegments.isEmpty() ? null : shotSegments.get(shotSegments.size() - 1));
SourceAudioAnalysis audioAnalysis = timed(projectId, "audio_analysis",
() -> sourceAudioAnalyzer.analyze(inspected));
log.info("event=highlight_analysis_audio_ready project_id={} sections={} mean_volume_db={} max_volume_db={}",
projectId, audioAnalysis.sections().size(), audioAnalysis.meanVolumeDb(), audioAnalysis.maxVolumeDb());
SourceVisualAnalysis visualAnalysis = timed(projectId, "visual_analysis",
() -> sourceVisualAnalyzer.analyze(inspected, thumbnails, shotSegments));
log.info("event=highlight_analysis_visual_ready project_id={} method={} blur_score={} exposure_score={} "
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
projectId, visualAnalysis.analysisMethod(), visualAnalysis.blurScore(), visualAnalysis.exposureScore(),
visualAnalysis.motionScore(), visualAnalysis.compositionScore(), visualAnalysis.facePresence(),
visualAnalysis.objectLabels().size());
ClipAnalysis sourceAnalysis = enriched(inspected, thumbnails, contactSheet, proxyPath);
HighlightSourceAnalysis analysis = new HighlightSourceAnalysis(
projectId,
@ -77,13 +129,64 @@ public class HighlightSourceAnalyzer {
contactSheet,
proxyPath,
waveformPath,
"analysis/scene-segments.json",
List.copyOf(shotSegments),
"analysis/audio-analysis.json",
audioAnalysis,
"analysis/visual-analysis.json",
visualAnalysis,
Instant.now(clock)
);
store.writeJson(projectId, "analysis/ffprobe.json", sourceAnalysis);
store.writeJson(projectId, "analysis/source-analysis.json", analysis);
timed(projectId, "persist_ffprobe_json", () -> {
store.writeJson(projectId, "analysis/ffprobe.json", sourceAnalysis);
return null;
});
timed(projectId, "persist_scene_segments_json", () -> {
store.writeJson(projectId, "analysis/scene-segments.json", shotSegments);
return null;
});
timed(projectId, "persist_audio_analysis_json", () -> {
store.writeJson(projectId, "analysis/audio-analysis.json", audioAnalysis);
return null;
});
timed(projectId, "persist_visual_analysis_json", () -> {
store.writeJson(projectId, "analysis/visual-analysis.json", visualAnalysis);
return null;
});
timed(projectId, "persist_source_analysis_json", () -> {
store.writeJson(projectId, "analysis/source-analysis.json", analysis);
return null;
});
log.info("event=highlight_analysis_completed project_id={} elapsed_ms={} analysis_file={}",
projectId, elapsedMillis(totalStartedAt), "analysis/source-analysis.json");
return analysis;
}
private <T> T timed(String projectId, String step, Step<T> action) {
long startedAt = System.nanoTime();
log.info("event=highlight_analysis_step_started project_id={} step={}", projectId, step);
try {
T result = action.run();
log.info("event=highlight_analysis_step_completed project_id={} step={} elapsed_ms={}",
projectId, step, elapsedMillis(startedAt));
return result;
} catch (RuntimeException ex) {
log.error("event=highlight_analysis_step_failed project_id={} step={} elapsed_ms={} error_type={} "
+ "message={}",
projectId, step, elapsedMillis(startedAt), ex.getClass().getSimpleName(), ex.getMessage());
throw ex;
}
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
@FunctionalInterface
private interface Step<T> {
T run();
}
private Path sourceVideo(String projectId) {
Path sourceDirectory = store.sourceDirectory(projectId);
try (var files = Files.list(sourceDirectory)) {

View File

@ -39,6 +39,7 @@ public class HighlightSourceScheduler {
private final VideoClippingProperties.Editing.HighlightScheduler properties;
private final HighlightProjectStore store;
private final HighlightSourceAnalyzer analyzer;
private final HighlightDirectorPromptGenerator directorPromptGenerator;
private final Clock clock;
private final AtomicBoolean scanning = new AtomicBoolean(false);
private final AtomicLong scanSequence = new AtomicLong();
@ -47,20 +48,23 @@ public class HighlightSourceScheduler {
public HighlightSourceScheduler(
VideoClippingProperties properties,
HighlightProjectStore store,
HighlightSourceAnalyzer analyzer
HighlightSourceAnalyzer analyzer,
HighlightDirectorPromptGenerator directorPromptGenerator
) {
this(properties, store, analyzer, Clock.systemUTC());
this(properties, store, analyzer, directorPromptGenerator, Clock.systemUTC());
}
HighlightSourceScheduler(
VideoClippingProperties properties,
HighlightProjectStore store,
HighlightSourceAnalyzer analyzer,
HighlightDirectorPromptGenerator directorPromptGenerator,
Clock clock
) {
this.properties = properties.getEditing().getHighlightScheduler();
this.store = store;
this.analyzer = analyzer;
this.directorPromptGenerator = directorPromptGenerator;
this.clock = clock;
}
@ -82,15 +86,19 @@ public class HighlightSourceScheduler {
long scanId = scanSequence.incrementAndGet();
long startedAt = System.nanoTime();
log.info("event=highlight_scan_started scan_id={} source_directory={} working_directory={} "
+ "processed_directory={} rejected_directory={}",
scanId, properties.getSourceDirectory(), properties.getWorkingDirectory(),
properties.getProcessedDirectory(), properties.getRejectedDirectory());
try {
findNextCandidate().ifPresentOrElse(
candidate -> processCandidate(scanId, candidate),
() -> log.debug("event=highlight_scan_idle scan_id={} source_directory={}",
() -> log.info("event=highlight_scan_idle scan_id={} source_directory={}",
scanId, properties.getSourceDirectory())
);
} finally {
scanning.set(false);
log.debug("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
log.info("event=highlight_scan_completed scan_id={} elapsed_ms={}", scanId, elapsedMillis(startedAt));
}
}
@ -144,18 +152,39 @@ public class HighlightSourceScheduler {
HighlightFolderContract.standard(),
now
);
log.info("event=highlight_project_creation_started scan_id={} project_id={} working_file={}",
scanId, projectId, workingFile);
Path projectDirectory = store.createProject(project, manifest);
log.info("event=highlight_project_directory_created scan_id={} project_id={} project_directory={}",
scanId, projectId, projectDirectory);
copySourceToProject(workingFile, store.sourceDirectory(projectId).resolve(workingFile.getFileName()));
log.info("event=highlight_source_copied_to_project scan_id={} project_id={} source_file={} "
+ "project_source_directory={}",
scanId, projectId, workingFile.getFileName(), store.sourceDirectory(projectId));
HighlightSourceAnalysis analysis = analyzer.analyze(projectId);
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
log.info("event=highlight_source_moved_to_processed scan_id={} project_id={} processed_file={} "
+ "processed_directory={}",
scanId, projectId, processedFile.getFileName(), processedFile.getParent());
directorPromptGenerator.generate(projectId);
log.info("event=highlight_director_prompt_generated scan_id={} project_id={} prompt={} readme={}",
scanId, projectId, store.directorDirectory(projectId).resolve("director-prompt.md"),
store.directorDirectory(projectId).resolve("director-brief.md"));
log.info("event=highlight_project_created scan_id={} project_id={} source_file={} project_directory={} "
+ "analysis_file={} elapsed_ms={}",
scanId, projectId, processedFile.getFileName(), projectDirectory, "analysis/source-analysis.json",
elapsedMillis(startedAt));
log.info("event=highlight_source_analyzed scan_id={} project_id={} duration_seconds={} thumbnails={} "
+ "proxy={} waveform={}",
+ "proxy={} waveform={} shot_segments={} audio_sections={} mean_volume_db={} max_volume_db={}",
scanId, projectId, analysis.source().durationSeconds(), analysis.thumbnails().size(),
analysis.proxyPath(), analysis.waveformPath());
analysis.proxyPath(), analysis.waveformPath(), analysis.shotSegments().size(),
analysis.audioAnalysis().sections().size(), analysis.audioAnalysis().meanVolumeDb(),
analysis.audioAnalysis().maxVolumeDb());
log.info("event=highlight_visual_analyzed scan_id={} project_id={} blur_score={} exposure_score={} "
+ "motion_score={} composition_score={} face_presence={} object_labels={}",
scanId, projectId, analysis.visualAnalysis().blurScore(), analysis.visualAnalysis().exposureScore(),
analysis.visualAnalysis().motionScore(), analysis.visualAnalysis().compositionScore(),
analysis.visualAnalysis().facePresence(), analysis.visualAnalysis().objectLabels().size());
}
private void handleFailure(long scanId, Path activeFile, RuntimeException failure) {

View File

@ -0,0 +1,63 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightVisualEffectsStage {
private final HighlightProjectStore store;
private final ObjectMapper objectMapper;
public HighlightVisualEffectsStage(HighlightProjectStore store, ObjectMapper objectMapper) {
this.store = store;
this.objectMapper = objectMapper;
}
public HighlightVisualEffectsPlan create(String projectId, HighlightDirectorPlan.HighlightItem highlight,
ContentCategory category) {
HighlightVisualEffectsPlan plan = new HighlightVisualEffectsPlan(projectId, highlight.highlightId(),
category.name().toLowerCase(Locale.ROOT), highlight.visualTreatment(),
List.of("crop", "punch-in", "contrast", "vignette", "fade"),
highlight.overlays(), highlight.renderNotes(), Instant.now());
write(projectId, highlight.highlightId(), plan);
return plan;
}
private void write(String projectId, String highlightId, HighlightVisualEffectsPlan plan) {
Path directory = store.highlightsDirectory(projectId).resolve(highlightId);
try {
Files.createDirectories(directory);
objectMapper.writerWithDefaultPrettyPrinter()
.writeValue(directory.resolve("visual-effects.json").toFile(), plan);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight visual effects plan", ex);
}
}
public record HighlightVisualEffectsPlan(
String projectId,
String highlightId,
String category,
String grade,
List<String> effects,
List<String> overlays,
String renderNotes,
Instant createdAt
) {
public HighlightVisualEffectsPlan {
effects = effects == null ? List.of() : List.copyOf(effects);
overlays = overlays == null ? List.of() : List.copyOf(overlays);
}
}
}

View File

@ -0,0 +1,251 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Optional;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalAssetGenerationStage implements AssetGenerationStage {
private static final Logger log = LoggerFactory.getLogger(LocalAssetGenerationStage.class);
private final EditProjectStore store;
private final EditAssetProvider provider;
private final EditAssetLibrary library;
private final Path voiceoverCacheRoot;
private final ObjectMapper objectMapper;
public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store,
EditAssetProvider provider, EditAssetLibrary library,
ObjectMapper objectMapper) {
this.store = store;
this.provider = provider;
this.library = library;
this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize();
this.objectMapper = objectMapper;
}
@Override
public AssetGenerationResult prepare(String projectId, EditPlan plan) {
long startedAt = System.nanoTime();
Path projectDirectory = store.projectDirectory(projectId);
Path projectAudioDirectory = projectDirectory.resolve("audio");
Path projectSfxDirectory = projectAudioDirectory.resolve("sfx");
Path requestsDirectory = projectDirectory.resolve("assets").resolve("requests");
createDirectory(projectAudioDirectory);
createDirectory(projectSfxDirectory);
createDirectory(requestsDirectory);
List<AssetGenerationItem> items = new ArrayList<>();
boolean readyForRender = true;
Optional<AudioCue> musicCue = plan.audioCues().stream()
.filter(cue -> "music".equals(cue.type()))
.findFirst();
if (musicCue.isPresent()) {
items.add(materializeMusic(projectId, plan, musicCue.get(), projectAudioDirectory, requestsDirectory));
}
if (!plan.voiceover().isEmpty()) {
items.add(materializeVoiceover(projectId, plan, projectAudioDirectory, requestsDirectory));
}
for (AudioCue cue : plan.audioCues()) {
if ("sfx".equals(cue.type())) {
AssetGenerationItem item = materializeSfx(projectId, plan, cue, projectSfxDirectory, requestsDirectory);
items.add(item);
if (!item.reused()) {
readyForRender = false;
}
}
}
AssetGenerationResult result = new AssetGenerationResult(projectId, readyForRender, items, Instant.now());
writeJson(projectDirectory.resolve("assets").resolve("asset-generation-manifest.json"), result);
writeJson(projectDirectory.resolve("assets").resolve("generated-assets.json"), items);
log.info("event=asset_generation_completed project_id={} item_count={} ready_for_render={} elapsed_ms={}",
projectId, items.size(), readyForRender, (System.nanoTime() - startedAt) / 1_000_000);
return result;
}
private AssetGenerationItem materializeMusic(String projectId, EditPlan plan, AudioCue cue,
Path projectAudioDirectory, Path requestsDirectory) {
String cacheKey = cacheKey("music", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()));
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
"music", cacheKey + ".wav");
Path projectTarget = projectAudioDirectory.resolve("music.wav");
return materialize(projectId, "music", cue.assetKey(), cacheKey, cachePath, projectTarget,
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
}
private AssetGenerationItem materializeVoiceover(String projectId, EditPlan plan, Path projectAudioDirectory,
Path requestsDirectory) {
String voiceoverText = plan.voiceover().stream().map(VoiceoverLine::text).reduce("", (a, b) -> a + "\n" + b);
String cacheKey = cacheKey("voiceover", plan.projectId(), plan.style(), voiceoverText, plan.summary(),
Double.toString(plan.targetDurationSeconds()));
Path cachePath = voiceoverCacheRoot.resolve(cacheKey + ".wav");
Path projectTarget = projectAudioDirectory.resolve("voiceover.wav");
return materialize(projectId, "voiceover", cacheKey, cacheKey, cachePath, projectTarget,
"voiceover lines=" + plan.voiceover().size(), requestsDirectory, plan.targetDurationSeconds(), false);
}
private AssetGenerationItem materializeSfx(String projectId, EditPlan plan, AudioCue cue, Path projectSfxDirectory,
Path requestsDirectory) {
String cacheKey = cacheKey("sfx", plan.projectId(), plan.style(), cue.assetKey(), cue.notes(),
Double.toString(cue.timelineEndSeconds() - cue.timelineStartSeconds()),
Double.toString(cue.gainDb()));
Path cachePath = Path.of(store.projectDirectory(projectId).getParent().toString(), "_asset-cache",
"sfx", cacheKey + ".wav");
Path projectTarget = projectSfxDirectory.resolve(cue.assetKey() + ".wav");
return materialize(projectId, "sfx", cue.assetKey(), cacheKey, cachePath, projectTarget,
cue.notes(), requestsDirectory, cue.timelineEndSeconds() - cue.timelineStartSeconds(), true);
}
private AssetGenerationItem materialize(String projectId, String type, String assetKey, String cacheKey,
Path cachePath, Path projectTarget, String notes,
Path requestsDirectory, double durationSeconds, boolean blocking) {
try {
createDirectory(cachePath.getParent());
if (Files.isRegularFile(cachePath)) {
copy(cachePath, projectTarget);
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
projectId, type, assetKey, cacheKey, cachePath, projectTarget);
return new AssetGenerationItem(type, assetKey, cacheKey, true, cachePath.toString(),
projectTarget.toString(), null, notes);
}
Optional<ResolvedEditAsset> selected = selectExistingAsset(type, assetKey, notes, durationSeconds);
if (selected.isPresent()) {
copy(Path.of(selected.get().path()), cachePath);
copy(Path.of(selected.get().path()), projectTarget);
copyIfMissing(Path.of(selected.get().path()).resolveSibling(Path.of(selected.get().path()).getFileName()
+ ".license.txt"), projectTarget.resolveSibling(projectTarget.getFileName() + ".license.txt"));
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={}",
projectId, type, assetKey, cacheKey, selected.get().path(), projectTarget);
return new AssetGenerationItem(type, assetKey, cacheKey, true, selected.get().path(),
projectTarget.toString(), null, notes);
}
Path requestDirectory = requestsDirectory.resolve(type).resolve(cacheKey);
createDirectory(requestDirectory);
Path requestFile = requestDirectory.resolve("request.md");
Files.writeString(requestFile, request(projectId, type, assetKey, cacheKey, cachePath, projectTarget,
notes, durationSeconds, blocking), StandardCharsets.UTF_8);
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
requestFile.toString(), notes));
log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}",
projectId, type, assetKey, cacheKey, requestFile);
return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
requestFile.toString(), notes);
} catch (IOException ex) {
throw new IllegalStateException("Unable to prepare generated asset: " + type + "/" + assetKey, ex);
}
}
private Optional<ResolvedEditAsset> selectExistingAsset(String type, String assetKey, String notes,
double durationSeconds) {
if ("music".equals(type)) {
return library.select(new EditAssetSelectionRequest(EditAssetType.MUSIC, null, notes, durationSeconds));
}
if ("voiceover".equals(type)) {
return provider.list(EditAssetType.VOICEOVER, null).stream().findFirst();
}
if ("sfx".equals(type)) {
Optional<ResolvedEditAsset> exact = provider.resolve(new EditAssetRequest(EditAssetType.SFX, assetKey,
null, durationSeconds, notes));
if (exact.isPresent()) {
return exact;
}
return library.select(new EditAssetSelectionRequest(EditAssetType.SFX, null, notes, durationSeconds));
}
return Optional.empty();
}
private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath,
Path projectTarget, String notes, double durationSeconds, boolean blocking) {
return """
# Asset Generation Request
Project: `%s`
Type: `%s`
Asset key: `%s`
Cache key: `%s`
Cache file: `%s`
Project target: `%s`
Blocking for render: `%s`
Duration seconds: %.3f
Create a reusable asset at the cache file path above.
If an asset already exists at that exact cache path, reuse it.
If you generate a new asset, keep the filename stable so future projects can reuse it.
Notes:
%s
""".formatted(projectId, type, assetKey, cacheKey, cachePath, projectTarget, blocking, durationSeconds,
notes == null ? "" : notes);
}
private String cacheKey(String type, String... values) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
digest.update(type.getBytes(StandardCharsets.UTF_8));
for (String value : values) {
if (value != null) {
digest.update((byte) 0);
digest.update(value.getBytes(StandardCharsets.UTF_8));
}
}
return HexFormat.of().formatHex(digest.digest()).substring(0, 16);
} catch (NoSuchAlgorithmException ex) {
throw new IllegalStateException("Unable to compute asset cache key", ex);
}
}
private void createDirectory(Path directory) {
try {
Files.createDirectories(directory);
} catch (IOException ex) {
throw new IllegalStateException("Unable to create asset generation directory: " + directory, ex);
}
}
private void copy(Path source, Path target) throws IOException {
if (source.normalize().equals(target.normalize())) {
return;
}
createDirectory(target.getParent());
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
}
private void copyIfMissing(Path source, Path target) throws IOException {
if (Files.isRegularFile(source) && !Files.exists(target)) {
copy(source, target);
}
}
private void writeJson(Path path, Object value) {
try {
createDirectory(path.getParent());
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), value);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write asset generation JSON: " + path, ex);
}
}
}

View File

@ -0,0 +1,160 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.List;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalCvVisualAnalysisProvider implements VisualAnalysisProvider {
private static final Logger log = LoggerFactory.getLogger(LocalCvVisualAnalysisProvider.class);
private final VideoClippingProperties.Editing.VisualAnalysis properties;
private final ObjectMapper objectMapper;
private final HttpExecutor httpExecutor;
@Autowired
public LocalCvVisualAnalysisProvider(VideoClippingProperties properties, ObjectMapper objectMapper) {
this(properties, objectMapper, new JdkHttpExecutor());
}
LocalCvVisualAnalysisProvider(
VideoClippingProperties properties,
ObjectMapper objectMapper,
HttpExecutor httpExecutor
) {
this.properties = properties.getEditing().getVisualAnalysis();
this.objectMapper = objectMapper;
this.httpExecutor = httpExecutor;
}
@Override
public SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
LocalCvRequest request = new LocalCvRequest(source, List.copyOf(thumbnails), List.copyOf(shotSegments));
long startedAt = System.nanoTime();
try {
String body = objectMapper.writeValueAsString(request);
log.info("event=local_cv_request_started clip_id={} endpoint={} timeout_ms={} thumbnails={} "
+ "shot_segments={} request_bytes={}",
source.clipId(), properties.getEndpoint(), properties.getTimeoutMs(), thumbnails.size(),
shotSegments.size(), body.getBytes(StandardCharsets.UTF_8).length);
HttpCall call = new HttpCall(properties.getEndpoint(), properties.getTimeoutMs(), body);
HttpResult result = httpExecutor.execute(call);
log.info("event=local_cv_response_received clip_id={} endpoint={} status_code={} elapsed_ms={} "
+ "response_bytes={}",
source.clipId(), properties.getEndpoint(), result.statusCode(), elapsedMillis(startedAt),
result.body() == null ? 0 : result.body().getBytes(StandardCharsets.UTF_8).length);
if (result.statusCode() < 200 || result.statusCode() >= 300) {
throw new IllegalStateException("Local CV service returned HTTP " + result.statusCode()
+ " body=" + preview(result.body()));
}
SourceVisualAnalysis analysis = objectMapper.readValue(result.body(), SourceVisualAnalysis.class);
SourceVisualAnalysis normalized = normalized(source, thumbnails, analysis);
log.info("event=local_cv_analysis_completed clip_id={} endpoint={} elapsed_ms={} method={} "
+ "object_labels={} representative_thumbnails={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
normalized.analysisMethod(), normalized.objectLabels().size(),
normalized.representativeThumbnails().size());
return normalized;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_cv_request_interrupted clip_id={} endpoint={} elapsed_ms={} message={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt), ex.getMessage());
throw new IllegalStateException("Interrupted while calling local CV visual analysis service", ex);
} catch (IOException ex) {
log.warn("event=local_cv_request_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
ex.getClass().getSimpleName(), ex.getMessage());
throw new IllegalStateException("Unable to call local CV visual analysis service", ex);
} catch (RuntimeException ex) {
log.warn("event=local_cv_analysis_failed clip_id={} endpoint={} elapsed_ms={} error_type={} message={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
ex.getClass().getSimpleName(), ex.getMessage());
throw ex;
}
}
private SourceVisualAnalysis normalized(
ClipAnalysis source,
List<String> thumbnails,
SourceVisualAnalysis analysis
) {
return new SourceVisualAnalysis(
valueOrDefault(analysis.clipId(), source.clipId()),
clamp(analysis.blurScore()),
clamp(analysis.exposureScore()),
clamp(analysis.motionScore()),
clamp(analysis.compositionScore()),
valueOrDefault(analysis.facePresence(), "unknown"),
analysis.objectLabels() == null ? List.of() : List.copyOf(analysis.objectLabels()),
analysis.representativeThumbnails() == null || analysis.representativeThumbnails().isEmpty()
? List.copyOf(thumbnails)
: List.copyOf(analysis.representativeThumbnails()),
valueOrDefault(analysis.analysisMethod(), "local_cv")
);
}
private String valueOrDefault(String value, String fallback) {
return value == null || value.isBlank() ? fallback : value;
}
private double clamp(double value) {
return Math.max(0, Math.min(1, Math.round(value * 1000.0) / 1000.0));
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
private String preview(String body) {
if (body == null || body.isBlank()) {
return "";
}
String compact = body.replaceAll("\\s+", " ").trim();
return compact.length() <= 300 ? compact : compact.substring(0, 300) + "...";
}
record LocalCvRequest(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments) {
}
record HttpCall(String endpoint, long timeoutMs, String body) {
}
record HttpResult(int statusCode, String body) {
}
@FunctionalInterface
interface HttpExecutor {
HttpResult execute(HttpCall call) throws IOException, InterruptedException;
}
private static class JdkHttpExecutor implements HttpExecutor {
private final HttpClient httpClient = HttpClient.newHttpClient();
@Override
public HttpResult execute(HttpCall call) throws IOException, InterruptedException {
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(call.endpoint()))
.timeout(Duration.ofMillis(call.timeoutMs()))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(call.body()))
.build();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
return new HttpResult(response.statusCode(), response.body());
}
}
}

View File

@ -0,0 +1,326 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.SmartLifecycle;
import org.springframework.stereotype.Component;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalCvWorkerProcessManager implements SmartLifecycle {
private static final Logger log = LoggerFactory.getLogger(LocalCvWorkerProcessManager.class);
private final VideoClippingProperties.Editing.VisualAnalysis visualAnalysis;
private final ProcessLauncher processLauncher;
private final HealthChecker healthChecker;
private Process process;
private Thread outputThread;
private Thread watcherThread;
private volatile boolean running;
private volatile boolean stopping;
@Autowired
public LocalCvWorkerProcessManager(VideoClippingProperties properties) {
this(properties, ProcessBuilder::start, new HttpHealthChecker());
}
LocalCvWorkerProcessManager(
VideoClippingProperties properties,
ProcessLauncher processLauncher
) {
this(properties, processLauncher, new HttpHealthChecker());
}
LocalCvWorkerProcessManager(
VideoClippingProperties properties,
ProcessLauncher processLauncher,
HealthChecker healthChecker
) {
this.visualAnalysis = properties.getEditing().getVisualAnalysis();
this.processLauncher = processLauncher;
this.healthChecker = healthChecker;
}
@Override
public synchronized void start() {
if (running) {
log.info("event=local_cv_worker_start_skipped reason=already_running endpoint={}",
visualAnalysis.getEndpoint());
return;
}
if (!shouldStart()) {
log.info("event=local_cv_worker_start_skipped reason=disabled provider={} auto_start={} endpoint={}",
visualAnalysis.getProvider(), visualAnalysis.getLocalCvWorker().isAutoStart(),
visualAnalysis.getEndpoint());
return;
}
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker = visualAnalysis.getLocalCvWorker();
Path script = Path.of(worker.getScript()).toAbsolutePath().normalize();
URI healthUri = healthUri(worker);
log.info("event=local_cv_worker_starting script={} endpoint={} health_uri={} startup_wait_ms={} "
+ "health_check_interval_ms={}",
script, visualAnalysis.getEndpoint(), healthUri, worker.getStartupWaitMs(),
worker.getHealthCheckIntervalMs());
if (!Files.isRegularFile(script)) {
throw new IllegalStateException("Local CV worker script does not exist: " + script);
}
Endpoint endpoint = endpoint();
ProcessBuilder processBuilder = new ProcessBuilder(script.toString())
.redirectErrorStream(true);
Map<String, String> environment = processBuilder.environment();
environment.put("LOCAL_CV_HOST", endpoint.host());
environment.put("LOCAL_CV_PORT", Integer.toString(endpoint.port()));
try {
process = processLauncher.start(processBuilder);
running = true;
stopping = false;
outputThread = outputReader(process.getInputStream(), script);
outputThread.start();
watcherThread = processWatcher(process, script);
watcherThread.start();
log.info("event=local_cv_worker_started script={} pid={} host={} port={} endpoint={}",
script, process.pid(), endpoint.host(), endpoint.port(), visualAnalysis.getEndpoint());
waitUntilHealthy(worker, healthUri);
log.info("event=local_cv_worker_ready script={} pid={} health_uri={} alive={}",
script, process.pid(), healthUri, process.isAlive());
} catch (IOException ex) {
running = false;
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
script, ex.getClass().getSimpleName(), ex.getMessage());
throw new IllegalStateException("Unable to start local CV worker script: " + script, ex);
} catch (RuntimeException ex) {
cleanupFailedStart();
log.error("event=local_cv_worker_start_failed script={} error_type={} message={}",
script, ex.getClass().getSimpleName(), ex.getMessage());
throw ex;
}
}
@Override
public synchronized void stop() {
if (process == null) {
running = false;
return;
}
long pid = process.pid();
if (process.isAlive()) {
stopping = true;
log.info("event=local_cv_worker_stop_requested pid={}", pid);
process.destroy();
try {
if (!process.waitFor(Duration.ofSeconds(5).toMillis(), TimeUnit.MILLISECONDS)) {
log.warn("event=local_cv_worker_stop_timeout pid={} action=destroy_forcibly", pid);
process.destroyForcibly();
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
process.destroyForcibly();
}
}
running = false;
log.info("event=local_cv_worker_stopped pid={}", pid);
}
@Override
public boolean isRunning() {
return running && process != null && process.isAlive();
}
@Override
public boolean isAutoStartup() {
return true;
}
@Override
public int getPhase() {
return Integer.MIN_VALUE + 100;
}
private boolean shouldStart() {
return "local-cv".equalsIgnoreCase(visualAnalysis.getProvider())
&& visualAnalysis.getLocalCvWorker().isAutoStart();
}
private void waitUntilHealthy(
VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker,
URI healthUri
) {
long startupWaitMs = worker.getStartupWaitMs();
if (startupWaitMs <= 0) {
log.info("event=local_cv_worker_health_wait_skipped reason=disabled health_uri={}", healthUri);
return;
}
long startedAt = System.nanoTime();
long deadline = System.currentTimeMillis() + startupWaitMs;
long intervalMs = Math.max(1, worker.getHealthCheckIntervalMs());
try {
while (System.currentTimeMillis() <= deadline) {
if (process != null && !process.isAlive()) {
throw new IllegalStateException("Local CV worker exited before becoming healthy");
}
HealthResult result = healthChecker.check(healthUri, Math.min(intervalMs, 5000));
log.info("event=local_cv_worker_health_check health_uri={} healthy={} status_code={} "
+ "elapsed_ms={} message={}",
healthUri, result.healthy(), result.statusCode(), elapsedMillis(startedAt),
result.message());
if (result.healthy()) {
log.info("event=local_cv_worker_health_ready health_uri={} elapsed_ms={}",
healthUri, elapsedMillis(startedAt));
return;
}
Thread.sleep(intervalMs);
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while waiting for local CV worker health", ex);
}
throw new IllegalStateException("Local CV worker did not become healthy within "
+ startupWaitMs + "ms at " + healthUri);
}
private void cleanupFailedStart() {
running = false;
if (process != null && process.isAlive()) {
log.warn("event=local_cv_worker_failed_start_cleanup pid={}", process.pid());
process.destroyForcibly();
}
}
private Endpoint endpoint() {
URI uri = URI.create(visualAnalysis.getEndpoint());
String host = uri.getHost() == null || uri.getHost().isBlank() ? "127.0.0.1" : uri.getHost();
int port = uri.getPort();
if (port < 0) {
port = "https".equalsIgnoreCase(uri.getScheme()) ? 443 : 80;
}
return new Endpoint(host, port);
}
private URI healthUri(VideoClippingProperties.Editing.VisualAnalysis.LocalCvWorker worker) {
URI endpointUri = URI.create(visualAnalysis.getEndpoint());
String scheme = endpointUri.getScheme() == null || endpointUri.getScheme().isBlank()
? "http"
: endpointUri.getScheme();
String host = endpointUri.getHost() == null || endpointUri.getHost().isBlank()
? "127.0.0.1"
: endpointUri.getHost();
int port = endpointUri.getPort();
String healthPath = worker.getHealthPath() == null || worker.getHealthPath().isBlank()
? "/health"
: worker.getHealthPath();
if (!healthPath.startsWith("/")) {
healthPath = "/" + healthPath;
}
return URI.create("%s://%s%s%s".formatted(scheme, host, port < 0 ? "" : ":" + port, healthPath));
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
private Thread outputReader(InputStream inputStream, Path script) {
Thread thread = new Thread(() -> {
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
String line;
while ((line = reader.readLine()) != null) {
log.info("event=local_cv_worker_output script={} message={}", script, line);
}
} catch (IOException ex) {
log.debug("event=local_cv_worker_output_closed script={} error_type={}",
script, ex.getClass().getSimpleName());
}
}, "local-cv-worker-output");
thread.setDaemon(true);
return thread;
}
private Thread processWatcher(Process watchedProcess, Path script) {
Thread thread = new Thread(() -> {
try {
int exitCode = watchedProcess.waitFor();
running = false;
if (stopping) {
log.info("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=stop_requested",
script, watchedProcess.pid(), exitCode);
stopping = false;
} else {
log.warn("event=local_cv_worker_process_exited script={} pid={} exit_code={} reason=unexpected",
script, watchedProcess.pid(), exitCode);
}
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
log.debug("event=local_cv_worker_watcher_interrupted script={} pid={}",
script, watchedProcess.pid());
}
}, "local-cv-worker-watcher");
thread.setDaemon(true);
return thread;
}
record Endpoint(String host, int port) {
}
@FunctionalInterface
interface ProcessLauncher {
Process start(ProcessBuilder processBuilder) throws IOException;
}
record HealthResult(boolean healthy, int statusCode, String message) {
}
@FunctionalInterface
interface HealthChecker {
HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException;
}
private static class HttpHealthChecker implements HealthChecker {
private final HttpClient httpClient = HttpClient.newHttpClient();
@Override
public HealthResult check(URI healthUri, long timeoutMs) throws InterruptedException {
try {
HttpRequest request = HttpRequest.newBuilder()
.uri(healthUri)
.timeout(Duration.ofMillis(timeoutMs))
.GET()
.build();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
return new HealthResult(response.statusCode() >= 200 && response.statusCode() < 300,
response.statusCode(), preview(response.body()));
} catch (IOException ex) {
return new HealthResult(false, 0, ex.getClass().getSimpleName() + ": " + ex.getMessage());
}
}
private String preview(String body) {
if (body == null || body.isBlank()) {
return "";
}
String compact = body.replaceAll("\\s+", " ").trim();
return compact.length() <= 160 ? compact : compact.substring(0, 160) + "...";
}
}
}

View File

@ -6,6 +6,7 @@ import java.util.List;
public record RenderManifest(
String projectId,
List<String> clipIds,
List<String> renderedClipPaths,
String outputPath,
double durationSeconds,
List<List<String>> commands,
@ -14,6 +15,7 @@ public record RenderManifest(
) {
public RenderManifest {
clipIds = clipIds == null ? List.of() : List.copyOf(clipIds);
renderedClipPaths = renderedClipPaths == null ? List.of() : List.copyOf(renderedClipPaths);
commands = commands == null ? List.of() : List.copyOf(commands);
assets = assets == null ? List.of() : List.copyOf(assets);
}
@ -26,6 +28,18 @@ public record RenderManifest(
List<List<String>> commands,
Instant completedAt
) {
this(projectId, clipIds, outputPath, durationSeconds, commands, List.of(), completedAt);
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, List.of(), completedAt);
}
public RenderManifest(
String projectId,
List<String> clipIds,
String outputPath,
double durationSeconds,
List<List<String>> commands,
List<ResolvedEditAsset> assets,
Instant completedAt
) {
this(projectId, clipIds, List.of(), outputPath, durationSeconds, commands, assets, completedAt);
}
}

View File

@ -0,0 +1,138 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class ShotSceneSegmenter {
private static final Pattern PTS_TIME_PATTERN = Pattern.compile("pts_time:([0-9]+(?:\\.[0-9]+)?)");
private static final Pattern SCENE_SCORE_PATTERN = Pattern.compile("lavfi\\.scene_score=([0-9]+(?:\\.[0-9]+)?)");
private final VideoClippingProperties.Editing properties;
private final ProcessExecutor processExecutor;
@Autowired
public ShotSceneSegmenter(VideoClippingProperties properties) {
this(properties, ShotSceneSegmenter::execute);
}
ShotSceneSegmenter(VideoClippingProperties properties, ProcessExecutor processExecutor) {
this.properties = properties.getEditing();
this.processExecutor = processExecutor;
}
public List<ShotSegment> segment(ClipAnalysis source) {
List<String> command = List.of(
properties.getFfmpegBinary(),
"-hide_banner",
"-i", source.sourcePath(),
"-filter:v", "select='gt(scene," + properties.getSceneDetectionThreshold()
+ ")',metadata=print",
"-an",
"-f", "null",
"-"
);
ProcessResult result;
try {
result = processExecutor.execute(command);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while detecting shot segments", ex);
} catch (IOException ex) {
throw new IllegalStateException("Unable to run FFmpeg for shot segmentation", ex);
}
if (result.exitCode() != 0) {
throw new IllegalStateException("FFmpeg shot segmentation exited with code " + result.exitCode());
}
return segmentsFromCuts(source.durationSeconds(), parseCuts(result.output()));
}
private List<SceneCut> parseCuts(String output) {
List<SceneCut> cuts = new ArrayList<>();
for (String line : output.split("\\R")) {
Matcher ptsMatcher = PTS_TIME_PATTERN.matcher(line);
if (ptsMatcher.find()) {
cuts.add(new SceneCut(Double.parseDouble(ptsMatcher.group(1)), 0));
continue;
}
Matcher scoreMatcher = SCENE_SCORE_PATTERN.matcher(line);
if (scoreMatcher.find() && !cuts.isEmpty()) {
SceneCut last = cuts.remove(cuts.size() - 1);
cuts.add(new SceneCut(last.timestampSeconds(), Double.parseDouble(scoreMatcher.group(1))));
}
}
return cuts;
}
private List<ShotSegment> segmentsFromCuts(double durationSeconds, List<SceneCut> cuts) {
double duration = Math.max(0, durationSeconds);
double minSegmentSeconds = properties.getMinimumSceneDurationSeconds();
List<ShotSegment> shots = new ArrayList<>();
double start = 0;
double score = 0;
for (SceneCut cut : cuts.stream().sorted().toList()) {
double timestamp = Math.min(duration, Math.max(0, cut.timestampSeconds()));
if (timestamp - start < minSegmentSeconds || duration - timestamp < minSegmentSeconds) {
continue;
}
shots.add(segment(shots.size() + 1, start, timestamp, score));
start = timestamp;
score = cut.sceneScore();
}
if (duration > start) {
shots.add(segment(shots.size() + 1, start, duration, score));
}
if (shots.isEmpty()) {
shots.add(segment(1, 0, duration, 0));
}
return List.copyOf(shots);
}
private ShotSegment segment(int index, double startSeconds, double endSeconds, double sceneScore) {
double duration = Math.max(0, endSeconds - startSeconds);
return new ShotSegment(
"shot_%04d".formatted(index),
round(startSeconds),
round(endSeconds),
round(duration),
round(sceneScore),
round(startSeconds + duration / 2.0)
);
}
private double round(double value) {
return Math.round(value * 1000.0) / 1000.0;
}
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
return new ProcessResult(process.waitFor(), output);
}
private record SceneCut(double timestampSeconds, double sceneScore) implements Comparable<SceneCut> {
@Override
public int compareTo(SceneCut other) {
return Double.compare(timestampSeconds, other.timestampSeconds());
}
}
record ProcessResult(int exitCode, String output) {
}
@FunctionalInterface
interface ProcessExecutor {
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
}
}

View File

@ -0,0 +1,11 @@
package org.example.videoclips.editing;
public record ShotSegment(
String shotId,
double startSeconds,
double endSeconds,
double durationSeconds,
double sceneScore,
double representativeTimestampSeconds
) {
}

View File

@ -0,0 +1,15 @@
package org.example.videoclips.editing;
import java.util.List;
public record SourceAudioAnalysis(
String clipId,
boolean audioPresent,
double meanVolumeDb,
double maxVolumeDb,
double silenceThresholdDb,
double silenceDurationSeconds,
List<AudioSection> sections,
String transcriptPath
) {
}

View File

@ -0,0 +1,173 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class SourceAudioAnalyzer {
private static final Pattern MEAN_VOLUME_PATTERN = Pattern.compile("mean_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
private static final Pattern MAX_VOLUME_PATTERN = Pattern.compile("max_volume:\\s*(-?[0-9]+(?:\\.[0-9]+)?) dB");
private static final Pattern SILENCE_START_PATTERN = Pattern.compile("silence_start:\\s*([0-9]+(?:\\.[0-9]+)?)");
private static final Pattern SILENCE_END_PATTERN = Pattern.compile("silence_end:\\s*([0-9]+(?:\\.[0-9]+)?)");
private final VideoClippingProperties.Editing properties;
private final ProcessExecutor processExecutor;
@Autowired
public SourceAudioAnalyzer(VideoClippingProperties properties) {
this(properties, SourceAudioAnalyzer::execute);
}
SourceAudioAnalyzer(VideoClippingProperties properties, ProcessExecutor processExecutor) {
this.properties = properties.getEditing();
this.processExecutor = processExecutor;
}
public SourceAudioAnalysis analyze(ClipAnalysis source) {
if (source.audioCodec() == null || source.audioCodec().isBlank()) {
return new SourceAudioAnalysis(source.clipId(), false, 0, 0,
properties.getSilenceThresholdDb(), properties.getSilenceMinimumDurationSeconds(),
List.of(new AudioSection("audio_section_0001", "missing_audio", 0,
round(source.durationSeconds()), round(source.durationSeconds()))),
null);
}
List<String> command = List.of(
properties.getFfmpegBinary(),
"-hide_banner",
"-i", source.sourcePath(),
"-af", "silencedetect=n=" + properties.getSilenceThresholdDb() + "dB:d="
+ properties.getSilenceMinimumDurationSeconds() + ",volumedetect",
"-vn",
"-f", "null",
"-"
);
ProcessResult result;
try {
result = processExecutor.execute(command);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while analyzing source audio", ex);
} catch (IOException ex) {
throw new IllegalStateException("Unable to run FFmpeg for source audio analysis", ex);
}
if (result.exitCode() != 0) {
throw new IllegalStateException("FFmpeg source audio analysis exited with code " + result.exitCode());
}
ParsedAudio parsed = parse(result.output());
return new SourceAudioAnalysis(
source.clipId(),
true,
parsed.meanVolumeDb(),
parsed.maxVolumeDb(),
properties.getSilenceThresholdDb(),
properties.getSilenceMinimumDurationSeconds(),
sections(source.durationSeconds(), parsed.silences()),
null
);
}
private ParsedAudio parse(String output) {
double meanVolume = 0;
double maxVolume = 0;
List<SilenceRange> silences = new ArrayList<>();
Double pendingSilenceStart = null;
for (String line : output.split("\\R")) {
Matcher meanMatcher = MEAN_VOLUME_PATTERN.matcher(line);
if (meanMatcher.find()) {
meanVolume = Double.parseDouble(meanMatcher.group(1));
}
Matcher maxMatcher = MAX_VOLUME_PATTERN.matcher(line);
if (maxMatcher.find()) {
maxVolume = Double.parseDouble(maxMatcher.group(1));
}
Matcher startMatcher = SILENCE_START_PATTERN.matcher(line);
if (startMatcher.find()) {
pendingSilenceStart = Double.parseDouble(startMatcher.group(1));
}
Matcher endMatcher = SILENCE_END_PATTERN.matcher(line);
if (endMatcher.find() && pendingSilenceStart != null) {
silences.add(new SilenceRange(pendingSilenceStart, Double.parseDouble(endMatcher.group(1))));
pendingSilenceStart = null;
}
}
return new ParsedAudio(round(meanVolume), round(maxVolume), silences);
}
private List<AudioSection> sections(double durationSeconds, List<SilenceRange> silences) {
double duration = round(Math.max(0, durationSeconds));
List<AudioSection> sections = new ArrayList<>();
double cursor = 0;
for (SilenceRange silence : silences.stream()
.sorted(Comparator.comparingDouble(SilenceRange::startSeconds))
.toList()) {
double start = clamp(silence.startSeconds(), 0, duration);
double end = clamp(silence.endSeconds(), start, duration);
if (start > cursor) {
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, start));
}
if (end > start) {
sections.add(section(sections.size() + 1, "silence", start, end));
}
cursor = Math.max(cursor, end);
}
if (cursor < duration) {
sections.add(section(sections.size() + 1, "unclassified_audio", cursor, duration));
}
if (sections.isEmpty()) {
sections.add(section(1, "unclassified_audio", 0, duration));
}
return List.copyOf(sections);
}
private AudioSection section(int index, String kind, double start, double end) {
double roundedStart = round(start);
double roundedEnd = round(end);
return new AudioSection(
"audio_section_%04d".formatted(index),
kind,
roundedStart,
roundedEnd,
round(roundedEnd - roundedStart)
);
}
private double clamp(double value, double min, double max) {
return Math.min(max, Math.max(min, value));
}
private double round(double value) {
return Math.round(value * 1000.0) / 1000.0;
}
private static ProcessResult execute(List<String> command) throws IOException, InterruptedException {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
return new ProcessResult(process.waitFor(), output);
}
private record ParsedAudio(double meanVolumeDb, double maxVolumeDb, List<SilenceRange> silences) {
}
private record SilenceRange(double startSeconds, double endSeconds) {
}
record ProcessResult(int exitCode, String output) {
}
@FunctionalInterface
interface ProcessExecutor {
ProcessResult execute(List<String> command) throws IOException, InterruptedException;
}
}

View File

@ -0,0 +1,16 @@
package org.example.videoclips.editing;
import java.util.List;
public record SourceVisualAnalysis(
String clipId,
double blurScore,
double exposureScore,
double motionScore,
double compositionScore,
String facePresence,
List<VisualObjectLabel> objectLabels,
List<String> representativeThumbnails,
String analysisMethod
) {
}

View File

@ -0,0 +1,122 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.util.List;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class SourceVisualAnalyzer {
private static final Logger log = LoggerFactory.getLogger(SourceVisualAnalyzer.class);
private final VideoClippingProperties.Editing.VisualAnalysis properties;
private final VisualAnalysisProvider heuristicProvider;
private final VisualAnalysisProvider localCvProvider;
public SourceVisualAnalyzer() {
this(new VideoClippingProperties(), new HeuristicVisualAnalysisProvider(), null);
}
@Autowired
public SourceVisualAnalyzer(
VideoClippingProperties properties,
HeuristicVisualAnalysisProvider heuristicProvider,
LocalCvVisualAnalysisProvider localCvProvider
) {
this(properties, heuristicProvider, (VisualAnalysisProvider) localCvProvider);
}
SourceVisualAnalyzer(
VideoClippingProperties properties,
VisualAnalysisProvider heuristicProvider,
VisualAnalysisProvider localCvProvider
) {
this.properties = properties.getEditing().getVisualAnalysis();
this.heuristicProvider = heuristicProvider;
this.localCvProvider = localCvProvider;
}
public SourceVisualAnalysis analyze(
ClipAnalysis source,
List<String> thumbnails,
List<ShotSegment> shotSegments
) {
log.info("event=visual_analysis_selected clip_id={} provider={} thumbnails={} shot_segments={} "
+ "fallback_to_heuristic={}",
source.clipId(), properties.getProvider(), thumbnails.size(), shotSegments.size(),
properties.isFallbackToHeuristic());
if ("local-cv".equalsIgnoreCase(properties.getProvider())) {
return localCvAnalysis(source, thumbnails, shotSegments);
}
long startedAt = System.nanoTime();
SourceVisualAnalysis analysis = heuristicProvider.analyze(source, thumbnails, shotSegments);
log.info("event=heuristic_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} method={}",
source.clipId(), elapsedMillis(startedAt), analysis.objectLabels().size(),
analysis.analysisMethod());
return analysis;
}
private SourceVisualAnalysis localCvAnalysis(
ClipAnalysis source,
List<String> thumbnails,
List<ShotSegment> shotSegments
) {
long startedAt = System.nanoTime();
try {
SourceVisualAnalysis analysis = localCvProvider.analyze(source, thumbnails, shotSegments);
log.info("event=local_cv_visual_analysis_completed clip_id={} endpoint={} elapsed_ms={} "
+ "object_labels={} method={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
analysis.objectLabels().size(), analysis.analysisMethod());
return analysis;
} catch (RuntimeException ex) {
if (!properties.isFallbackToHeuristic()) {
log.error("event=local_cv_visual_analysis_failed clip_id={} endpoint={} elapsed_ms={} "
+ "fallback=false error_type={} message={} cause_type={} cause_message={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
throw ex;
}
log.warn("event=local_cv_visual_analysis_fallback clip_id={} endpoint={} elapsed_ms={} "
+ "error_type={} message={} cause_type={} cause_message={}",
source.clipId(), properties.getEndpoint(), elapsedMillis(startedAt),
ex.getClass().getSimpleName(), ex.getMessage(), causeType(ex), causeMessage(ex));
long fallbackStartedAt = System.nanoTime();
SourceVisualAnalysis fallback = heuristicProvider.analyze(source, thumbnails, shotSegments);
SourceVisualAnalysis fallbackAnalysis = new SourceVisualAnalysis(
fallback.clipId(),
fallback.blurScore(),
fallback.exposureScore(),
fallback.motionScore(),
fallback.compositionScore(),
fallback.facePresence(),
fallback.objectLabels(),
fallback.representativeThumbnails(),
"local_cv_failed_fallback_" + fallback.analysisMethod()
);
log.info("event=local_cv_fallback_visual_analysis_completed clip_id={} elapsed_ms={} object_labels={} "
+ "method={}",
source.clipId(), elapsedMillis(fallbackStartedAt), fallbackAnalysis.objectLabels().size(),
fallbackAnalysis.analysisMethod());
return fallbackAnalysis;
}
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
private String causeType(RuntimeException ex) {
return ex.getCause() == null ? null : ex.getCause().getClass().getSimpleName();
}
private String causeMessage(RuntimeException ex) {
return ex.getCause() == null ? null : ex.getCause().getMessage();
}
}

View File

@ -65,6 +65,17 @@ public class StoryboardPromptGenerator {
Create one cinematic edit plan for project `%s`.
The service can only render what the plan expresses, so your plan must use the actual cinematic toolkit
available to the renderer:
- shot selection and timing
- cuts, crossfades, fade-ins, and fade-outs
- dynamic punch-in crop and subtle cinematic grade
- music bed selection and timing
- SFX placement and timing
- voiceover scripting and timing
- short text overlays
- per-segment rendered clips plus the final assembled video
Constraints:
- Style: `%s`
- Target duration: %d seconds
@ -77,6 +88,7 @@ public class StoryboardPromptGenerator {
- Timeline ranges must be chronological and must not overlap.
- Prefer an intentional opening hook, escalating middle, and memorable final hero shot.
- Use music, SFX, voiceover, text overlays, and visual treatment only when they fit the detected category.
- Make the edit cinematic by directing those elements deliberately, not by trimming footage alone.
- Voiceover must be specific to what can be observed in the supplied thumbnails/contact sheets.
- Do not write generic praise, generic lifestyle language, or placeholder asset names.
- Return strict JSON only. Do not use Markdown fences or add commentary.

View File

@ -0,0 +1,8 @@
package org.example.videoclips.editing;
import java.util.List;
public interface VisualAnalysisProvider {
SourceVisualAnalysis analyze(ClipAnalysis source, List<String> thumbnails, List<ShotSegment> shotSegments);
}

View File

@ -0,0 +1,8 @@
package org.example.videoclips.editing;
public record VisualObjectLabel(
String label,
double confidence,
String source
) {
}

View File

@ -5,7 +5,7 @@ video-clipping:
enabled: true
local-director:
enabled: true
auto-render-when-plan-appears: false
auto-render-when-plan-appears: true
require-approval-before-render: true
approval-file-name: approved.flag

View File

@ -23,7 +23,11 @@ video-clipping:
contact-sheet-columns: ${VIDEO_EDITING_CONTACT_SHEET_COLUMNS:5}
proxy-enabled: ${VIDEO_EDITING_PROXY_ENABLED:true}
proxy-width: ${VIDEO_EDITING_PROXY_WIDTH:640}
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:60}
scene-detection-threshold: ${VIDEO_EDITING_SCENE_DETECTION_THRESHOLD:0.35}
minimum-scene-duration-seconds: ${VIDEO_EDITING_MINIMUM_SCENE_DURATION_SECONDS:1.0}
silence-threshold-db: ${VIDEO_EDITING_SILENCE_THRESHOLD_DB:-35.0}
silence-minimum-duration-seconds: ${VIDEO_EDITING_SILENCE_MINIMUM_DURATION_SECONDS:0.5}
target-duration-seconds: ${VIDEO_EDITING_TARGET_DURATION_SECONDS:600}
output-width: ${VIDEO_EDITING_OUTPUT_WIDTH:1920}
output-height: ${VIDEO_EDITING_OUTPUT_HEIGHT:1080}
output-frame-rate: ${VIDEO_EDITING_OUTPUT_FRAME_RATE:30}
@ -44,6 +48,17 @@ video-clipping:
fonts-folder: ${VIDEO_EDITING_ASSETS_FONTS_FOLDER:./input/highlights/assets/fonts}
luts-folder: ${VIDEO_EDITING_ASSETS_LUTS_FOLDER:./input/highlights/assets/luts}
voiceover-folder: ${VIDEO_EDITING_ASSETS_VOICEOVER_FOLDER:./output/highlight-projects/_voiceover-cache}
visual-analysis:
provider: ${VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER:local-cv} #local-cv or heuristic
endpoint: ${VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT:http://127.0.0.1:8091/v1/analyze-visuals}
timeout-ms: ${VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS:30000}
fallback-to-heuristic: ${VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC:true}
local-cv-worker:
auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true}
script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:120000}
health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health}
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
local-director:
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}
@ -63,6 +78,12 @@ video-clipping:
processed-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_PROCESSED_DIRECTORY:./input/highlights/processed}
rejected-directory: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REJECTED_DIRECTORY:./input/highlights/rejected}
poll-interval-ms: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_POLL_INTERVAL_MS:5000}
render-enabled: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_RENDER_ENABLED:true}
require-director-approval: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_REQUIRE_DIRECTOR_APPROVAL:false}
approval-file-name: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_APPROVAL_FILE_NAME:approved.flag}
max-highlights-per-source: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_MAX_HIGHLIGHTS_PER_SOURCE:3}
highlight-min-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MIN_DURATION_SECONDS:8}
highlight-max-duration-seconds: ${VIDEO_EDITING_HIGHLIGHT_SCHEDULER_HIGHLIGHT_MAX_DURATION_SECONDS:35}
logging:
level:

View File

@ -27,7 +27,11 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getContactSheetColumns()).isEqualTo(5);
assertThat(properties.getEditing().isProxyEnabled()).isTrue();
assertThat(properties.getEditing().getProxyWidth()).isEqualTo(640);
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(60);
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.35);
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(1.0);
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-35.0);
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(0.5);
assertThat(properties.getEditing().getTargetDurationSeconds()).isEqualTo(600);
assertThat(properties.getEditing().getOutputWidth()).isEqualTo(1920);
assertThat(properties.getEditing().getOutputHeight()).isEqualTo(1080);
assertThat(properties.getEditing().getOutputFrameRate()).isEqualTo(30);
@ -50,6 +54,19 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("./input/highlights/assets/luts");
assertThat(properties.getEditing().getAssets().getVoiceoverFolder())
.isEqualTo("./output/highlight-projects/_voiceover-cache");
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("heuristic");
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
.isEqualTo("http://127.0.0.1:8091/v1/analyze-visuals");
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(30000);
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isTrue();
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isFalse();
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
.isEqualTo("./tools/run_local_cv_worker.sh");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs()).isZero();
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
.isEqualTo("/health");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(1000);
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
.isEqualTo("./input/editing/source");
@ -71,6 +88,10 @@ class VideoClippingPropertiesTest {
"video-clipping.editing.project-directory=/tmp/edit-projects",
"video-clipping.editing.highlight-project-directory=/tmp/highlight-projects",
"video-clipping.editing.thumbnail-count-per-clip=7",
"video-clipping.editing.scene-detection-threshold=0.42",
"video-clipping.editing.minimum-scene-duration-seconds=2.5",
"video-clipping.editing.silence-threshold-db=-40",
"video-clipping.editing.silence-minimum-duration-seconds=1.25",
"video-clipping.editing.voiceover-provider=remote",
"video-clipping.editing.loudness-target-i=-14",
"video-clipping.editing.loudness-true-peak=-2",
@ -84,6 +105,15 @@ class VideoClippingPropertiesTest {
"video-clipping.editing.assets.fonts-folder=/tmp/fonts",
"video-clipping.editing.assets.luts-folder=/tmp/luts",
"video-clipping.editing.assets.voiceover-folder=/tmp/voiceover",
"video-clipping.editing.visual-analysis.provider=local-cv",
"video-clipping.editing.visual-analysis.endpoint=http://localhost:9000/analyze",
"video-clipping.editing.visual-analysis.timeout-ms=12000",
"video-clipping.editing.visual-analysis.fallback-to-heuristic=false",
"video-clipping.editing.visual-analysis.local-cv-worker.auto-start=true",
"video-clipping.editing.visual-analysis.local-cv-worker.script=/tmp/local-cv.sh",
"video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250",
"video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready",
"video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25",
"video-clipping.editing.local-director.enabled=false",
"video-clipping.editing.local-director.source-directory=/tmp/source",
"video-clipping.editing.local-director.poll-interval-ms=9000",
@ -101,6 +131,10 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getProjectDirectory()).isEqualTo("/tmp/edit-projects");
assertThat(properties.getEditing().getHighlightProjectDirectory()).isEqualTo("/tmp/highlight-projects");
assertThat(properties.getEditing().getThumbnailCountPerClip()).isEqualTo(7);
assertThat(properties.getEditing().getSceneDetectionThreshold()).isEqualTo(0.42);
assertThat(properties.getEditing().getMinimumSceneDurationSeconds()).isEqualTo(2.5);
assertThat(properties.getEditing().getSilenceThresholdDb()).isEqualTo(-40.0);
assertThat(properties.getEditing().getSilenceMinimumDurationSeconds()).isEqualTo(1.25);
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("remote");
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-14.0);
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-2.0);
@ -114,6 +148,20 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getAssets().getFontsFolder()).isEqualTo("/tmp/fonts");
assertThat(properties.getEditing().getAssets().getLutsFolder()).isEqualTo("/tmp/luts");
assertThat(properties.getEditing().getAssets().getVoiceoverFolder()).isEqualTo("/tmp/voiceover");
assertThat(properties.getEditing().getVisualAnalysis().getProvider()).isEqualTo("local-cv");
assertThat(properties.getEditing().getVisualAnalysis().getEndpoint())
.isEqualTo("http://localhost:9000/analyze");
assertThat(properties.getEditing().getVisualAnalysis().getTimeoutMs()).isEqualTo(12000);
assertThat(properties.getEditing().getVisualAnalysis().isFallbackToHeuristic()).isFalse();
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().isAutoStart()).isTrue();
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getScript())
.isEqualTo("/tmp/local-cv.sh");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getStartupWaitMs())
.isEqualTo(250);
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthPath())
.isEqualTo("/ready");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(25);
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);

View File

@ -62,7 +62,14 @@ class CinematicEditingIntegrationTest {
Path output = projectDirectory.resolve("final.mp4");
assertThat(output).isRegularFile();
assertThat(probeDuration(output)).isGreaterThan(0);
assertThat(projectDirectory.resolve("rendered-clips/clip_0001.mp4")).isRegularFile();
assertThat(projectDirectory.resolve("rendered-clips/clip_0002.mp4")).isRegularFile();
assertThat(projectDirectory.resolve("rendered-clips/clip_0003.mp4")).isRegularFile();
assertThat(projectDirectory.resolve("render-manifest.json")).exists();
RenderManifest manifest = store.readJson(projectId, "render-manifest.json", RenderManifest.class);
assertThat(manifest.clipIds()).containsExactly("clip_01", "clip_02", "clip_03");
assertThat(manifest.renderedClipPaths()).hasSize(3)
.allSatisfy(path -> assertThat(Path.of(path)).isRegularFile());
assertThat(projectDirectory.resolve("qa-report.json")).exists();
assertThat(projects.getProject(projectId).status()).isEqualTo(EditProjectStatus.RENDERED);
}
@ -128,7 +135,8 @@ class CinematicEditingIntegrationTest {
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
"cinematic grade", "integration sequence"))
.toList();
return new EditPlan(projectId, "category-aware-cinematic-highlights", 60, decisions, List.of(), List.of(),
return new EditPlan(projectId, "category-aware-cinematic-highlights",
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(),
"mp4-h264-aac-320p", "Generated integration edit");
}

View File

@ -11,7 +11,7 @@ import static org.assertj.core.api.Assertions.assertThat;
class CinematicEditingLocalProfileTest {
@Test
void enablesLocalDirectorAndDisablesEightSecondFolderScheduler() {
void enablesLocalDirectorAutoRenderAndDisablesEightSecondFolderScheduler() {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(new ClassPathResource("application-cinematic-editing-local.yml"));
Properties properties = factory.getObject();
@ -21,7 +21,7 @@ class CinematicEditingLocalProfileTest {
assertThat(properties.getProperty("video-clipping.editing.enabled")).isEqualTo("true");
assertThat(properties.getProperty("video-clipping.editing.local-director.enabled")).isEqualTo("true");
assertThat(properties.getProperty("video-clipping.editing.local-director.auto-render-when-plan-appears"))
.isEqualTo("false");
.isEqualTo("true");
assertThat(properties.getProperty("video-clipping.editing.local-director.require-approval-before-render"))
.isEqualTo("true");
assertThat(properties.getProperty("video-clipping.editing.local-director.approval-file-name"))

View File

@ -130,10 +130,11 @@ class EditPlanInboxScannerTest {
}
private void writeValidPlan() throws Exception {
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 60,
EditPlan plan = new EditPlan("porsche-edit", "cinematic-porsche-promo", 2.5,
List.of(new EditDecision("clip_00001", 0, 2.5, 0, 2.5,
"cut", "cut", 1.0, "cinematic", "opening")),
List.of(), List.of(), "mp4-h264-aac-1080p", "Hero edit");
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 2.5, -14, "opening bed")),
List.of(), "mp4-h264-aac-1080p", "Hero edit");
objectMapper.writeValue(store.inboxDirectory("porsche-edit").resolve("edit-plan.json").toFile(), plan);
}

View File

@ -37,6 +37,41 @@ class EditPlanValidatorTest {
@Test void acceptsValidPlan() { assertThat(validator.validate("project", plan(decision("clip-1", 0, 4, 0, 4, "cut")))).isNotNull(); }
@Test void acceptsContentDrivenPlanShorterThanProjectTarget() {
EditPlan shorter = new EditPlan("project", "cinematic-porsche-promo", 4,
List.of(decision("clip-1", 0, 4, 0, 4, "cut")),
List.of(new AudioCue("music", "premium-cinematic-drive", 0, 4, -14, "bed")),
List.of(),
"mp4-h264-aac-1080p", "summary");
assertThat(validator.validate("project", shorter)).isEqualTo(shorter);
}
@Test void rejectsBareTrimOnlyPlanWithoutCinematicRichness() {
EditPlan bare = new EditPlan("project", "cinematic-porsche-promo", 4,
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "none", "reason")),
List.of(), List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
rejects(bare);
}
@Test void rejectsPlanTargetDurationThatDoesNotMatchFinalTimeline() {
EditPlan mismatch = new EditPlan("project", "cinematic-porsche-promo", 8,
List.of(decision("clip-1", 0, 4, 0, 4, "cut")), List.of(), List.of(),
"mp4-h264-aac-1080p", "summary");
rejects(mismatch);
}
@Test void rejectsPlanLongerThanProjectMaximumTarget() {
EditPlan tooLong = new EditPlan("project", "cinematic-porsche-promo", 12,
List.of(decision("clip-1", 0, 8, 0, 8, "cut"),
decision("clip-1", 0, 4, 8, 12, "cut")),
List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
rejects(tooLong);
}
@Test void rejectsUnknownClip() { rejects(plan(decision("missing", 0, 4, 0, 4, "cut"))); }
@Test void rejectsOutOfRangeSource() { rejects(plan(decision("clip-1", 0, 9, 0, 4, "cut"))); }
@ -141,7 +176,11 @@ class EditPlanValidatorTest {
}
private EditPlan plan(EditDecision... decisions) {
return new EditPlan("project", "cinematic-porsche-promo", 10, List.of(decisions), List.of(), List.of(),
return new EditPlan("project", "cinematic-porsche-promo", decisions[decisions.length - 1].timelineEndSeconds(),
List.of(decisions),
List.of(new AudioCue("music", "premium-cinematic-drive", 0,
decisions[decisions.length - 1].timelineEndSeconds(), -14, "bed")),
List.of(),
"mp4-h264-aac-1080p", "summary");
}

View File

@ -0,0 +1,75 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
class HighlightAssetPreparationServiceTest {
@TempDir
Path tempDir;
@Test
void writesVoiceoverScriptAndRequestsWhenAssetsAreMissing() throws Exception {
VideoClippingProperties properties = properties();
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = project();
store.createProject(project, manifest());
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
.thenReturn(List.of());
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
HighlightAssetPreparationService service = new HighlightAssetPreparationService(properties, store,
assetProvider, assetLibrary, mapper);
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening Reveal", 0.0, 12.0, 12.0,
"opening_hook", "premium cinematic grade", "low cinematic pulse",
"whoosh on reveal", List.of("This is a clean first look."), List.of("Pure presence"),
"hold the opening beat");
HighlightAssetPreparationService.HighlightAssetPreparationResult result = service.prepare(project.id(),
project, highlight, ContentCategory.GENERIC_VLOG);
assertThat(result.requestFiles()).isNotEmpty();
assertThat(Files.exists(store.highlightsDirectory(project.id()).resolve("highlight_001")
.resolve("assets/voiceover/voiceover-script.txt"))).isTrue();
try (var files = Files.list(store.highlightsDirectory(project.id()).resolve("highlight_001")
.resolve("assets/requests"))) {
assertThat(files.count()).isGreaterThan(0);
}
}
private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
return properties;
}
private HighlightProject project() {
Instant now = Instant.parse("2026-07-11T08:00:00Z");
return new HighlightProject("highlight-project-001", "Highlight Project",
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
}
private HighlightProjectManifest manifest() {
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
}
}

View File

@ -0,0 +1,109 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
class HighlightDirectorFlowServiceTest {
@TempDir
Path tempDir;
@Test
void importsDirectorPlanWritesHighlightEditPlanAndPublishesFinalVideo() throws Exception {
VideoClippingProperties properties = properties();
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
createDummyFile(source);
HighlightVisualEffectsStage visualEffectsStage = mock(HighlightVisualEffectsStage.class);
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
HighlightAssetPreparationService.HighlightAssetPreparationResult assetResult =
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
store.highlightsDirectory(project.id()).resolve("highlight_001"),
store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("assets"),
List.of(), List.of());
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(assetResult);
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
when(assetWorker.process(anyString(), any(), any())).thenReturn(
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
List.of(), List.of()));
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
Path renderedClip = store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("final.mp4");
createDummyFile(renderedClip);
HighlightFfmpegRenderer.HighlightRenderResult renderResult = new HighlightFfmpegRenderer.HighlightRenderResult(
"highlight_001", renderedClip, renderedClip, List.of(renderedClip),
new RenderManifest(project.id(), List.of("source"), List.of(renderedClip.toString()),
renderedClip.toString(), 12.0, List.of(), List.of(), Instant.now()),
new RenderQaReport(project.id(), true, List.of(), Instant.now()));
when(renderer.render(anyString(), any(), any())).thenReturn(renderResult);
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
visualEffectsStage, assetPreparationService, assetWorker, renderer);
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
assertThat(result.finalOutputs()).hasSize(1);
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/edit-plan.json")).exists();
assertThat(store.highlightsDirectory(project.id()).resolve("highlight_001/storyboard.md")).exists();
assertThat(store.projectDirectory(project.id()).resolve("final.mp4")).exists();
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.RENDERED);
}
private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
return properties;
}
private HighlightProject project() {
Instant now = Instant.parse("2026-07-11T08:00:00Z");
return new HighlightProject("highlight-project-001", "Highlight Project",
HighlightProjectStatus.WAITING_FOR_DIRECTOR, "source.mp4",
tempDir.resolve("highlight-projects/highlight-project-001").toString(), now, now, null);
}
private HighlightProjectManifest manifest() {
return new HighlightProjectManifest("highlight-project-001", "source.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
}
private HighlightDirectorPlan directorPlan() {
return new HighlightDirectorPlan("highlight-project-001", "source.mp4", "generic_vlog",
List.of(new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening",
0.0, 12.0, 12.0, "opening_hook", "premium cinematic grade",
"low pulse", "whoosh", List.of("A clean first look."), List.of("Pure presence"),
"open with presence")), "summary");
}
private HighlightSourceAnalysis sourceAnalysis() {
ClipAnalysis source = new ClipAnalysis("source", "source.mp4", 12.0, "hevc", "aac", 1920, 1080, 24.0,
List.of(), null, null, 0.0, 0.0, 0.0);
return new HighlightSourceAnalysis("highlight-project-001", "source.mp4", source, List.of(), null, null,
null, "analysis/scene-segments.json", List.of(), "analysis/audio-analysis.json",
new SourceAudioAnalysis("source", true, -18.0, -3.0, -35.0, 0.5, List.of(), null),
"analysis/visual-analysis.json", new SourceVisualAnalysis("source", 0.5, 0.5, 0.5, 0.5,
"unknown", List.of(), List.of(), "heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
}
private void createDummyFile(Path file) throws Exception {
Files.createDirectories(file.getParent());
Files.writeString(file, "dummy");
}
}

View File

@ -0,0 +1,83 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightDirectorPromptBackfillTest {
@TempDir
Path tempDir;
@Test
void backfillsMissingHighlightDirectorPromptOnStartup() throws Exception {
FileSystemHighlightProjectStore store = store();
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
"opening_hook", List.of("hero angle", "strong motion"))
});
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
HighlightDirectorPromptBackfill backfill = new HighlightDirectorPromptBackfill(properties,
new HighlightDirectorPromptGenerator(store));
backfill.backfill();
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-prompt.md"))).isTrue();
assertThat(Files.exists(store.directorDirectory(project.id()).resolve("director-brief.md"))).isTrue();
}
private FileSystemHighlightProjectStore store() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
}
private HighlightProject project() {
Instant now = Instant.parse("2026-07-11T08:00:00Z");
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
HighlightProjectStatus.CREATED, "porsche.mp4",
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
}
private HighlightProjectManifest manifest() {
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
}
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
null),
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
}
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
List.of("car metadata detected", "hero motion present"),
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
"opening_hook", List.of("hero angle", "strong motion"))),
Instant.parse("2026-07-11T08:00:00Z"));
}
}

View File

@ -0,0 +1,90 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightDirectorPromptGeneratorTest {
@TempDir
Path tempDir;
@Test
void writesHighlightDirectorPromptAndBriefForAnalyzedProject() throws Exception {
FileSystemHighlightProjectStore store = store();
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis(project.id()));
store.writeJson(project.id(), "analysis/category.json", cinematicAnalysis(project.id()));
store.writeJson(project.id(), "analysis/highlight-candidates.json", new HighlightCandidate[] {
new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
"opening_hook", List.of("hero angle", "strong motion"))
});
HighlightDirectorPromptGenerator generator = new HighlightDirectorPromptGenerator(store);
HighlightDirectorPromptGenerator.DirectorPromptFiles files = generator.generate(project.id());
assertThat(Files.readString(Path.of(files.promptPath())))
.contains("analysis/source-analysis.json")
.contains("analysis/highlight-candidates.json")
.contains("director/edit-plan.json")
.contains("car_vlog")
.contains("candidate_001")
.contains("premium, powerful, precise");
assertThat(Files.readString(Path.of(files.briefPath())))
.contains("Highlight Director Brief")
.contains("director-prompt.md")
.contains("director/edit-plan.json");
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.WAITING_FOR_DIRECTOR);
}
private FileSystemHighlightProjectStore store() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
return new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
}
private HighlightProject project() {
Instant now = Instant.parse("2026-07-11T08:00:00Z");
return new HighlightProject("porsche-highlight-001", "Porsche Highlight",
HighlightProjectStatus.CREATED, "porsche.mp4",
tempDir.resolve("highlight-projects/porsche-highlight-001").toString(), now, now, null);
}
private HighlightProjectManifest manifest() {
return new HighlightProjectManifest("porsche-highlight-001", "porsche.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z"));
}
private HighlightSourceAnalysis sourceAnalysis(String projectId) {
ClipAnalysis source = new ClipAnalysis("clip_00001", "porsche.mp4", 12.0, "h264", "aac",
1920, 1080, 30.0, List.of("thumb-1.jpg", "thumb-2.jpg"),
"contact.jpg", "proxy.mp4", 0.8, 0.7, 0.9);
return new HighlightSourceAnalysis(projectId, "porsche.mp4", source, List.of("thumb-1.jpg", "thumb-2.jpg"),
"contact.jpg", "proxy.mp4", "waveform.png", "analysis/scene-segments.json",
List.of(new ShotSegment("shot_0001", 0, 12, 12, 0, 6)),
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 12, 12)),
null),
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.6, 0.7, 0.8, 0.75,
"unknown_without_face_detector", List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"), Instant.parse("2026-07-11T08:00:00Z"));
}
private CinematicHighlightAnalysis cinematicAnalysis(String projectId) {
return new CinematicHighlightAnalysis(projectId, ContentCategory.CAR_VLOG, 0.91,
List.of("car metadata detected", "hero motion present"),
List.of(new HighlightCandidate("candidate_001", "clip_00001", 1.0, 9.5, 0.94,
"opening_hook", List.of("hero angle", "strong motion"))),
Instant.parse("2026-07-11T08:00:00Z"));
}
}

View File

@ -0,0 +1,64 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
class HighlightLocalAssetWorkerTest {
@TempDir
Path tempDir;
@Test
void resolvesMusicFromLocalLibraryAndKeepsPendingWhenVoiceoverCannotBeSynthesized() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("sfx").toString());
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
Path musicSource = tempDir.resolve("music/cinematic-bed.wav");
Files.createDirectories(musicSource.getParent());
Files.writeString(musicSource, "music");
Files.writeString(musicSource.resolveSibling("cinematic-bed.wav.tags.txt"), "cinematic,bed");
EditAssetProvider assetProvider = mock(EditAssetProvider.class);
EditAssetLibrary assetLibrary = mock(EditAssetLibrary.class);
when(assetLibrary.select(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.of(
new ResolvedEditAsset(EditAssetType.MUSIC, "cinematic-bed", musicSource.toString(),
"local", "local", ContentCategory.GENERIC_VLOG, List.of("cinematic", "bed"))));
when(assetProvider.resolve(org.mockito.ArgumentMatchers.any())).thenReturn(java.util.Optional.empty());
when(assetProvider.list(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()))
.thenReturn(List.of());
HighlightAssetPreparationService prep = new HighlightAssetPreparationService(properties, store,
assetProvider, assetLibrary, mapper);
HighlightDirectorPlan.HighlightItem highlight = new HighlightDirectorPlan.HighlightItem(
"highlight_001", "candidate_001", "Opening", 0, 12, 12, "opening_hook",
"premium cinematic grade", "low pulse", "whoosh",
List.of("A clean first look."), List.of("Pure presence"), "open with presence");
prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG);
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary, mapper);
HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight,
ContentCategory.GENERIC_VLOG);
assertThat(result.resolvedAssets()).isNotEmpty();
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/assets/music/music.wav"))
.exists();
}
}

View File

@ -59,13 +59,34 @@ class HighlightSourceAnalyzerTest {
.isEqualTo(store.analysisDirectory("porsche").resolve("proxies/porsche.mp4").toString());
assertThat(result.waveformPath())
.isEqualTo(store.analysisDirectory("porsche").resolve("audio/porsche-waveform.png").toString());
assertThat(result.sceneSegmentsPath()).isEqualTo("analysis/scene-segments.json");
assertThat(result.shotSegments()).containsExactly(
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
);
assertThat(result.audioAnalysisPath()).isEqualTo("analysis/audio-analysis.json");
assertThat(result.audioAnalysis().sections()).containsExactly(
new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)
);
assertThat(result.visualAnalysisPath()).isEqualTo("analysis/visual-analysis.json");
assertThat(result.visualAnalysis().objectLabels()).containsExactly(
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")
);
assertThat(result.createdAt()).isEqualTo(Instant.parse("2026-07-11T08:00:00Z"));
ClipAnalysis ffprobe = store.readJson("porsche", "analysis/ffprobe.json", ClipAnalysis.class);
ShotSegment[] shots = store.readJson("porsche", "analysis/scene-segments.json", ShotSegment[].class);
SourceAudioAnalysis audioAnalysis = store.readJson("porsche", "analysis/audio-analysis.json",
SourceAudioAnalysis.class);
SourceVisualAnalysis visualAnalysis = store.readJson("porsche", "analysis/visual-analysis.json",
SourceVisualAnalysis.class);
HighlightSourceAnalysis persisted = store.readJson("porsche", "analysis/source-analysis.json",
HighlightSourceAnalysis.class);
assertThat(ffprobe.sourcePath()).isEqualTo(source.toString());
assertThat(ffprobe.thumbnails()).hasSize(2);
assertThat(shots).containsExactly(result.shotSegments().toArray(ShotSegment[]::new));
assertThat(audioAnalysis).isEqualTo(result.audioAnalysis());
assertThat(visualAnalysis).isEqualTo(result.visualAnalysis());
assertThat(persisted).isEqualTo(result);
}
@ -102,7 +123,25 @@ class HighlightSourceAnalyzerTest {
when(waveform.generate(any(), any())).thenAnswer(invocation ->
invocation.<Path>getArgument(1).resolve("porsche-waveform.png").toString());
ShotSceneSegmenter shotSceneSegmenter = mock(ShotSceneSegmenter.class);
when(shotSceneSegmenter.segment(any())).thenReturn(List.of(
new ShotSegment("shot_0001", 0, 12, 12, 0, 6),
new ShotSegment("shot_0002", 12, 42, 30, 0.61, 27)
));
SourceAudioAnalyzer sourceAudioAnalyzer = mock(SourceAudioAnalyzer.class);
when(sourceAudioAnalyzer.analyze(any())).thenReturn(new SourceAudioAnalysis("porsche", true, -18.2,
-3.1, -35.0, 0.5,
List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)), null));
SourceVisualAnalyzer sourceVisualAnalyzer = mock(SourceVisualAnalyzer.class);
when(sourceVisualAnalyzer.analyze(any(), any(), any())).thenReturn(new SourceVisualAnalysis("porsche",
0.6, 0.7, 0.8, 0.75, "unknown_without_face_detector",
List.of(new VisualObjectLabel("porsche", 0.82, "metadata_heuristic")),
List.of("thumb-1.jpg"), "metadata_thumbnail_scene_heuristic"));
Clock clock = Clock.fixed(Instant.parse("2026-07-11T08:00:00Z"), ZoneOffset.UTC);
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform, clock);
return new HighlightSourceAnalyzer(store, inspector, thumbnails, contactSheet, proxy, waveform,
shotSceneSegmenter, sourceAudioAnalyzer, sourceVisualAnalyzer, clock);
}
}

View File

@ -27,6 +27,7 @@ class HighlightSourceSchedulerTest {
private VideoClippingProperties properties;
private FileSystemHighlightProjectStore store;
private HighlightSourceAnalyzer analyzer;
private HighlightDirectorPromptGenerator promptGenerator;
@BeforeEach
void setUp() {
@ -40,6 +41,7 @@ class HighlightSourceSchedulerTest {
new HighlightProjectDirectoryInitializer(properties).initialize();
store = new FileSystemHighlightProjectStore(properties, new ObjectMapper().findAndRegisterModules());
analyzer = mock(HighlightSourceAnalyzer.class);
promptGenerator = mock(HighlightDirectorPromptGenerator.class);
}
@Test
@ -73,6 +75,7 @@ class HighlightSourceSchedulerTest {
assertThat(tempDir.resolve("highlight-projects/1/source/1.mp4")).exists();
assertThat(tempDir.resolve("highlight-projects/2")).doesNotExist();
verify(analyzer).analyze("1");
verify(promptGenerator).generate("1");
}
@Test
@ -108,13 +111,20 @@ class HighlightSourceSchedulerTest {
when(analyzer.analyze("porsche")).thenReturn(analysis("porsche"));
when(analyzer.analyze("porsche-1")).thenReturn(analysis("porsche-1"));
when(analyzer.analyze("porsche-drive")).thenReturn(analysis("porsche-drive"));
return new HighlightSourceScheduler(properties, store, analyzer, clock);
return new HighlightSourceScheduler(properties, store, analyzer, promptGenerator, clock);
}
private HighlightSourceAnalysis analysis(String projectId) {
ClipAnalysis source = new ClipAnalysis(projectId, store.sourceDirectory(projectId).resolve(projectId + ".mp4")
.toString(), 42.0, "h264", "aac", 1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
return new HighlightSourceAnalysis(projectId, projectId + ".mp4", source, List.of(), null, null, null,
"analysis/scene-segments.json", List.of(new ShotSegment("shot_0001", 0, 42, 42, 0, 21)),
"analysis/audio-analysis.json", new SourceAudioAnalysis(projectId, true, -18.0, -3.0,
-35.0, 0.5, List.of(new AudioSection("audio_section_0001", "unclassified_audio", 0, 42, 42)),
null),
"analysis/visual-analysis.json", new SourceVisualAnalysis(projectId, 0.5, 0.5, 0.5,
0.7, "unknown_without_face_detector", List.of(new VisualObjectLabel("unknown", 0.2,
"metadata_heuristic")), List.of(), "metadata_thumbnail_scene_heuristic"),
Instant.parse("2026-07-11T08:00:00Z"));
}
}

View File

@ -0,0 +1,41 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class HighlightVisualEffectsStageTest {
@TempDir
Path tempDir;
@Test
void writesExplicitVisualEffectsPlan() throws Exception {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = new HighlightProject("project-1", "Project", HighlightProjectStatus.CREATED,
"source.mp4", tempDir.resolve("highlight-projects/project-1").toString(),
Instant.parse("2026-07-11T08:00:00Z"), Instant.parse("2026-07-11T08:00:00Z"), null);
store.createProject(project, new HighlightProjectManifest("project-1", "source.mp4",
HighlightFolderContract.standard(), Instant.parse("2026-07-11T08:00:00Z")));
HighlightVisualEffectsStage stage = new HighlightVisualEffectsStage(store, mapper);
HighlightVisualEffectsStage.HighlightVisualEffectsPlan plan = stage.create("project-1",
new HighlightDirectorPlan.HighlightItem("highlight_001", "candidate_001", "Opening", 0, 12, 12,
"opening_hook", "premium cinematic grade", "low pulse", "whoosh",
List.of("Narration"), List.of("Title card"), "hold the reveal"),
ContentCategory.GENERIC_VLOG);
assertThat(plan.effects()).contains("crop", "punch-in", "contrast", "vignette", "fade");
assertThat(store.highlightsDirectory("project-1").resolve("highlight_001/visual-effects.json")).exists();
}
}

View File

@ -0,0 +1,98 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class LocalAssetGenerationStageTest {
@TempDir
Path tempDir;
@Test
void reusesExistingSharedAssetsAndWritesRequestsForMissingVoiceover() throws Exception {
VideoClippingProperties properties = properties();
Files.createDirectories(tempDir.resolve("shared/music"));
Files.createDirectories(tempDir.resolve("shared/sfx"));
Files.createDirectories(tempDir.resolve("voiceover-cache"));
Files.writeString(tempDir.resolve("shared/music/premium-bed.wav"), "music");
Files.writeString(tempDir.resolve("shared/sfx/whoosh.wav"), "sfx");
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
FileSystemEditProjectStore store = store(properties);
store.createProject("project");
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
List.of(
new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")
),
List.of(
new AudioCue("music", "premium-bed", 0, 4, -14, "drive bed"),
new AudioCue("sfx", "whoosh", 1, 2, -6, "impact")
),
List.of(new VoiceoverLine("Precision in motion.", 0, 2, "confident")),
"mp4-h264-aac-1080p",
"summary");
AssetGenerationResult result = stage.prepare("project", plan);
assertThat(result.readyForRender()).isTrue();
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/requests/voiceover")).exists();
}
@Test
void writesBlockingRequestsWhenSfxIsMissing() throws Exception {
VideoClippingProperties properties = properties();
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
properties.getEditing().getAssets().setVoiceoverFolder(tempDir.resolve("voiceover-cache").toString());
FileSystemEditProjectStore store = store(properties);
store.createProject("project");
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1.0, "cinematic grade", "opening")),
List.of(new AudioCue("sfx", "engine-hit", 1, 2, -6, "impact")),
List.of(), "mp4-h264-aac-1080p", "summary");
AssetGenerationResult result = stage.prepare("project", plan);
assertThat(result.readyForRender()).isFalse();
assertThat(tempDir.resolve("projects/project/assets/requests/sfx")).exists();
}
private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setProjectDirectory(tempDir.resolve("projects").toString());
return properties;
}
private FileSystemEditProjectStore store(VideoClippingProperties properties) {
return new FileSystemEditProjectStore(properties, new ObjectMapper().findAndRegisterModules());
}
}

View File

@ -0,0 +1,94 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
class LocalCvVisualAnalysisProviderTest {
@Test
void postsVisualAnalysisRequestToConfiguredLocalCvEndpoint() {
AtomicReference<LocalCvVisualAnalysisProvider.HttpCall> call = new AtomicReference<>();
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9000/analyze");
properties.getEditing().getVisualAnalysis().setTimeoutMs(12000);
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
properties,
new ObjectMapper().findAndRegisterModules(),
request -> {
call.set(request);
return new LocalCvVisualAnalysisProvider.HttpResult(200, """
{
"clipId": "source",
"blurScore": 0.81,
"exposureScore": 0.67,
"motionScore": 0.73,
"compositionScore": 0.79,
"facePresence": "faces_detected",
"objectLabels": [
{"label": "car", "confidence": 0.91, "source": "yolo"}
],
"representativeThumbnails": ["thumb-2.jpg"],
"analysisMethod": "local_cv_yolo_clip"
}
""");
}
);
SourceVisualAnalysis analysis = provider.analyze(clip(), List.of("thumb-1.jpg", "thumb-2.jpg"),
List.of(new ShotSegment("shot_0001", 0, 8, 8, 0, 4)));
assertThat(call.get().endpoint()).isEqualTo("http://localhost:9000/analyze");
assertThat(call.get().timeoutMs()).isEqualTo(12000);
assertThat(call.get().body()).contains("\"clipId\":\"source\"", "\"thumbnails\":[\"thumb-1.jpg\"");
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("car", 0.91, "yolo"));
assertThat(analysis.facePresence()).isEqualTo("faces_detected");
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_yolo_clip");
}
@Test
void rejectsNonSuccessfulLocalCvResponses() {
LocalCvVisualAnalysisProvider provider = new LocalCvVisualAnalysisProvider(
new VideoClippingProperties(),
new ObjectMapper().findAndRegisterModules(),
request -> new LocalCvVisualAnalysisProvider.HttpResult(500, "{}")
);
assertThrows(IllegalStateException.class, () -> provider.analyze(clip(), List.of(), List.of()));
}
@Test
void reportsLocalCvIoAndInterruptionFailures() {
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
new VideoClippingProperties(),
new ObjectMapper().findAndRegisterModules(),
request -> {
throw new IOException("connection refused");
}).analyze(clip(), List.of(), List.of()));
try {
assertThrows(IllegalStateException.class, () -> new LocalCvVisualAnalysisProvider(
new VideoClippingProperties(),
new ObjectMapper().findAndRegisterModules(),
request -> {
throw new InterruptedException("stopped");
}).analyze(clip(), List.of(), List.of()));
assertTrue(Thread.currentThread().isInterrupted());
} finally {
Thread.interrupted();
}
}
private ClipAnalysis clip() {
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", "aac",
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
}
}

View File

@ -0,0 +1,250 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class LocalCvWorkerProcessManagerTest {
@TempDir
Path tempDir;
@Test
void doesNotStartWhenProviderIsNotLocalCv() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("heuristic");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
throw new AssertionError("worker should not start");
});
manager.start();
assertThat(manager.isRunning()).isFalse();
}
@Test
void doesNotStartWhenAutoStartIsDisabled() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(false);
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
throw new AssertionError("worker should not start");
});
manager.start();
assertThat(manager.isRunning()).isFalse();
}
@Test
void startsWorkerScriptWithEndpointHostAndPort() throws Exception {
Path script = tempDir.resolve("run-worker.sh");
Files.writeString(script, "#!/usr/bin/env bash\n");
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().setEndpoint("http://localhost:9005/v1/analyze-visuals");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
AtomicReference<ProcessBuilder> captured = new AtomicReference<>();
AtomicReference<URI> healthUri = new AtomicReference<>();
TestProcess process = new TestProcess();
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
properties,
processBuilder -> {
captured.set(processBuilder);
return process;
},
(uri, timeoutMs) -> {
healthUri.set(uri);
return new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok");
});
manager.start();
assertThat(manager.isRunning()).isTrue();
assertThat(captured.get().command()).containsExactly(script.toAbsolutePath().normalize().toString());
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_HOST", "localhost");
assertThat(captured.get().environment()).containsEntry("LOCAL_CV_PORT", "9005");
assertThat(healthUri.get()).hasToString("http://localhost:9005/health");
manager.stop();
assertThat(process.destroyed).isTrue();
assertThat(manager.isRunning()).isFalse();
}
@Test
void rejectsMissingWorkerScriptWhenAutoStartIsEnabled() {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(tempDir.resolve("missing.sh").toString());
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(properties, processBuilder -> {
throw new AssertionError("worker should not start");
});
assertThatThrownBy(manager::start)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("Local CV worker script does not exist");
}
@Test
void resetsRunningStateWhenWorkerExitsBeforeHealthCheckPasses() throws Exception {
Path script = tempDir.resolve("run-worker.sh");
Files.writeString(script, "#!/usr/bin/env bash\n");
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
TestProcess process = new TestProcess();
process.alive = false;
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
properties,
processBuilder -> process,
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused"));
assertThatThrownBy(manager::start)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("Local CV worker exited before becoming healthy");
assertThat(manager.isRunning()).isFalse();
}
@Test
void failsWhenWorkerDoesNotBecomeHealthyBeforeTimeout() throws Exception {
Path script = tempDir.resolve("run-worker.sh");
Files.writeString(script, "#!/usr/bin/env bash\n");
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(1);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
TestProcess process = new TestProcess();
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
properties,
processBuilder -> process,
(uri, timeoutMs) -> new LocalCvWorkerProcessManager.HealthResult(false, 503, "starting"));
assertThatThrownBy(manager::start)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("Local CV worker did not become healthy");
assertThat(process.destroyed).isTrue();
assertThat(manager.isRunning()).isFalse();
}
@Test
void retriesHealthChecksUntilWorkerIsReady() throws Exception {
Path script = tempDir.resolve("run-worker.sh");
Files.writeString(script, "#!/usr/bin/env bash\n");
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setAutoStart(true);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setScript(script.toString());
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setStartupWaitMs(100);
properties.getEditing().getVisualAnalysis().getLocalCvWorker().setHealthCheckIntervalMs(1);
AtomicInteger attempts = new AtomicInteger();
TestProcess process = new TestProcess();
LocalCvWorkerProcessManager manager = new LocalCvWorkerProcessManager(
properties,
processBuilder -> process,
(uri, timeoutMs) -> attempts.incrementAndGet() < 2
? new LocalCvWorkerProcessManager.HealthResult(false, 0, "connection refused")
: new LocalCvWorkerProcessManager.HealthResult(true, 200, "ok"));
manager.start();
assertThat(attempts.get()).isEqualTo(2);
assertThat(manager.isRunning()).isTrue();
manager.stop();
}
private static class TestProcess extends Process {
private boolean alive = true;
private boolean destroyed;
@Override
public OutputStream getOutputStream() {
return OutputStream.nullOutputStream();
}
@Override
public InputStream getInputStream() {
return new ByteArrayInputStream("worker ready\n".getBytes());
}
@Override
public InputStream getErrorStream() {
return InputStream.nullInputStream();
}
@Override
public synchronized int waitFor() throws InterruptedException {
while (alive) {
wait();
}
return 0;
}
@Override
public synchronized boolean waitFor(long timeout, TimeUnit unit) throws InterruptedException {
if (alive) {
wait(unit.toMillis(timeout));
}
return !alive;
}
@Override
public int exitValue() {
return alive ? 1 : 0;
}
@Override
public synchronized void destroy() {
destroyed = true;
alive = false;
notifyAll();
}
@Override
public synchronized Process destroyForcibly() {
destroyed = true;
alive = false;
notifyAll();
return this;
}
@Override
public synchronized boolean isAlive() {
return alive;
}
@Override
public long pid() {
return 1234;
}
}
}

View File

@ -0,0 +1,90 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
class ShotSceneSegmenterTest {
@Test
void detectsShotSegmentsFromFfmpegSceneMetadata() {
AtomicReference<List<String>> command = new AtomicReference<>();
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setSceneDetectionThreshold(0.42);
properties.getEditing().setMinimumSceneDurationSeconds(1.5);
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(properties, arguments -> {
command.set(arguments);
return new ShotSceneSegmenter.ProcessResult(0, """
frame:1 pts:72000 pts_time:2.400
lavfi.scene_score=0.510
frame:2 pts:180000 pts_time:6.000
lavfi.scene_score=0.730
""");
});
List<ShotSegment> shots = segmenter.segment(clip(9));
assertThat(shots).containsExactly(
new ShotSegment("shot_0001", 0, 2.4, 2.4, 0, 1.2),
new ShotSegment("shot_0002", 2.4, 6, 3.6, 0.51, 4.2),
new ShotSegment("shot_0003", 6, 9, 3, 0.73, 7.5)
);
assertCommandPair(command.get(), "-i", "source.mp4");
assertCommandPair(command.get(), "-filter:v", "select='gt(scene,0.42)',metadata=print");
}
@Test
void returnsSingleShotWhenNoSceneCutPassesMinimumDurationRules() {
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), arguments ->
new ShotSceneSegmenter.ProcessResult(0, """
frame:1 pts:12000 pts_time:0.400
lavfi.scene_score=0.900
"""));
assertThat(segmenter.segment(clip(4))).containsExactly(
new ShotSegment("shot_0001", 0, 4, 4, 0, 2)
);
}
@Test
void rejectsFfmpegFailures() {
ShotSceneSegmenter segmenter = new ShotSceneSegmenter(new VideoClippingProperties(), command ->
new ShotSceneSegmenter.ProcessResult(1, "failed"));
assertThrows(IllegalStateException.class, () -> segmenter.segment(clip(9)));
}
@Test
void reportsProcessStartupAndInterruptionFailures() {
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
throw new IOException("missing executable");
}).segment(clip(9)));
try {
assertThrows(IllegalStateException.class, () -> new ShotSceneSegmenter(new VideoClippingProperties(), command -> {
throw new InterruptedException("stopped");
}).segment(clip(9)));
assertTrue(Thread.currentThread().isInterrupted());
} finally {
Thread.interrupted();
}
}
private ClipAnalysis clip(double durationSeconds) {
return new ClipAnalysis("source", "source.mp4", durationSeconds, "h264", "aac",
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
}
private void assertCommandPair(List<String> command, String option, String value) {
int index = command.indexOf(option);
assertThat(index).isGreaterThanOrEqualTo(0);
assertThat(command.get(index + 1)).isEqualTo(value);
}
}

View File

@ -0,0 +1,96 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
class SourceAudioAnalyzerTest {
@Test
void analyzesLoudnessSilenceAndAudioSections() {
AtomicReference<List<String>> command = new AtomicReference<>();
VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setSilenceThresholdDb(-40);
properties.getEditing().setSilenceMinimumDurationSeconds(1.25);
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(properties, arguments -> {
command.set(arguments);
return new SourceAudioAnalyzer.ProcessResult(0, """
[silencedetect @ 0x1] silence_start: 2.5
[silencedetect @ 0x1] silence_end: 4.0 | silence_duration: 1.5
[Parsed_volumedetect_1 @ 0x2] mean_volume: -19.2 dB
[Parsed_volumedetect_1 @ 0x2] max_volume: -2.7 dB
""");
});
SourceAudioAnalysis analysis = analyzer.analyze(clip("aac"));
assertThat(analysis.audioPresent()).isTrue();
assertThat(analysis.meanVolumeDb()).isEqualTo(-19.2);
assertThat(analysis.maxVolumeDb()).isEqualTo(-2.7);
assertThat(analysis.silenceThresholdDb()).isEqualTo(-40);
assertThat(analysis.silenceDurationSeconds()).isEqualTo(1.25);
assertThat(analysis.sections()).containsExactly(
new AudioSection("audio_section_0001", "unclassified_audio", 0, 2.5, 2.5),
new AudioSection("audio_section_0002", "silence", 2.5, 4, 1.5),
new AudioSection("audio_section_0003", "unclassified_audio", 4, 8, 4)
);
assertCommandPair(command.get(), "-i", "source.mp4");
assertCommandPair(command.get(), "-af", "silencedetect=n=-40.0dB:d=1.25,volumedetect");
}
@Test
void returnsMissingAudioAnalysisWhenSourceHasNoAudioCodec() {
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
throw new AssertionError("ffmpeg should not run without an audio stream");
});
SourceAudioAnalysis analysis = analyzer.analyze(clip(null));
assertThat(analysis.audioPresent()).isFalse();
assertThat(analysis.sections()).containsExactly(
new AudioSection("audio_section_0001", "missing_audio", 0, 8, 8)
);
}
@Test
void rejectsFfmpegFailures() {
SourceAudioAnalyzer analyzer = new SourceAudioAnalyzer(new VideoClippingProperties(), command ->
new SourceAudioAnalyzer.ProcessResult(1, "failed"));
assertThrows(IllegalStateException.class, () -> analyzer.analyze(clip("aac")));
}
@Test
void reportsProcessStartupAndInterruptionFailures() {
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
throw new IOException("missing executable");
}).analyze(clip("aac")));
try {
assertThrows(IllegalStateException.class, () -> new SourceAudioAnalyzer(new VideoClippingProperties(), command -> {
throw new InterruptedException("stopped");
}).analyze(clip("aac")));
assertTrue(Thread.currentThread().isInterrupted());
} finally {
Thread.interrupted();
}
}
private ClipAnalysis clip(String audioCodec) {
return new ClipAnalysis("source", "source.mp4", 8.0, "h264", audioCodec,
1920, 1080, 30.0, List.of(), null, null, 0, 0, 0);
}
private void assertCommandPair(List<String> command, String option, String value) {
int index = command.indexOf(option);
assertThat(index).isGreaterThanOrEqualTo(0);
assertThat(command.get(index + 1)).isEqualTo(value);
}
}

View File

@ -0,0 +1,130 @@
package org.example.videoclips.editing;
import org.junit.jupiter.api.Test;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
class SourceVisualAnalyzerTest {
private final SourceVisualAnalyzer analyzer = new SourceVisualAnalyzer();
@Test
void createsVisualAnalysisFromScoresThumbnailsAndSceneDensity() {
SourceVisualAnalysis analysis = analyzer.analyze(
clip("porsche-drive", "source/Porsche Drive.mp4", 0.72, 0.61, 0.0),
List.of("thumb-1.jpg", "thumb-2.jpg", "thumb-3.jpg", "thumb-4.jpg", "thumb-5.jpg"),
List.of(
new ShotSegment("shot_0001", 0, 8, 8, 0, 4),
new ShotSegment("shot_0002", 8, 16, 8, 0.5, 12),
new ShotSegment("shot_0003", 16, 24, 8, 0.6, 20)
)
);
assertThat(analysis.blurScore()).isEqualTo(0.72);
assertThat(analysis.exposureScore()).isEqualTo(0.61);
assertThat(analysis.motionScore()).isEqualTo(0.5);
assertThat(analysis.compositionScore()).isEqualTo(0.75);
assertThat(analysis.facePresence()).isEqualTo("unknown_without_face_detector");
assertThat(analysis.objectLabels()).containsExactly(
new VisualObjectLabel("porsche", 0.82, "metadata_heuristic"),
new VisualObjectLabel("car", 0.62, "metadata_heuristic")
);
assertThat(analysis.representativeThumbnails()).containsExactly("thumb-1.jpg", "thumb-3.jpg", "thumb-5.jpg");
assertThat(analysis.analysisMethod()).isEqualTo("metadata_thumbnail_scene_heuristic");
}
@Test
void fallsBackWhenNoVisualSignalsAreAvailable() {
SourceVisualAnalysis analysis = analyzer.analyze(
clip("unknown", "source/video.mp4", 0, 0, 0),
List.of(),
List.of()
);
assertThat(analysis.blurScore()).isEqualTo(0.5);
assertThat(analysis.exposureScore()).isEqualTo(0.5);
assertThat(analysis.motionScore()).isEqualTo(0.3);
assertThat(analysis.compositionScore()).isEqualTo(0.625);
assertThat(analysis.objectLabels()).containsExactly(
new VisualObjectLabel("unknown", 0.2, "metadata_heuristic")
);
assertThat(analysis.representativeThumbnails()).isEmpty();
}
@Test
void usesLocalCvProviderWhenConfigured() {
org.example.videoclips.config.VideoClippingProperties properties =
new org.example.videoclips.config.VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
properties,
(source, thumbnails, shotSegments) -> {
throw new AssertionError("heuristic fallback should not run");
},
(source, thumbnails, shotSegments) -> new SourceVisualAnalysis(source.clipId(), 0.9, 0.8,
0.7, 0.6, "faces_detected",
List.of(new VisualObjectLabel("person", 0.91, "mediapipe")), thumbnails, "local_cv")
);
SourceVisualAnalysis analysis = localAnalyzer.analyze(
clip("family", "family.mp4", 0, 0, 0),
List.of("thumb.jpg"),
List.of()
);
assertThat(analysis.analysisMethod()).isEqualTo("local_cv");
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("person", 0.91, "mediapipe"));
}
@Test
void fallsBackToHeuristicWhenLocalCvFailsAndFallbackIsEnabled() {
org.example.videoclips.config.VideoClippingProperties properties =
new org.example.videoclips.config.VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
properties,
new HeuristicVisualAnalysisProvider(),
(source, thumbnails, shotSegments) -> {
throw new IllegalStateException("local cv unavailable");
}
);
SourceVisualAnalysis analysis = localAnalyzer.analyze(
clip("porsche", "porsche.mp4", 0, 0, 0),
List.of("thumb.jpg"),
List.of()
);
assertThat(analysis.analysisMethod()).isEqualTo("local_cv_failed_fallback_metadata_thumbnail_scene_heuristic");
assertThat(analysis.objectLabels()).containsExactly(new VisualObjectLabel("porsche", 0.82,
"metadata_heuristic"));
}
@Test
void failsWhenLocalCvFailsAndFallbackIsDisabled() {
org.example.videoclips.config.VideoClippingProperties properties =
new org.example.videoclips.config.VideoClippingProperties();
properties.getEditing().getVisualAnalysis().setProvider("local-cv");
properties.getEditing().getVisualAnalysis().setFallbackToHeuristic(false);
SourceVisualAnalyzer localAnalyzer = new SourceVisualAnalyzer(
properties,
new HeuristicVisualAnalysisProvider(),
(source, thumbnails, shotSegments) -> {
throw new IllegalStateException("local cv unavailable");
}
);
org.junit.jupiter.api.Assertions.assertThrows(IllegalStateException.class, () -> localAnalyzer.analyze(
clip("porsche", "porsche.mp4", 0, 0, 0),
List.of("thumb.jpg"),
List.of()
));
}
private ClipAnalysis clip(String clipId, String sourcePath, double sharpness, double brightness, double motion) {
return new ClipAnalysis(clipId, sourcePath, 24.0, "h264", "aac", 1920, 1080, 30.0,
List.of(), null, null, motion, brightness, sharpness);
}
}

View File

@ -30,6 +30,7 @@ class StoryboardPromptGeneratorTest {
assertThat(prompt)
.contains("clipId: clip_00001")
.contains("durationSeconds: 8.0")
.contains("The service can only render what the plan expresses")
.contains("Target duration: 60 seconds")
.contains("Style: `cinematic-porsche-promo`")
.contains("\"decisions\"")

View File

@ -0,0 +1,4 @@
video-clipping.folder-scheduler.enabled=false
video-clipping.editing.local-director.enabled=false
video-clipping.editing.highlight-scheduler.enabled=false
video-clipping.editing.visual-analysis.local-cv-worker.auto-start=false

View File

@ -0,0 +1,4 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
opencv-python==4.10.0.84
ultralytics==8.3.57

235
tools/local_cv_worker.py Normal file
View File

@ -0,0 +1,235 @@
#!/usr/bin/env python3
"""Optional local CV worker for the Spring visual-analysis provider.
Run:
pip install -r tools/local_cv_requirements.txt
uvicorn tools.local_cv_worker:app --host 127.0.0.1 --port 8091
"""
from __future__ import annotations
import os
import logging
import time
from pathlib import Path
from typing import Any
try:
import cv2 # type: ignore
except ImportError: # pragma: no cover - optional runtime dependency
cv2 = None
try:
from fastapi import FastAPI
except ImportError as exc: # pragma: no cover - fail clearly at worker startup
raise SystemExit("Install FastAPI first: pip install fastapi uvicorn") from exc
try:
from ultralytics import YOLO # type: ignore
except ImportError: # pragma: no cover - optional runtime dependency
YOLO = None
app = FastAPI(title="Local CV Visual Analysis Worker")
_yolo_model: Any | None = None
logging.basicConfig(level=os.getenv("LOCAL_CV_LOG_LEVEL", "INFO"))
log = logging.getLogger("local_cv_worker")
@app.get("/health")
def health() -> dict[str, Any]:
return {
"status": "ok",
"opencvAvailable": cv2 is not None,
"yoloAvailable": YOLO is not None,
"yoloModel": configured_yolo_model(),
}
@app.post("/v1/analyze-visuals")
def analyze_visuals(payload: dict[str, Any]) -> dict[str, Any]:
started_at = time.monotonic()
source = payload.get("source", {})
thumbnails = [str(item) for item in payload.get("thumbnails", [])]
readable_thumbnails = [path for path in thumbnails if Path(path).is_file()]
clip_id = source.get("clipId", "source")
log.info(
"event=local_cv_worker_request_started clip_id=%s thumbnails=%d readable_thumbnails=%d "
"shot_segments=%d duration_seconds=%s",
clip_id,
len(thumbnails),
len(readable_thumbnails),
len(payload.get("shotSegments", [])),
source.get("durationSeconds", 0),
)
if len(readable_thumbnails) != len(thumbnails):
unreadable = [path for path in thumbnails if path not in readable_thumbnails]
log.warning(
"event=local_cv_worker_unreadable_thumbnails clip_id=%s unreadable_count=%d unreadable=%s",
clip_id,
len(unreadable),
unreadable[:10],
)
quality = quality_scores(readable_thumbnails)
labels = object_labels(readable_thumbnails)
response = {
"clipId": clip_id,
"blurScore": quality["blurScore"],
"exposureScore": quality["exposureScore"],
"motionScore": motion_score(payload.get("shotSegments", []), source.get("durationSeconds", 0)),
"compositionScore": composition_score(source, readable_thumbnails),
"facePresence": face_presence(readable_thumbnails),
"objectLabels": labels or [{"label": "unknown", "confidence": 0.2, "source": "local_cv_worker"}],
"representativeThumbnails": representative_thumbnails(readable_thumbnails or thumbnails),
"analysisMethod": "local_cv_worker_opencv_yolo" if labels else "local_cv_worker_opencv",
}
log.info(
"event=local_cv_worker_request_completed clip_id=%s elapsed_ms=%d blur_score=%s "
"exposure_score=%s object_labels=%d method=%s",
clip_id,
elapsed_ms(started_at),
response["blurScore"],
response["exposureScore"],
len(response["objectLabels"]),
response["analysisMethod"],
)
return response
def quality_scores(thumbnails: list[str]) -> dict[str, float]:
if cv2 is None or not thumbnails:
log.info(
"event=local_cv_worker_quality_defaulted reason=%s",
"opencv_unavailable" if cv2 is None else "no_readable_thumbnails",
)
return {"blurScore": 0.5, "exposureScore": 0.5}
blur_scores: list[float] = []
exposure_scores: list[float] = []
for thumbnail in thumbnails:
image = cv2.imread(thumbnail)
if image is None:
continue
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
laplacian_variance = float(cv2.Laplacian(gray, cv2.CV_64F).var())
blur_scores.append(clamp(laplacian_variance / 500.0))
mean_brightness = float(gray.mean()) / 255.0
exposure_scores.append(clamp(1.0 - abs(mean_brightness - 0.5) * 2.0))
return {
"blurScore": average_or_default(blur_scores, 0.5),
"exposureScore": average_or_default(exposure_scores, 0.5),
}
def object_labels(thumbnails: list[str]) -> list[dict[str, Any]]:
started_at = time.monotonic()
model = yolo_model()
if model is None or not thumbnails:
log.info(
"event=local_cv_worker_object_detection_skipped reason=%s thumbnails=%d",
"model_unavailable" if model is None else "no_readable_thumbnails",
len(thumbnails),
)
return []
labels: dict[str, float] = {}
for result in model(thumbnails, verbose=False):
names = getattr(result, "names", {})
boxes = getattr(result, "boxes", None)
if boxes is None:
continue
for box in boxes:
label = str(names.get(int(box.cls[0]), "object"))
confidence = float(box.conf[0])
labels[label] = max(labels.get(label, 0.0), confidence)
detected = [
{"label": label, "confidence": round(clamp(confidence), 3), "source": "yolo"}
for label, confidence in sorted(labels.items(), key=lambda item: item[1], reverse=True)[:10]
]
log.info(
"event=local_cv_worker_object_detection_completed elapsed_ms=%d thumbnails=%d labels=%d",
elapsed_ms(started_at),
len(thumbnails),
len(detected),
)
return detected
def yolo_model() -> Any | None:
global _yolo_model
model_path = configured_yolo_model()
if YOLO is None or not model_path:
return None
if _yolo_model is None:
started_at = time.monotonic()
log.info("event=local_cv_worker_yolo_model_loading model=%s", model_path)
_yolo_model = YOLO(model_path)
log.info("event=local_cv_worker_yolo_model_loaded model=%s elapsed_ms=%d", model_path, elapsed_ms(started_at))
return _yolo_model
def configured_yolo_model() -> str:
if os.getenv("LOCAL_CV_DISABLE_YOLO", "false").lower() == "true":
return ""
return os.getenv("LOCAL_CV_YOLO_MODEL", "yolov8n.pt")
def face_presence(thumbnails: list[str]) -> str:
if cv2 is None or not thumbnails:
return "unknown_without_face_detector"
cascade_path = getattr(cv2.data, "haarcascades", "") + "haarcascade_frontalface_default.xml"
if not Path(cascade_path).is_file():
return "unknown_without_face_detector"
cascade = cv2.CascadeClassifier(cascade_path)
for thumbnail in thumbnails:
image = cv2.imread(thumbnail)
if image is None:
continue
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
faces = cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=4)
if len(faces) > 0:
return "faces_detected"
return "no_faces_detected"
def motion_score(shot_segments: list[dict[str, Any]], duration_seconds: float) -> float:
try:
duration = float(duration_seconds)
except (TypeError, ValueError):
duration = 0.0
if duration <= 0:
return 0.3
cuts_per_minute = max(0, len(shot_segments) - 1) / duration * 60.0
return round(clamp(0.25 + cuts_per_minute / 20.0), 3)
def composition_score(source: dict[str, Any], thumbnails: list[str]) -> float:
width = float(source.get("width", 0) or 0)
height = float(source.get("height", 0) or 0)
aspect_ratio = width / height if height else 0
aspect_score = 0.8 if 1.70 <= aspect_ratio <= 1.90 else 0.55
thumbnail_score = 0.7 if thumbnails else 0.45
return round((aspect_score + thumbnail_score) / 2.0, 3)
def representative_thumbnails(thumbnails: list[str]) -> list[str]:
if len(thumbnails) <= 3:
return thumbnails
return [thumbnails[0], thumbnails[len(thumbnails) // 2], thumbnails[-1]]
def average_or_default(values: list[float], default: float) -> float:
if not values:
return default
return round(sum(values) / len(values), 3)
def clamp(value: float) -> float:
return max(0.0, min(1.0, value))
def elapsed_ms(started_at: float) -> int:
return int((time.monotonic() - started_at) * 1000)

25
tools/run_local_cv_worker.sh Executable file
View File

@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
HOST="${LOCAL_CV_HOST:-127.0.0.1}"
PORT="${LOCAL_CV_PORT:-8091}"
MODEL="${LOCAL_CV_YOLO_MODEL:-yolov8n.pt}"
if [ ! -d ".venv-local-cv" ]; then
python3 -m venv .venv-local-cv
fi
. .venv-local-cv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r tools/local_cv_requirements.txt
if [ "${LOCAL_CV_DISABLE_YOLO:-false}" != "true" ]; then
LOCAL_CV_YOLO_MODEL="$MODEL" python -c "import os; from ultralytics import YOLO; YOLO(os.environ['LOCAL_CV_YOLO_MODEL'])"
fi
if [ "${LOCAL_CV_PRELOAD_ONLY:-false}" = "true" ]; then
echo "Local CV dependencies and model are ready."
exit 0
fi
exec uvicorn tools.local_cv_worker:app --host "$HOST" --port "$PORT"