add asset generation

This commit is contained in:
JSLMPR 2026-07-21 10:03:27 +02:00
parent 618ba9af49
commit 97ba827d50
77 changed files with 1689 additions and 150 deletions

View File

@ -0,0 +1,98 @@
# Cinematic Highlight Operator Checklist
Use this when you want one source video turned into a cinematic highlight project.
## Start Here
1. Start the service with the cinematic local profile:
```bash
mvn spring-boot:run -Dspring-boot.run.profiles=cinematic-editing-local
```
2. Put one valid source video in:
```text
input/highlights/source/
```
3. Watch the logs for:
- `event=highlight_scan_started`
- `event=highlight_candidate_selected`
- `event=highlight_project_created`
- `event=highlight_director_prompt_generated`
- `event=highlight_scan_completed`
## Exact Project Folders
Project root:
```text
output/highlight-projects/<project-id>/
```
Files and folders the service writes:
- `project.json`
- `source/`
- `analysis/source-analysis.json`
- `analysis/category.json`
- `analysis/highlight-candidates.json`
- `director/director-brief.md`
- `director/director-prompt.md`
- `director/edit-plan.json`
- `highlights/<highlight-id>/storyboard.md`
- `highlights/<highlight-id>/visual-effects.json`
- `highlights/<highlight-id>/assets/requests/`
- `highlights/<highlight-id>/assets/music/`
- `highlights/<highlight-id>/assets/sfx/`
- `highlights/<highlight-id>/assets/voiceover/`
- `highlights/<highlight-id>/rendered-clips/`
- `highlights/<highlight-id>/preview.mp4`
- `highlights/<highlight-id>/final.mp4`
- `highlights/<highlight-id>/render-manifest.json`
- `highlights/<highlight-id>/qa-report.json`
## Exact Operator Flow
1. Wait for the service to create the highlight project.
2. Open:
```text
output/highlight-projects/<project-id>/director/director-prompt.md
```
3. Run Codex, Claude, or another filesystem-capable AI instance on that prompt.
4. Make the AI write the finished plan to:
```text
output/highlight-projects/<project-id>/director/edit-plan.json
```
5. Confirm `project.json` moved to `WAITING_FOR_DIRECTOR` after prompt generation.
6. The service now turns the director plan into `highlights/<highlight-id>/visual-effects.json`.
7. The asset bridge writes request files and the local asset worker resolves what it can into `highlights/<highlight-id>/assets/`.
8. The renderer publishes `rendered-clips/clip_*.mp4`, `preview.mp4`, `final.mp4`, `render-manifest.json`, and `qa-report.json`.
## What Is Automatic Today
- The scheduler analyzes the source video.
- The service writes the director prompt and brief.
- Existing analyzed projects are backfilled on startup if the prompt is missing.
- The visual-effects stage writes explicit renderer instructions.
- The local asset worker tries to resolve or generate reusable music, SFX, and voiceover assets.
## What Is Still Manual
- Running the AI director.
- Triggering any downstream render step that your current flow requires.
## If Output Is Missing
- No file in `input/highlights/source/`
- Unsupported video file
- The scheduler has not run yet
- `output/highlight-projects/<project-id>/director/director-prompt.md` is missing
- `director/edit-plan.json` was not written by the AI
- Rendering is not wired for the current highlight flow yet

View File

@ -0,0 +1,204 @@
# Local Cinematic Asset Worker Prompt
Use this prompt for the local asset worker that runs on the VPS after the director plan has been written.
## Purpose
Generate production-grade local assets for a cinematic video edit:
- spoken voiceover audio
- cinematic music bed
- sound effects
- a machine-readable asset manifest
The worker must use the best local model available for each job:
- `Piper` or another local neural TTS engine for voiceover
- `MusicGen` via `AudioCraft` for music
- `AudioGen` via `AudioCraft` for SFX
If a preferred model is unavailable, fall back to the next best local option, but do not invent fake assets or silently skip required outputs.
## Inputs
Work only with the current project folder and its generated plan:
- `director/edit-plan.json`
- `highlights/<highlight-id>/edit-plan.json`
- `highlights/<highlight-id>/storyboard.md`
- `highlights/<highlight-id>/visual-effects.json`
- `analysis/`
- `assets/requests/`
- `assets/music/`
- `assets/sfx/`
- `assets/voiceover/`
Use the analysis artifacts and director plan as the source of truth. Do not invent scene facts, car specs, people, locations, or actions that are not visible in the footage or explicitly stated in the plan.
## Quality Bar
The generated assets must feel intentional and cinematic:
- voiceover must sound natural, premium, and grounded in visible footage
- music must match the project category and energy curve
- SFX must support cuts, reveals, transitions, and emphasis without sounding noisy or random
- assets must be reusable in future projects when possible
- the result must be good enough for public upload, not placeholder quality
## Model Selection
Prefer this local model split:
1. Voiceover
- Use a local neural TTS model first.
- Prefer `Piper` for fast, reliable, local synthesis.
- If a higher-quality local TTS engine is available on the VPS, use it only if it clearly improves speech naturalness and pronunciation.
- Keep delivery human, clear, and premium.
2. Music
- Use `MusicGen` through `AudioCraft` for text-to-music generation.
- Generate a loopable or long-form bed that can cover the timeline cleanly.
- Preserve room for voiceover and key SFX.
3. SFX
- Use `AudioGen` through `AudioCraft` for text-to-sound generation.
- If a specific SFX prompt produces weak results, retry with a shorter, more concrete prompt.
- Use category-aware effects such as whooshes, rises, impacts, camera accents, engine tones, texture hits, and ambience.
## Instructions
You are the local cinematic asset worker.
Your job is to create the best possible reusable asset set for this project.
Follow these rules:
- Read the edit plan first.
- Read the storyboard and visual effects plan second.
- Use the project category to drive style, pacing, tone, and sound design.
- Keep voiceover grounded in visible footage.
- Keep music and SFX supportive, not distracting.
- Generate assets in stable filenames so the renderer can reuse them.
- Write a manifest that records every generated, reused, and skipped asset.
- Prefer generating one strong asset over several weak variants.
- If the prompt asks for a voiceover line, synthesize it to WAV.
- If the prompt asks for music, synthesize a WAV bed that spans the intended duration.
- If the prompt asks for SFX, synthesize one WAV per cue.
- If generation fails, retry once with a tighter prompt.
- If it still fails, write a request record and mark the asset as pending.
## Voiceover Requirements
When voiceover is enabled:
- synthesize every required line to `assets/voiceover/voiceover.wav`
- if the plan contains multiple lines, synthesize them into a single clean narration track unless the renderer expects split assets
- choose a voice that fits the category
- use confident, concise, polished delivery
- do not sound salesy, robotic, or exaggerated
- keep wording faithful to visible footage
Voiceover style by category:
- `car_vlog`: confident, premium, controlled, slightly aspirational
- `family_vlog`: warm, human, intimate, gentle
- `food_vlog`: sensory, warm, appetizing, precise
- `generic_vlog`: clean, observational, polished, grounded
## Music Requirements
When music is needed:
- generate a cinematic bed that matches the story arc
- build energy from intro to payoff
- leave a pocket for voiceover clarity
- avoid constant high-intensity density
- choose tone based on category and scene
Music style by category:
- `car_vlog`: confident pulse, modern cinematic drive, controlled bass, sleek tension
- `family_vlog`: emotional, warm, reflective, soft rise
- `food_vlog`: light groove, tactile warmth, subtle motion
- `generic_vlog`: tasteful atmospheric score with restrained lift
## SFX Requirements
When SFX are needed:
- align each effect to a cut, reveal, beat change, or visual emphasis
- use short, clean transients rather than heavy clutter
- keep effects category-aware
- avoid overusing impacts
Useful SFX prompts by category:
- `car_vlog`: soft whoosh, bass hit, subtle rise, mechanical accent, road ambience
- `family_vlog`: gentle whoosh, soft room tone, warm ambience, subtle scene transition
- `food_vlog`: tactile foley, pour, chop, sizzle, plate tap, light texture accents
- `generic_vlog`: clean whoosh, understated hit, transition swell, ambient bed
## Output Files
Create these files when applicable:
- `assets/voiceover/voiceover.wav`
- `assets/music/music.wav`
- `assets/sfx/<asset-key>.wav`
- `assets/generated-assets.json`
- `assets/asset-generation-manifest.json`
Also create request files for anything still missing:
- `assets/requests/<type>-<asset-key>.md`
- `assets/requests/<type>-<asset-key>.json`
## Manifest Requirements
The manifest must include:
- project id
- highlight id
- asset type
- asset key
- source model or provider
- prompt text
- output path
- status: `generated`, `reused`, or `pending`
- reason if not generated
## Suggested Prompt Templates
### Voiceover Prompt
Write a short, polished narration for this footage.
Keep it grounded in what is visibly shown.
Match the category tone.
Avoid unsupported facts.
Avoid generic praise.
Make the result sound premium and human.
### Music Prompt
Create a cinematic background track for this edit.
Match the category tone and the pacing curve in the director plan.
Keep the track polished, modern, and supportive of voiceover.
Use a clear intro, a controlled build, and a satisfying payoff.
### SFX Prompt
Create a precise cinematic sound effect for the described edit moment.
Make it short, clean, and category-appropriate.
Use the effect to support the cut or reveal, not to overwhelm the scene.
## Final Acceptance Criteria
The asset stage is done only when:
- voiceover audio exists if voiceover is enabled
- music audio exists if music is enabled
- required SFX files exist or are explicitly marked pending
- the manifest records what was generated and what was reused
- filenames are stable and project-local
- assets are ready for the renderer without manual cleanup

View File

@ -530,7 +530,7 @@ public class VideoClippingProperties {
private String audioBitrate = "192k"; private String audioBitrate = "192k";
private String voiceoverProvider = "noop"; private String voiceoverProvider = "local";
private double loudnessTargetI = -16.0; private double loudnessTargetI = -16.0;
@ -552,6 +552,8 @@ public class VideoClippingProperties {
private final VisualAnalysis visualAnalysis = new VisualAnalysis(); private final VisualAnalysis visualAnalysis = new VisualAnalysis();
private final LocalAssetWorker localAssetWorker = new LocalAssetWorker();
private final LocalDirector localDirector = new LocalDirector(); private final LocalDirector localDirector = new LocalDirector();
private final HighlightScheduler highlightScheduler = new HighlightScheduler(); private final HighlightScheduler highlightScheduler = new HighlightScheduler();
@ -788,6 +790,10 @@ public class VideoClippingProperties {
return visualAnalysis; return visualAnalysis;
} }
public LocalAssetWorker getLocalAssetWorker() {
return localAssetWorker;
}
public LocalDirector getLocalDirector() { public LocalDirector getLocalDirector() {
return localDirector; return localDirector;
} }
@ -951,6 +957,130 @@ public class VideoClippingProperties {
} }
} }
public static class LocalAssetWorker {
private boolean autoStart = false;
private boolean strictRuntime = false;
private String bootstrapScript = "./tools/run_local_asset_worker.sh";
private String script = "./tools/local_asset_worker.py";
@Min(0)
private long startupWaitMs = 0;
private String healthPath = "/health";
@Min(1)
private long healthCheckIntervalMs = 1000;
private String pythonBinary = "./.venv-local-asset/bin/python";
private String piperBinary = "piper";
private String piperModelPath = "";
private String musicModel = "musicgen-small";
private String sfxModel = "audiogen-medium";
public boolean isAutoStart() {
return autoStart;
}
public void setAutoStart(boolean autoStart) {
this.autoStart = autoStart;
}
public boolean isStrictRuntime() {
return strictRuntime;
}
public void setStrictRuntime(boolean strictRuntime) {
this.strictRuntime = strictRuntime;
}
public String getBootstrapScript() {
return bootstrapScript;
}
public void setBootstrapScript(String bootstrapScript) {
this.bootstrapScript = bootstrapScript;
}
public String getScript() {
return script;
}
public void setScript(String script) {
this.script = script;
}
public long getStartupWaitMs() {
return startupWaitMs;
}
public void setStartupWaitMs(long startupWaitMs) {
this.startupWaitMs = startupWaitMs;
}
public String getHealthPath() {
return healthPath;
}
public void setHealthPath(String healthPath) {
this.healthPath = healthPath;
}
public long getHealthCheckIntervalMs() {
return healthCheckIntervalMs;
}
public void setHealthCheckIntervalMs(long healthCheckIntervalMs) {
this.healthCheckIntervalMs = healthCheckIntervalMs;
}
public String getPythonBinary() {
return pythonBinary;
}
public void setPythonBinary(String pythonBinary) {
this.pythonBinary = pythonBinary;
}
public String getPiperBinary() {
return piperBinary;
}
public void setPiperBinary(String piperBinary) {
this.piperBinary = piperBinary;
}
public String getPiperModelPath() {
return piperModelPath;
}
public void setPiperModelPath(String piperModelPath) {
this.piperModelPath = piperModelPath;
}
public String getMusicModel() {
return musicModel;
}
public void setMusicModel(String musicModel) {
this.musicModel = musicModel;
}
public String getSfxModel() {
return sfxModel;
}
public void setSfxModel(String sfxModel) {
this.sfxModel = sfxModel;
}
}
public static class LocalDirector { public static class LocalDirector {
private boolean enabled = true; private boolean enabled = true;

View File

@ -0,0 +1,13 @@
package org.example.videoclips.editing;
public record AssetGenerationItem(
String type,
String assetKey,
String cacheKey,
boolean reused,
String sourcePath,
String targetPath,
String requestPath,
String notes
) {
}

View File

@ -0,0 +1,15 @@
package org.example.videoclips.editing;
import java.time.Instant;
import java.util.List;
public record AssetGenerationResult(
String projectId,
boolean readyForRender,
List<AssetGenerationItem> items,
Instant createdAt
) {
public AssetGenerationResult {
items = items == null ? List.of() : List.copyOf(items);
}
}

View File

@ -0,0 +1,6 @@
package org.example.videoclips.editing;
public interface AssetGenerationStage {
AssetGenerationResult prepare(String projectId, EditPlan plan);
}

View File

@ -105,7 +105,7 @@ public class FfmpegEditRenderer implements EditRenderer {
} }
private void renderInternal(String projectId, long startedAt) { private void renderInternal(String projectId, long startedAt) {
EditPlan plan = validator.validate(projectId, store.readJson(projectId, "edit-plan.json", EditPlan.class)); EditPlan plan = validator.validate(projectId, readEditPlan(projectId));
if (observability != null) { if (observability != null) {
observability.renderStarted(projectId, plan.decisions().size()); observability.renderStarted(projectId, plan.decisions().size());
} }
@ -167,6 +167,23 @@ public class FfmpegEditRenderer implements EditRenderer {
} }
} }
private EditPlan readEditPlan(String projectId) {
Path editPlan = store.editPlanFile(projectId);
if (Files.isRegularFile(editPlan)) {
return store.readJson(projectId, "edit-plan.json", EditPlan.class);
}
Path inboxPlan = store.inboxDirectory(projectId).resolve("edit-plan.json");
if (Files.isRegularFile(inboxPlan)) {
try {
return new com.fasterxml.jackson.databind.ObjectMapper().findAndRegisterModules()
.readValue(inboxPlan.toFile(), EditPlan.class);
} catch (IOException ex) {
throw new IllegalStateException("Unable to read edit plan from inbox: " + inboxPlan, ex);
}
}
throw new IllegalStateException("Unable to locate edit plan for project: " + projectId);
}
List<String> segmentCommand(String source, EditDecision decision, Path output) { List<String> segmentCommand(String source, EditDecision decision, Path output) {
double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed(); double outputDuration = (decision.sourceEndSeconds() - decision.sourceStartSeconds()) / decision.playbackSpeed();
StringBuilder filter = new StringBuilder(); StringBuilder filter = new StringBuilder();

View File

@ -0,0 +1,190 @@
package org.example.videoclips.editing;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class HighlightAssetPreparationService {
private static final Logger log = LoggerFactory.getLogger(HighlightAssetPreparationService.class);
private final HighlightProjectStore store;
private final EditAssetProvider assetProvider;
private final EditAssetLibrary assetLibrary;
private final VideoClippingProperties.Editing.Assets assets;
private final ObjectMapper objectMapper;
public HighlightAssetPreparationService(VideoClippingProperties properties, HighlightProjectStore store,
EditAssetProvider assetProvider, EditAssetLibrary assetLibrary,
ObjectMapper objectMapper) {
this.store = store;
this.assetProvider = assetProvider;
this.assetLibrary = assetLibrary;
this.assets = properties.getEditing().getAssets();
this.objectMapper = objectMapper;
}
public HighlightAssetPreparationResult prepare(String projectId, HighlightProject project,
HighlightDirectorPlan.HighlightItem highlight,
ContentCategory category) {
log.info("event=highlight_asset_preparation_started project_id={} highlight_id={} category={} target_duration_seconds={}",
projectId, highlight.highlightId(), category, highlight.targetDurationSeconds());
Path highlightDirectory = store.highlightsDirectory(projectId).resolve(highlight.highlightId());
Path assetsDirectory = highlightDirectory.resolve("assets");
Path musicDirectory = assetsDirectory.resolve("music");
Path sfxDirectory = assetsDirectory.resolve("sfx");
Path voiceoverDirectory = assetsDirectory.resolve("voiceover");
Path requestDirectory = assetsDirectory.resolve("requests");
createDirectories(musicDirectory, sfxDirectory, voiceoverDirectory, requestDirectory);
List<String> requestFiles = new ArrayList<>();
List<String> resolvedAssets = new ArrayList<>();
Optional<ResolvedEditAsset> music = assetLibrary.select(new EditAssetSelectionRequest(EditAssetType.MUSIC,
category, highlight.musicDirection(), highlight.targetDurationSeconds()));
if (music.isPresent()) {
Path target = musicDirectory.resolve("music.wav");
copy(Path.of(music.get().path()), target);
resolvedAssets.add(target.toString());
} else {
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "music",
"music_bed", highlight.musicDirection(), musicDirectory.resolve("music.wav"),
highlight.targetDurationSeconds(), false));
}
if (!highlight.voiceover().isEmpty()) {
Path script = voiceoverDirectory.resolve("voiceover-script.txt");
writeString(script, String.join(System.lineSeparator(), highlight.voiceover()));
Optional<ResolvedEditAsset> voiceover = assetProvider.list(EditAssetType.VOICEOVER, null).stream()
.findFirst();
if (voiceover.isPresent()) {
Path target = voiceoverDirectory.resolve("voiceover.wav");
copy(Path.of(voiceover.get().path()), target);
resolvedAssets.add(target.toString());
} else {
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "voiceover",
"voiceover_script", String.join(System.lineSeparator(), highlight.voiceover()),
voiceoverDirectory.resolve("voiceover.wav"), highlight.targetDurationSeconds(), false));
}
}
List<AudioCue> sfxCues = sfxCues(highlight);
for (AudioCue cue : sfxCues) {
Optional<ResolvedEditAsset> resolved = assetProvider.resolve(new EditAssetRequest(EditAssetType.SFX,
cue.assetKey(), category, cue.timelineEndSeconds() - cue.timelineStartSeconds(), cue.notes()));
Path target = sfxDirectory.resolve(cue.assetKey() + ".wav");
if (resolved.isPresent()) {
copy(Path.of(resolved.get().path()), target);
resolvedAssets.add(target.toString());
} else {
requestFiles.add(writeRequest(requestDirectory, projectId, highlight.highlightId(), "sfx",
cue.assetKey(), cue.notes(), target, cue.timelineEndSeconds() - cue.timelineStartSeconds(),
true));
}
}
log.info("event=highlight_asset_preparation_completed project_id={} highlight_id={} resolved={} requests={}",
projectId, highlight.highlightId(), resolvedAssets.size(), requestFiles.size());
return new HighlightAssetPreparationResult(highlightDirectory, assetsDirectory, requestFiles, resolvedAssets);
}
private List<AudioCue> sfxCues(HighlightDirectorPlan.HighlightItem highlight) {
double duration = Math.max(1.0, highlight.targetDurationSeconds());
List<AudioCue> cues = new ArrayList<>();
if (highlight.storyPurpose() != null && highlight.storyPurpose().contains("opening")) {
cues.add(new AudioCue("sfx", "whoosh_soft", 0.0, Math.min(0.8, duration), -6.0,
highlight.sfxDirection()));
}
cues.add(new AudioCue("sfx", "impact_hit", Math.max(0.0, duration - 0.8), duration, -3.0,
highlight.sfxDirection()));
return cues;
}
private String writeRequest(Path requestDirectory, String projectId, String highlightId, String type,
String assetKey, String notes, Path target, double durationSeconds, boolean blocking) {
Path requestFile = requestDirectory.resolve(type + "-" + assetKey + ".md");
Path requestJson = requestDirectory.resolve(type + "-" + assetKey + ".json");
String content = """
# Highlight Asset Request
Project: `%s`
Highlight: `%s`
Type: `%s`
Asset key: `%s`
Project target: `%s`
Blocking for render: `%s`
Duration seconds: %.3f
Notes:
%s
""".formatted(projectId, highlightId, type, assetKey, target, blocking, durationSeconds,
notes == null ? "" : notes);
writeString(requestFile, content);
writeRequestJson(requestJson, new HighlightAssetRequest(projectId, highlightId, type, assetKey,
target.toString(), requestFile.toString(), notes, durationSeconds, blocking));
return requestFile.toString();
}
private void writeRequestJson(Path path, HighlightAssetRequest request) {
try {
Files.createDirectories(path.getParent());
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), request);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight asset request JSON: " + path, ex);
}
}
private void writeString(Path path, String value) {
try {
Files.createDirectories(path.getParent());
Files.writeString(path, value, StandardCharsets.UTF_8);
} catch (IOException ex) {
throw new IllegalStateException("Unable to write highlight asset file: " + path, ex);
}
}
private void copy(Path source, Path target) {
try {
Files.createDirectories(target.getParent());
Files.copy(source, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
} catch (IOException ex) {
throw new IllegalStateException("Unable to copy highlight asset: " + source, ex);
}
}
private void createDirectories(Path... directories) {
for (Path directory : directories) {
try {
Files.createDirectories(directory);
} catch (IOException ex) {
throw new IllegalStateException("Unable to create highlight asset directory: " + directory, ex);
}
}
}
public record HighlightAssetPreparationResult(
Path highlightDirectory,
Path assetsDirectory,
List<String> requestFiles,
List<String> resolvedAssets
) {
public HighlightAssetPreparationResult {
requestFiles = requestFiles == null ? List.of() : List.copyOf(requestFiles);
resolvedAssets = resolvedAssets == null ? List.of() : List.copyOf(resolvedAssets);
}
}
}

View File

@ -65,7 +65,6 @@ public class HighlightDirectorFlowService {
markFailed(project, "No highlights were provided by the director plan"); markFailed(project, "No highlights were provided by the director plan");
throw new IllegalStateException("Director plan contains no highlights"); throw new IllegalStateException("Director plan contains no highlights");
} }
markStatus(project, HighlightProjectStatus.RENDERING, null);
log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}", log.info("event=highlight_flow_started flow_id={} scan_id={} project_id={} source_file={} highlights={}",
flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size()); flowId, scanId, projectId, project.sourceVideoFileName(), plan.highlights().size());
List<Path> finalOutputs = new ArrayList<>(); List<Path> finalOutputs = new ArrayList<>();
@ -89,6 +88,12 @@ public class HighlightDirectorFlowService {
log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}", log.info("event=highlight_asset_worker_completed flow_id={} project_id={} highlight_id={} resolved={} pending={}",
flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(), flowId, projectId, highlight.highlightId(), assetWorkerResult.resolvedAssets().size(),
assetWorkerResult.pendingRequests().size()); assetWorkerResult.pendingRequests().size());
if (!assetWorkerResult.pendingRequests().isEmpty()) {
log.warn("event=highlight_flow_waiting_for_assets flow_id={} project_id={} highlight_id={} pending_requests={}",
flowId, projectId, highlight.highlightId(), assetWorkerResult.pendingRequests());
return HighlightFlowResult.skipped(projectId, flowId, "assets_pending");
}
markStatus(project, HighlightProjectStatus.RENDERING, null);
HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan); HighlightFfmpegRenderer.HighlightRenderResult result = renderer.render(projectId, highlight, editPlan);
finalOutputs.add(result.finalOutput()); finalOutputs.add(result.finalOutput());
renderedHighlightIds.add(result.highlightId()); renderedHighlightIds.add(result.highlightId());

View File

@ -2,6 +2,7 @@ package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties; import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
import java.io.IOException; import java.io.IOException;
@ -27,6 +28,7 @@ public class HighlightFfmpegRenderer {
private final EditObservability observability; private final EditObservability observability;
private final ProcessExecutor executor; private final ProcessExecutor executor;
@Autowired
public HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store, public HighlightFfmpegRenderer(VideoClippingProperties properties, HighlightProjectStore store,
EditAssetProvider assetProvider, EditObservability observability) { EditAssetProvider assetProvider, EditObservability observability) {
this(properties, store, assetProvider, observability, HighlightFfmpegRenderer::execute); this(properties, store, assetProvider, observability, HighlightFfmpegRenderer::execute);
@ -119,12 +121,26 @@ public class HighlightFfmpegRenderer {
"sfx", Integer.toString(sfx.size()), "sfx", Integer.toString(sfx.size()),
"voiceover", Boolean.toString(Files.isRegularFile(voiceover))); "voiceover", Boolean.toString(Files.isRegularFile(voiceover)));
if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) { if (Files.isRegularFile(music) || Files.isRegularFile(voiceover) || !sfx.isEmpty()) {
run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null, try {
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands); run(audioMixCommand(postTimeline, Files.isRegularFile(music) ? music : null,
Files.isRegularFile(voiceover) ? voiceover : null, sfx, output), commands);
} catch (IllegalStateException ex) {
log(projectId, highlight.highlightId(), "highlight_render_audio_failed",
"error", summarizeMessage(ex.getMessage()),
"fallback", "copy_post_timeline");
copy(postTimeline, output);
}
} else { } else {
copy(postTimeline, output); copy(postTimeline, output);
} }
run(previewCommand(output, preview), commands); try {
run(previewCommand(output, preview), commands);
} catch (IllegalStateException ex) {
log(projectId, highlight.highlightId(), "highlight_render_preview_failed",
"error", summarizeMessage(ex.getMessage()),
"fallback", "copy_final_output");
copy(output, preview);
}
double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1) double duration = plan.decisions().isEmpty() ? 0 : plan.decisions().get(plan.decisions().size() - 1)
.timelineEndSeconds(); .timelineEndSeconds();
RenderManifest manifest = new RenderManifest(projectId, RenderManifest manifest = new RenderManifest(projectId,
@ -403,10 +419,11 @@ public class HighlightFfmpegRenderer {
private void run(List<String> command, List<List<String>> commands) { private void run(List<String> command, List<List<String>> commands) {
try { try {
ProcessResult result = executor.execute(command);
commands.add(command); commands.add(command);
ProcessResult result = executor.execute(command);
if (result.exitCode() != 0) { if (result.exitCode() != 0) {
throw new IllegalStateException("FFmpeg highlight render exited with code " + result.exitCode()); throw new IllegalStateException("FFmpeg highlight render exited with code " + result.exitCode()
+ messageSuffix(result.output()));
} }
} catch (InterruptedException ex) { } catch (InterruptedException ex) {
Thread.currentThread().interrupt(); Thread.currentThread().interrupt();
@ -416,6 +433,20 @@ public class HighlightFfmpegRenderer {
} }
} }
private String summarizeMessage(String message) {
if (message == null || message.isBlank()) {
return "n/a";
}
return message.length() <= 500 ? message : message.substring(0, 500) + "...";
}
private String messageSuffix(String output) {
if (output == null || output.isBlank()) {
return "";
}
return ": " + summarizeMessage(output);
}
private void log(String projectId, String highlightId, String event, String... kvPairs) { private void log(String projectId, String highlightId, String event, String... kvPairs) {
StringBuilder builder = new StringBuilder("event=").append(event).append(" project_id=").append(projectId) StringBuilder builder = new StringBuilder("event=").append(event).append(" project_id=").append(projectId)
.append(" highlight_id=").append(highlightId); .append(" highlight_id=").append(highlightId);

View File

@ -26,13 +26,16 @@ public class HighlightLocalAssetWorker {
private final HighlightProjectStore store; private final HighlightProjectStore store;
private final EditAssetProvider assetProvider; private final EditAssetProvider assetProvider;
private final EditAssetLibrary assetLibrary; private final EditAssetLibrary assetLibrary;
private final LocalAssetSynthesizer synthesizer;
private final ObjectMapper objectMapper; private final ObjectMapper objectMapper;
public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider, public HighlightLocalAssetWorker(HighlightProjectStore store, EditAssetProvider assetProvider,
EditAssetLibrary assetLibrary, ObjectMapper objectMapper) { EditAssetLibrary assetLibrary, LocalAssetSynthesizer synthesizer,
ObjectMapper objectMapper) {
this.store = store; this.store = store;
this.assetProvider = assetProvider; this.assetProvider = assetProvider;
this.assetLibrary = assetLibrary; this.assetLibrary = assetLibrary;
this.synthesizer = synthesizer;
this.objectMapper = objectMapper; this.objectMapper = objectMapper;
} }
@ -44,9 +47,14 @@ public class HighlightLocalAssetWorker {
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of()); return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), List.of(), List.of());
} }
List<HighlightAssetRequest> requests = readRequests(requestsDirectory); List<HighlightAssetRequest> requests = readRequests(requestsDirectory);
log.info("event=highlight_asset_worker_started project_id={} highlight_id={} request_count={} request_directory={} category={}",
projectId, highlight.highlightId(), requests.size(), requestsDirectory, category);
List<String> resolved = new ArrayList<>(); List<String> resolved = new ArrayList<>();
List<String> pending = new ArrayList<>(); List<String> pending = new ArrayList<>();
for (HighlightAssetRequest request : requests) { for (HighlightAssetRequest request : requests) {
log.info("event=highlight_asset_generation_requested project_id={} highlight_id={} type={} asset_key={} target={} blocking={} duration_seconds={} request={}",
projectId, highlight.highlightId(), request.type(), request.assetKey(), request.targetPath(),
request.blocking(), request.durationSeconds(), request.requestPath());
Optional<Path> materialized = materialize(request, category); Optional<Path> materialized = materialize(request, category);
if (materialized.isPresent()) { if (materialized.isPresent()) {
resolved.add(materialized.get().toString()); resolved.add(materialized.get().toString());
@ -58,6 +66,8 @@ public class HighlightLocalAssetWorker {
projectId, highlight.highlightId(), request.type(), request.requestPath()); projectId, highlight.highlightId(), request.type(), request.requestPath());
} }
} }
log.info("event=highlight_asset_worker_completed project_id={} highlight_id={} resolved={} pending={}",
projectId, highlight.highlightId(), resolved.size(), pending.size());
return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending); return new HighlightAssetWorkerResult(projectId, highlight.highlightId(), resolved, pending);
} }
@ -101,7 +111,7 @@ public class HighlightLocalAssetWorker {
copy(Path.of(asset.get().path()), target); copy(Path.of(asset.get().path()), target);
return Optional.of(target); return Optional.of(target);
} }
return generateToneBed(target, request.durationSeconds()); return synthesizer.synthesizeMusic(request.projectId(), request.notes(), request.durationSeconds(), target);
} }
private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) { private Optional<Path> resolveOrGenerateSfx(HighlightAssetRequest request, ContentCategory category) {
@ -118,7 +128,7 @@ public class HighlightLocalAssetWorker {
copy(Path.of(fallback.get().path()), target); copy(Path.of(fallback.get().path()), target);
return Optional.of(target); return Optional.of(target);
} }
return generateImpactTone(target, request.durationSeconds()); return synthesizer.synthesizeSfx(request.projectId(), request.notes(), request.durationSeconds(), target);
} }
private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) { private Optional<Path> resolveOrGenerateVoiceover(HighlightAssetRequest request) {
@ -127,70 +137,8 @@ public class HighlightLocalAssetWorker {
return Optional.empty(); return Optional.empty();
} }
Path target = Path.of(request.targetPath()); Path target = Path.of(request.targetPath());
if (runSay(text, target)) { return synthesizer.synthesizeVoiceover(request.projectId(), List.of(new VoiceoverLine(text, 0.0,
return Optional.of(target); Math.max(1.0, request.durationSeconds()), "cinematic_narration")), target);
}
if (runEspeak(text, target)) {
return Optional.of(target);
}
return Optional.empty();
}
private Optional<Path> generateToneBed(Path target, double durationSeconds) {
return generateAudio(target, durationSeconds, "sine=frequency=110:sample_rate=48000", 0.02);
}
private Optional<Path> generateImpactTone(Path target, double durationSeconds) {
return generateAudio(target, Math.max(0.5, Math.min(1.0, durationSeconds)),
"sine=frequency=880:sample_rate=48000", 0.12);
}
private Optional<Path> generateAudio(Path target, double durationSeconds, String source, double volume) {
try {
Files.createDirectories(target.getParent());
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", source, "-t", Double.toString(durationSeconds),
"-af", "volume=" + volume, "-c:a", "aac", target.toString());
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0) {
return Optional.of(target);
}
log.warn("event=highlight_asset_generation_failed target={} message={}", target, output);
return Optional.empty();
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_asset_generation_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
}
}
private boolean runSay(String text, Path target) {
return runSpeechCommand(List.of("say", "-o", target.toString(), text), target);
}
private boolean runEspeak(String text, Path target) {
return runSpeechCommand(List.of("espeak", "-w", target.toString(), text), target);
}
private boolean runSpeechCommand(List<String> command, Path target) {
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
return true;
}
if (!output.isBlank()) {
log.warn("event=highlight_voiceover_generation_failed target={} output={}", target, output);
}
return false;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=highlight_voiceover_generation_failed target={} error_type={} message={}", target,
ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
} }
private void copy(Path source, Path target) { private void copy(Path source, Path target) {

View File

@ -28,15 +28,17 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
private final EditProjectStore store; private final EditProjectStore store;
private final EditAssetProvider provider; private final EditAssetProvider provider;
private final EditAssetLibrary library; private final EditAssetLibrary library;
private final LocalAssetSynthesizer synthesizer;
private final Path voiceoverCacheRoot; private final Path voiceoverCacheRoot;
private final ObjectMapper objectMapper; private final ObjectMapper objectMapper;
public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store, public LocalAssetGenerationStage(VideoClippingProperties properties, EditProjectStore store,
EditAssetProvider provider, EditAssetLibrary library, EditAssetProvider provider, EditAssetLibrary library,
ObjectMapper objectMapper) { LocalAssetSynthesizer synthesizer, ObjectMapper objectMapper) {
this.store = store; this.store = store;
this.provider = provider; this.provider = provider;
this.library = library; this.library = library;
this.synthesizer = synthesizer;
this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize(); this.voiceoverCacheRoot = Path.of(properties.getEditing().getAssets().getVoiceoverFolder()).normalize();
this.objectMapper = objectMapper; this.objectMapper = objectMapper;
} }
@ -151,6 +153,13 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(), objectMapper.writerWithDefaultPrettyPrinter().writeValue(requestDirectory.resolve("request.json").toFile(),
new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(), new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
requestFile.toString(), notes)); requestFile.toString(), notes));
Optional<Path> generated = generate(type, projectId, notes, durationSeconds, projectTarget);
if (generated.isPresent()) {
log.info("event=asset_materialized project_id={} type={} asset_key={} cache_key={} source={} target={} strategy=local-worker",
projectId, type, assetKey, cacheKey, generated.get(), projectTarget);
return new AssetGenerationItem(type, assetKey, cacheKey, true, generated.get().toString(),
projectTarget.toString(), requestFile.toString(), notes);
}
log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}", log.info("event=asset_generation_requested project_id={} type={} asset_key={} cache_key={} request={}",
projectId, type, assetKey, cacheKey, requestFile); projectId, type, assetKey, cacheKey, requestFile);
return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(), return new AssetGenerationItem(type, assetKey, cacheKey, false, null, projectTarget.toString(),
@ -179,6 +188,23 @@ public class LocalAssetGenerationStage implements AssetGenerationStage {
return Optional.empty(); return Optional.empty();
} }
private Optional<Path> generate(String type, String projectId, String notes, double durationSeconds,
Path target) {
if ("music".equals(type)) {
return synthesizer.synthesizeMusic(projectId, notes, durationSeconds, target);
}
if ("voiceover".equals(type)) {
List<VoiceoverLine> lines = notes == null || notes.isBlank()
? List.of()
: List.of(new VoiceoverLine(notes, 0.0, Math.max(1.0, durationSeconds), "cinematic_narration"));
return synthesizer.synthesizeVoiceover(projectId, lines, target);
}
if ("sfx".equals(type)) {
return synthesizer.synthesizeSfx(projectId, notes, durationSeconds, target);
}
return Optional.empty();
}
private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath, private String request(String projectId, String type, String assetKey, String cacheKey, Path cachePath,
Path projectTarget, String notes, double durationSeconds, boolean blocking) { Path projectTarget, String notes, double durationSeconds, boolean blocking) {
return """ return """

View File

@ -0,0 +1,205 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalAssetRuntimeVerifier implements ApplicationRunner {
private static final Logger log = LoggerFactory.getLogger(LocalAssetRuntimeVerifier.class);
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
public LocalAssetRuntimeVerifier(VideoClippingProperties properties) {
this.worker = properties.getEditing().getLocalAssetWorker();
}
@Override
public void run(ApplicationArguments args) {
log.info("event=local_asset_runtime_check_started auto_start={} strict_runtime={} script={} python={} "
+ "bootstrap_script={} piper_binary={} piper_model_path={} music_model={} sfx_model={}",
worker.isAutoStart(), worker.isStrictRuntime(), worker.getScript(), worker.getPythonBinary(),
worker.getBootstrapScript(),
worker.getPiperBinary(), worker.getPiperModelPath(), worker.getMusicModel(), worker.getSfxModel());
if (!worker.isAutoStart()) {
log.info("event=local_asset_runtime_check_skipped reason=auto_start_disabled");
return;
}
bootstrapEnvironment();
RuntimeReadiness readiness = verifyRuntime();
if (!readiness.ready()) {
log.error("event=local_asset_runtime_check_failed missing={} voiceover_ready={} music_ready={} sfx_ready={}",
readiness.missing(), readiness.voiceoverReady(), readiness.musicReady(),
readiness.sfxReady());
if (worker.isStrictRuntime()) {
log.warn("event=local_asset_runtime_degraded strict_runtime_enabled=true missing={}",
readiness.missing());
}
return;
}
log.info("event=local_asset_runtime_check_completed status=ready voiceover_ready=true music_ready=true "
+ "sfx_ready=true python={} piper_binary={} piper_model_path={}",
worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
}
private void bootstrapEnvironment() {
Path bootstrapScript = Path.of(worker.getBootstrapScript()).toAbsolutePath().normalize();
if (!Files.isRegularFile(bootstrapScript)) {
throw new IllegalStateException("Local asset bootstrap script does not exist: " + bootstrapScript);
}
log.info("event=local_asset_bootstrap_started script={} bootstrap_mode=auto preload_only=true python={} "
+ "piper_binary={} piper_model_path={}",
bootstrapScript, worker.getPythonBinary(), worker.getPiperBinary(), worker.getPiperModelPath());
ProcessBuilder builder = new ProcessBuilder("bash", bootstrapScript.toString())
.redirectErrorStream(true);
Map<String, String> environment = builder.environment();
environment.put("LOCAL_ASSET_BOOTSTRAP_MODE", "auto");
environment.put("LOCAL_ASSET_PRELOAD_ONLY", "true");
environment.put("LOCAL_ASSET_PIPER_BINARY", worker.getPiperBinary());
if (worker.getPiperModelPath() != null && !worker.getPiperModelPath().isBlank()) {
environment.put("LOCAL_ASSET_PIPER_MODEL_PATH", worker.getPiperModelPath());
}
long startedAt = System.nanoTime();
try {
Process process = builder.start();
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {
String line;
while ((line = reader.readLine()) != null) {
log.info("event=local_asset_bootstrap_output script={} message={}", bootstrapScript, line);
}
}
int exitCode = process.waitFor();
if (exitCode != 0) {
log.error("event=local_asset_bootstrap_failed script={} exit_code={} reason=non_blocking",
bootstrapScript, exitCode);
return;
}
log.info("event=local_asset_bootstrap_completed script={} elapsed_ms={}", bootstrapScript,
elapsedMillis(startedAt));
} catch (IOException ex) {
log.error("event=local_asset_bootstrap_failed script={} error_type={} message={}", bootstrapScript,
ex.getClass().getSimpleName(), ex.getMessage());
throw new IllegalStateException("Unable to bootstrap local asset runtime", ex);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
log.error("event=local_asset_bootstrap_failed script={} error_type={} message={}", bootstrapScript,
ex.getClass().getSimpleName(), ex.getMessage());
throw new IllegalStateException("Interrupted while bootstrapping local asset runtime", ex);
}
}
private RuntimeReadiness verifyRuntime() {
List<String> missing = new ArrayList<>();
boolean pythonReady = isPythonAudioStackReady();
if (!Files.isRegularFile(Path.of(worker.getPythonBinary()))) {
missing.add("python_binary:" + worker.getPythonBinary());
}
if (!Files.isRegularFile(Path.of(worker.getBootstrapScript()))) {
missing.add("bootstrap_script:" + worker.getBootstrapScript());
}
if (!Files.isRegularFile(Path.of(worker.getScript()))) {
missing.add("worker_script:" + worker.getScript());
}
if (!isExecutablePresent(worker.getPiperBinary())) {
missing.add("piper_binary:" + worker.getPiperBinary());
}
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
missing.add("piper_model_path:missing");
} else if (!Files.isRegularFile(Path.of(worker.getPiperModelPath()))) {
missing.add("piper_model_path:" + worker.getPiperModelPath());
}
if (!pythonReady) {
missing.add("python_audio_stack:torch,audiocraft,soundfile,numpy");
}
return new RuntimeReadiness(
missing.isEmpty(),
isVoiceoverReady(pythonReady),
pythonReady,
pythonReady,
List.copyOf(missing)
);
}
private boolean isVoiceoverReady(boolean pythonReady) {
if (!pythonReady) {
return false;
}
if (worker.getPiperModelPath() == null || worker.getPiperModelPath().isBlank()) {
return false;
}
return isExecutablePresent(worker.getPiperBinary());
}
private boolean isPythonAudioStackReady() {
List<String> command = List.of(worker.getPythonBinary(), "-c",
"import torch, audiocraft, soundfile, numpy");
try {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
int exitCode = process.waitFor();
if (exitCode != 0 && !output.isBlank()) {
log.warn("event=local_asset_python_stack_check_failed python={} exit_code={} output={}",
worker.getPythonBinary(), exitCode, output);
}
return exitCode == 0;
} catch (IOException ex) {
log.warn("event=local_asset_python_stack_check_failed python={} error_type={} message={}",
worker.getPythonBinary(), ex.getClass().getSimpleName(), ex.getMessage());
return false;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_python_stack_check_failed python={} error_type={} message={}",
worker.getPythonBinary(), ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private boolean isExecutablePresent(String executable) {
try {
Process process = new ProcessBuilder(executable, "--help").redirectErrorStream(true).start();
process.waitFor();
return true;
} catch (IOException ex) {
return false;
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
return false;
}
}
private long elapsedMillis(long startedAt) {
return (System.nanoTime() - startedAt) / 1_000_000;
}
private record RuntimeReadiness(
boolean ready,
boolean voiceoverReady,
boolean musicReady,
boolean sfxReady,
List<String> missing
) {
private RuntimeReadiness {
missing = missing == null ? List.of() : List.copyOf(missing);
}
}
}

View File

@ -0,0 +1,316 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.UnsupportedAudioFileException;
@Component
@ConditionalOnProperty(name = "video-clipping.editing.enabled", havingValue = "true", matchIfMissing = true)
public class LocalAssetSynthesizer {
private static final Logger log = LoggerFactory.getLogger(LocalAssetSynthesizer.class);
private final VideoClippingProperties.Editing.LocalAssetWorker worker;
private final ProcessExecutor processExecutor;
@Autowired
public LocalAssetSynthesizer(VideoClippingProperties properties) {
this(properties, command -> new ProcessBuilder(command).redirectErrorStream(true).start());
}
LocalAssetSynthesizer(VideoClippingProperties properties, ProcessExecutor processExecutor) {
this.worker = properties.getEditing().getLocalAssetWorker();
this.processExecutor = processExecutor;
}
public Optional<Path> synthesizeVoiceover(String projectId, List<VoiceoverLine> lines, Path target) {
if (lines == null || lines.isEmpty()) {
return Optional.empty();
}
String prompt = String.join(System.lineSeparator(),
lines.stream().map(VoiceoverLine::text).toList());
return synthesize(projectId, "voiceover", prompt, target, Math.max(1.0, lines.size() * 2.5), true);
}
public Optional<Path> synthesizeMusic(String projectId, String prompt, double durationSeconds, Path target) {
return synthesize(projectId, "music", prompt, target, durationSeconds, false);
}
public Optional<Path> synthesizeSfx(String projectId, String prompt, double durationSeconds, Path target) {
return synthesize(projectId, "sfx", prompt, target, durationSeconds, false);
}
private Optional<Path> synthesize(String projectId, String kind, String prompt, Path target,
double durationSeconds, boolean allowSpeechFallback) {
if (prompt == null || prompt.isBlank()) {
return Optional.empty();
}
try {
String model = selectedModel(kind);
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=local-worker",
projectId, kind, model, target, durationSeconds);
Files.createDirectories(target.getParent());
Path promptFile = Files.createTempFile(target.getParent(),
sanitize(kind) + "-prompt-", ".txt");
Files.writeString(promptFile, prompt, StandardCharsets.UTF_8);
List<String> command = new ArrayList<>();
command.add(worker.getPythonBinary());
command.add(Path.of(worker.getScript()).toAbsolutePath().normalize().toString());
command.add(kind);
command.add("--prompt-file");
command.add(promptFile.toString());
command.add("--output");
command.add(target.toString());
command.add("--duration");
command.add(Double.toString(durationSeconds));
if ("music".equals(kind)) {
command.add("--model");
command.add(worker.getMusicModel());
} else if ("sfx".equals(kind)) {
command.add("--model");
command.add(worker.getSfxModel());
} else if ("voiceover".equals(kind) && !worker.getPiperModelPath().isBlank()) {
command.add("--model");
command.add(worker.getPiperModelPath());
command.add("--piper-binary");
command.add(worker.getPiperBinary());
}
if (runWorker(command, target)) {
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=local-worker",
projectId, kind, model, target);
return Optional.of(target);
}
if ("voiceover".equals(kind) && allowSpeechFallback) {
log.info("event=local_asset_generation_started project_id={} type={} model={} target={} duration_seconds={} strategy=speech-fallback",
projectId, kind, selectedVoiceoverFallback(), target, durationSeconds);
if (runSpeechFallback(prompt, target)) {
if (!hasAudibleSignal(target)) {
log.warn("event=local_asset_voiceover_silence_detected project_id={} target={} strategy=speech-fallback",
projectId, target);
} else {
log.info("event=local_asset_generation_completed project_id={} type={} model={} target={} strategy=speech-fallback",
projectId, kind, selectedVoiceoverFallback(), target);
return Optional.of(target);
}
}
}
if ("voiceover".equals(kind)) {
log.error("event=local_asset_voiceover_unavailable project_id={} target={} reason=no_audible_speech_generated",
projectId, target);
return Optional.empty();
}
if ("music".equals(kind)) {
if (worker.isStrictRuntime()) {
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
projectId, kind, model, target);
return Optional.empty();
}
return fallbackTone(target, durationSeconds, 110.0, 0.02, projectId, kind);
}
if ("sfx".equals(kind)) {
if (worker.isStrictRuntime()) {
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
projectId, kind, model, target);
return Optional.empty();
}
return fallbackTone(target, Math.max(0.5, Math.min(1.0, durationSeconds)), 880.0, 0.12,
projectId, kind);
}
if (worker.isStrictRuntime()) {
log.error("event=local_asset_generation_failed project_id={} type={} model={} target={} reason=strict_runtime_required",
projectId, kind, model, target);
return Optional.empty();
}
writeSilence(target, Math.max(1.0, durationSeconds));
log.warn("event=local_asset_generation_completed project_id={} type={} model=silence target={} strategy=silence",
projectId, kind, target);
return Optional.of(target);
} catch (IOException ex) {
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
}
}
private boolean runWorker(List<String> command, Path target) {
try {
log.info("event=local_asset_worker_invoked script={} target={}", worker.getScript(), target);
Process process = processExecutor.execute(command);
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
int exitCode = process.waitFor();
if (exitCode == 0 && Files.isRegularFile(target)) {
log.info("event=local_asset_worker_completed script={} target={} exit_code={}",
worker.getScript(), target, exitCode);
return true;
}
if (!output.isBlank()) {
log.warn("event=local_asset_worker_failed target={} output={}", target, output);
}
return false;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_worker_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private boolean runSpeechFallback(String text, Path target) {
List<List<String>> candidates = List.of(
List.of("say", "-o", target.toString(), text),
List.of("espeak", "-w", target.toString(), text)
);
for (List<String> command : candidates) {
if (command.isEmpty() || command.get(0).isBlank()) {
continue;
}
if (!isAvailable(command.get(0))) {
continue;
}
try {
log.info("event=local_asset_voiceover_fallback_started tool={} target={}", command.get(0), target);
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
log.info("event=local_asset_voiceover_fallback_completed tool={} target={}",
command.get(0), target);
return true;
}
if (!output.isBlank()) {
log.warn("event=local_asset_voiceover_fallback_failed target={} output={}", target, output);
}
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_voiceover_fallback_failed target={} error_type={} message={}",
target, ex.getClass().getSimpleName(), ex.getMessage());
}
}
return false;
}
private Optional<Path> fallbackTone(Path target, double durationSeconds, double frequency, double volume,
String projectId, String kind) {
try {
log.info("event=local_asset_generation_started project_id={} type={} model=ffmpeg-sine target={} duration_seconds={} strategy=fallback-tone",
projectId, kind, target, durationSeconds);
Files.createDirectories(target.getParent());
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", "sine=frequency=" + (int) frequency + ":sample_rate=48000",
"-t", Double.toString(durationSeconds), "-af", "volume=" + volume, "-c:a", "pcm_s16le",
target.toString());
Process process = processExecutor.execute(command);
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
if (process.waitFor() == 0 && Files.isRegularFile(target)) {
log.info("event=local_asset_generation_completed project_id={} type={} model=ffmpeg-sine target={} strategy=fallback-tone",
projectId, kind, target);
return Optional.of(target);
}
if (!output.isBlank()) {
log.warn("event=local_asset_generation_failed project_id={} type={} target={} output={}",
projectId, kind, target, output);
}
return Optional.empty();
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
log.warn("event=local_asset_generation_failed project_id={} type={} target={} error_type={} message={}",
projectId, kind, target, ex.getClass().getSimpleName(), ex.getMessage());
return Optional.empty();
}
}
private void writeSilence(Path target, double durationSeconds) {
try {
log.info("event=local_asset_generation_started type=voiceover model=silence target={} duration_seconds={} strategy=silence",
target, durationSeconds);
Files.createDirectories(target.getParent());
List<String> command = List.of("ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "lavfi", "-i", "anullsrc=channel_layout=mono:sample_rate=24000",
"-t", Double.toString(durationSeconds), "-c:a", "pcm_s16le", target.toString());
Process process = processExecutor.execute(command);
process.waitFor();
log.info("event=local_asset_generation_completed type=voiceover model=silence target={} strategy=silence",
target);
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
private String selectedModel(String kind) {
if ("music".equals(kind)) {
return worker.getMusicModel();
}
if ("sfx".equals(kind)) {
return worker.getSfxModel();
}
if ("voiceover".equals(kind)) {
return worker.getPiperModelPath().isBlank() ? selectedVoiceoverFallback() : worker.getPiperModelPath();
}
return "unknown";
}
private String selectedVoiceoverFallback() {
return "piper-fallback";
}
private boolean hasAudibleSignal(Path audioFile) {
if (!Files.isRegularFile(audioFile)) {
return false;
}
try (AudioInputStream input = AudioSystem.getAudioInputStream(audioFile.toFile())) {
byte[] buffer = new byte[8192];
int read;
while ((read = input.read(buffer)) != -1) {
for (int index = 0; index < read; index++) {
if (buffer[index] != 0) {
return true;
}
}
}
return false;
} catch (UnsupportedAudioFileException | IOException ex) {
log.warn("event=local_asset_voiceover_audibility_check_failed target={} error_type={} message={}",
audioFile, ex.getClass().getSimpleName(), ex.getMessage());
return false;
}
}
private boolean isAvailable(String executable) {
try {
Process process = new ProcessBuilder(executable, "--help").redirectErrorStream(true).start();
process.waitFor();
return true;
} catch (IOException | InterruptedException ex) {
Thread.currentThread().interrupt();
return false;
}
}
private String sanitize(String value) {
return value == null ? "asset" : value.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9._-]+", "-");
}
@FunctionalInterface
interface ProcessExecutor {
Process execute(List<String> command) throws IOException;
}
}

View File

@ -0,0 +1,43 @@
package org.example.videoclips.editing;
import org.example.videoclips.config.VideoClippingProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.stereotype.Component;
import java.nio.file.Path;
import java.util.List;
@Component
@ConditionalOnExpression("${video-clipping.editing.enabled:true} and "
+ "'${video-clipping.editing.voiceover-provider:local}' == 'local'")
public class LocalVoiceoverGenerator implements VoiceoverGenerator {
private final EditProjectStore store;
private final LocalAssetSynthesizer synthesizer;
public LocalVoiceoverGenerator(EditProjectStore store, LocalAssetSynthesizer synthesizer) {
this.store = store;
this.synthesizer = synthesizer;
}
@Override
public Path generateVoiceover(String projectId, List<VoiceoverLine> lines) {
Path script = store.projectDirectory(projectId).resolve("voiceover-script.txt");
Path target = store.projectDirectory(projectId).resolve("audio").resolve("voiceover.wav");
StringBuilder content = new StringBuilder("Voiceover script for project ").append(projectId).append('\n');
for (VoiceoverLine line : lines) {
content.append("[%.3f-%.3f] %s%nDelivery: %s%n%n".formatted(
line.timelineStartSeconds(), line.timelineEndSeconds(), line.text(), line.delivery()));
}
try {
java.nio.file.Files.createDirectories(script.getParent());
java.nio.file.Files.writeString(script, content.toString());
} catch (java.io.IOException ex) {
throw new IllegalStateException("Unable to write voiceover script for project: " + projectId, ex);
}
synthesizer.synthesizeVoiceover(projectId, lines, target)
.orElseThrow(() -> new IllegalStateException("Unable to synthesize voiceover for project: "
+ projectId));
return target;
}
}

View File

@ -34,7 +34,7 @@ video-clipping:
audio-sample-rate: ${VIDEO_EDITING_AUDIO_SAMPLE_RATE:48000} audio-sample-rate: ${VIDEO_EDITING_AUDIO_SAMPLE_RATE:48000}
video-bitrate: ${VIDEO_EDITING_VIDEO_BITRATE:12000k} video-bitrate: ${VIDEO_EDITING_VIDEO_BITRATE:12000k}
audio-bitrate: ${VIDEO_EDITING_AUDIO_BITRATE:192k} audio-bitrate: ${VIDEO_EDITING_AUDIO_BITRATE:192k}
voiceover-provider: ${VIDEO_EDITING_VOICEOVER_PROVIDER:noop} voiceover-provider: ${VIDEO_EDITING_VOICEOVER_PROVIDER:local}
loudness-target-i: ${VIDEO_EDITING_LOUDNESS_TARGET_I:-16.0} loudness-target-i: ${VIDEO_EDITING_LOUDNESS_TARGET_I:-16.0}
loudness-true-peak: ${VIDEO_EDITING_LOUDNESS_TRUE_PEAK:-1.5} loudness-true-peak: ${VIDEO_EDITING_LOUDNESS_TRUE_PEAK:-1.5}
loudness-range: ${VIDEO_EDITING_LOUDNESS_RANGE:11.0} loudness-range: ${VIDEO_EDITING_LOUDNESS_RANGE:11.0}
@ -56,9 +56,22 @@ video-clipping:
local-cv-worker: local-cv-worker:
auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true} auto-start: ${VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START:true}
script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh} script: ${VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT:./tools/run_local_cv_worker.sh}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:120000} startup-wait-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS:0}
health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health} health-path: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH:/health}
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000} health-check-interval-ms: ${VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
local-asset-worker:
auto-start: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_AUTO_START:true}
strict-runtime: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_STRICT_RUNTIME:true}
bootstrap-script: ${VIDEO_EDITING_LOCAL_ASSET_BOOTSTRAP_SCRIPT:./tools/run_local_asset_worker.sh}
script: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_SCRIPT:./tools/local_asset_worker.py}
startup-wait-ms: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_STARTUP_WAIT_MS:0}
health-path: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_PATH:/health}
health-check-interval-ms: ${VIDEO_EDITING_LOCAL_ASSET_WORKER_HEALTH_CHECK_INTERVAL_MS:1000}
python-binary: ${VIDEO_EDITING_LOCAL_ASSET_PYTHON_BINARY:./.venv-local-asset/bin/python}
piper-binary: ${VIDEO_EDITING_LOCAL_ASSET_PIPER_BINARY:piper}
piper-model-path: ${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:}
music-model: ${VIDEO_EDITING_LOCAL_ASSET_MUSIC_MODEL:musicgen-small}
sfx-model: ${VIDEO_EDITING_LOCAL_ASSET_SFX_MODEL:audiogen-medium}
local-director: local-director:
enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true} enabled: ${VIDEO_EDITING_LOCAL_DIRECTOR_ENABLED:true}
source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source} source-directory: ${VIDEO_EDITING_LOCAL_DIRECTOR_SOURCE_DIRECTORY:./input/editing/source}

View File

@ -123,11 +123,15 @@ class EditProjectControllerTest {
mockMvc.perform(put("/v1/edit-projects/{projectId}/plan", projectId) mockMvc.perform(put("/v1/edit-projects/{projectId}/plan", projectId)
.contentType(MediaType.APPLICATION_JSON) .contentType(MediaType.APPLICATION_JSON)
.content(""" .content("""
{"style":"cinematic-porsche-promo","targetDurationSeconds":45, {"style":"cinematic-porsche-promo","targetDurationSeconds":4,
"decisions":[{"clipId":"clip-1","sourceStartSeconds":0,"sourceEndSeconds":4, "decisions":[{"clipId":"clip-1","sourceStartSeconds":0,"sourceEndSeconds":4,
"timelineStartSeconds":0,"timelineEndSeconds":4,"transitionIn":"cut", "timelineStartSeconds":0,"timelineEndSeconds":4,"transitionIn":"cut",
"transitionOut":"cut","playbackSpeed":1,"visualTreatment":"grade","reason":"hero"}], "transitionOut":"cut","playbackSpeed":1,"visualTreatment":"grade","reason":"hero"}],
"audioCues":[],"voiceover":[],"renderProfile":"mp4-h264-aac-1080p"} "audioCues":[{"type":"music","assetKey":"cinematic-bed","timelineStartSeconds":0,
"timelineEndSeconds":4,"gainDb":-12,"notes":"bed"}],"voiceover":[],
"renderProfile":"mp4-h264-aac-1080p","overlays":[{"text":"Hero shot",
"timelineStartSeconds":0,"timelineEndSeconds":4,"placement":"lower_center_safe",
"animation":"fade_in","reason":"title"}]}
""")) """))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(jsonPath("$.projectId").value(projectId)); .andExpect(jsonPath("$.projectId").value(projectId));

View File

@ -10,7 +10,8 @@ import org.springframework.test.web.servlet.MvcResult;
import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.hamcrest.Matchers.greaterThanOrEqualTo; import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.isOneOf; import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.is;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
@ -106,7 +107,7 @@ class VideoAssetControllerTest {
mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId)) mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(jsonPath("$.assetId").value(assetId)) .andExpect(jsonPath("$.assetId").value(assetId))
.andExpect(jsonPath("$.status", isOneOf("QUEUED", "RUNNING", "SUCCEEDED"))); .andExpect(jsonPath("$.status", anyOf(is("QUEUED"), is("RUNNING"), is("SUCCEEDED"))));
for (int attempt = 0; attempt < 20; attempt++) { for (int attempt = 0; attempt < 20; attempt++) {
String statusBody = mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId)) String statusBody = mockMvc.perform(get("/v1/clip-jobs/{jobId}", jobId))

View File

@ -38,7 +38,7 @@ class VideoClippingPropertiesTest {
assertThat(properties.getEditing().getAudioSampleRate()).isEqualTo(48000); assertThat(properties.getEditing().getAudioSampleRate()).isEqualTo(48000);
assertThat(properties.getEditing().getVideoBitrate()).isEqualTo("12000k"); assertThat(properties.getEditing().getVideoBitrate()).isEqualTo("12000k");
assertThat(properties.getEditing().getAudioBitrate()).isEqualTo("192k"); assertThat(properties.getEditing().getAudioBitrate()).isEqualTo("192k");
assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("noop"); assertThat(properties.getEditing().getVoiceoverProvider()).isEqualTo("local");
assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-16.0); assertThat(properties.getEditing().getLoudnessTargetI()).isEqualTo(-16.0);
assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-1.5); assertThat(properties.getEditing().getLoudnessTruePeak()).isEqualTo(-1.5);
assertThat(properties.getEditing().getLoudnessRange()).isEqualTo(11.0); assertThat(properties.getEditing().getLoudnessRange()).isEqualTo(11.0);
@ -67,6 +67,21 @@ class VideoClippingPropertiesTest {
.isEqualTo("/health"); .isEqualTo("/health");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs()) assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(1000); .isEqualTo(1000);
assertThat(properties.getEditing().getLocalAssetWorker().isAutoStart()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getBootstrapScript())
.isEqualTo("./tools/run_local_asset_worker.sh");
assertThat(properties.getEditing().getLocalAssetWorker().getScript())
.isEqualTo("./tools/local_asset_worker.py");
assertThat(properties.getEditing().getLocalAssetWorker().getStartupWaitMs()).isZero();
assertThat(properties.getEditing().getLocalAssetWorker().getHealthPath()).isEqualTo("/health");
assertThat(properties.getEditing().getLocalAssetWorker().getHealthCheckIntervalMs()).isEqualTo(1000);
assertThat(properties.getEditing().getLocalAssetWorker().isStrictRuntime()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getPythonBinary())
.isEqualTo("./.venv-local-asset/bin/python");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperBinary()).isEqualTo("piper");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperModelPath()).isEmpty();
assertThat(properties.getEditing().getLocalAssetWorker().getMusicModel()).isEqualTo("musicgen-small");
assertThat(properties.getEditing().getLocalAssetWorker().getSfxModel()).isEqualTo("audiogen-medium");
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue(); assertThat(properties.getEditing().getLocalDirector().isEnabled()).isTrue();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()) assertThat(properties.getEditing().getLocalDirector().getSourceDirectory())
.isEqualTo("./input/editing/source"); .isEqualTo("./input/editing/source");
@ -114,6 +129,17 @@ class VideoClippingPropertiesTest {
"video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250", "video-clipping.editing.visual-analysis.local-cv-worker.startup-wait-ms=250",
"video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready", "video-clipping.editing.visual-analysis.local-cv-worker.health-path=/ready",
"video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25", "video-clipping.editing.visual-analysis.local-cv-worker.health-check-interval-ms=25",
"video-clipping.editing.local-asset-worker.auto-start=true",
"video-clipping.editing.local-asset-worker.bootstrap-script=/tmp/local-asset-bootstrap.sh",
"video-clipping.editing.local-asset-worker.script=/tmp/local-asset.py",
"video-clipping.editing.local-asset-worker.startup-wait-ms=250",
"video-clipping.editing.local-asset-worker.health-path=/asset-health",
"video-clipping.editing.local-asset-worker.health-check-interval-ms=50",
"video-clipping.editing.local-asset-worker.python-binary=/tmp/python",
"video-clipping.editing.local-asset-worker.piper-binary=/tmp/piper",
"video-clipping.editing.local-asset-worker.piper-model-path=/tmp/voice.onnx",
"video-clipping.editing.local-asset-worker.music-model=musicgen-large",
"video-clipping.editing.local-asset-worker.sfx-model=audiogen-large",
"video-clipping.editing.local-director.enabled=false", "video-clipping.editing.local-director.enabled=false",
"video-clipping.editing.local-director.source-directory=/tmp/source", "video-clipping.editing.local-director.source-directory=/tmp/source",
"video-clipping.editing.local-director.poll-interval-ms=9000", "video-clipping.editing.local-director.poll-interval-ms=9000",
@ -162,6 +188,22 @@ class VideoClippingPropertiesTest {
.isEqualTo("/ready"); .isEqualTo("/ready");
assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs()) assertThat(properties.getEditing().getVisualAnalysis().getLocalCvWorker().getHealthCheckIntervalMs())
.isEqualTo(25); .isEqualTo(25);
assertThat(properties.getEditing().getLocalAssetWorker().isAutoStart()).isTrue();
assertThat(properties.getEditing().getLocalAssetWorker().getBootstrapScript())
.isEqualTo("/tmp/local-asset-bootstrap.sh");
assertThat(properties.getEditing().getLocalAssetWorker().getScript()).isEqualTo("/tmp/local-asset.py");
assertThat(properties.getEditing().getLocalAssetWorker().getStartupWaitMs()).isEqualTo(250);
assertThat(properties.getEditing().getLocalAssetWorker().getHealthPath()).isEqualTo("/asset-health");
assertThat(properties.getEditing().getLocalAssetWorker().getHealthCheckIntervalMs()).isEqualTo(50);
assertThat(properties.getEditing().getLocalAssetWorker().isStrictRuntime()).isFalse();
assertThat(properties.getEditing().getLocalAssetWorker().getPythonBinary()).isEqualTo("/tmp/python");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperBinary()).isEqualTo("/tmp/piper");
assertThat(properties.getEditing().getLocalAssetWorker().getPiperModelPath())
.isEqualTo("/tmp/voice.onnx");
assertThat(properties.getEditing().getLocalAssetWorker().getMusicModel())
.isEqualTo("musicgen-large");
assertThat(properties.getEditing().getLocalAssetWorker().getSfxModel())
.isEqualTo("audiogen-large");
assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse(); assertThat(properties.getEditing().getLocalDirector().isEnabled()).isFalse();
assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source"); assertThat(properties.getEditing().getLocalDirector().getSourceDirectory()).isEqualTo("/tmp/source");
assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000); assertThat(properties.getEditing().getLocalDirector().getPollIntervalMs()).isEqualTo(9000);

View File

@ -135,8 +135,13 @@ class CinematicEditingIntegrationTest {
index * 2.0, (index + 1) * 2.0, "cut", "cut", 1, index * 2.0, (index + 1) * 2.0, "cut", "cut", 1,
"cinematic grade", "integration sequence")) "cinematic grade", "integration sequence"))
.toList(); .toList();
List<AudioCue> audioCues = List.of(new AudioCue("music", "cinematic-bed", 0.0,
decisions.get(decisions.size() - 1).timelineEndSeconds(), -12.0, "integration score"));
List<TextOverlay> overlays = List.of(new TextOverlay("Cinematic highlight", 0.0,
decisions.get(0).timelineEndSeconds(), "lower_center_safe", "fade_in", "integration title"));
return new EditPlan(projectId, "category-aware-cinematic-highlights", return new EditPlan(projectId, "category-aware-cinematic-highlights",
decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, List.of(), List.of(), decisions.get(decisions.size() - 1).timelineEndSeconds(), decisions, audioCues, List.of(),
overlays,
"mp4-h264-aac-320p", "Generated integration edit"); "mp4-h264-aac-320p", "Generated integration edit");
} }

View File

@ -34,7 +34,7 @@ class ContactSheetGeneratorTest {
assertCommandPair(command.get(), "-i", "./clips/clip_00001.mp4"); assertCommandPair(command.get(), "-i", "./clips/clip_00001.mp4");
assertCommandPair(command.get(), "-vf", "fps=1/2,scale=320:-1,tile=4x4"); assertCommandPair(command.get(), "-vf", "fps=1/2,scale=320:-1,tile=4x4");
assertCommandPair(command.get(), "-frames:v", "1"); assertCommandPair(command.get(), "-frames:v", "1");
assertThat(command.get().getLast()).endsWith("clip_00001.jpg"); assertThat(command.get().get(command.get().size() - 1)).endsWith("clip_00001.jpg");
} }
@Test @Test

View File

@ -32,9 +32,10 @@ class EditPlanServiceTest {
new ClipAnalysis("clip-1", "clip.mp4", 8, "h264", "aac", 1920, 1080, 30, new ClipAnalysis("clip-1", "clip.mp4", 8, "h264", "aac", 1920, 1080, 30,
List.of(), null, null, 0, 0, 0)), List.of(), Instant.now())); List.of(), null, null, 0, 0, 0)), List.of(), Instant.now()));
EditPlanService service = new EditPlanService(store, projects, new EditPlanValidator(store)); EditPlanService service = new EditPlanService(store, projects, new EditPlanValidator(store));
SaveEditPlanRequest request = new SaveEditPlanRequest("cinematic-porsche-promo", 60, List<AudioCue> audioCues = List.of(new AudioCue("music", "cinematic-bed", 0, 4, -12, "bed"));
SaveEditPlanRequest request = new SaveEditPlanRequest("cinematic-porsche-promo", 4,
List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "grade", "hero")), List.of(new EditDecision("clip-1", 0, 4, 0, 4, "cut", "cut", 1, "grade", "hero")),
List.of(), List.of(), "mp4-h264-aac-1080p", "summary"); audioCues, List.of(), List.of(), "mp4-h264-aac-1080p", "summary");
EditPlan saved = service.save("project", request); EditPlan saved = service.save("project", request);

View File

@ -53,9 +53,9 @@ class EditProjectAnalyzerTest {
assertThat(analysis.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z")); assertThat(analysis.createdAt()).isEqualTo(Instant.parse("2026-07-10T10:00:00Z"));
assertThat(analysis.errors()).isEmpty(); assertThat(analysis.errors()).isEmpty();
assertThat(analysis.clips()).hasSize(1); assertThat(analysis.clips()).hasSize(1);
assertThat(analysis.clips().getFirst().thumbnails()).containsExactly("thumb-1.jpg"); assertThat(analysis.clips().get(0).thumbnails()).containsExactly("thumb-1.jpg");
assertThat(analysis.clips().getFirst().contactSheet()).isEqualTo("contact-sheet.jpg"); assertThat(analysis.clips().get(0).contactSheet()).isEqualTo("contact-sheet.jpg");
assertThat(analysis.clips().getFirst().proxyPath()).isEqualTo("proxy.mp4"); assertThat(analysis.clips().get(0).proxyPath()).isEqualTo("proxy.mp4");
assertThat(Files.exists(tempDir.resolve("projects/project-1/analysis.json"))).isTrue(); assertThat(Files.exists(tempDir.resolve("projects/project-1/analysis.json"))).isTrue();
assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis); assertThat(analyzer.readAnalysis("project-1")).isEqualTo(analysis);
} }

View File

@ -14,7 +14,9 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock; import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.when; import static org.mockito.Mockito.when;
import static org.mockito.Mockito.verify;
class HighlightDirectorFlowServiceTest { class HighlightDirectorFlowServiceTest {
@ -66,6 +68,42 @@ class HighlightDirectorFlowServiceTest {
.isEqualTo(HighlightProjectStatus.RENDERED); .isEqualTo(HighlightProjectStatus.RENDERED);
} }
@Test
void waitsForAssetsInsteadOfRenderingWhenLocalWorkerLeavesPendingRequests() throws Exception {
VideoClippingProperties properties = properties();
ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
FileSystemHighlightProjectStore store = new FileSystemHighlightProjectStore(properties, mapper);
HighlightProject project = project();
store.createProject(project, manifest());
store.writeJson(project.id(), "analysis/source-analysis.json", sourceAnalysis());
store.writeJson(project.id(), "director/edit-plan.json", directorPlan());
Path source = store.sourceDirectory(project.id()).resolve("source.mp4");
createDummyFile(source);
HighlightVisualEffectsStage visualEffectsStage = mock(HighlightVisualEffectsStage.class);
HighlightAssetPreparationService assetPreparationService = mock(HighlightAssetPreparationService.class);
HighlightAssetPreparationService.HighlightAssetPreparationResult assetResult =
new HighlightAssetPreparationService.HighlightAssetPreparationResult(
store.highlightsDirectory(project.id()).resolve("highlight_001"),
store.highlightsDirectory(project.id()).resolve("highlight_001").resolve("assets"),
List.of("requests/voiceover.json"), List.of());
when(assetPreparationService.prepare(anyString(), any(), any(), any())).thenReturn(assetResult);
HighlightLocalAssetWorker assetWorker = mock(HighlightLocalAssetWorker.class);
when(assetWorker.process(anyString(), any(), any())).thenReturn(
new HighlightLocalAssetWorker.HighlightAssetWorkerResult(project.id(), "highlight_001",
List.of(), List.of("requests/voiceover.json")));
HighlightFfmpegRenderer renderer = mock(HighlightFfmpegRenderer.class);
HighlightDirectorFlowService service = new HighlightDirectorFlowService(properties, mapper, store,
visualEffectsStage, assetPreparationService, assetWorker, renderer);
HighlightDirectorFlowService.HighlightFlowResult result = service.process(project.id(), 1);
assertThat(result.reason()).isEqualTo("assets_pending");
assertThat(result.finalOutputs()).isEmpty();
verify(renderer, never()).render(anyString(), any(), any());
assertThat(store.readJson(project.id(), "project.json", HighlightProject.class).status())
.isEqualTo(HighlightProjectStatus.WAITING_FOR_DIRECTOR);
}
private VideoClippingProperties properties() { private VideoClippingProperties properties() {
VideoClippingProperties properties = new VideoClippingProperties(); VideoClippingProperties properties = new VideoClippingProperties();
properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString()); properties.getEditing().setHighlightProjectDirectory(tempDir.resolve("highlight-projects").toString());

View File

@ -52,7 +52,8 @@ class HighlightLocalAssetWorkerTest {
"premium cinematic grade", "low pulse", "whoosh", "premium cinematic grade", "low pulse", "whoosh",
List.of("A clean first look."), List.of("Pure presence"), "open with presence"); List.of("A clean first look."), List.of("Pure presence"), "open with presence");
prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG); prep.prepare("project-1", project, highlight, ContentCategory.GENERIC_VLOG);
HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary, mapper); HighlightLocalAssetWorker worker = new HighlightLocalAssetWorker(store, assetProvider, assetLibrary,
new LocalAssetSynthesizer(properties), mapper);
HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight, HighlightLocalAssetWorker.HighlightAssetWorkerResult result = worker.process("project-1", highlight,
ContentCategory.GENERIC_VLOG); ContentCategory.GENERIC_VLOG);

View File

@ -18,7 +18,7 @@ class LocalAssetGenerationStageTest {
Path tempDir; Path tempDir;
@Test @Test
void reusesExistingSharedAssetsAndWritesRequestsForMissingVoiceover() throws Exception { void reusesExistingSharedAssetsAndSynthesizesMissingVoiceover() throws Exception {
VideoClippingProperties properties = properties(); VideoClippingProperties properties = properties();
Files.createDirectories(tempDir.resolve("shared/music")); Files.createDirectories(tempDir.resolve("shared/music"));
Files.createDirectories(tempDir.resolve("shared/sfx")); Files.createDirectories(tempDir.resolve("shared/sfx"));
@ -36,6 +36,7 @@ class LocalAssetGenerationStageTest {
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store, LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties), new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)), new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new LocalAssetSynthesizer(properties),
new ObjectMapper().findAndRegisterModules()); new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4, EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
@ -55,13 +56,13 @@ class LocalAssetGenerationStageTest {
assertThat(result.readyForRender()).isTrue(); assertThat(result.readyForRender()).isTrue();
assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists(); assertThat(tempDir.resolve("projects/project/audio/music.wav")).exists();
assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists(); assertThat(tempDir.resolve("projects/project/audio/sfx/whoosh.wav")).exists();
assertThat(tempDir.resolve("projects/project/audio/voiceover.wav")).exists();
assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists(); assertThat(tempDir.resolve("projects/project/assets/asset-generation-manifest.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists(); assertThat(tempDir.resolve("projects/project/assets/generated-assets.json")).exists();
assertThat(tempDir.resolve("projects/project/assets/requests/voiceover")).exists();
} }
@Test @Test
void writesBlockingRequestsWhenSfxIsMissing() throws Exception { void synthesizesMissingSfxAndKeepsProjectReadyForRender() throws Exception {
VideoClippingProperties properties = properties(); VideoClippingProperties properties = properties();
properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString()); properties.getEditing().getAssets().setMusicFolder(tempDir.resolve("shared/music").toString());
properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString()); properties.getEditing().getAssets().setSfxFolder(tempDir.resolve("shared/sfx").toString());
@ -73,6 +74,7 @@ class LocalAssetGenerationStageTest {
LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store, LocalAssetGenerationStage stage = new LocalAssetGenerationStage(properties, store,
new LocalEditAssetProvider(properties), new LocalEditAssetProvider(properties),
new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)), new LocalEditAssetLibrary(new LocalEditAssetProvider(properties)),
new LocalAssetSynthesizer(properties),
new ObjectMapper().findAndRegisterModules()); new ObjectMapper().findAndRegisterModules());
EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4, EditPlan plan = new EditPlan("project", "cinematic-porsche-promo", 4,
@ -82,8 +84,8 @@ class LocalAssetGenerationStageTest {
AssetGenerationResult result = stage.prepare("project", plan); AssetGenerationResult result = stage.prepare("project", plan);
assertThat(result.readyForRender()).isFalse(); assertThat(result.readyForRender()).isTrue();
assertThat(tempDir.resolve("projects/project/assets/requests/sfx")).exists(); assertThat(tempDir.resolve("projects/project/audio/sfx/engine-hit.wav")).exists();
} }
private VideoClippingProperties properties() { private VideoClippingProperties properties() {

View File

@ -80,7 +80,7 @@ class LocalDirectorSchedulerTest {
assertThat(project.status()).isEqualTo(EditProjectStatus.WAITING_FOR_DIRECTOR); assertThat(project.status()).isEqualTo(EditProjectStatus.WAITING_FOR_DIRECTOR);
assertThat(project.inputDirectory()).isEqualTo(tempDir.resolve("processed/01-porsche").toString()); assertThat(project.inputDirectory()).isEqualTo(tempDir.resolve("processed/01-porsche").toString());
EditProjectAnalysis analysis = store.readJson("01-porsche", "analysis.json", EditProjectAnalysis.class); EditProjectAnalysis analysis = store.readJson("01-porsche", "analysis.json", EditProjectAnalysis.class);
assertThat(analysis.clips().getFirst().sourcePath()) assertThat(analysis.clips().get(0).sourcePath())
.isEqualTo(tempDir.resolve("processed/01-porsche/clip_00001.mp4").toString()); .isEqualTo(tempDir.resolve("processed/01-porsche/clip_00001.mp4").toString());
assertThat(firstClip).doesNotExist(); assertThat(firstClip).doesNotExist();
} }

View File

@ -40,7 +40,7 @@ class ThumbnailExtractorTest {
assertCommandPair(commands.get(1), "-ss", "4.000"); assertCommandPair(commands.get(1), "-ss", "4.000");
assertCommandPair(commands.get(2), "-ss", "6.000"); assertCommandPair(commands.get(2), "-ss", "6.000");
assertCommandPair(commands.get(0), "-i", "./clips/clip_00001.mp4"); assertCommandPair(commands.get(0), "-i", "./clips/clip_00001.mp4");
assertThat(commands.get(0).getLast()).endsWith("clip_00001_0001.jpg"); assertThat(commands.get(0).get(commands.get(0).size() - 1)).endsWith("clip_00001_0001.jpg");
} }
@Test @Test

View File

@ -24,7 +24,7 @@ class FolderFfmpegClipperTest {
AtomicReference<List<String>> command = new AtomicReference<>(); AtomicReference<List<String>> command = new AtomicReference<>();
FolderFfmpegClipper clipper = clipper(arguments -> { FolderFfmpegClipper clipper = clipper(arguments -> {
command.set(arguments); command.set(arguments);
Path outputDirectory = Path.of(arguments.getLast()).getParent(); Path outputDirectory = Path.of(arguments.get(arguments.size() - 1)).getParent();
Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip"); Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip");
Files.writeString(outputDirectory.resolve("clip_00001.mp4"), "clip"); Files.writeString(outputDirectory.resolve("clip_00001.mp4"), "clip");
Files.writeString(outputDirectory.resolve("ignored.txt"), "ignored"); Files.writeString(outputDirectory.resolve("ignored.txt"), "ignored");
@ -38,7 +38,7 @@ class FolderFfmpegClipperTest {
assertEquals(2, result.clipCount()); assertEquals(2, result.clipCount());
assertCommandPair(command.get(), "-c", "copy"); assertCommandPair(command.get(), "-c", "copy");
assertCommandPair(command.get(), "-segment_time", "8"); assertCommandPair(command.get(), "-segment_time", "8");
assertTrue(command.get().getLast().endsWith("clip_%05d.mp4")); assertTrue(command.get().get(command.get().size() - 1).endsWith("clip_%05d.mp4"));
} }
@Test @Test
@ -48,7 +48,7 @@ class FolderFfmpegClipperTest {
properties.getFolderScheduler().setPreserveInputQuality(false); properties.getFolderScheduler().setPreserveInputQuality(false);
FolderFfmpegClipper clipper = new FolderFfmpegClipper(properties, arguments -> { FolderFfmpegClipper clipper = new FolderFfmpegClipper(properties, arguments -> {
command.set(arguments); command.set(arguments);
Path outputDirectory = Path.of(arguments.getLast()).getParent(); Path outputDirectory = Path.of(arguments.get(arguments.size() - 1)).getParent();
Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip"); Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip");
return new FolderFfmpegClipper.ProcessResult(0, "ok"); return new FolderFfmpegClipper.ProcessResult(0, "ok");
}); });
@ -66,7 +66,7 @@ class FolderFfmpegClipperTest {
Files.createDirectory(tempDir.resolve("output/movie")); Files.createDirectory(tempDir.resolve("output/movie"));
Files.createDirectory(tempDir.resolve("output/movie-1")); Files.createDirectory(tempDir.resolve("output/movie-1"));
FolderFfmpegClipper clipper = clipper(arguments -> { FolderFfmpegClipper clipper = clipper(arguments -> {
Path outputDirectory = Path.of(arguments.getLast()).getParent(); Path outputDirectory = Path.of(arguments.get(arguments.size() - 1)).getParent();
Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip"); Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip");
return new FolderFfmpegClipper.ProcessResult(0, "ok"); return new FolderFfmpegClipper.ProcessResult(0, "ok");
}); });
@ -77,7 +77,7 @@ class FolderFfmpegClipperTest {
@Test @Test
void usesWholeFilenameAsStemWhenThereIsNoExtension() throws Exception { void usesWholeFilenameAsStemWhenThereIsNoExtension() throws Exception {
FolderFfmpegClipper clipper = clipper(arguments -> { FolderFfmpegClipper clipper = clipper(arguments -> {
Path outputDirectory = Path.of(arguments.getLast()).getParent(); Path outputDirectory = Path.of(arguments.get(arguments.size() - 1)).getParent();
Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip"); Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip");
return new FolderFfmpegClipper.ProcessResult(0, "ok"); return new FolderFfmpegClipper.ProcessResult(0, "ok");
}); });
@ -96,7 +96,7 @@ class FolderFfmpegClipperTest {
@Test @Test
void reportsGeneratedClipListingFailure() { void reportsGeneratedClipListingFailure() {
assertThrows(IllegalStateException.class, () -> clipper(arguments -> { assertThrows(IllegalStateException.class, () -> clipper(arguments -> {
Files.delete(Path.of(arguments.getLast()).getParent()); Files.delete(Path.of(arguments.get(arguments.size() - 1)).getParent());
return new FolderFfmpegClipper.ProcessResult(0, "ok"); return new FolderFfmpegClipper.ProcessResult(0, "ok");
}).clip(Path.of("deleted-output.mp4"))); }).clip(Path.of("deleted-output.mp4")));
} }

View File

@ -32,8 +32,8 @@ class FolderVideoValidatorTest {
assertTrue(result.valid()); assertTrue(result.valid());
assertEquals("valid video", result.reason()); assertEquals("valid video", result.reason());
assertEquals("ffprobe", command.get().getFirst()); assertEquals("ffprobe", command.get().get(0));
assertEquals("movie.MP4", command.get().getLast()); assertEquals("movie.MP4", command.get().get(command.get().size() - 1));
} }
@Test @Test

View File

@ -2,3 +2,5 @@ video-clipping.folder-scheduler.enabled=false
video-clipping.editing.local-director.enabled=false video-clipping.editing.local-director.enabled=false
video-clipping.editing.highlight-scheduler.enabled=false video-clipping.editing.highlight-scheduler.enabled=false
video-clipping.editing.visual-analysis.local-cv-worker.auto-start=false video-clipping.editing.visual-analysis.local-cv-worker.auto-start=false
video-clipping.editing.local-asset-worker.auto-start=false
video-clipping.editing.local-asset-worker.strict-runtime=false

View File

@ -1 +0,0 @@
stub-video-content

View File

@ -1 +0,0 @@
stub-video-content

View File

@ -1 +0,0 @@
stub-video-content

View File

@ -1 +0,0 @@
stub-clip-content-clip_55da9cbe7120

View File

@ -1 +0,0 @@
stub-clip-content-clip_4f379a7d5941

View File

@ -1 +0,0 @@
stub-clip-content-clip_12f6257786bf

View File

@ -1 +0,0 @@
stub-clip-content-clip_3db7ae28dd6e

View File

@ -1 +0,0 @@
stub-clip-content-clip_a87a659135b7

View File

@ -1 +0,0 @@
stub-clip-content-clip_f6d9f025b976

View File

@ -1 +0,0 @@
stub-clip-content-clip_4c3db1c00f58

View File

@ -1 +0,0 @@
stub-clip-content-clip_40f5c4bd1569

View File

@ -1 +0,0 @@
stub-clip-content-clip_9bc569665c0f

View File

@ -1 +0,0 @@
stub-clip-content-clip_10c8dc32bebf

View File

@ -1 +0,0 @@
stub-clip-content-clip_04750c742dae

View File

@ -1 +0,0 @@
stub-clip-content-clip_9c0062d6e046

View File

@ -1 +0,0 @@
stub-clip-content-clip_f205e44bd8cb

View File

@ -1 +0,0 @@
stub-clip-content-clip_0f9504972838

View File

@ -1 +0,0 @@
stub-clip-content-clip_26599ec473ec

View File

@ -1 +0,0 @@
stub-clip-content-clip_247296e47c86

View File

@ -1 +0,0 @@
stub-clip-content-clip_30378c1a8d2c

View File

@ -1 +0,0 @@
stub-clip-content-clip_f859bc77404b

View File

@ -1 +0,0 @@
stub-clip-content-clip_8469868182d1

View File

@ -1 +0,0 @@
stub-clip-content-clip_dedf0bdd8aec

View File

@ -1 +0,0 @@
stub-clip-content-clip_937f3ff7df71

View File

@ -1 +0,0 @@
stub-clip-content-clip_6610d92757c1

View File

@ -1 +0,0 @@
stub-clip-content-clip_f6612e1119d5

View File

@ -1 +0,0 @@
stub-clip-content-clip_b68e31fe0da6

View File

@ -1 +0,0 @@
stub-clip-content-clip_a58ccb424fa0

View File

@ -1 +0,0 @@
stub-clip-content-clip_59a1c0bee8ed

View File

@ -1 +0,0 @@
stub-clip-content-clip_aeaf1155bc22

View File

@ -1 +0,0 @@
stub-clip-content-clip_5806cd2fc91e

View File

@ -1 +0,0 @@
stub-clip-content-clip_297c6a9ab956

View File

@ -1 +0,0 @@
stub-clip-content-clip_8a000ebfe018

View File

@ -1 +0,0 @@
stub-clip-content-clip_4c3db1c00f58

View File

@ -1 +0,0 @@
stub-clip-content-clip_40f5c4bd1569

View File

@ -1 +0,0 @@
stub-clip-content-clip_9bc569665c0f

View File

@ -1 +0,0 @@
stub-clip-content-clip_247296e47c86

View File

@ -1 +0,0 @@
stub-clip-content-clip_30378c1a8d2c

View File

@ -1 +0,0 @@
stub-clip-content-clip_f859bc77404b

View File

@ -1 +0,0 @@
stub-clip-content-clip_8469868182d1

View File

@ -1 +0,0 @@
stub-clip-content-clip_dedf0bdd8aec

View File

@ -1 +0,0 @@
stub-clip-content-clip_937f3ff7df71

View File

@ -0,0 +1,4 @@
torch
audiocraft
soundfile
numpy

180
tools/local_asset_worker.py Normal file
View File

@ -0,0 +1,180 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import math
import os
import shutil
import subprocess
import wave
from array import array
from pathlib import Path
try:
import soundfile as sf # type: ignore
except Exception: # pragma: no cover - optional runtime dependency
sf = None
try:
import torch # type: ignore
except Exception: # pragma: no cover - optional runtime dependency
torch = None
try:
import numpy as np # type: ignore
except Exception: # pragma: no cover - optional runtime dependency
np = None
def main() -> int:
parser = argparse.ArgumentParser(description="Local cinematic asset synthesizer")
parser.add_argument("kind", choices=["voiceover", "music", "sfx"])
parser.add_argument("--prompt-file", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--duration", type=float, default=8.0)
parser.add_argument("--model", default="")
parser.add_argument("--piper-binary", default=os.getenv("LOCAL_ASSET_PIPER_BINARY", "piper"))
args = parser.parse_args()
prompt = Path(args.prompt_file).read_text(encoding="utf-8").strip()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
if args.kind == "voiceover":
if synthesize_voiceover(prompt, output, args.piper_binary, args.model):
return 0
write_silence(output, max(1.0, args.duration))
return 0
if args.kind == "music":
if synthesize_audiocraft("music", prompt, output, args.duration, args.model):
return 0
write_fallback_tone(output, max(4.0, args.duration), 110.0, 0.02)
return 0
if synthesize_audiocraft("sfx", prompt, output, args.duration, args.model):
return 0
write_fallback_tone(output, max(0.5, min(1.0, args.duration)), 880.0, 0.12)
return 0
def synthesize_voiceover(prompt: str, output: Path, piper_binary: str, model: str) -> bool:
model_path = model.strip()
if model_path and shutil.which(piper_binary):
command = [piper_binary, "--model", model_path, "--output_file", str(output)]
try:
process = subprocess.run(
command,
input=prompt.encode("utf-8"),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
check=False,
)
return process.returncode == 0 and output.is_file()
except Exception:
return False
say = shutil.which("say")
if say:
process = subprocess.run([say, "-o", str(output), prompt], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return process.returncode == 0 and output.is_file()
espeak = shutil.which("espeak")
if espeak:
process = subprocess.run([espeak, "-w", str(output), prompt], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return process.returncode == 0 and output.is_file()
return False
def synthesize_audiocraft(kind: str, prompt: str, output: Path, duration: float, model_name: str) -> bool:
if torch is None:
return False
try:
if kind == "music":
from audiocraft.models import MusicGen # type: ignore
model = MusicGen.get_pretrained(normalize_model_name(model_name, "musicgen-small"))
model.set_generation_params(duration=max(1.0, float(duration)))
audio = model.generate([prompt])[0]
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
return output.is_file()
from audiocraft.models import AudioGen # type: ignore
model = AudioGen.get_pretrained(normalize_model_name(model_name, "audiogen-medium"))
model.set_generation_params(duration=max(0.5, float(duration)))
audio = model.generate([prompt])[0]
write_tensor_audio(audio, output, getattr(model, "sample_rate", 32000))
return output.is_file()
except Exception:
return False
def normalize_model_name(model_name: str, default_name: str) -> str:
value = model_name.strip() if model_name else default_name
if "/" in value:
return value
return f"facebook/{value}"
def write_tensor_audio(audio, output: Path, sample_rate: int) -> None:
if np is None:
raise RuntimeError("numpy is required to write model-generated audio")
if hasattr(audio, "detach"):
audio = audio.detach().cpu().numpy()
elif torch is not None and isinstance(audio, torch.Tensor):
audio = audio.cpu().numpy()
audio = np.asarray(audio)
if audio.ndim == 3:
audio = audio[0]
if audio.ndim == 2 and audio.shape[0] < audio.shape[1]:
audio = audio.transpose(1, 0)
if sf is not None:
sf.write(str(output), audio, sample_rate)
return
write_wave_file(output, audio, sample_rate)
def write_fallback_tone(output: Path, duration: float, frequency: float, volume: float) -> None:
sample_rate = 48000
total_samples = int(sample_rate * duration)
frames = array(
"h",
[
int(max(-1.0, min(1.0, math.sin(2.0 * math.pi * frequency * (i / sample_rate))) * volume) * 32767.0)
for i in range(total_samples)
],
)
write_wave_file(output, frames, sample_rate, channels=1)
def write_silence(output: Path, duration: float) -> None:
sample_rate = 24000
total_samples = int(sample_rate * duration)
frames = array("h", [0] * total_samples)
write_wave_file(output, frames, sample_rate, channels=1)
def write_wave_file(output: Path, waveform, sample_rate: int, channels: int | None = None) -> None:
if hasattr(waveform, "ndim") and np is not None:
waveform = np.asarray(waveform)
if waveform.ndim == 1:
waveform = waveform[:, None]
waveform = np.clip(waveform, -1.0, 1.0)
pcm = (waveform * 32767.0).astype(np.int16)
channels = pcm.shape[1]
frames = array("h", pcm.reshape(-1).tolist())
else:
frames = waveform
if channels is None:
channels = 1
with wave.open(str(output), "wb") as wav:
wav.setnchannels(channels)
wav.setsampwidth(2)
wav.setframerate(sample_rate)
wav.writeframes(frames.tobytes())
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,32 @@
#!/usr/bin/env bash
set -euo pipefail
HOST="${LOCAL_ASSET_HOST:-127.0.0.1}"
PORT="${LOCAL_ASSET_PORT:-8092}"
BOOTSTRAP_MODE="${LOCAL_ASSET_BOOTSTRAP_MODE:-auto}"
if [ ! -d ".venv-local-asset" ]; then
python3 -m venv .venv-local-asset
fi
. .venv-local-asset/bin/activate
SENTINEL=".venv-local-asset/.bootstrap-complete"
if [ "${BOOTSTRAP_MODE}" = "always" ]; then
python -m pip install --upgrade pip
python -m pip install -r tools/local_asset_requirements.txt
touch "${SENTINEL}"
elif [ "${BOOTSTRAP_MODE}" = "auto" ] && [ ! -f "${SENTINEL}" ]; then
python -m pip install --upgrade pip
python -m pip install -r tools/local_asset_requirements.txt
touch "${SENTINEL}"
else
echo "Local asset bootstrap skipped; using existing virtualenv."
fi
if [ "${LOCAL_ASSET_PRELOAD_ONLY:-false}" = "true" ]; then
echo "Local asset dependencies are ready."
exit 0
fi
exec python tools/local_asset_worker.py "$@"

View File

@ -4,14 +4,23 @@ set -euo pipefail
HOST="${LOCAL_CV_HOST:-127.0.0.1}" HOST="${LOCAL_CV_HOST:-127.0.0.1}"
PORT="${LOCAL_CV_PORT:-8091}" PORT="${LOCAL_CV_PORT:-8091}"
MODEL="${LOCAL_CV_YOLO_MODEL:-yolov8n.pt}" MODEL="${LOCAL_CV_YOLO_MODEL:-yolov8n.pt}"
BOOTSTRAP_MODE="${LOCAL_CV_BOOTSTRAP_MODE:-auto}"
if [ ! -d ".venv-local-cv" ]; then if [ ! -d ".venv-local-cv" ]; then
python3 -m venv .venv-local-cv python3 -m venv .venv-local-cv
fi fi
. .venv-local-cv/bin/activate . .venv-local-cv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r tools/local_cv_requirements.txt if [ "${BOOTSTRAP_MODE}" = "always" ]; then
python -m pip install --upgrade pip
python -m pip install -r tools/local_cv_requirements.txt
elif [ "${BOOTSTRAP_MODE}" = "auto" ] && [ ! -x ".venv-local-cv/bin/uvicorn" ]; then
python -m pip install --upgrade pip
python -m pip install -r tools/local_cv_requirements.txt
else
echo "Local CV bootstrap skipped; using existing virtualenv."
fi
if [ "${LOCAL_CV_DISABLE_YOLO:-false}" != "true" ]; then if [ "${LOCAL_CV_DISABLE_YOLO:-false}" != "true" ]; then
LOCAL_CV_YOLO_MODEL="$MODEL" python -c "import os; from ultralytics import YOLO; YOLO(os.environ['LOCAL_CV_YOLO_MODEL'])" LOCAL_CV_YOLO_MODEL="$MODEL" python -c "import os; from ultralytics import YOLO; YOLO(os.environ['LOCAL_CV_YOLO_MODEL'])"