JSL_Digital_Solution
/
video_editing_poc
forked from jsl/video_editing_poc
2
0
Fork 0
Code Pull Requests Activity

Configure folder scheduler in YAML

This commit is contained in:
JSLMPR 2026-07-10 12:24:16 +02:00
parent b61ab0604a
commit 7fd92a72df
14 changed files with 220 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
docs/input-folder-scheduler-plan.md
View File

@ -58,21 +58,24 @@ Sources:
Add a new scheduler config group under `video-clipping.folder-scheduler`.
Recommended properties:
Recommended YAML:
```properties
video-clipping.folder-scheduler.enabled=true
video-clipping.folder-scheduler.input-directory=/input/source
video-clipping.folder-scheduler.output-directory=/output/clips
video-clipping.folder-scheduler.processed-directory=/input/processed
video-clipping.folder-scheduler.rejected-directory=/input/rejected
video-clipping.folder-scheduler.working-directory=/input/working
video-clipping.folder-scheduler.poll-interval-ms=5000
video-clipping.folder-scheduler.segment-duration-seconds=8
video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg
video-clipping.folder-scheduler.ffprobe-binary=ffprobe
video-clipping.folder-scheduler.output-container=mp4
video-clipping.folder-scheduler.exact-preset=veryfast
```yaml
video-clipping:
folder-scheduler:
enabled: true
input-directory: /input/source
output-directory: /output/clips
processed-directory: /input/processed
rejected-directory: /input/rejected
working-directory: /input/working
poll-interval-ms: 5000
segment-duration-seconds: 8
ffmpeg-binary: ffmpeg
ffprobe-binary: ffprobe
output-container: mp4
exact-preset: veryfast
preserve-input-quality: true
```
Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow.
@ -173,35 +176,29 @@ Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contain
## FFmpeg Command
Default to exact 8-second MP4/H.264/AAC output:
Default to quality-preserving MP4 stream copy when the source already has usable keyframes:
```bash
ffmpeg -hide_banner -y \
-i /input/working/1.mp4 \
-map 0 \
-c:v libx264 \
-preset veryfast \
-crf 20 \
-force_key_frames "expr:gte(t,n_forced*8)" \
-c:a aac \
-b:a 128k \
-map 0 -c copy \
-f segment \
-segment_time 8 \
-segment_format_options movflags=+faststart \
-reset_timestamps 1 \
/output/clips/1/clip_%05d.mp4
```
Expected behavior:
- each generated clip is approximately 8 seconds
- each generated clip preserves the input codec quality because no re-encode occurs
- clips are cut on keyframes, so boundaries may drift from exact 8-second marks
- the final clip may be shorter than 8 seconds
- output files are broadly playable MP4 files
Implementation note:
- If exact boundaries are more important than speed, always transcode with `libx264` and force keyframes.
- If speed is more important and keyframe-aligned cuts are acceptable, add an optional `FAST` mode using `-c copy`, but document that clips may not start exactly at 8-second boundaries.
- If exact 8-second boundaries are more important than preserving source quality, transcode with `libx264` and force keyframes.
- If preserving source quality is more important, use stream copy and accept keyframe-aligned cuts.
## Proposed Code Structure
@ -268,6 +265,19 @@ This ensures the scheduler never processes the same source file repeatedly.
The scheduler uses an initial delay and fixed delay of `5000` ms by default. The initial delay lets startup directory initialization finish; subsequent delays start after the previous scan finishes, so scans never intentionally overlap. Configure both with `video-clipping.folder-scheduler.poll-interval-ms` (minimum `1000` ms).
The base configuration enables repository-local folder processing for normal IDE starts. Production deployments can override it without changing the artifact:
```bash
FOLDER_SCHEDULER_ENABLED=true
FOLDER_SCHEDULER_INPUT_DIRECTORY=/input/source
FOLDER_SCHEDULER_OUTPUT_DIRECTORY=/output/clips
FOLDER_SCHEDULER_WORKING_DIRECTORY=/input/working
FOLDER_SCHEDULER_PROCESSED_DIRECTORY=/input/processed
FOLDER_SCHEDULER_REJECTED_DIRECTORY=/input/rejected
FOLDER_SCHEDULER_POLL_INTERVAL_MS=5000
FOLDER_SCHEDULER_LOG_LEVEL=INFO
```
Lifecycle logs use stable `event=...` fields and a per-run `scan_id` for correlation. Active work and state transitions log at info or warn/error level; scan start, idle, completion, directory readiness, and probe timing log at debug level.
Log these events:

9
pom.xml
View File

@ -87,6 +87,15 @@
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<spring.profiles.active>test</spring.profiles.active>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>

10
src/main/java/org/example/videoclips/config/VideoClippingProperties.java
View File

@ -368,6 +368,8 @@ public class VideoClippingProperties {
private String exactPreset = "veryfast";
private boolean preserveInputQuality = true;
public boolean isEnabled() {
return enabled;
}
@ -463,6 +465,14 @@ public class VideoClippingProperties {
public void setExactPreset(String exactPreset) {
this.exactPreset = exactPreset;
}
public boolean isPreserveInputQuality() {
return preserveInputQuality;
}
public void setPreserveInputQuality(boolean preserveInputQuality) {
this.preserveInputQuality = preserveInputQuality;
}
}
public static class Quotas {

70
src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java
View File

@ -38,30 +38,13 @@ public class FolderFfmpegClipper {
Path outputDirectory = createOutputDirectory(sourceFile);
String segmentDuration = Integer.toString(properties.getSegmentDurationSeconds());
Path outputPattern = outputDirectory.resolve("clip_%05d." + properties.getOutputContainer());
List<String> command = List.of(
properties.getFfmpegBinary(),
"-hide_banner", "-y",
"-i", sourceFile.toString(),
"-map", "0:v:0",
"-map", "0:a?",
"-c:v", "libx264",
"-preset", properties.getExactPreset(),
"-crf", "20",
"-force_key_frames", "expr:gte(t,n_forced*" + segmentDuration + ")",
"-c:a", "aac",
"-b:a", "128k",
"-f", "segment",
"-segment_time", segmentDuration,
"-segment_format_options", "movflags=+faststart",
"-reset_timestamps", "1",
outputPattern.toString()
);
List<String> command = buildCommand(sourceFile, segmentDuration, outputPattern);
ProcessResult processResult;
long startedAt = System.nanoTime();
log.info("event=ffmpeg_started file={} output_directory={} segment_duration_seconds={} preset={}",
log.info("event=ffmpeg_started file={} output_directory={} segment_duration_seconds={} quality_mode={} preset={}",
sourceFile.getFileName(), outputDirectory, properties.getSegmentDurationSeconds(),
properties.getExactPreset());
qualityMode(), properties.getExactPreset());
try {
processResult = processExecutor.execute(command);
} catch (InterruptedException ex) {
@ -83,6 +66,53 @@ public class FolderFfmpegClipper {
return new ClipResult(outputDirectory, clipCount);
}
private List<String> buildCommand(Path sourceFile, String segmentDuration, Path outputPattern) {
List<String> command = new java.util.ArrayList<>();
command.add(properties.getFfmpegBinary());
command.add("-hide_banner");
command.add("-y");
command.add("-i");
command.add(sourceFile.toString());
command.add("-map");
command.add("0");
if (properties.isPreserveInputQuality()) {
command.add("-c");
command.add("copy");
} else {
command.add("-c:v");
command.add("libx264");
command.add("-preset");
command.add(properties.getExactPreset());
command.add("-crf");
command.add("20");
command.add("-force_key_frames");
command.add("expr:gte(t,n_forced*" + segmentDuration + ")");
command.add("-c:a");
command.add("aac");
command.add("-b:a");
command.add("128k");
}
command.add("-f");
command.add("segment");
command.add("-segment_time");
command.add(segmentDuration);
if (properties.isPreserveInputQuality()) {
command.add("-reset_timestamps");
command.add("1");
} else {
command.add("-segment_format_options");
command.add("movflags=+faststart");
command.add("-reset_timestamps");
command.add("1");
}
command.add(outputPattern.toString());
return command;
}
private String qualityMode() {
return properties.isPreserveInputQuality() ? "copy" : "transcode";
}
private long elapsedMillis(long startedAt) {
return TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startedAt);
}

21
src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java
View File

@ -4,6 +4,8 @@ import org.example.videoclips.config.VideoClippingProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@ -41,6 +43,15 @@ public class FolderVideoScanScheduler {
this.clipper = clipper;
}
@EventListener(ApplicationReadyEvent.class)
public void logStartup() {
log.info("event=folder_scheduler_started enabled=true initial_delay_ms={} fixed_delay_ms={} "
+ "segment_duration_seconds={} input_directory={} output_directory={} quality_mode={}",
properties.getPollIntervalMs(), properties.getPollIntervalMs(),
properties.getSegmentDurationSeconds(), properties.getInputDirectory(),
properties.getOutputDirectory(), properties.isPreserveInputQuality() ? "copy" : "transcode");
}
@Scheduled(
initialDelayString = "${video-clipping.folder-scheduler.poll-interval-ms:5000}",
fixedDelayString = "${video-clipping.folder-scheduler.poll-interval-ms:5000}"
@ -115,9 +126,15 @@ public class FolderVideoScanScheduler {
FolderVideoValidator.ValidationResult result = validator.validate(workingFile);
if (result.valid()) {
log.info("event=validation_succeeded scan_id={} file={}", scanId, workingFile.getFileName());
long clippingStartedAt = System.nanoTime();
log.info("event=video_processing_started scan_id={} file={} segment_duration_seconds={} quality_mode={}",
scanId, workingFile.getFileName(), properties.getSegmentDurationSeconds(),
properties.isPreserveInputQuality() ? "copy" : "transcode");
FolderFfmpegClipper.ClipResult clipResult = clipper.clip(workingFile);
log.info("event=clipping_succeeded scan_id={} file={} clip_count={} output_directory={}",
scanId, workingFile.getFileName(), clipResult.clipCount(), clipResult.outputDirectory());
log.info("event=video_processing_completed scan_id={} file={} elapsed_ms={} clip_count={} "
+ "clips_folder_name={} clips_folder={}",
scanId, workingFile.getFileName(), elapsedMillis(clippingStartedAt), clipResult.clipCount(),
clipResult.outputDirectory().getFileName(), clipResult.outputDirectory());
Path processedFile = moveToDirectory(workingFile, Path.of(properties.getProcessedDirectory()));
log.info("event=processing_succeeded scan_id={} file={} processed_directory={}",
scanId, processedFile.getFileName(), processedFile.getParent());

13
src/main/resources/application-folder-scheduler-local.properties
View File

@ -1,13 +0,0 @@
video-clipping.folder-scheduler.enabled=true
video-clipping.folder-scheduler.input-directory=./input/source
video-clipping.folder-scheduler.output-directory=./output/clips
video-clipping.folder-scheduler.processed-directory=./input/processed
video-clipping.folder-scheduler.rejected-directory=./input/rejected
video-clipping.folder-scheduler.working-directory=./input/working
video-clipping.folder-scheduler.poll-interval-ms=5000
video-clipping.folder-scheduler.segment-duration-seconds=8
video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg
video-clipping.folder-scheduler.ffprobe-binary=ffprobe
video-clipping.folder-scheduler.output-container=mp4
video-clipping.folder-scheduler.exact-preset=veryfast
logging.level.org.example.videoclips.folder=DEBUG

19
src/main/resources/application-folder-scheduler-local.yml Normal file
View File

@ -0,0 +1,19 @@
video-clipping:
folder-scheduler:
enabled: true
input-directory: ./input/source
output-directory: ./output/clips
processed-directory: ./input/processed
rejected-directory: ./input/rejected
working-directory: ./input/working
poll-interval-ms: 5000
segment-duration-seconds: 8
ffmpeg-binary: ffmpeg
ffprobe-binary: ffprobe
output-container: mp4
exact-preset: veryfast
preserve-input-quality: true
logging:
level:
org.example.videoclips.folder: INFO

12
src/main/resources/application.properties
View File

@ -25,18 +25,6 @@ video-clipping.ffmpeg.ffmpeg-binary=ffmpeg
video-clipping.ffmpeg.input-directory=./tmp/ffmpeg-input
video-clipping.ffmpeg.output-directory=./tmp/ffmpeg-output
video-clipping.ffmpeg.cleanup-local-files=true
video-clipping.folder-scheduler.enabled=false
video-clipping.folder-scheduler.input-directory=/input/source
video-clipping.folder-scheduler.output-directory=/output/clips
video-clipping.folder-scheduler.processed-directory=/input/processed
video-clipping.folder-scheduler.rejected-directory=/input/rejected
video-clipping.folder-scheduler.working-directory=/input/working
video-clipping.folder-scheduler.poll-interval-ms=5000
video-clipping.folder-scheduler.segment-duration-seconds=8
video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg
video-clipping.folder-scheduler.ffprobe-binary=ffprobe
video-clipping.folder-scheduler.output-container=mp4
video-clipping.folder-scheduler.exact-preset=veryfast
video-clipping.quotas.max-active-jobs-per-tenant=5
video-clipping.cleanup.enabled=true
video-clipping.cleanup.local-artifact-poll-interval-ms=300000

19
src/main/resources/application.yml Normal file
View File

@ -0,0 +1,19 @@
video-clipping:
folder-scheduler:
enabled: ${FOLDER_SCHEDULER_ENABLED:true}
input-directory: ${FOLDER_SCHEDULER_INPUT_DIRECTORY:./input/source}
output-directory: ${FOLDER_SCHEDULER_OUTPUT_DIRECTORY:./output/clips}
processed-directory: ${FOLDER_SCHEDULER_PROCESSED_DIRECTORY:./input/processed}
rejected-directory: ${FOLDER_SCHEDULER_REJECTED_DIRECTORY:./input/rejected}
working-directory: ${FOLDER_SCHEDULER_WORKING_DIRECTORY:./input/working}
poll-interval-ms: ${FOLDER_SCHEDULER_POLL_INTERVAL_MS:5000}
segment-duration-seconds: 8
ffmpeg-binary: ffmpeg
ffprobe-binary: ffprobe
output-container: mp4
exact-preset: veryfast
preserve-input-quality: ${FOLDER_SCHEDULER_PRESERVE_INPUT_QUALITY:true}
logging:
level:
org.example.videoclips.folder: ${FOLDER_SCHEDULER_LOG_LEVEL:INFO}

73
src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
View File

@ -1,73 +0,0 @@
package org.example.videoclips.api;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.MediaType;
import org.springframework.test.web.servlet.MockMvc;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@SpringBootTest(properties = "video-clipping.quotas.max-active-jobs-per-tenant=0")
@AutoConfigureMockMvc
class TenantQuotaControllerTest {
@Autowired
private MockMvc mockMvc;
@Test
void rejectsClipJobCreationWhenTenantActiveJobQuotaIsExceeded() throws Exception {
String assetResponse = mockMvc.perform(post("/v1/video-assets")
.header("X-Tenant-Id", "tenant-a")
.contentType(MediaType.APPLICATION_JSON)
.content("""
{
"fileName": "quota.mp4",
"contentType": "video/mp4",
"contentLengthBytes": 4096,
"checksumSha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
}
"""))
.andExpect(status().isCreated())
.andReturn()
.getResponse()
.getContentAsString();
String assetId = JsonTestHelper.read(assetResponse, "$.assetId");
String uploadId = JsonTestHelper.read(assetResponse, "$.upload.uploadId");
mockMvc.perform(post("/v1/video-assets/{assetId}/uploads:complete", assetId)
.header("X-Tenant-Id", "tenant-a")
.contentType(MediaType.APPLICATION_JSON)
.content("""
{
"uploadId": "%s",
"sourceDurationSeconds": 17,
"parts": [
{
"partNumber": 1,
"etag": "\\"etag-1\\""
}
]
}
""".formatted(uploadId)))
.andExpect(status().isOk());
mockMvc.perform(post("/v1/video-assets/{assetId}/clip-jobs", assetId)
.header("X-Tenant-Id", "tenant-a")
.contentType(MediaType.APPLICATION_JSON)
.content("""
{
"segmentDurationSeconds": 8,
"accuracyMode": "FAST",
"outputContainer": "mp4",
"overwriteExisting": false
}
"""))
.andExpect(status().isTooManyRequests())
.andExpect(jsonPath("$.type").value("https://api.example.com/problems/quota-exceeded"));
}
}

23
src/test/java/org/example/videoclips/folder/FolderFfmpegClipperTest.java
View File

@ -20,7 +20,7 @@ class FolderFfmpegClipperTest {
Path tempDir;
@Test
void buildsExactEightSecondMp4CommandAndCountsClips() throws Exception {
void buildsQualityPreservingMp4CommandAndCountsClips() throws Exception {
AtomicReference<List<String>> command = new AtomicReference<>();
FolderFfmpegClipper clipper = clipper(arguments -> {
command.set(arguments);
@ -36,11 +36,28 @@ class FolderFfmpegClipperTest {
assertEquals(tempDir.resolve("output/movie"), result.outputDirectory());
assertEquals(2, result.clipCount());
assertCommandPair(command.get(), "-c", "copy");
assertCommandPair(command.get(), "-segment_time", "8");
assertTrue(command.get().getLast().endsWith("clip_%05d.mp4"));
}
@Test
void buildsTranscodingCommandWhenPreserveInputQualityIsDisabled() throws Exception {
AtomicReference<List<String>> command = new AtomicReference<>();
VideoClippingProperties properties = properties();
properties.getFolderScheduler().setPreserveInputQuality(false);
FolderFfmpegClipper clipper = new FolderFfmpegClipper(properties, arguments -> {
command.set(arguments);
Path outputDirectory = Path.of(arguments.getLast()).getParent();
Files.writeString(outputDirectory.resolve("clip_00000.mp4"), "clip");
return new FolderFfmpegClipper.ProcessResult(0, "ok");
});
clipper.clip(tempDir.resolve("source/movie.mp4"));
assertCommandPair(command.get(), "-c:v", "libx264");
assertCommandPair(command.get(), "-c:a", "aac");
assertCommandPair(command.get(), "-segment_time", "8");
assertCommandPair(command.get(), "-force_key_frames", "expr:gte(t,n_forced*8)");
assertTrue(command.get().getLast().endsWith("clip_%05d.mp4"));
}
@Test

17
src/test/java/org/example/videoclips/folder/FolderSchedulerLocalProfileTest.java
View File

@ -2,7 +2,7 @@ package org.example.videoclips.folder;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.support.PropertiesLoaderUtils;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import java.util.Properties;
@ -12,8 +12,7 @@ class FolderSchedulerLocalProfileTest {
@Test
void enablesSchedulerWithRepositoryLocalEightSecondPaths() throws Exception {
Properties properties = PropertiesLoaderUtils.loadProperties(
new ClassPathResource("application-folder-scheduler-local.properties"));
Properties properties = yamlProperties("application-folder-scheduler-local.yml");
assertEquals("true", properties.getProperty("video-clipping.folder-scheduler.enabled"));
assertEquals("./input/source", properties.getProperty("video-clipping.folder-scheduler.input-directory"));
@ -23,6 +22,16 @@ class FolderSchedulerLocalProfileTest {
assertEquals("./input/rejected", properties.getProperty("video-clipping.folder-scheduler.rejected-directory"));
assertEquals("8", properties.getProperty("video-clipping.folder-scheduler.segment-duration-seconds"));
assertEquals("mp4", properties.getProperty("video-clipping.folder-scheduler.output-container"));
assertEquals("DEBUG", properties.getProperty("logging.level.org.example.videoclips.folder"));
assertEquals("INFO", properties.getProperty("logging.level.org.example.videoclips.folder"));
}
private Properties yamlProperties(String classpathResource) {
YamlPropertiesFactoryBean factoryBean = new YamlPropertiesFactoryBean();
factoryBean.setResources(new ClassPathResource(classpathResource));
Properties properties = factoryBean.getObject();
if (properties == null) {
throw new IllegalStateException("Unable to load " + classpathResource);
}
return properties;
}
}

24
src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java
View File

@ -88,6 +88,20 @@ class FolderVideoScanSchedulerTest {
assertTrue(((AtomicBoolean) scanningField.get(scheduler)).get());
}
@Test
void logsSchedulerStartupConfiguration() {
scheduler(tempDir).logStartup();
}
@Test
void logsTranscodingQualityModeWhenConfigured() throws Exception {
Files.writeString(tempDir.resolve("1.mp4"), "video");
FolderVideoScanScheduler scheduler = scheduler(tempDir, acceptedValidator(), successfulClipper(), false);
scheduler.logStartup();
scheduler.scan();
}
@Test
void scanReleasesGuardAfterSelectingCandidate() throws Exception {
Files.writeString(tempDir.resolve("1.mp4"), "video");
@ -257,9 +271,19 @@ class FolderVideoScanSchedulerTest {
Path inputDirectory,
FolderVideoValidator validator,
FolderFfmpegClipper clipper
) {
return scheduler(inputDirectory, validator, clipper, true);
}
private FolderVideoScanScheduler scheduler(
Path inputDirectory,
FolderVideoValidator validator,
FolderFfmpegClipper clipper,
boolean preserveInputQuality
) {
VideoClippingProperties properties = new VideoClippingProperties();
properties.getFolderScheduler().setInputDirectory(inputDirectory.toString());
properties.getFolderScheduler().setPreserveInputQuality(preserveInputQuality);
Path workingDirectory = tempDir.resolve("working");
if (!Files.exists(workingDirectory)) {
try {

1
src/test/resources/application-test.properties Normal file
View File

@ -0,0 +1 @@
video-clipping.folder-scheduler.enabled=false