JSL_Digital_Solution
/
video_editing_poc
forked from jsl/video_editing_poc
2
0
Fork 0
Code Pull Requests Activity
video_editing_poc/docs/input-folder-scheduler-plan.md

12 KiB
Raw Blame History

Input Folder Scheduler Plan

Goal

Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder.

Requested runtime paths:

  • input: /input/source
  • output: /output/clips

Repository-local placeholder directories:

  • input/source
  • output/clips

Note: the local machine's filesystem root is read-only in this workspace, so absolute /input/source and /output/clips could not be created here. In production, mount or create those absolute paths and configure the service to use them.

Milestones

  1. Create repository-local placeholder directories input/source and output/clips.
  2. Add folder scheduler configuration properties.
  3. Create startup directory initialization for input, output, working, processed, and rejected folders.
  4. Add deterministic scanner that picks one candidate video at a time.
  5. Add repeat-prevention by moving selected inputs to a working folder before processing.
  6. Add valid-video detection using extension filtering and ffprobe.
  7. Add invalid-file handling that logs the reason and moves files to rejected storage.
  8. Add FFmpeg clipping into /output/clips/<video-name>/ using 8-second MP4/H.264/AAC segments.
  9. Add success handling that moves processed source files to processed storage.
  10. Add failure handling for FFmpeg or filesystem errors.
  11. Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions.
  12. Add optional FFmpeg integration test for a short fixture.
  13. Add local profile or documented runtime config for enabling the scheduler.
  14. Run verification and mark feature complete.

Format Recommendation

Use .mp4 clips containing:

  • video: H.264 / AVC via libx264
  • audio: AAC via aac
  • MIME type: video/mp4

Rationale:

  • MDN documents MP4 as a popular, broadly supported container.
  • MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4.
  • MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser.
  • FFmpeg's segment muxer supports segment_time; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode.

Sources:

Configuration

Add a new scheduler config group under video-clipping.folder-scheduler.

Recommended YAML:

video-clipping:
  folder-scheduler:
    enabled: true
    input-directory: /input/source
    output-directory: /output/clips
    processed-directory: /input/processed
    rejected-directory: /input/rejected
    working-directory: /input/working
    poll-interval-ms: 5000
    segment-duration-seconds: 8
    ffmpeg-binary: ffmpeg
    ffprobe-binary: ffprobe
    output-container: mp4
    exact-preset: veryfast
    preserve-input-quality: true

Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow.

Running Locally

Install ffmpeg and ffprobe, then start the service with the repository-local scheduler profile:

mvn spring-boot:run -Dspring-boot.run.profiles=folder-scheduler-local

Place source videos in input/source. The service creates input/working, input/processed, and input/rejected at startup and writes clips to output/clips/<video-name>.

Producers must copy files with a temporary .part, .tmp, or .download suffix and atomically rename them to the final video filename only after the write completes. The scheduler ignores these temporary suffixes, preventing partially written videos from being rejected by ffprobe.

For production, keep the default absolute paths and mount persistent writable volumes at /input and /output. Run only one scheduler replica unless a cross-process lease is added.

Completion Status

Completed on 2026-07-10.

  • mvn -q clean verify: passed with 57 tests, 0 failures, 0 errors, and 0 skipped tests.
  • Real FFmpeg integration: a generated 17-second H.264/AAC source produced 3 MP4 clips.
  • JaCoCo scheduler package coverage: 1,009/1,009 instructions, 52/52 branches, and 205/205 lines covered (100%).
  • Coverage enforcement and the HTML/XML/CSV report run automatically during Maven verify.
  • Final review fixed conditional Spring component creation and explicit constructor injection; the disabled default profile and enabled scheduler pipeline both have context tests.

Processing Rules

The scheduler should:

  1. Create all configured directories on startup if they do not exist.
  2. Scan input-directory on a fixed delay.
  3. Ignore directories, hidden files, temporary files, and files currently being written.
  4. Pick only one candidate at a time.
  5. Process candidates in deterministic order, preferably filename natural order so 1.mp4 is processed before 2.mp4.
  6. Validate a candidate with ffprobe, not only by file extension.
  7. Move invalid files to rejected-directory and log why they were rejected.
  8. Move a selected valid input to working-directory before processing, so the same file is not picked again while processing.
  9. Write clips to output-directory/<video-name>/.
  10. Move the source file to processed-directory only after all clips are successfully created.
  11. If processing fails, move the source file to rejected-directory or leave it in working-directory with a failure marker, depending on operator preference.

Use the source filename stem for the output directory:

/input/source/1.mp4 -> /output/clips/1/clip_00000.mp4
/input/source/1.mp4 -> /output/clips/1/clip_00001.mp4
/input/source/1.mp4 -> /output/clips/1/clip_00002.mp4

If two input files share the same stem, append a deterministic numeric suffix:

/output/clips/1
/output/clips/1-1

Valid Video Detection

Use a two-stage validation:

  1. Extension allowlist for cheap filtering:
    • .mp4
    • .mov
    • .m4v
    • .webm
    • .mkv
  2. ffprobe verification for real validation.

Suggested ffprobe command:

ffprobe -v error \
  -select_streams v:0 \
  -show_entries stream=codec_type,codec_name,width,height \
  -show_entries format=duration \
  -of json \
  /input/working/1.mp4

Accept the file only if:

  • ffprobe exits with code 0
  • at least one video stream exists
  • duration is present and greater than 0
  • width and height are positive

Log invalid files with:

  • source path
  • reason
  • ffprobe exit code
  • sanitized ffprobe output

Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details.

FFmpeg Command

Default to quality-preserving MP4 stream copy when the source already has usable keyframes:

ffmpeg -hide_banner -y \
  -i /input/working/1.mp4 \
  -map 0 -c copy \
  -f segment \
  -segment_time 8 \
  -reset_timestamps 1 \
  /output/clips/1/clip_%05d.mp4

Expected behavior:

  • each generated clip preserves the input codec quality because no re-encode occurs
  • clips are cut on keyframes, so boundaries may drift from exact 8-second marks
  • the final clip may be shorter than 8 seconds
  • output files are broadly playable MP4 files

Implementation note:

  • If exact 8-second boundaries are more important than preserving source quality, transcode with libx264 and force keyframes.
  • If preserving source quality is more important, use stream copy and accept keyframe-aligned cuts.

Proposed Code Structure

Add:

  • src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java
  • src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java
  • src/main/java/org/example/videoclips/folder/FolderVideoValidator.java
  • src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java
  • src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java

Responsibilities:

  • FolderSchedulerProperties: typed config for directories, binaries, poll interval, segment length, and output format.
  • FolderVideoScanScheduler: scheduled scanner, single-file lock, ordering, state moves.
  • FolderVideoValidator: extension filtering and ffprobe validation.
  • FolderFfmpegClipper: FFmpeg command construction and process execution.
  • Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction.

Concurrency Control

Use an in-process guard to prevent overlapping scheduled runs:

  • AtomicBoolean processing
  • or ReentrantLock tryLock()

Only one file should process at a time. This directly satisfies:

1.mp4 completes before 2.mp4 starts

If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance.

File State Flow

Recommended state transitions:

/input/source/1.mp4
  -> /input/working/1.mp4
  -> /output/clips/1/clip_00000.mp4
  -> /output/clips/1/clip_00001.mp4
  -> /input/processed/1.mp4

Invalid file:

/input/source/not-video.txt
  -> /input/rejected/not-video.txt

Failed processing:

/input/working/broken.mp4
  -> /input/rejected/broken.mp4

This ensures the scheduler never processes the same source file repeatedly.

Operational Logging

The scheduler uses an initial delay and fixed delay of 5000 ms by default. The initial delay lets startup directory initialization finish; subsequent delays start after the previous scan finishes, so scans never intentionally overlap. Configure both with video-clipping.folder-scheduler.poll-interval-ms (minimum 1000 ms).

The base configuration enables repository-local folder processing for normal IDE starts. Production deployments can override it without changing the artifact:

FOLDER_SCHEDULER_ENABLED=true
FOLDER_SCHEDULER_INPUT_DIRECTORY=/input/source
FOLDER_SCHEDULER_OUTPUT_DIRECTORY=/output/clips
FOLDER_SCHEDULER_WORKING_DIRECTORY=/input/working
FOLDER_SCHEDULER_PROCESSED_DIRECTORY=/input/processed
FOLDER_SCHEDULER_REJECTED_DIRECTORY=/input/rejected
FOLDER_SCHEDULER_POLL_INTERVAL_MS=5000
FOLDER_SCHEDULER_LOG_LEVEL=INFO

Lifecycle logs use stable event=... fields and a per-run scan_id for correlation. Active work and state transitions log at info or warn/error level; scan start, idle, completion, directory readiness, and probe timing log at debug level.

Log these events:

  • scheduler started and configured directories
  • no candidate found at debug level
  • selected candidate
  • invalid candidate rejected
  • FFmpeg command started, without sensitive metadata
  • clip generation succeeded with clip count
  • source moved to processed directory
  • processing failed with sanitized error

Testing Plan

Unit tests:

  • creates configured folders
  • sorts 1.mp4 before 2.mp4
  • ignores hidden and temporary files
  • rejects invalid extension before ffprobe
  • rejects ffprobe failure
  • moves valid input to working before processing
  • moves successful source to processed
  • does not pick the same input twice

Integration tests:

  • run FFmpeg against a short generated fixture if ffmpeg and ffprobe are installed
  • verify a 17-second input produces 3 clips
  • verify output path is output/clips/<video-name>/clip_%05d.mp4

Rollout Plan

  1. Add config properties and disabled-by-default scheduler.
  2. Add directory creation and scanning behavior.
  3. Add validation and rejection flow.
  4. Add FFmpeg clipping flow.
  5. Add unit tests for file-state transitions.
  6. Add optional integration test gated on local FFmpeg availability.
  7. Enable in a local profile first.
  8. Enable in production only with mounted /input and /output volumes.

Acceptance Criteria

The feature is complete when:

  • /input/source and /output/clips are configurable and created on startup
  • valid video files are processed one at a time
  • 1.mp4 is processed before 2.mp4 under deterministic ordering
  • clips are written to /output/clips/<video-name>/
  • each clip is MP4/H.264/AAC by default
  • invalid files are ignored for processing, moved to rejected storage, and logged
  • processed source files are not picked up again
  • tests cover ordering, invalid file handling, and repeat-prevention behavior