# Input Folder Scheduler Plan ## Goal Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder. Requested runtime paths: - input: `/input/source` - output: `/output/clips` Repository-local placeholder directories: - `input/source` - `output/clips` Note: the local machine's filesystem root is read-only in this workspace, so absolute `/input/source` and `/output/clips` could not be created here. In production, mount or create those absolute paths and configure the service to use them. ## Milestones 1. [x] Create repository-local placeholder directories `input/source` and `output/clips`. 2. [x] Add folder scheduler configuration properties. 3. [x] Create startup directory initialization for input, output, working, processed, and rejected folders. 4. [x] Add deterministic scanner that picks one candidate video at a time. 5. [x] Add repeat-prevention by moving selected inputs to a working folder before processing. 6. [x] Add valid-video detection using extension filtering and `ffprobe`. 7. [x] Add invalid-file handling that logs the reason and moves files to rejected storage. 8. [x] Add FFmpeg clipping into `/output/clips//` using 8-second MP4/H.264/AAC segments. 9. [x] Add success handling that moves processed source files to processed storage. 10. [x] Add failure handling for FFmpeg or filesystem errors. 11. [x] Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions. 12. [x] Add optional FFmpeg integration test for a short fixture. 13. [x] Add local profile or documented runtime config for enabling the scheduler. 14. [x] Run verification and mark feature complete. ## Format Recommendation Use `.mp4` clips containing: - video: H.264 / AVC via `libx264` - audio: AAC via `aac` - MIME type: `video/mp4` Rationale: - MDN documents MP4 as a popular, broadly supported container. - MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4. - MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser. - FFmpeg's segment muxer supports `segment_time`; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode. Sources: - - - ## Configuration Add a new scheduler config group under `video-clipping.folder-scheduler`. Recommended YAML: ```yaml video-clipping: folder-scheduler: enabled: true input-directory: /input/source output-directory: /output/clips processed-directory: /input/processed rejected-directory: /input/rejected working-directory: /input/working poll-interval-ms: 5000 segment-duration-seconds: 8 ffmpeg-binary: ffmpeg ffprobe-binary: ffprobe output-container: mp4 exact-preset: veryfast preserve-input-quality: true ``` Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow. ### Running Locally Install `ffmpeg` and `ffprobe`, then start the service with the repository-local scheduler profile: ```bash mvn spring-boot:run -Dspring-boot.run.profiles=folder-scheduler-local ``` Place source videos in `input/source`. The service creates `input/working`, `input/processed`, and `input/rejected` at startup and writes clips to `output/clips/`. Producers must copy files with a temporary `.part`, `.tmp`, or `.download` suffix and atomically rename them to the final video filename only after the write completes. The scheduler ignores these temporary suffixes, preventing partially written videos from being rejected by `ffprobe`. For production, keep the default absolute paths and mount persistent writable volumes at `/input` and `/output`. Run only one scheduler replica unless a cross-process lease is added. ## Completion Status Completed on 2026-07-10. - `mvn -q clean verify`: passed with 57 tests, 0 failures, 0 errors, and 0 skipped tests. - Real FFmpeg integration: a generated 17-second H.264/AAC source produced 3 MP4 clips. - JaCoCo scheduler package coverage: 1,009/1,009 instructions, 52/52 branches, and 205/205 lines covered (100%). - Coverage enforcement and the HTML/XML/CSV report run automatically during Maven `verify`. - Final review fixed conditional Spring component creation and explicit constructor injection; the disabled default profile and enabled scheduler pipeline both have context tests. ## Processing Rules The scheduler should: 1. Create all configured directories on startup if they do not exist. 2. Scan `input-directory` on a fixed delay. 3. Ignore directories, hidden files, temporary files, and files currently being written. 4. Pick only one candidate at a time. 5. Process candidates in deterministic order, preferably filename natural order so `1.mp4` is processed before `2.mp4`. 6. Validate a candidate with `ffprobe`, not only by file extension. 7. Move invalid files to `rejected-directory` and log why they were rejected. 8. Move a selected valid input to `working-directory` before processing, so the same file is not picked again while processing. 9. Write clips to `output-directory//`. 10. Move the source file to `processed-directory` only after all clips are successfully created. 11. If processing fails, move the source file to `rejected-directory` or leave it in `working-directory` with a failure marker, depending on operator preference. Use the source filename stem for the output directory: ```text /input/source/1.mp4 -> /output/clips/1/clip_00000.mp4 /input/source/1.mp4 -> /output/clips/1/clip_00001.mp4 /input/source/1.mp4 -> /output/clips/1/clip_00002.mp4 ``` If two input files share the same stem, append a deterministic numeric suffix: ```text /output/clips/1 /output/clips/1-1 ``` ## Valid Video Detection Use a two-stage validation: 1. Extension allowlist for cheap filtering: - `.mp4` - `.mov` - `.m4v` - `.webm` - `.mkv` 2. `ffprobe` verification for real validation. Suggested `ffprobe` command: ```bash ffprobe -v error \ -select_streams v:0 \ -show_entries stream=codec_type,codec_name,width,height \ -show_entries format=duration \ -of json \ /input/working/1.mp4 ``` Accept the file only if: - `ffprobe` exits with code `0` - at least one video stream exists - duration is present and greater than `0` - width and height are positive Log invalid files with: - source path - reason - ffprobe exit code - sanitized ffprobe output Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details. ## FFmpeg Command Default to quality-preserving MP4 stream copy when the source already has usable keyframes: ```bash ffmpeg -hide_banner -y \ -i /input/working/1.mp4 \ -map 0 -c copy \ -f segment \ -segment_time 8 \ -reset_timestamps 1 \ /output/clips/1/clip_%05d.mp4 ``` Expected behavior: - each generated clip preserves the input codec quality because no re-encode occurs - clips are cut on keyframes, so boundaries may drift from exact 8-second marks - the final clip may be shorter than 8 seconds - output files are broadly playable MP4 files Implementation note: - If exact 8-second boundaries are more important than preserving source quality, transcode with `libx264` and force keyframes. - If preserving source quality is more important, use stream copy and accept keyframe-aligned cuts. ## Proposed Code Structure Add: - `src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java` - `src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java` - `src/main/java/org/example/videoclips/folder/FolderVideoValidator.java` - `src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java` - `src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java` Responsibilities: - `FolderSchedulerProperties`: typed config for directories, binaries, poll interval, segment length, and output format. - `FolderVideoScanScheduler`: scheduled scanner, single-file lock, ordering, state moves. - `FolderVideoValidator`: extension filtering and `ffprobe` validation. - `FolderFfmpegClipper`: FFmpeg command construction and process execution. - Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction. ## Concurrency Control Use an in-process guard to prevent overlapping scheduled runs: - `AtomicBoolean processing` - or `ReentrantLock tryLock()` Only one file should process at a time. This directly satisfies: ```text 1.mp4 completes before 2.mp4 starts ``` If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance. ## File State Flow Recommended state transitions: ```text /input/source/1.mp4 -> /input/working/1.mp4 -> /output/clips/1/clip_00000.mp4 -> /output/clips/1/clip_00001.mp4 -> /input/processed/1.mp4 ``` Invalid file: ```text /input/source/not-video.txt -> /input/rejected/not-video.txt ``` Failed processing: ```text /input/working/broken.mp4 -> /input/rejected/broken.mp4 ``` This ensures the scheduler never processes the same source file repeatedly. ## Operational Logging The scheduler uses an initial delay and fixed delay of `5000` ms by default. The initial delay lets startup directory initialization finish; subsequent delays start after the previous scan finishes, so scans never intentionally overlap. Configure both with `video-clipping.folder-scheduler.poll-interval-ms` (minimum `1000` ms). The base configuration enables repository-local folder processing for normal IDE starts. Production deployments can override it without changing the artifact: ```bash FOLDER_SCHEDULER_ENABLED=true FOLDER_SCHEDULER_INPUT_DIRECTORY=/input/source FOLDER_SCHEDULER_OUTPUT_DIRECTORY=/output/clips FOLDER_SCHEDULER_WORKING_DIRECTORY=/input/working FOLDER_SCHEDULER_PROCESSED_DIRECTORY=/input/processed FOLDER_SCHEDULER_REJECTED_DIRECTORY=/input/rejected FOLDER_SCHEDULER_POLL_INTERVAL_MS=5000 FOLDER_SCHEDULER_LOG_LEVEL=INFO ``` Lifecycle logs use stable `event=...` fields and a per-run `scan_id` for correlation. Active work and state transitions log at info or warn/error level; scan start, idle, completion, directory readiness, and probe timing log at debug level. Log these events: - scheduler started and configured directories - no candidate found at debug level - selected candidate - invalid candidate rejected - FFmpeg command started, without sensitive metadata - clip generation succeeded with clip count - source moved to processed directory - processing failed with sanitized error ## Testing Plan Unit tests: - creates configured folders - sorts `1.mp4` before `2.mp4` - ignores hidden and temporary files - rejects invalid extension before `ffprobe` - rejects `ffprobe` failure - moves valid input to working before processing - moves successful source to processed - does not pick the same input twice Integration tests: - run FFmpeg against a short generated fixture if `ffmpeg` and `ffprobe` are installed - verify a 17-second input produces 3 clips - verify output path is `output/clips//clip_%05d.mp4` ## Rollout Plan 1. Add config properties and disabled-by-default scheduler. 2. Add directory creation and scanning behavior. 3. Add validation and rejection flow. 4. Add FFmpeg clipping flow. 5. Add unit tests for file-state transitions. 6. Add optional integration test gated on local FFmpeg availability. 7. Enable in a local profile first. 8. Enable in production only with mounted `/input` and `/output` volumes. ## Acceptance Criteria The feature is complete when: - `/input/source` and `/output/clips` are configurable and created on startup - valid video files are processed one at a time - `1.mp4` is processed before `2.mp4` under deterministic ordering - clips are written to `/output/clips//` - each clip is MP4/H.264/AAC by default - invalid files are ignored for processing, moved to rejected storage, and logged - processed source files are not picked up again - tests cover ordering, invalid file handling, and repeat-prevention behavior