# Input Folder Scheduler Plan ## Goal Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder. Requested runtime paths: - input: `/input/source` - output: `/output/clips` Repository-local placeholder directories: - `input/source` - `output/clips` Note: the local machine's filesystem root is read-only in this workspace, so absolute `/input/source` and `/output/clips` could not be created here. In production, mount or create those absolute paths and configure the service to use them. ## Milestones 1. [x] Create repository-local placeholder directories `input/source` and `output/clips`. 2. [x] Add folder scheduler configuration properties. 3. [x] Create startup directory initialization for input, output, working, processed, and rejected folders. 4. [x] Add deterministic scanner that picks one candidate video at a time. 5. [x] Add repeat-prevention by moving selected inputs to a working folder before processing. 6. [x] Add valid-video detection using extension filtering and `ffprobe`. 7. [x] Add invalid-file handling that logs the reason and moves files to rejected storage. 8. [x] Add FFmpeg clipping into `/output/clips//` using 8-second MP4/H.264/AAC segments. 9. [ ] Add success handling that moves processed source files to processed storage. 10. [ ] Add failure handling for FFmpeg or filesystem errors. 11. [ ] Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions. 12. [ ] Add optional FFmpeg integration test for a short fixture. 13. [ ] Add local profile or documented runtime config for enabling the scheduler. 14. [ ] Run verification and mark feature complete. ## Format Recommendation Use `.mp4` clips containing: - video: H.264 / AVC via `libx264` - audio: AAC via `aac` - MIME type: `video/mp4` Rationale: - MDN documents MP4 as a popular, broadly supported container. - MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4. - MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser. - FFmpeg's segment muxer supports `segment_time`; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode. Sources: - - - ## Configuration Add a new scheduler config group under `video-clipping.folder-scheduler`. Recommended properties: ```properties video-clipping.folder-scheduler.enabled=true video-clipping.folder-scheduler.input-directory=/input/source video-clipping.folder-scheduler.output-directory=/output/clips video-clipping.folder-scheduler.processed-directory=/input/processed video-clipping.folder-scheduler.rejected-directory=/input/rejected video-clipping.folder-scheduler.working-directory=/input/working video-clipping.folder-scheduler.poll-interval-ms=5000 video-clipping.folder-scheduler.segment-duration-seconds=8 video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg video-clipping.folder-scheduler.ffprobe-binary=ffprobe video-clipping.folder-scheduler.output-container=mp4 video-clipping.folder-scheduler.exact-preset=veryfast ``` Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow. ## Processing Rules The scheduler should: 1. Create all configured directories on startup if they do not exist. 2. Scan `input-directory` on a fixed delay. 3. Ignore directories, hidden files, temporary files, and files currently being written. 4. Pick only one candidate at a time. 5. Process candidates in deterministic order, preferably filename natural order so `1.mp4` is processed before `2.mp4`. 6. Validate a candidate with `ffprobe`, not only by file extension. 7. Move invalid files to `rejected-directory` and log why they were rejected. 8. Move a selected valid input to `working-directory` before processing, so the same file is not picked again while processing. 9. Write clips to `output-directory//`. 10. Move the source file to `processed-directory` only after all clips are successfully created. 11. If processing fails, move the source file to `rejected-directory` or leave it in `working-directory` with a failure marker, depending on operator preference. Use the source filename stem for the output directory: ```text /input/source/1.mp4 -> /output/clips/1/clip_00000.mp4 /input/source/1.mp4 -> /output/clips/1/clip_00001.mp4 /input/source/1.mp4 -> /output/clips/1/clip_00002.mp4 ``` If two input files share the same stem, append a deterministic numeric suffix: ```text /output/clips/1 /output/clips/1-1 ``` ## Valid Video Detection Use a two-stage validation: 1. Extension allowlist for cheap filtering: - `.mp4` - `.mov` - `.m4v` - `.webm` - `.mkv` 2. `ffprobe` verification for real validation. Suggested `ffprobe` command: ```bash ffprobe -v error \ -select_streams v:0 \ -show_entries stream=codec_type,codec_name,width,height \ -show_entries format=duration \ -of json \ /input/working/1.mp4 ``` Accept the file only if: - `ffprobe` exits with code `0` - at least one video stream exists - duration is present and greater than `0` - width and height are positive Log invalid files with: - source path - reason - ffprobe exit code - sanitized ffprobe output Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details. ## FFmpeg Command Default to exact 8-second MP4/H.264/AAC output: ```bash ffmpeg -hide_banner -y \ -i /input/working/1.mp4 \ -map 0 \ -c:v libx264 \ -preset veryfast \ -crf 20 \ -force_key_frames "expr:gte(t,n_forced*8)" \ -c:a aac \ -b:a 128k \ -f segment \ -segment_time 8 \ -segment_format_options movflags=+faststart \ -reset_timestamps 1 \ /output/clips/1/clip_%05d.mp4 ``` Expected behavior: - each generated clip is approximately 8 seconds - the final clip may be shorter than 8 seconds - output files are broadly playable MP4 files Implementation note: - If exact boundaries are more important than speed, always transcode with `libx264` and force keyframes. - If speed is more important and keyframe-aligned cuts are acceptable, add an optional `FAST` mode using `-c copy`, but document that clips may not start exactly at 8-second boundaries. ## Proposed Code Structure Add: - `src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java` - `src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java` - `src/main/java/org/example/videoclips/folder/FolderVideoValidator.java` - `src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java` - `src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java` Responsibilities: - `FolderSchedulerProperties`: typed config for directories, binaries, poll interval, segment length, and output format. - `FolderVideoScanScheduler`: scheduled scanner, single-file lock, ordering, state moves. - `FolderVideoValidator`: extension filtering and `ffprobe` validation. - `FolderFfmpegClipper`: FFmpeg command construction and process execution. - Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction. ## Concurrency Control Use an in-process guard to prevent overlapping scheduled runs: - `AtomicBoolean processing` - or `ReentrantLock tryLock()` Only one file should process at a time. This directly satisfies: ```text 1.mp4 completes before 2.mp4 starts ``` If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance. ## File State Flow Recommended state transitions: ```text /input/source/1.mp4 -> /input/working/1.mp4 -> /output/clips/1/clip_00000.mp4 -> /output/clips/1/clip_00001.mp4 -> /input/processed/1.mp4 ``` Invalid file: ```text /input/source/not-video.txt -> /input/rejected/not-video.txt ``` Failed processing: ```text /input/working/broken.mp4 -> /input/rejected/broken.mp4 ``` This ensures the scheduler never processes the same source file repeatedly. ## Operational Logging Log these events: - scheduler started and configured directories - no candidate found at debug level - selected candidate - invalid candidate rejected - FFmpeg command started, without sensitive metadata - clip generation succeeded with clip count - source moved to processed directory - processing failed with sanitized error ## Testing Plan Unit tests: - creates configured folders - sorts `1.mp4` before `2.mp4` - ignores hidden and temporary files - rejects invalid extension before `ffprobe` - rejects `ffprobe` failure - moves valid input to working before processing - moves successful source to processed - does not pick the same input twice Integration tests: - run FFmpeg against a short generated fixture if `ffmpeg` and `ffprobe` are installed - verify a 17-second input produces 3 clips - verify output path is `output/clips//clip_%05d.mp4` ## Rollout Plan 1. Add config properties and disabled-by-default scheduler. 2. Add directory creation and scanning behavior. 3. Add validation and rejection flow. 4. Add FFmpeg clipping flow. 5. Add unit tests for file-state transitions. 6. Add optional integration test gated on local FFmpeg availability. 7. Enable in a local profile first. 8. Enable in production only with mounted `/input` and `/output` volumes. ## Acceptance Criteria The feature is complete when: - `/input/source` and `/output/clips` are configurable and created on startup - valid video files are processed one at a time - `1.mp4` is processed before `2.mp4` under deterministic ordering - clips are written to `/output/clips//` - each clip is MP4/H.264/AAC by default - invalid files are ignored for processing, moved to rejected storage, and logged - processed source files are not picked up again - tests cover ordering, invalid file handling, and repeat-prevention behavior