From 1cda571c302c5fd0c20860c54694edb8ff668022 Mon Sep 17 00:00:00 2001 From: JSLMPR Date: Fri, 10 Jul 2026 00:49:29 +0200 Subject: [PATCH] =?UTF-8?q?=E2=80=A2=20clip=20video=20locally=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/input-folder-scheduler-plan.md | 297 ++++++++++++++++++ ...eo-clipping-service-implementation-plan.md | 10 + 2 files changed, 307 insertions(+) create mode 100644 docs/input-folder-scheduler-plan.md diff --git a/docs/input-folder-scheduler-plan.md b/docs/input-folder-scheduler-plan.md new file mode 100644 index 0000000..f188c07 --- /dev/null +++ b/docs/input-folder-scheduler-plan.md @@ -0,0 +1,297 @@ +# Input Folder Scheduler Plan + +## Goal + +Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder. + +Requested runtime paths: + +- input: `/input/source` +- output: `/output/clips` + +Repository-local placeholder directories: + +- `input/source` +- `output/clips` + +Note: the local machine's filesystem root is read-only in this workspace, so absolute `/input/source` and `/output/clips` could not be created here. In production, mount or create those absolute paths and configure the service to use them. + +## Milestones + +1. [x] Create repository-local placeholder directories `input/source` and `output/clips`. +2. [ ] Add folder scheduler configuration properties. +3. [ ] Create startup directory initialization for input, output, working, processed, and rejected folders. +4. [ ] Add deterministic scanner that picks one candidate video at a time. +5. [ ] Add repeat-prevention by moving selected inputs to a working folder before processing. +6. [ ] Add valid-video detection using extension filtering and `ffprobe`. +7. [ ] Add invalid-file handling that logs the reason and moves files to rejected storage. +8. [ ] Add FFmpeg clipping into `/output/clips//` using 8-second MP4/H.264/AAC segments. +9. [ ] Add success handling that moves processed source files to processed storage. +10. [ ] Add failure handling for FFmpeg or filesystem errors. +11. [ ] Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions. +12. [ ] Add optional FFmpeg integration test for a short fixture. +13. [ ] Add local profile or documented runtime config for enabling the scheduler. +14. [ ] Run verification and mark feature complete. + +## Format Recommendation + +Use `.mp4` clips containing: + +- video: H.264 / AVC via `libx264` +- audio: AAC via `aac` +- MIME type: `video/mp4` + +Rationale: + +- MDN documents MP4 as a popular, broadly supported container. +- MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4. +- MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser. +- FFmpeg's segment muxer supports `segment_time`; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode. + +Sources: + +- +- +- + +## Configuration + +Add a new scheduler config group under `video-clipping.folder-scheduler`. + +Recommended properties: + +```properties +video-clipping.folder-scheduler.enabled=true +video-clipping.folder-scheduler.input-directory=/input/source +video-clipping.folder-scheduler.output-directory=/output/clips +video-clipping.folder-scheduler.processed-directory=/input/processed +video-clipping.folder-scheduler.rejected-directory=/input/rejected +video-clipping.folder-scheduler.working-directory=/input/working +video-clipping.folder-scheduler.poll-interval-ms=5000 +video-clipping.folder-scheduler.segment-duration-seconds=8 +video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg +video-clipping.folder-scheduler.ffprobe-binary=ffprobe +video-clipping.folder-scheduler.output-container=mp4 +video-clipping.folder-scheduler.exact-preset=veryfast +``` + +Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow. + +## Processing Rules + +The scheduler should: + +1. Create all configured directories on startup if they do not exist. +2. Scan `input-directory` on a fixed delay. +3. Ignore directories, hidden files, temporary files, and files currently being written. +4. Pick only one candidate at a time. +5. Process candidates in deterministic order, preferably filename natural order so `1.mp4` is processed before `2.mp4`. +6. Validate a candidate with `ffprobe`, not only by file extension. +7. Move invalid files to `rejected-directory` and log why they were rejected. +8. Move a selected valid input to `working-directory` before processing, so the same file is not picked again while processing. +9. Write clips to `output-directory//`. +10. Move the source file to `processed-directory` only after all clips are successfully created. +11. If processing fails, move the source file to `rejected-directory` or leave it in `working-directory` with a failure marker, depending on operator preference. + +Use the source filename stem for the output directory: + +```text +/input/source/1.mp4 -> /output/clips/1/clip_00000.mp4 +/input/source/1.mp4 -> /output/clips/1/clip_00001.mp4 +/input/source/1.mp4 -> /output/clips/1/clip_00002.mp4 +``` + +If two input files share the same stem, append a stable suffix: + +```text +/output/clips/1 +/output/clips/1-20260710T141500Z +``` + +## Valid Video Detection + +Use a two-stage validation: + +1. Extension allowlist for cheap filtering: + - `.mp4` + - `.mov` + - `.m4v` + - `.webm` + - `.mkv` +2. `ffprobe` verification for real validation. + +Suggested `ffprobe` command: + +```bash +ffprobe -v error \ + -select_streams v:0 \ + -show_entries stream=codec_type,codec_name,width,height \ + -show_entries format=duration \ + -of json \ + /input/working/1.mp4 +``` + +Accept the file only if: + +- `ffprobe` exits with code `0` +- at least one video stream exists +- duration is present and greater than `0` +- width and height are positive + +Log invalid files with: + +- source path +- reason +- ffprobe exit code +- sanitized ffprobe output + +Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details. + +## FFmpeg Command + +Default to exact 8-second MP4/H.264/AAC output: + +```bash +ffmpeg -hide_banner -y \ + -i /input/working/1.mp4 \ + -map 0 \ + -c:v libx264 \ + -preset veryfast \ + -crf 20 \ + -force_key_frames "expr:gte(t,n_forced*8)" \ + -c:a aac \ + -b:a 128k \ + -f segment \ + -segment_time 8 \ + -segment_format_options movflags=+faststart \ + -reset_timestamps 1 \ + /output/clips/1/clip_%05d.mp4 +``` + +Expected behavior: + +- each generated clip is approximately 8 seconds +- the final clip may be shorter than 8 seconds +- output files are broadly playable MP4 files + +Implementation note: + +- If exact boundaries are more important than speed, always transcode with `libx264` and force keyframes. +- If speed is more important and keyframe-aligned cuts are acceptable, add an optional `FAST` mode using `-c copy`, but document that clips may not start exactly at 8-second boundaries. + +## Proposed Code Structure + +Add: + +- `src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java` +- `src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java` +- `src/main/java/org/example/videoclips/folder/FolderVideoValidator.java` +- `src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java` +- `src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java` + +Responsibilities: + +- `FolderSchedulerProperties`: typed config for directories, binaries, poll interval, segment length, and output format. +- `FolderVideoScanScheduler`: scheduled scanner, single-file lock, ordering, state moves. +- `FolderVideoValidator`: extension filtering and `ffprobe` validation. +- `FolderFfmpegClipper`: FFmpeg command construction and process execution. +- Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction. + +## Concurrency Control + +Use an in-process guard to prevent overlapping scheduled runs: + +- `AtomicBoolean processing` +- or `ReentrantLock tryLock()` + +Only one file should process at a time. This directly satisfies: + +```text +1.mp4 completes before 2.mp4 starts +``` + +If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance. + +## File State Flow + +Recommended state transitions: + +```text +/input/source/1.mp4 + -> /input/working/1.mp4 + -> /output/clips/1/clip_00000.mp4 + -> /output/clips/1/clip_00001.mp4 + -> /input/processed/1.mp4 +``` + +Invalid file: + +```text +/input/source/not-video.txt + -> /input/rejected/not-video.txt +``` + +Failed processing: + +```text +/input/working/broken.mp4 + -> /input/rejected/broken.mp4 +``` + +This ensures the scheduler never processes the same source file repeatedly. + +## Operational Logging + +Log these events: + +- scheduler started and configured directories +- no candidate found at debug level +- selected candidate +- invalid candidate rejected +- FFmpeg command started, without sensitive metadata +- clip generation succeeded with clip count +- source moved to processed directory +- processing failed with sanitized error + +## Testing Plan + +Unit tests: + +- creates configured folders +- sorts `1.mp4` before `2.mp4` +- ignores hidden and temporary files +- rejects invalid extension before `ffprobe` +- rejects `ffprobe` failure +- moves valid input to working before processing +- moves successful source to processed +- does not pick the same input twice + +Integration tests: + +- run FFmpeg against a short generated fixture if `ffmpeg` and `ffprobe` are installed +- verify a 17-second input produces 3 clips +- verify output path is `output/clips//clip_%05d.mp4` + +## Rollout Plan + +1. Add config properties and disabled-by-default scheduler. +2. Add directory creation and scanning behavior. +3. Add validation and rejection flow. +4. Add FFmpeg clipping flow. +5. Add unit tests for file-state transitions. +6. Add optional integration test gated on local FFmpeg availability. +7. Enable in a local profile first. +8. Enable in production only with mounted `/input` and `/output` volumes. + +## Acceptance Criteria + +The feature is complete when: + +- `/input/source` and `/output/clips` are configurable and created on startup +- valid video files are processed one at a time +- `1.mp4` is processed before `2.mp4` under deterministic ordering +- clips are written to `/output/clips//` +- each clip is MP4/H.264/AAC by default +- invalid files are ignored for processing, moved to rejected storage, and logged +- processed source files are not picked up again +- tests cover ordering, invalid file handling, and repeat-prevention behavior diff --git a/docs/video-clipping-service-implementation-plan.md b/docs/video-clipping-service-implementation-plan.md index d6ca35a..5303a8d 100644 --- a/docs/video-clipping-service-implementation-plan.md +++ b/docs/video-clipping-service-implementation-plan.md @@ -726,6 +726,16 @@ Keep the domain independent from Spring framework details. Adapters implement st 5. [x] DLQ redrive procedure. 6. [x] Load-test signoff. +### Plan Status + +All repository-scoped implementation-plan artifacts are now complete. + +Remaining work is execution work outside this repository, primarily: + +- deploy the documented alerts and dashboard into the target monitoring stack +- run a real backup-and-restore drill against infrastructure snapshots +- run a real production-like end-to-end load test and record the signoff outcome + ## 14. Open Questions - Must clips be exactly 8 seconds, or is keyframe-aligned clipping acceptable?