• clip video locally plan
This commit is contained in:
JSLMPR
parent
06cf76e318
commit
1cda571c30
|
|
@ -0,0 +1,297 @@
|
|||
# Input Folder Scheduler Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder.
|
||||
|
||||
Requested runtime paths:
|
||||
|
||||
- input: `/input/source`
|
||||
- output: `/output/clips`
|
||||
|
||||
Repository-local placeholder directories:
|
||||
|
||||
- `input/source`
|
||||
- `output/clips`
|
||||
|
||||
Note: the local machine's filesystem root is read-only in this workspace, so absolute `/input/source` and `/output/clips` could not be created here. In production, mount or create those absolute paths and configure the service to use them.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. [x] Create repository-local placeholder directories `input/source` and `output/clips`.
|
||||
2. [ ] Add folder scheduler configuration properties.
|
||||
3. [ ] Create startup directory initialization for input, output, working, processed, and rejected folders.
|
||||
4. [ ] Add deterministic scanner that picks one candidate video at a time.
|
||||
5. [ ] Add repeat-prevention by moving selected inputs to a working folder before processing.
|
||||
6. [ ] Add valid-video detection using extension filtering and `ffprobe`.
|
||||
7. [ ] Add invalid-file handling that logs the reason and moves files to rejected storage.
|
||||
8. [ ] Add FFmpeg clipping into `/output/clips/<video-name>/` using 8-second MP4/H.264/AAC segments.
|
||||
9. [ ] Add success handling that moves processed source files to processed storage.
|
||||
10. [ ] Add failure handling for FFmpeg or filesystem errors.
|
||||
11. [ ] Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions.
|
||||
12. [ ] Add optional FFmpeg integration test for a short fixture.
|
||||
13. [ ] Add local profile or documented runtime config for enabling the scheduler.
|
||||
14. [ ] Run verification and mark feature complete.
|
||||
|
||||
## Format Recommendation
|
||||
|
||||
Use `.mp4` clips containing:
|
||||
|
||||
- video: H.264 / AVC via `libx264`
|
||||
- audio: AAC via `aac`
|
||||
- MIME type: `video/mp4`
|
||||
|
||||
Rationale:
|
||||
|
||||
- MDN documents MP4 as a popular, broadly supported container.
|
||||
- MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4.
|
||||
- MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser.
|
||||
- FFmpeg's segment muxer supports `segment_time`; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode.
|
||||
|
||||
Sources:
|
||||
|
||||
- <https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Containers>
|
||||
- <https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Video_codecs>
|
||||
- <https://ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment>
|
||||
|
||||
## Configuration
|
||||
|
||||
Add a new scheduler config group under `video-clipping.folder-scheduler`.
|
||||
|
||||
Recommended properties:
|
||||
|
||||
```properties
|
||||
video-clipping.folder-scheduler.enabled=true
|
||||
video-clipping.folder-scheduler.input-directory=/input/source
|
||||
video-clipping.folder-scheduler.output-directory=/output/clips
|
||||
video-clipping.folder-scheduler.processed-directory=/input/processed
|
||||
video-clipping.folder-scheduler.rejected-directory=/input/rejected
|
||||
video-clipping.folder-scheduler.working-directory=/input/working
|
||||
video-clipping.folder-scheduler.poll-interval-ms=5000
|
||||
video-clipping.folder-scheduler.segment-duration-seconds=8
|
||||
video-clipping.folder-scheduler.ffmpeg-binary=ffmpeg
|
||||
video-clipping.folder-scheduler.ffprobe-binary=ffprobe
|
||||
video-clipping.folder-scheduler.output-container=mp4
|
||||
video-clipping.folder-scheduler.exact-preset=veryfast
|
||||
```
|
||||
|
||||
Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow.
|
||||
|
||||
## Processing Rules
|
||||
|
||||
The scheduler should:
|
||||
|
||||
1. Create all configured directories on startup if they do not exist.
|
||||
2. Scan `input-directory` on a fixed delay.
|
||||
3. Ignore directories, hidden files, temporary files, and files currently being written.
|
||||
4. Pick only one candidate at a time.
|
||||
5. Process candidates in deterministic order, preferably filename natural order so `1.mp4` is processed before `2.mp4`.
|
||||
6. Validate a candidate with `ffprobe`, not only by file extension.
|
||||
7. Move invalid files to `rejected-directory` and log why they were rejected.
|
||||
8. Move a selected valid input to `working-directory` before processing, so the same file is not picked again while processing.
|
||||
9. Write clips to `output-directory/<video-name>/`.
|
||||
10. Move the source file to `processed-directory` only after all clips are successfully created.
|
||||
11. If processing fails, move the source file to `rejected-directory` or leave it in `working-directory` with a failure marker, depending on operator preference.
|
||||
|
||||
Use the source filename stem for the output directory:
|
||||
|
||||
```text
|
||||
/input/source/1.mp4 -> /output/clips/1/clip_00000.mp4
|
||||
/input/source/1.mp4 -> /output/clips/1/clip_00001.mp4
|
||||
/input/source/1.mp4 -> /output/clips/1/clip_00002.mp4
|
||||
```
|
||||
|
||||
If two input files share the same stem, append a stable suffix:
|
||||
|
||||
```text
|
||||
/output/clips/1
|
||||
/output/clips/1-20260710T141500Z
|
||||
```
|
||||
|
||||
## Valid Video Detection
|
||||
|
||||
Use a two-stage validation:
|
||||
|
||||
1. Extension allowlist for cheap filtering:
|
||||
- `.mp4`
|
||||
- `.mov`
|
||||
- `.m4v`
|
||||
- `.webm`
|
||||
- `.mkv`
|
||||
2. `ffprobe` verification for real validation.
|
||||
|
||||
Suggested `ffprobe` command:
|
||||
|
||||
```bash
|
||||
ffprobe -v error \
|
||||
-select_streams v:0 \
|
||||
-show_entries stream=codec_type,codec_name,width,height \
|
||||
-show_entries format=duration \
|
||||
-of json \
|
||||
/input/working/1.mp4
|
||||
```
|
||||
|
||||
Accept the file only if:
|
||||
|
||||
- `ffprobe` exits with code `0`
|
||||
- at least one video stream exists
|
||||
- duration is present and greater than `0`
|
||||
- width and height are positive
|
||||
|
||||
Log invalid files with:
|
||||
|
||||
- source path
|
||||
- reason
|
||||
- ffprobe exit code
|
||||
- sanitized ffprobe output
|
||||
|
||||
Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details.
|
||||
|
||||
## FFmpeg Command
|
||||
|
||||
Default to exact 8-second MP4/H.264/AAC output:
|
||||
|
||||
```bash
|
||||
ffmpeg -hide_banner -y \
|
||||
-i /input/working/1.mp4 \
|
||||
-map 0 \
|
||||
-c:v libx264 \
|
||||
-preset veryfast \
|
||||
-crf 20 \
|
||||
-force_key_frames "expr:gte(t,n_forced*8)" \
|
||||
-c:a aac \
|
||||
-b:a 128k \
|
||||
-f segment \
|
||||
-segment_time 8 \
|
||||
-segment_format_options movflags=+faststart \
|
||||
-reset_timestamps 1 \
|
||||
/output/clips/1/clip_%05d.mp4
|
||||
```
|
||||
|
||||
Expected behavior:
|
||||
|
||||
- each generated clip is approximately 8 seconds
|
||||
- the final clip may be shorter than 8 seconds
|
||||
- output files are broadly playable MP4 files
|
||||
|
||||
Implementation note:
|
||||
|
||||
- If exact boundaries are more important than speed, always transcode with `libx264` and force keyframes.
|
||||
- If speed is more important and keyframe-aligned cuts are acceptable, add an optional `FAST` mode using `-c copy`, but document that clips may not start exactly at 8-second boundaries.
|
||||
|
||||
## Proposed Code Structure
|
||||
|
||||
Add:
|
||||
|
||||
- `src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java`
|
||||
- `src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java`
|
||||
- `src/main/java/org/example/videoclips/folder/FolderVideoValidator.java`
|
||||
- `src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java`
|
||||
- `src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- `FolderSchedulerProperties`: typed config for directories, binaries, poll interval, segment length, and output format.
|
||||
- `FolderVideoScanScheduler`: scheduled scanner, single-file lock, ordering, state moves.
|
||||
- `FolderVideoValidator`: extension filtering and `ffprobe` validation.
|
||||
- `FolderFfmpegClipper`: FFmpeg command construction and process execution.
|
||||
- Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction.
|
||||
|
||||
## Concurrency Control
|
||||
|
||||
Use an in-process guard to prevent overlapping scheduled runs:
|
||||
|
||||
- `AtomicBoolean processing`
|
||||
- or `ReentrantLock tryLock()`
|
||||
|
||||
Only one file should process at a time. This directly satisfies:
|
||||
|
||||
```text
|
||||
1.mp4 completes before 2.mp4 starts
|
||||
```
|
||||
|
||||
If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance.
|
||||
|
||||
## File State Flow
|
||||
|
||||
Recommended state transitions:
|
||||
|
||||
```text
|
||||
/input/source/1.mp4
|
||||
-> /input/working/1.mp4
|
||||
-> /output/clips/1/clip_00000.mp4
|
||||
-> /output/clips/1/clip_00001.mp4
|
||||
-> /input/processed/1.mp4
|
||||
```
|
||||
|
||||
Invalid file:
|
||||
|
||||
```text
|
||||
/input/source/not-video.txt
|
||||
-> /input/rejected/not-video.txt
|
||||
```
|
||||
|
||||
Failed processing:
|
||||
|
||||
```text
|
||||
/input/working/broken.mp4
|
||||
-> /input/rejected/broken.mp4
|
||||
```
|
||||
|
||||
This ensures the scheduler never processes the same source file repeatedly.
|
||||
|
||||
## Operational Logging
|
||||
|
||||
Log these events:
|
||||
|
||||
- scheduler started and configured directories
|
||||
- no candidate found at debug level
|
||||
- selected candidate
|
||||
- invalid candidate rejected
|
||||
- FFmpeg command started, without sensitive metadata
|
||||
- clip generation succeeded with clip count
|
||||
- source moved to processed directory
|
||||
- processing failed with sanitized error
|
||||
|
||||
## Testing Plan
|
||||
|
||||
Unit tests:
|
||||
|
||||
- creates configured folders
|
||||
- sorts `1.mp4` before `2.mp4`
|
||||
- ignores hidden and temporary files
|
||||
- rejects invalid extension before `ffprobe`
|
||||
- rejects `ffprobe` failure
|
||||
- moves valid input to working before processing
|
||||
- moves successful source to processed
|
||||
- does not pick the same input twice
|
||||
|
||||
Integration tests:
|
||||
|
||||
- run FFmpeg against a short generated fixture if `ffmpeg` and `ffprobe` are installed
|
||||
- verify a 17-second input produces 3 clips
|
||||
- verify output path is `output/clips/<video-name>/clip_%05d.mp4`
|
||||
|
||||
## Rollout Plan
|
||||
|
||||
1. Add config properties and disabled-by-default scheduler.
|
||||
2. Add directory creation and scanning behavior.
|
||||
3. Add validation and rejection flow.
|
||||
4. Add FFmpeg clipping flow.
|
||||
5. Add unit tests for file-state transitions.
|
||||
6. Add optional integration test gated on local FFmpeg availability.
|
||||
7. Enable in a local profile first.
|
||||
8. Enable in production only with mounted `/input` and `/output` volumes.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- `/input/source` and `/output/clips` are configurable and created on startup
|
||||
- valid video files are processed one at a time
|
||||
- `1.mp4` is processed before `2.mp4` under deterministic ordering
|
||||
- clips are written to `/output/clips/<video-name>/`
|
||||
- each clip is MP4/H.264/AAC by default
|
||||
- invalid files are ignored for processing, moved to rejected storage, and logged
|
||||
- processed source files are not picked up again
|
||||
- tests cover ordering, invalid file handling, and repeat-prevention behavior
|
||||
|
|
@ -726,6 +726,16 @@ Keep the domain independent from Spring framework details. Adapters implement st
|
|||
5. [x] DLQ redrive procedure.
|
||||
6. [x] Load-test signoff.
|
||||
|
||||
### Plan Status
|
||||
|
||||
All repository-scoped implementation-plan artifacts are now complete.
|
||||
|
||||
Remaining work is execution work outside this repository, primarily:
|
||||
|
||||
- deploy the documented alerts and dashboard into the target monitoring stack
|
||||
- run a real backup-and-restore drill against infrastructure snapshots
|
||||
- run a real production-like end-to-end load test and record the signoff outcome
|
||||
|
||||
## 14. Open Questions
|
||||
|
||||
- Must clips be exactly 8 seconds, or is keyframe-aligned clipping acceptable?
|
||||
|
|
|
|||
Loading…
Reference in New Issue