video_editing_poc/docs/input-folder-scheduler-plan.md

# Input Folder Scheduler Plan

## Goal

Add a scheduler that scans an input folder for video files and creates 8-second clips into an output folder.

Requested runtime paths:

- input: `/input/source`
- output: `/output/clips`

Repository-local placeholder directories:

- `input/source`
- `output/clips`

Note: the local machine's filesystem root is read-only in this workspace, so absolute `/input/source` and `/output/clips` could not be created here. In production, mount or create those absolute paths and configure the service to use them.

## Milestones

1. [x] Create repository-local placeholder directories `input/source` and `output/clips`.
2. [x] Add folder scheduler configuration properties.
3. [x] Create startup directory initialization for input, output, working, processed, and rejected folders.
4. [x] Add deterministic scanner that picks one candidate video at a time.
5. [x] Add repeat-prevention by moving selected inputs to a working folder before processing.
6. [x] Add valid-video detection using extension filtering and `ffprobe`.
7. [x] Add invalid-file handling that logs the reason and moves files to rejected storage.
8. [x] Add FFmpeg clipping into `/output/clips/<video-name>/` using 8-second MP4/H.264/AAC segments.
9. [x] Add success handling that moves processed source files to processed storage.
10. [x] Add failure handling for FFmpeg or filesystem errors.
11. [x] Add unit tests for ordering, invalid-file handling, repeat prevention, and state transitions.
12. [x] Add optional FFmpeg integration test for a short fixture.
13. [x] Add local profile or documented runtime config for enabling the scheduler.
14. [x] Run verification and mark feature complete.

## Format Recommendation

Use `.mp4` clips containing:

- video: H.264 / AVC via `libx264`
- audio: AAC via `aac`
- MIME type: `video/mp4`

Rationale:

- MDN documents MP4 as a popular, broadly supported container.
- MDN documents MPEG-4 / MP4 as supported by all major browsers and lists H.264 and AAC support for MP4.
- MDN specifically recommends MP4 with H.264 video and AAC audio as a broadly supported combination across every major browser.
- FFmpeg's segment muxer supports `segment_time`; its documentation notes that accurate splitting needs keyframes at the split points, so exact 8-second clips should force keyframes or transcode.

Sources:

- <https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Containers>
- <https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Video_codecs>
- <https://ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment>

## Configuration

Add a new scheduler config group under `video-clipping.folder-scheduler`.

Recommended YAML:

```yaml
video-clipping:
  folder-scheduler:
    enabled: true
    input-directory: /input/source
    output-directory: /output/clips
    processed-directory: /input/processed
    rejected-directory: /input/rejected
    working-directory: /input/working
    poll-interval-ms: 5000
    segment-duration-seconds: 8
    ffmpeg-binary: ffmpeg
    ffprobe-binary: ffprobe
    output-container: mp4
    exact-preset: veryfast
    preserve-input-quality: true
```

Keep the existing API-driven clipper config separate from this folder scheduler. The folder scheduler is a direct filesystem workflow, not an API/upload/session workflow.

### Running Locally

Install `ffmpeg` and `ffprobe`, then start the service with the repository-local scheduler profile:

```bash
mvn spring-boot:run -Dspring-boot.run.profiles=folder-scheduler-local
```

Place source videos in `input/source`. The service creates `input/working`, `input/processed`, and `input/rejected` at startup and writes clips to `output/clips/<video-name>`.

Producers must copy files with a temporary `.part`, `.tmp`, or `.download` suffix and atomically rename them to the final video filename only after the write completes. The scheduler ignores these temporary suffixes, preventing partially written videos from being rejected by `ffprobe`.

For production, keep the default absolute paths and mount persistent writable volumes at `/input` and `/output`. Run only one scheduler replica unless a cross-process lease is added.

## Completion Status

Completed on 2026-07-10.

- `mvn -q clean verify`: passed with 57 tests, 0 failures, 0 errors, and 0 skipped tests.
- Real FFmpeg integration: a generated 17-second H.264/AAC source produced 3 MP4 clips.
- JaCoCo scheduler package coverage: 1,009/1,009 instructions, 52/52 branches, and 205/205 lines covered (100%).
- Coverage enforcement and the HTML/XML/CSV report run automatically during Maven `verify`.
- Final review fixed conditional Spring component creation and explicit constructor injection; the disabled default profile and enabled scheduler pipeline both have context tests.

## Processing Rules

The scheduler should:

1. Create all configured directories on startup if they do not exist.
2. Scan `input-directory` on a fixed delay.
3. Ignore directories, hidden files, temporary files, and files currently being written.
4. Pick only one candidate at a time.
5. Process candidates in deterministic order, preferably filename natural order so `1.mp4` is processed before `2.mp4`.
6. Validate a candidate with `ffprobe`, not only by file extension.
7. Move invalid files to `rejected-directory` and log why they were rejected.
8. Move a selected valid input to `working-directory` before processing, so the same file is not picked again while processing.
9. Write clips to `output-directory/<video-name>/`.
10. Move the source file to `processed-directory` only after all clips are successfully created.
11. If processing fails, move the source file to `rejected-directory` or leave it in `working-directory` with a failure marker, depending on operator preference.

Use the source filename stem for the output directory:

```text
/input/source/1.mp4 -> /output/clips/1/clip_00000.mp4
/input/source/1.mp4 -> /output/clips/1/clip_00001.mp4
/input/source/1.mp4 -> /output/clips/1/clip_00002.mp4
```

If two input files share the same stem, append a deterministic numeric suffix:

```text
/output/clips/1
/output/clips/1-1
```

## Valid Video Detection

Use a two-stage validation:

1. Extension allowlist for cheap filtering:
   - `.mp4`
   - `.mov`
   - `.m4v`
   - `.webm`
   - `.mkv`
2. `ffprobe` verification for real validation.

Suggested `ffprobe` command:

```bash
ffprobe -v error \
  -select_streams v:0 \
  -show_entries stream=codec_type,codec_name,width,height \
  -show_entries format=duration \
  -of json \
  /input/working/1.mp4
```

Accept the file only if:

- `ffprobe` exits with code `0`
- at least one video stream exists
- duration is present and greater than `0`
- width and height are positive

Log invalid files with:

- source path
- reason
- ffprobe exit code
- sanitized ffprobe output

Do not log raw paths from untrusted metadata or full FFmpeg stderr if it contains sensitive filesystem details.

## FFmpeg Command

Default to quality-preserving MP4 stream copy when the source already has usable keyframes:

```bash
ffmpeg -hide_banner -y \
  -i /input/working/1.mp4 \
  -map 0 -c copy \
  -f segment \
  -segment_time 8 \
  -reset_timestamps 1 \
  /output/clips/1/clip_%05d.mp4
```

Expected behavior:

- each generated clip preserves the input codec quality because no re-encode occurs
- clips are cut on keyframes, so boundaries may drift from exact 8-second marks
- the final clip may be shorter than 8 seconds
- output files are broadly playable MP4 files

Implementation note:

- If exact 8-second boundaries are more important than preserving source quality, transcode with `libx264` and force keyframes.
- If preserving source quality is more important, use stream copy and accept keyframe-aligned cuts.

## Proposed Code Structure

Add:

- `src/main/java/org/example/videoclips/folder/FolderSchedulerProperties.java`
- `src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java`
- `src/main/java/org/example/videoclips/folder/FolderVideoValidator.java`
- `src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java`
- `src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java`

Responsibilities:

- `FolderSchedulerProperties`: typed config for directories, binaries, poll interval, segment length, and output format.
- `FolderVideoScanScheduler`: scheduled scanner, single-file lock, ordering, state moves.
- `FolderVideoValidator`: extension filtering and `ffprobe` validation.
- `FolderFfmpegClipper`: FFmpeg command construction and process execution.
- Tests: validate ordering, invalid-file rejection, no repeat processing, and command construction.

## Concurrency Control

Use an in-process guard to prevent overlapping scheduled runs:

- `AtomicBoolean processing`
- or `ReentrantLock tryLock()`

Only one file should process at a time. This directly satisfies:

```text
1.mp4 completes before 2.mp4 starts
```

If the service will run multiple replicas against the same mounted folder, add a filesystem lock or database lease before enabling more than one scheduler instance.

## File State Flow

Recommended state transitions:

```text
/input/source/1.mp4
  -> /input/working/1.mp4
  -> /output/clips/1/clip_00000.mp4
  -> /output/clips/1/clip_00001.mp4
  -> /input/processed/1.mp4
```

Invalid file:

```text
/input/source/not-video.txt
  -> /input/rejected/not-video.txt
```

Failed processing:

```text
/input/working/broken.mp4
  -> /input/rejected/broken.mp4
```

This ensures the scheduler never processes the same source file repeatedly.

## Operational Logging

The scheduler uses an initial delay and fixed delay of `5000` ms by default. The initial delay lets startup directory initialization finish; subsequent delays start after the previous scan finishes, so scans never intentionally overlap. Configure both with `video-clipping.folder-scheduler.poll-interval-ms` (minimum `1000` ms).

The base configuration enables repository-local folder processing for normal IDE starts. Production deployments can override it without changing the artifact:

```bash
FOLDER_SCHEDULER_ENABLED=true
FOLDER_SCHEDULER_INPUT_DIRECTORY=/input/source
FOLDER_SCHEDULER_OUTPUT_DIRECTORY=/output/clips
FOLDER_SCHEDULER_WORKING_DIRECTORY=/input/working
FOLDER_SCHEDULER_PROCESSED_DIRECTORY=/input/processed
FOLDER_SCHEDULER_REJECTED_DIRECTORY=/input/rejected
FOLDER_SCHEDULER_POLL_INTERVAL_MS=5000
FOLDER_SCHEDULER_LOG_LEVEL=INFO
```

Lifecycle logs use stable `event=...` fields and a per-run `scan_id` for correlation. Active work and state transitions log at info or warn/error level; scan start, idle, completion, directory readiness, and probe timing log at debug level.

Log these events:

- scheduler started and configured directories
- no candidate found at debug level
- selected candidate
- invalid candidate rejected
- FFmpeg command started, without sensitive metadata
- clip generation succeeded with clip count
- source moved to processed directory
- processing failed with sanitized error

## Testing Plan

Unit tests:

- creates configured folders
- sorts `1.mp4` before `2.mp4`
- ignores hidden and temporary files
- rejects invalid extension before `ffprobe`
- rejects `ffprobe` failure
- moves valid input to working before processing
- moves successful source to processed
- does not pick the same input twice

Integration tests:

- run FFmpeg against a short generated fixture if `ffmpeg` and `ffprobe` are installed
- verify a 17-second input produces 3 clips
- verify output path is `output/clips/<video-name>/clip_%05d.mp4`

## Rollout Plan

1. Add config properties and disabled-by-default scheduler.
2. Add directory creation and scanning behavior.
3. Add validation and rejection flow.
4. Add FFmpeg clipping flow.
5. Add unit tests for file-state transitions.
6. Add optional integration test gated on local FFmpeg availability.
7. Enable in a local profile first.
8. Enable in production only with mounted `/input` and `/output` volumes.

## Acceptance Criteria

The feature is complete when:

- `/input/source` and `/output/clips` are configurable and created on startup
- valid video files are processed one at a time
- `1.mp4` is processed before `2.mp4` under deterministic ordering
- clips are written to `/output/clips/<video-name>/`
- each clip is MP4/H.264/AAC by default
- invalid files are ignored for processing, moved to rejected storage, and logged
- processed source files are not picked up again
- tests cover ordering, invalid file handling, and repeat-prevention behavior