Changes:
- Added JaCoCo to pom.xml with a 100% instruction, line, and branch coverage rule scoped to org.example.videoclips.folder.
- Added src/test/java/org/example/videoclips/folder/FolderSchedulerDirectoryInitializerTest.java.
- Added src/test/java/org/example/videoclips/folder/FolderVideoScanSchedulerTest.java.
- Adjusted the scanner comparator in src/main/java/org/example/videoclips/folder/FolderVideoScanScheduler.java to keep deterministic ordering without an uncovered comparator branch.
Verification: mvn -q verify passed. The generated JaCoCo report shows org.example.videoclips.folder at 100% instruction, line, and branch coverage.
Scope note: this enforces 100% coverage for the new scheduler package, not the whole existing service. Project-wide 100% would be a much larger pass across pre-existing classes. The next
scheduler milestone is repeat-prevention by moving selected inputs to the working folder before processing.
Added src/main/java/org/example/videoclips/folder/FolderSchedulerDirectoryInitializer.java, which creates the configured input, output, working, processed, and rejected directories when video-clipping.folder-
scheduler.enabled=true. It stays inactive by default, so normal startup will not fail on machines where /input and /output cannot be created.
Updated docs/input-folder-scheduler-plan.md to mark milestone 3 complete.
The implementation plan is now fully checked off. The signoff doc is explicit about the remaining real-world gap: the repo has benchmark baselines and readiness documentation, but no checked-in evidence of an executed
production-like end-to-end load test.
The procedure is aligned to the actual db queue behavior in this repo: manual operator-driven reset of queue_messages and clip_jobs, staged replay, audit requirements, and explicit warnings about retry storms and stale
PROCESSING reclamation.
The doc defines what must be backed up together for this service, the restore order, SQL and API verification checks, queue replay safety checks, and cleanup-related caveats. It also states the important limitation: this repo
does not include production backup automation, so a real infra restore drill still needs to be executed outside the repo.
The runbook covers alert-driven triage for queue backlog, DLQ growth, API 5xx, instance-down, CPU saturation, and cleanup/retention issues, using the actual queue retry and cleanup behavior in the codebase.
The alert doc includes a concrete PrometheusRule for queue age, DLQ presence, API 5xx ratio, download-url failures, target down, and sustained CPU saturation, plus the instrumentation gaps for alerts the service cannot
support yet from current metrics alone.
under explicit planning assumptions. The checked-in companion doc is docs/cost-model-by-video-minute.md:1, and it now includes the modeled baseline table:
- exact-veryfast-720p: 0.000754 USD / source min
- exact-medium-720p: 0.000816 USD / source min
- exact-veryfast-720p-slow-storage: 0.000802 USD / source min
I added autoscaling-ready queue metrics in src/main/java/org/example/videoclips/observability/DatabaseQueueMetricsBinder.java:13, backed by new queue repository counters, and enabled Prometheus scraping via src/main/
resources/application.properties:3 plus the Prometheus registry dependency in pom.xml:43. The policy itself is documented in docs/autoscaling-policies.md:1, including worker/API scaling rules and a sample KEDA ScaledObject.
Coverage for the new gauges is in src/test/java/org/example/videoclips/observability/DatabaseQueueMetricsBinderTest.java:14.
The main change is in src/main/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java:52: the DB queue now reclaims stale PROCESSING messages after a configurable visibility timeout instead of leaving them
stuck forever after a worker crash. I added video-clipping.database-queue.visibility-timeout-ms with a default of 900000 ms in src/main/java/org/example/videoclips/config/VideoClippingProperties.java:171 and src/main/
resources/application.properties:19, plus coverage for stale-claim recovery in src/test/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapterTest.java:91. I also added the tuning note and sizing rule in docs/
queue-visibility-timeout-tuning.md:1.
I added a reproducible benchmark harness in src/test/java/org/example/videoclips/perf/ObjectStorageBandwidthBenchmarkHarness.java:17 that exercises ObjectStoragePort materialization and upload paths across a local-disk
baseline plus throttled 100/50/25 MiB/s profiles. I also checked in the measured baseline and rerun command in docs/object-storage-bandwidth-benchmark-baseline.md:1. On this workspace, the measured throughput came out at
roughly 1282/825 MiB/s for local disk, 93/89 MiB/s for the 100 MiB/s cap, 48/47 MiB/s for 50 MiB/s, and 24/24 MiB/s for 25 MiB/s on materialize/upload respectively.
I completed Worker CPU/disk benchmarks by adding a reproducible benchmark harness at src/test/java/org/example/videoclips/perf/WorkerBenchmarkHarness.java:1 and checking in the resulting baseline at docs/worker-cpu-disk-
benchmark-baseline.md:1. The harness measures synchronous worker wall time, current-thread CPU time, clip count, and local/uploaded output bytes for FAST and EXACT stub-processing scenarios, and writes the generated report to
target/benchmarks/worker-cpu-disk-benchmark.md.
The plan now marks that item complete in docs/video-clipping-service-implementation-plan.md:711. Verification also stayed green: mvn -q -Dtest=WorkerBenchmarkHarness test passed, and mvn -q test passed with the default suite
plus the benchmark harness.
The security hardening I implemented was tenant isolation across the API surface. X-Tenant-Id is now enforced on asset reads, upload completion, asset/job event reads, clip listing, job reads, cancellation, deletion, and
signed download URL creation. Cross-tenant access returns 403 via the new src/main/java/org/example/videoclips/application/ForbiddenException.java:1 and handler wiring in src/main/java/org/example/videoclips/api/
ApiExceptionHandler.java:1. I also stopped exposing providerUploadId from the upload-session read response in src/main/java/org/example/videoclips/application/VideoAssetService.java:224, since that provider identifier is
internal metadata.
Verification: mvn -q test passed.
- TenantQuotaControllerTest: 1/1
- VideoAssetControllerTest: 11/11
- RetentionCleanupJobTest: 2/2
- DatabaseBackedClipJobQueueAdapterTest: 2/2
The production-hardening section of the plan is now fully checked off.
The main change is in src/main/java/org/example/videoclips/api/ApiExceptionHandler.java:1. It now consistently returns problem details for:
- validation errors
- malformed JSON
- unsupported request content type
- method not allowed
- quota/conflict/not-found/media-type domain exceptions
- unexpected internal errors
I also extended src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java:1 to assert application/problem+json and cover malformed JSON, unsupported request content type, and method-not-allowed responses.
marks the job FAILED when the message actually moves to DLQ, and retry/DLQ events are recorded at the job level. The key changes are in src/main/java/org/example/videoclips/processing/ClipProcessor.java:37 and src/main/java/
org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java:18.
I also kept the queue tests aligned in src/test/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapterTest.java:1, reran mvn -q test, and the suite passed:
- TenantQuotaControllerTest: 1/1
- VideoAssetControllerTest: 7/7
- RetentionCleanupJobTest: 2/2
- DatabaseBackedClipJobQueueAdapterTest: 2/2
The plan is updated accordingly in docs/video-clipping-service-implementation-plan.md:701, with Retry and DLQ handling now marked complete. The next unfinished step is Full application/problem+json error model.
The implementation adds tenant-aware assets/jobs plus a configurable per-tenant active-job quota. The main enforcement is in src/main/java/org/example/videoclips/application/VideoAssetService.java:165, backed by tenant fields
added to the domain/persistence model and migration src/main/resources/db/migration/V6__tenant_quota_support.sql:1. Requests can set X-Tenant-Id on asset creation, quota config lives in src/main/java/org/example/videoclips/
config/VideoClippingProperties.java:39, and quota violations now return 429 via src/main/java/org/example/videoclips/api/ApiExceptionHandler.java:44.
I also started that next step in code. The database-backed queue now retries failed jobs with configurable backoff and moves messages to a terminal DLQ status after the configured max attempts in src/main/java/org/example/
videoclips/queue/DatabaseBackedClipJobQueueAdapter.java:48. The new queue settings live in src/main/java/org/example/videoclips/config/VideoClippingProperties.java:133, and I added focused coverage in src/test/java/org/
example/videoclips/queue/DatabaseBackedClipJobQueueAdapterTest.java:1.
Verification: mvn -q test passed.
- VideoAssetControllerTest: 7/7
- RetentionCleanupJobTest: 2/2
- DatabaseBackedClipJobQueueAdapterTest: 2/2
Changes:
- src/main/java/org/example/videoclips/application/VideoAssetService.java:401 no longer exposes sourceObjectKey on asset reads or objectKey on clip listings.
- src/main/java/org/example/videoclips/processing/ClipProcessor.java:64 now records a sanitized SOURCE_MATERIALIZED event message instead of logging the worker-local path.
- src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java:67 now asserts those fields are absent and that job events do not contain uploads/ or tmp/.
VideoAssetService.java, src/main/java/org/example/videoclips/storage/ObjectStoragePort.java, src/main/java/org/example/videoclips/storage/InMemoryObjectStorageAdapter.java, and src/main/java/org/
example/videoclips/storage/S3ObjectStorageAdapter.java.
I also added focused coverage for this slice:
- API test for generating a signed URL for an actual produced clip in src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java
- cleanup job tests for expired clip/object deletion and disabled cleanup in src/test/java/org/example/videoclips/operations/RetentionCleanupJobTest.java
- src/main/java/org/example/videoclips/domain/VideoAsset.java
- src/main/java/org/example/videoclips/domain/Clip.java
That metadata is mapped through the in-memory and JPA repository paths, including:
- src/main/java/org/example/videoclips/infrastructure/InMemoryVideoAssetRepository.java
- src/main/java/org/example/videoclips/persistence/JpaVideoClippingRepository.java
- src/main/java/org/example/videoclips/persistence/entity/VideoAssetEntity.java
- src/main/java/org/example/videoclips/persistence/entity/ClipEntity.java
I also added scheduled retention cleanup in src/main/java/org/example/videoclips/operations/RetentionCleanupJob.java, with configuration in src/main/java/org/example/videoclips/config/
VideoClippingProperties.java and src/main/resources/application.properties. Right now it soft-expires assets by marking them deleted when their retention time passes. Clip retention timestamps
are now assigned by both clipper adapters so the service has real expiration data to work with.
What this does not do yet is physically delete expired clip objects or original source objects from object storage. The retention job is metadata-aware now, but object-store deletion still needs
to be added through the storage port as the next step.
- the FFmpeg input directory
- the FFmpeg output directory
- tmp/in-memory-storage
- tmp/stub-output
The schedule and retention window are now configurable through src/main/java/org/example/videoclips/config/VideoClippingProperties.java and src/main/resources/application.properties via:
- video-clipping.cleanup.enabled
- video-clipping.cleanup.local-artifact-poll-interval-ms
- video-clipping.cleanup.local-artifact-retention-hours
This closes the first retention slice from the plan on the worker side. It does not yet delete expired original videos or clips from real object storage, and it does not do database-driven
retention policies. That would be the next cleanup step.
instead of accumulating. Cleanup is controlled by the new video-clipping.ffmpeg.cleanup-local-files property wired through src/main/java/org/example/videoclips/config/VideoClippingProperties.java
and src/main/resources/application.properties.
I also improved FFmpeg failure diagnostics in src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java. It now captures the merged process output and includes a trimmed
version in the exception instead of only reporting the exit code, which makes job failures much easier to debug.
The root cause was that ClipProcessor had been updated to call objectStoragePort.uploadGeneratedClip(...), but src/main/java/org/example/videoclips/storage/ObjectStoragePort.java did not yet
declare that method. I added it to the port and confirmed both adapters implement it:
- src/main/java/org/example/videoclips/storage/InMemoryObjectStorageAdapter.java
- src/main/java/org/example/videoclips/storage/S3ObjectStorageAdapter.java
I also verified the generated-clip contract is aligned across:
- src/main/java/org/example/videoclips/processing/GeneratedClip.java
- src/main/java/org/example/videoclips/processing/VideoClipperPort.java
- src/main/java/org/example/videoclips/processing/StubVideoClipperAdapter.java
- src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java
java/org/example/videoclips/persistence/entity/ClipEntity.java and src/main/java/org/example/videoclips/persistence/JpaVideoClippingMapper.java, and added the schema migration at src/main/
resources/db/migration/V5__clip_object_key.sql.
The clipper contract now returns both metadata and the local file path for each generated clip through src/main/java/org/example/videoclips/processing/GeneratedClip.java and the updated src/main/
java/org/example/videoclips/processing/VideoClipperPort.java. Both adapters were updated:
- src/main/java/org/example/videoclips/processing/StubVideoClipperAdapter.java now creates stub local output files
- src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java now returns deterministic local output paths for generated segments
On the storage side, ObjectStoragePort gained uploadGeneratedClip(...), with implementations in:
- src/main/java/org/example/videoclips/storage/InMemoryObjectStorageAdapter.java
- src/main/java/org/example/videoclips/storage/S3ObjectStorageAdapter.java
And src/main/java/org/example/videoclips/processing/ClipProcessor.java now uploads each generated local file through storage before saving the clip record, so the job path is finally staged input
-> process -> uploaded output -> persisted clip metadata.
I also exposed objectKey in the clip response via src/main/java/org/example/videoclips/application/VideoAssetService.java and extended the MockMvc test to assert it exists in src/test/java/org/
example/videoclips/api/VideoAssetControllerTest.java.
- materializeSourceObject(String sourceObjectKey, Path targetPath) in src/main/java/org/example/videoclips/storage/ObjectStoragePort.java
Both storage adapters implement it:
- src/main/java/org/example/videoclips/storage/InMemoryObjectStorageAdapter.java creates a placeholder local file for the default stub flow
- src/main/java/org/example/videoclips/storage/S3ObjectStorageAdapter.java downloads the object from S3-compatible storage to the worker filesystem
The processor now uses that seam before clip generation starts. In src/main/java/org/example/videoclips/processing/ClipProcessor.java it resolves the local input path under the configured FFmpeg
input directory, materializes the source object there, and records a SOURCE_MATERIALIZED job event before invoking the clipper.
That means the FFmpeg path is no longer relying on a file appearing magically in the input directory. The next concrete step is to finish the output side the same way: persist generated clip
object keys and push FFmpeg outputs back through the storage port instead of only creating clip metadata in memory.
- src/main/java/org/example/videoclips/domain/VideoAsset.java
- src/main/java/org/example/videoclips/persistence/entity/VideoAssetEntity.java
- src/main/java/org/example/videoclips/persistence/JpaVideoClippingMapper.java
- src/main/resources/db/migration/V4__video_asset_source_object_key.sql
createAsset(...) now assigns a stable source object key of the form uploads/{uploadId}/source, and that value is exposed back through the asset read response in src/main/java/org/example/
videoclips/application/VideoAssetService.java. I also threaded that source-object reference through the clipper contract in src/main/java/org/example/videoclips/processing/VideoClipperPort.java,
so src/main/java/org/example/videoclips/processing/ClipProcessor.java passes a real persisted object reference into the processing adapter. The FFmpeg adapter now resolves its input path from
that source object key instead of assuming {assetId}.mp4.
I also completed the multipart upload seam:
- UploadSession now stores providerUploadId
- the storage port gained completeMultipartUpload(...)
- the service now calls storage completion during uploads:complete
- the new metadata is migrated in src/main/resources/db/migration/V3__upload_session_provider_metadata.sql
Finally, I extended the MockMvc test to assert that the asset read response now includes the expected sourceObjectKey in src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java.
application/VideoAssetService.java, src/main/java/org/example/videoclips/storage/ObjectStoragePort.java, and the JPA upload-session mapping, and added a migration at src/main/resources/db/
migration/V3__upload_session_provider_metadata.sql. The storage adapters now expose completeMultipartUpload(...), with the in-memory adapter as a no-op and the S3 adapter implementing the AWS
multipart completion call.
I also added an FFmpeg-backed clipper adapter in src/main/java/org/example/videoclips/processing/FfmpegVideoClipperAdapter.java, controlled by video-clipping.processing=ffmpeg. The stub clipper
remains the default via video-clipping.processing=stub, so the app still has a safe fallback. The new FFmpeg configuration lives under src/main/java/org/example/videoclips/config/
VideoClippingProperties.java and src/main/resources/application.properties.
I also extended the test expectations so the upload-session response now verifies providerUploadId in src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java.
upload part URLs and signed clip download URLs when video-clipping.storage=s3. The current in-memory adapter remains the default. I also added the supporting configuration fields in src/main/
java/org/example/videoclips/config/VideoClippingProperties.java and default values in src/main/resources/application.properties.
The queue port now has a durable database-backed implementation in src/main/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java. It persists queue messages in a new
queue_messages table from src/main/resources/db/migration/V2__queue_messages.sql, polls them on a schedule, and dispatches them through the existing processor. I updated src/main/java/org/
example/videoclips/processing/ClipProcessor.java so processing failures propagate back to the queue adapter instead of being swallowed.
I also enabled scheduling in src/main/java/org/example/videoclips/VideoClippingApplication.java, added the AWS SDK dependency in pom.xml, and updated src/main/resources/application-jpa.properties
so the JPA profile uses the database queue by default.
main/java/org/example/videoclips/infrastructure/InMemoryVideoAssetRepository.java and the new src/main/java/org/example/videoclips/persistence/JpaVideoClippingRepository.java behind it. The JPA
path includes entity models, Spring Data repositories, a mapper, and a Flyway baseline migration at src/main/resources/db/migration/V1__init_video_clipping.sql. I also added the required
dependencies in pom.xml.
The default runtime still uses the in-memory repository via video-clipping.repository=memory, so existing behavior stays intact. There’s now a ready-to-use JPA profile in src/main/resources/
application-jpa.properties that uses H2 in PostgreSQL mode for local bring-up. That gives the codebase a real persistence lane without forcing the switch yet.
- cancels any active jobs for that asset
- aborts an open upload session
- records an asset deletion event
- prevents clip download URLs from being created for clips that belong to deleted assets
I also added a regression test for that behavior in src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java, along with the earlier repository-port and operations work. The
application and worker layers now depend on VideoClippingRepository instead of the concrete in-memory adapter, which is the right base for a future PostgreSQL adapter.
VideoAssetController.java, backed by richer UploadSession metadata in src/main/java/org/example/videoclips/domain/UploadSession.java. The service now stores part size, part count, expiry, and
completion time, and returns them through src/main/java/org/example/videoclips/application/VideoAssetService.java. I also added a configurable media-type allowlist in src/main/java/org/example/
videoclips/config/VideoClippingProperties.java and reject unsupported uploads with 415 via src/main/java/org/example/videoclips/application/UnsupportedMediaTypeException.java and src/main/java/
org/example/videoclips/api/ApiExceptionHandler.java.
I also added adapter-level health visibility for operations. src/main/java/org/example/videoclips/observability/ObjectStorageHealthIndicator.java and src/main/java/org/example/videoclips/
observability/QueueHealthIndicator.java now expose basic readiness details through Actuator. Tests were extended in src/test/java/org/example/videoclips/api/VideoAssetControllerTest.java to cover
the upload-session read path and the unsupported-media-type case.
JSLMPR