forked from jsl/video_editing_poc
33 lines
1.7 KiB
Markdown
33 lines
1.7 KiB
Markdown
# Queue Visibility Timeout Tuning
|
|
|
|
This document captures the implementation and sizing guidance for performance plan item `Queue visibility-timeout tuning`.
|
|
|
|
Implemented behavior:
|
|
|
|
- Database-backed queue messages now use a configurable visibility timeout via `video-clipping.database-queue.visibility-timeout-ms`.
|
|
- The default is `900000` ms (`15` minutes).
|
|
- Messages left in `PROCESSING` past the visibility timeout are reclaimed on the next poll and retried.
|
|
- This prevents permanently stuck jobs after worker crashes or process termination between claim and completion.
|
|
|
|
Sizing rule:
|
|
|
|
- Start with `visibility-timeout-ms = 3x` the observed `p99` end-to-end job runtime for the dominant workload.
|
|
- Include source-object materialization, FFmpeg runtime, clip upload, and persistence overhead.
|
|
- Keep `retry-backoff-ms` materially smaller than the visibility timeout so normal retries happen before stale-claim recovery becomes the dominant path.
|
|
|
|
Current baseline rationale:
|
|
|
|
- The checked-in FFmpeg and object-storage benchmarks show local workloads completing in seconds, not minutes.
|
|
- The database queue does not extend visibility during active processing, so the default is intentionally conservative.
|
|
- `15` minutes gives room for slower `EXACT` jobs, degraded object-storage bandwidth, and moderate node contention without causing premature duplicate delivery.
|
|
|
|
How to validate:
|
|
|
|
```bash
|
|
mvn -q -Dtest=DatabaseBackedClipJobQueueAdapterTest test
|
|
```
|
|
|
|
Operational note:
|
|
|
|
- If production measurements show long-running jobs regularly exceeding the configured timeout, either raise `visibility-timeout-ms` or implement active lease extension before increasing worker concurrency.
|