video_editing_poc/docs/queue-visibility-timeout-tuning.md

Queue Visibility Timeout Tuning

This document captures the implementation and sizing guidance for performance plan item Queue visibility-timeout tuning.

Implemented behavior:

• Database-backed queue messages now use a configurable visibility timeout via video-clipping.database-queue.visibility-timeout-ms.
• The default is 900000 ms (15 minutes).
• Messages left in PROCESSING past the visibility timeout are reclaimed on the next poll and retried.
• This prevents permanently stuck jobs after worker crashes or process termination between claim and completion.

Sizing rule:

• Start with visibility-timeout-ms = 3x the observed p99 end-to-end job runtime for the dominant workload.
• Include source-object materialization, FFmpeg runtime, clip upload, and persistence overhead.
• Keep retry-backoff-ms materially smaller than the visibility timeout so normal retries happen before stale-claim recovery becomes the dominant path.

Current baseline rationale:

• The checked-in FFmpeg and object-storage benchmarks show local workloads completing in seconds, not minutes.
• The database queue does not extend visibility during active processing, so the default is intentionally conservative.
• 15 minutes gives room for slower EXACT jobs, degraded object-storage bandwidth, and moderate node contention without causing premature duplicate delivery.

How to validate:

mvn -q -Dtest=DatabaseBackedClipJobQueueAdapterTest test

Operational note:

• If production measurements show long-running jobs regularly exceeding the configured timeout, either raise visibility-timeout-ms or implement active lease extension before increasing worker concurrency.