video_editing_poc/docs/load-test-signoff.md

Load-Test Signoff

This document captures the operational readiness artifact for plan item Load-test signoff.

Scope

This signoff covers production-readiness expectations for:

• API request handling
• queue-backed worker throughput
• object-storage transfer behavior
• clip-processing saturation
• autoscaling response

It is based on artifacts already checked into this repository plus the execution criteria required for a real environment load test.

Existing Evidence In Repo

The following performance inputs already exist:

• Worker CPU/disk baseline: docs/worker-cpu-disk-benchmark-baseline.md
• FFmpeg preset baseline: docs/ffmpeg-preset-benchmark-baseline.md
• Object-storage bandwidth baseline: docs/object-storage-bandwidth-benchmark-baseline.md
• Queue timeout tuning: docs/queue-visibility-timeout-tuning.md
• Autoscaling policy: docs/autoscaling-policies.md
• Cost model: docs/cost-model-by-video-minute.md

What these artifacts prove:

• the codebase has reproducible local benchmark harnesses
• CPU, FFmpeg preset, and storage-throughput tradeoffs have been analyzed
• queue timeout and autoscaling thresholds have an initial rationale

What they do not prove:

• production API throughput
• production worker throughput
• multi-tenant contention behavior
• Kubernetes autoscaling effectiveness under real traffic
• end-to-end stability of upload, queue, processing, and clip retrieval at target load

Signoff Standard

Load-test signoff is only valid if all of the following are true:

• representative API and worker traffic has been exercised in a production-like environment
• object storage, database, and queue dependencies are real or equivalently provisioned
• sustained load and burst load have both been tested
• alerts, dashboards, and runbooks have been used during the exercise
• no unbounded DLQ growth, queue-age runaway, or data-corruption behavior occurred

Required Test Scenarios

At minimum, execute these scenarios in the target environment:

1. Steady-state asset creation and job submission.
2. Sustained worker processing with representative source durations.
3. Mixed FAST and EXACT job distribution.
4. Burst traffic that exceeds normal steady-state API rate.
5. Worker scale-out under queue backlog.
6. Clip download URL generation during active processing load.
7. Failure injection for at least one dependency or worker restart case.

Required Metrics During Load Test

Collect and retain:

• API request rate
• API p95 latency
• API 5xx ratio
• queue pending depth
• queue oldest pending age
• queue DLQ count
• process CPU
• JVM heap usage
• worker replica count
• object-storage transfer latency if available from platform metrics

Pass Criteria

A load test is considered acceptable for signoff when:

• API p95 latency remains within the agreed product target under steady-state load
• worker backlog clears within the expected recovery window after burst load
• video_clipping_queue_oldest_pending_age_seconds returns toward baseline after burst traffic
• DLQ count remains 0 for normal traffic or any DLQ growth is fully explained by intentional failure injection
• autoscaling reacts as designed without oscillation or runaway scaling
• no data loss, duplicate clip-index corruption, or stuck job accumulation is observed
• recovery after worker restart or transient dependency interruption is acceptable

Recommended Execution Shape

Use a production-like environment with:

• video-clipping.repository=jpa
• video-clipping.storage=s3
• video-clipping.queue=db or the intended production queue provider
• real Kubernetes autoscaling configuration
• real Prometheus and Grafana telemetry

Suggested workload shape:

• start with a low steady-state submission rate
• increase gradually to expected daily peak
• add a burst phase at 2x expected peak
• hold sustained load long enough to observe autoscaling and queue recovery

Suggested clip mix:

• short videos
• medium-length videos
• at least one longer-running EXACT workload class

Pre-Signoff Checklist

Before calling the system signed off:

• benchmark baselines have been reviewed
• alert rules are deployed
• Grafana dashboard is imported
• runbooks are available to operators
• backup/restore and DLQ procedures exist
• retention settings are appropriate for the test environment

Current Status

Current repository status is:

• benchmark-backed planning baselines exist
• operational readiness documents exist
• no checked-in evidence of an executed end-to-end environment load test exists in this repository

Because of that, current signoff status is:

• Documentation complete
• Execution evidence pending

Signoff Record Template

Use this template when the real load test is run:

Environment:
Date:
Build or commit:
Queue mode:
Storage backend:
Database backend:
Traffic generator:

Steady-state target:
Burst target:
Duration:

Observed API p95:
Observed max queue age:
Observed max pending depth:
Observed DLQ count:
Observed max worker replicas:

Incidents during test:
Recovery behavior:
Open issues:

Decision:
- Approved
- Approved with follow-up
- Rejected

Current Repo Limitation

This repository does not include:

• a checked-in load generator
• an executed production-like load-test report
• measured end-to-end throughput or latency results from a real deployment

This artifact defines the signoff gate and records that the documentation milestone is complete, but a real environment test still must be executed before claiming final production load readiness.