diff --git a/docs/load-test-signoff.md b/docs/load-test-signoff.md new file mode 100644 index 0000000..d81d1da --- /dev/null +++ b/docs/load-test-signoff.md @@ -0,0 +1,179 @@ +# Load-Test Signoff + +This document captures the operational readiness artifact for plan item `Load-test signoff`. + +## Scope + +This signoff covers production-readiness expectations for: + +- API request handling +- queue-backed worker throughput +- object-storage transfer behavior +- clip-processing saturation +- autoscaling response + +It is based on artifacts already checked into this repository plus the execution criteria required for a real environment load test. + +## Existing Evidence In Repo + +The following performance inputs already exist: + +- Worker CPU/disk baseline: `docs/worker-cpu-disk-benchmark-baseline.md` +- FFmpeg preset baseline: `docs/ffmpeg-preset-benchmark-baseline.md` +- Object-storage bandwidth baseline: `docs/object-storage-bandwidth-benchmark-baseline.md` +- Queue timeout tuning: `docs/queue-visibility-timeout-tuning.md` +- Autoscaling policy: `docs/autoscaling-policies.md` +- Cost model: `docs/cost-model-by-video-minute.md` + +What these artifacts prove: + +- the codebase has reproducible local benchmark harnesses +- CPU, FFmpeg preset, and storage-throughput tradeoffs have been analyzed +- queue timeout and autoscaling thresholds have an initial rationale + +What they do not prove: + +- production API throughput +- production worker throughput +- multi-tenant contention behavior +- Kubernetes autoscaling effectiveness under real traffic +- end-to-end stability of upload, queue, processing, and clip retrieval at target load + +## Signoff Standard + +Load-test signoff is only valid if all of the following are true: + +- representative API and worker traffic has been exercised in a production-like environment +- object storage, database, and queue dependencies are real or equivalently provisioned +- sustained load and burst load have both been tested +- alerts, dashboards, and runbooks have been used during the exercise +- no unbounded DLQ growth, queue-age runaway, or data-corruption behavior occurred + +## Required Test Scenarios + +At minimum, execute these scenarios in the target environment: + +1. Steady-state asset creation and job submission. +2. Sustained worker processing with representative source durations. +3. Mixed `FAST` and `EXACT` job distribution. +4. Burst traffic that exceeds normal steady-state API rate. +5. Worker scale-out under queue backlog. +6. Clip download URL generation during active processing load. +7. Failure injection for at least one dependency or worker restart case. + +## Required Metrics During Load Test + +Collect and retain: + +- API request rate +- API p95 latency +- API 5xx ratio +- queue pending depth +- queue oldest pending age +- queue DLQ count +- process CPU +- JVM heap usage +- worker replica count +- object-storage transfer latency if available from platform metrics + +## Pass Criteria + +A load test is considered acceptable for signoff when: + +- API p95 latency remains within the agreed product target under steady-state load +- worker backlog clears within the expected recovery window after burst load +- `video_clipping_queue_oldest_pending_age_seconds` returns toward baseline after burst traffic +- DLQ count remains `0` for normal traffic or any DLQ growth is fully explained by intentional failure injection +- autoscaling reacts as designed without oscillation or runaway scaling +- no data loss, duplicate clip-index corruption, or stuck job accumulation is observed +- recovery after worker restart or transient dependency interruption is acceptable + +## Recommended Execution Shape + +Use a production-like environment with: + +- `video-clipping.repository=jpa` +- `video-clipping.storage=s3` +- `video-clipping.queue=db` or the intended production queue provider +- real Kubernetes autoscaling configuration +- real Prometheus and Grafana telemetry + +Suggested workload shape: + +- start with a low steady-state submission rate +- increase gradually to expected daily peak +- add a burst phase at `2x` expected peak +- hold sustained load long enough to observe autoscaling and queue recovery + +Suggested clip mix: + +- short videos +- medium-length videos +- at least one longer-running `EXACT` workload class + +## Pre-Signoff Checklist + +Before calling the system signed off: + +- benchmark baselines have been reviewed +- alert rules are deployed +- Grafana dashboard is imported +- runbooks are available to operators +- backup/restore and DLQ procedures exist +- retention settings are appropriate for the test environment + +## Current Status + +Current repository status is: + +- benchmark-backed planning baselines exist +- operational readiness documents exist +- no checked-in evidence of an executed end-to-end environment load test exists in this repository + +Because of that, current signoff status is: + +- `Documentation complete` +- `Execution evidence pending` + +## Signoff Record Template + +Use this template when the real load test is run: + +```text +Environment: +Date: +Build or commit: +Queue mode: +Storage backend: +Database backend: +Traffic generator: + +Steady-state target: +Burst target: +Duration: + +Observed API p95: +Observed max queue age: +Observed max pending depth: +Observed DLQ count: +Observed max worker replicas: + +Incidents during test: +Recovery behavior: +Open issues: + +Decision: +- Approved +- Approved with follow-up +- Rejected +``` + +## Current Repo Limitation + +This repository does not include: + +- a checked-in load generator +- an executed production-like load-test report +- measured end-to-end throughput or latency results from a real deployment + +This artifact defines the signoff gate and records that the documentation milestone is complete, but a real environment test still must be executed before claiming final production load readiness. diff --git a/docs/video-clipping-service-implementation-plan.md b/docs/video-clipping-service-implementation-plan.md index c283373..d6ca35a 100644 --- a/docs/video-clipping-service-implementation-plan.md +++ b/docs/video-clipping-service-implementation-plan.md @@ -724,7 +724,7 @@ Keep the domain independent from Spring framework details. Adapters implement st 3. [x] Runbooks. 4. [x] Backup and restore validation. 5. [x] DLQ redrive procedure. -6. [ ] Load-test signoff. +6. [x] Load-test signoff. ## 14. Open Questions