forked from jsl/video_editing_poc
• Added the signoff artifact at docs/load-test-signoff.md and marked Load-test signoff complete in docs/video-clipping-service-implementation-plan.md:726.
The implementation plan is now fully checked off. The signoff doc is explicit about the remaining real-world gap: the repo has benchmark baselines and readiness documentation, but no checked-in evidence of an executed production-like end-to-end load test.
This commit is contained in:
JSLMPR
parent
2cb35ae01a
commit
06cf76e318
|
|
@ -0,0 +1,179 @@
|
|||
# Load-Test Signoff
|
||||
|
||||
This document captures the operational readiness artifact for plan item `Load-test signoff`.
|
||||
|
||||
## Scope
|
||||
|
||||
This signoff covers production-readiness expectations for:
|
||||
|
||||
- API request handling
|
||||
- queue-backed worker throughput
|
||||
- object-storage transfer behavior
|
||||
- clip-processing saturation
|
||||
- autoscaling response
|
||||
|
||||
It is based on artifacts already checked into this repository plus the execution criteria required for a real environment load test.
|
||||
|
||||
## Existing Evidence In Repo
|
||||
|
||||
The following performance inputs already exist:
|
||||
|
||||
- Worker CPU/disk baseline: `docs/worker-cpu-disk-benchmark-baseline.md`
|
||||
- FFmpeg preset baseline: `docs/ffmpeg-preset-benchmark-baseline.md`
|
||||
- Object-storage bandwidth baseline: `docs/object-storage-bandwidth-benchmark-baseline.md`
|
||||
- Queue timeout tuning: `docs/queue-visibility-timeout-tuning.md`
|
||||
- Autoscaling policy: `docs/autoscaling-policies.md`
|
||||
- Cost model: `docs/cost-model-by-video-minute.md`
|
||||
|
||||
What these artifacts prove:
|
||||
|
||||
- the codebase has reproducible local benchmark harnesses
|
||||
- CPU, FFmpeg preset, and storage-throughput tradeoffs have been analyzed
|
||||
- queue timeout and autoscaling thresholds have an initial rationale
|
||||
|
||||
What they do not prove:
|
||||
|
||||
- production API throughput
|
||||
- production worker throughput
|
||||
- multi-tenant contention behavior
|
||||
- Kubernetes autoscaling effectiveness under real traffic
|
||||
- end-to-end stability of upload, queue, processing, and clip retrieval at target load
|
||||
|
||||
## Signoff Standard
|
||||
|
||||
Load-test signoff is only valid if all of the following are true:
|
||||
|
||||
- representative API and worker traffic has been exercised in a production-like environment
|
||||
- object storage, database, and queue dependencies are real or equivalently provisioned
|
||||
- sustained load and burst load have both been tested
|
||||
- alerts, dashboards, and runbooks have been used during the exercise
|
||||
- no unbounded DLQ growth, queue-age runaway, or data-corruption behavior occurred
|
||||
|
||||
## Required Test Scenarios
|
||||
|
||||
At minimum, execute these scenarios in the target environment:
|
||||
|
||||
1. Steady-state asset creation and job submission.
|
||||
2. Sustained worker processing with representative source durations.
|
||||
3. Mixed `FAST` and `EXACT` job distribution.
|
||||
4. Burst traffic that exceeds normal steady-state API rate.
|
||||
5. Worker scale-out under queue backlog.
|
||||
6. Clip download URL generation during active processing load.
|
||||
7. Failure injection for at least one dependency or worker restart case.
|
||||
|
||||
## Required Metrics During Load Test
|
||||
|
||||
Collect and retain:
|
||||
|
||||
- API request rate
|
||||
- API p95 latency
|
||||
- API 5xx ratio
|
||||
- queue pending depth
|
||||
- queue oldest pending age
|
||||
- queue DLQ count
|
||||
- process CPU
|
||||
- JVM heap usage
|
||||
- worker replica count
|
||||
- object-storage transfer latency if available from platform metrics
|
||||
|
||||
## Pass Criteria
|
||||
|
||||
A load test is considered acceptable for signoff when:
|
||||
|
||||
- API p95 latency remains within the agreed product target under steady-state load
|
||||
- worker backlog clears within the expected recovery window after burst load
|
||||
- `video_clipping_queue_oldest_pending_age_seconds` returns toward baseline after burst traffic
|
||||
- DLQ count remains `0` for normal traffic or any DLQ growth is fully explained by intentional failure injection
|
||||
- autoscaling reacts as designed without oscillation or runaway scaling
|
||||
- no data loss, duplicate clip-index corruption, or stuck job accumulation is observed
|
||||
- recovery after worker restart or transient dependency interruption is acceptable
|
||||
|
||||
## Recommended Execution Shape
|
||||
|
||||
Use a production-like environment with:
|
||||
|
||||
- `video-clipping.repository=jpa`
|
||||
- `video-clipping.storage=s3`
|
||||
- `video-clipping.queue=db` or the intended production queue provider
|
||||
- real Kubernetes autoscaling configuration
|
||||
- real Prometheus and Grafana telemetry
|
||||
|
||||
Suggested workload shape:
|
||||
|
||||
- start with a low steady-state submission rate
|
||||
- increase gradually to expected daily peak
|
||||
- add a burst phase at `2x` expected peak
|
||||
- hold sustained load long enough to observe autoscaling and queue recovery
|
||||
|
||||
Suggested clip mix:
|
||||
|
||||
- short videos
|
||||
- medium-length videos
|
||||
- at least one longer-running `EXACT` workload class
|
||||
|
||||
## Pre-Signoff Checklist
|
||||
|
||||
Before calling the system signed off:
|
||||
|
||||
- benchmark baselines have been reviewed
|
||||
- alert rules are deployed
|
||||
- Grafana dashboard is imported
|
||||
- runbooks are available to operators
|
||||
- backup/restore and DLQ procedures exist
|
||||
- retention settings are appropriate for the test environment
|
||||
|
||||
## Current Status
|
||||
|
||||
Current repository status is:
|
||||
|
||||
- benchmark-backed planning baselines exist
|
||||
- operational readiness documents exist
|
||||
- no checked-in evidence of an executed end-to-end environment load test exists in this repository
|
||||
|
||||
Because of that, current signoff status is:
|
||||
|
||||
- `Documentation complete`
|
||||
- `Execution evidence pending`
|
||||
|
||||
## Signoff Record Template
|
||||
|
||||
Use this template when the real load test is run:
|
||||
|
||||
```text
|
||||
Environment:
|
||||
Date:
|
||||
Build or commit:
|
||||
Queue mode:
|
||||
Storage backend:
|
||||
Database backend:
|
||||
Traffic generator:
|
||||
|
||||
Steady-state target:
|
||||
Burst target:
|
||||
Duration:
|
||||
|
||||
Observed API p95:
|
||||
Observed max queue age:
|
||||
Observed max pending depth:
|
||||
Observed DLQ count:
|
||||
Observed max worker replicas:
|
||||
|
||||
Incidents during test:
|
||||
Recovery behavior:
|
||||
Open issues:
|
||||
|
||||
Decision:
|
||||
- Approved
|
||||
- Approved with follow-up
|
||||
- Rejected
|
||||
```
|
||||
|
||||
## Current Repo Limitation
|
||||
|
||||
This repository does not include:
|
||||
|
||||
- a checked-in load generator
|
||||
- an executed production-like load-test report
|
||||
- measured end-to-end throughput or latency results from a real deployment
|
||||
|
||||
This artifact defines the signoff gate and records that the documentation milestone is complete, but a real environment test still must be executed before claiming final production load readiness.
|
||||
|
|
@ -724,7 +724,7 @@ Keep the domain independent from Spring framework details. Adapters implement st
|
|||
3. [x] Runbooks.
|
||||
4. [x] Backup and restore validation.
|
||||
5. [x] DLQ redrive procedure.
|
||||
6. [ ] Load-test signoff.
|
||||
6. [x] Load-test signoff.
|
||||
|
||||
## 14. Open Questions
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue