video_editing_poc/docs/load-test-signoff.md

# Load-Test Signoff

This document captures the operational readiness artifact for plan item `Load-test signoff`.

## Scope

This signoff covers production-readiness expectations for:

- API request handling
- queue-backed worker throughput
- object-storage transfer behavior
- clip-processing saturation
- autoscaling response

It is based on artifacts already checked into this repository plus the execution criteria required for a real environment load test.

## Existing Evidence In Repo

The following performance inputs already exist:

- Worker CPU/disk baseline: `docs/worker-cpu-disk-benchmark-baseline.md`
- FFmpeg preset baseline: `docs/ffmpeg-preset-benchmark-baseline.md`
- Object-storage bandwidth baseline: `docs/object-storage-bandwidth-benchmark-baseline.md`
- Queue timeout tuning: `docs/queue-visibility-timeout-tuning.md`
- Autoscaling policy: `docs/autoscaling-policies.md`
- Cost model: `docs/cost-model-by-video-minute.md`

What these artifacts prove:

- the codebase has reproducible local benchmark harnesses
- CPU, FFmpeg preset, and storage-throughput tradeoffs have been analyzed
- queue timeout and autoscaling thresholds have an initial rationale

What they do not prove:

- production API throughput
- production worker throughput
- multi-tenant contention behavior
- Kubernetes autoscaling effectiveness under real traffic
- end-to-end stability of upload, queue, processing, and clip retrieval at target load

## Signoff Standard

Load-test signoff is only valid if all of the following are true:

- representative API and worker traffic has been exercised in a production-like environment
- object storage, database, and queue dependencies are real or equivalently provisioned
- sustained load and burst load have both been tested
- alerts, dashboards, and runbooks have been used during the exercise
- no unbounded DLQ growth, queue-age runaway, or data-corruption behavior occurred

## Required Test Scenarios

At minimum, execute these scenarios in the target environment:

1. Steady-state asset creation and job submission.
2. Sustained worker processing with representative source durations.
3. Mixed `FAST` and `EXACT` job distribution.
4. Burst traffic that exceeds normal steady-state API rate.
5. Worker scale-out under queue backlog.
6. Clip download URL generation during active processing load.
7. Failure injection for at least one dependency or worker restart case.

## Required Metrics During Load Test

Collect and retain:

- API request rate
- API p95 latency
- API 5xx ratio
- queue pending depth
- queue oldest pending age
- queue DLQ count
- process CPU
- JVM heap usage
- worker replica count
- object-storage transfer latency if available from platform metrics

## Pass Criteria

A load test is considered acceptable for signoff when:

- API p95 latency remains within the agreed product target under steady-state load
- worker backlog clears within the expected recovery window after burst load
- `video_clipping_queue_oldest_pending_age_seconds` returns toward baseline after burst traffic
- DLQ count remains `0` for normal traffic or any DLQ growth is fully explained by intentional failure injection
- autoscaling reacts as designed without oscillation or runaway scaling
- no data loss, duplicate clip-index corruption, or stuck job accumulation is observed
- recovery after worker restart or transient dependency interruption is acceptable

## Recommended Execution Shape

Use a production-like environment with:

- `video-clipping.repository=jpa`
- `video-clipping.storage=s3`
- `video-clipping.queue=db` or the intended production queue provider
- real Kubernetes autoscaling configuration
- real Prometheus and Grafana telemetry

Suggested workload shape:

- start with a low steady-state submission rate
- increase gradually to expected daily peak
- add a burst phase at `2x` expected peak
- hold sustained load long enough to observe autoscaling and queue recovery

Suggested clip mix:

- short videos
- medium-length videos
- at least one longer-running `EXACT` workload class

## Pre-Signoff Checklist

Before calling the system signed off:

- benchmark baselines have been reviewed
- alert rules are deployed
- Grafana dashboard is imported
- runbooks are available to operators
- backup/restore and DLQ procedures exist
- retention settings are appropriate for the test environment

## Current Status

Current repository status is:

- benchmark-backed planning baselines exist
- operational readiness documents exist
- no checked-in evidence of an executed end-to-end environment load test exists in this repository

Because of that, current signoff status is:

- `Documentation complete`
- `Execution evidence pending`

## Signoff Record Template

Use this template when the real load test is run:

```text
Environment:
Date:
Build or commit:
Queue mode:
Storage backend:
Database backend:
Traffic generator:

Steady-state target:
Burst target:
Duration:

Observed API p95:
Observed max queue age:
Observed max pending depth:
Observed DLQ count:
Observed max worker replicas:

Incidents during test:
Recovery behavior:
Open issues:

Decision:
- Approved
- Approved with follow-up
- Rejected
```

## Current Repo Limitation

This repository does not include:

- a checked-in load generator
- an executed production-like load-test report
- measured end-to-end throughput or latency results from a real deployment

This artifact defines the signoff gate and records that the documentation milestone is complete, but a real environment test still must be executed before claiming final production load readiness.