forked from jsl/video_editing_poc
66 lines
2.4 KiB
Markdown
66 lines
2.4 KiB
Markdown
# Autoscaling Policies
|
|
|
|
This document captures the implementation and rollout guidance for performance plan item `Autoscaling policies`.
|
|
|
|
Exposed metrics:
|
|
|
|
- `/actuator/prometheus` is enabled for scraping.
|
|
- `video.clipping.queue.pending`
|
|
- `video.clipping.queue.processing`
|
|
- `video.clipping.queue.dlq`
|
|
- `video.clipping.queue.oldest.pending.age.seconds`
|
|
|
|
Worker policy:
|
|
|
|
- Start with one FFmpeg process per worker pod.
|
|
- Use `minReplicas: 1` and `maxReplicas: 20`.
|
|
- Scale out when either queue depth or queue age indicates backlog:
|
|
- `video_clipping_queue_pending > 5` per worker target.
|
|
- `video_clipping_queue_oldest_pending_age_seconds > 30`.
|
|
- Keep CPU as a secondary safeguard with average utilization near `70%`.
|
|
|
|
API policy:
|
|
|
|
- Use a separate HPA from workers.
|
|
- Scale API pods on CPU and request latency or concurrent requests if those metrics are available from the ingress/controller layer.
|
|
- Start with `minReplicas: 2`, `maxReplicas: 10`, and CPU target near `60%`.
|
|
|
|
Sizing rationale:
|
|
|
|
- The worker CPU benchmark and FFmpeg preset benchmark show clip generation is CPU-bound once source staging is local.
|
|
- The object-storage bandwidth benchmark shows throughput degradation becomes material below `50 MiB/s`, so queue-age should remain part of the worker policy even when CPU is not yet saturated.
|
|
- The visibility-timeout default is conservative, so scaling should respond to queue buildup before stale-claim recovery becomes common.
|
|
|
|
Suggested worker HPA or KEDA shape:
|
|
|
|
```yaml
|
|
apiVersion: keda.sh/v1alpha1
|
|
kind: ScaledObject
|
|
metadata:
|
|
name: video-clipping-worker
|
|
spec:
|
|
scaleTargetRef:
|
|
name: video-clipping-worker
|
|
minReplicaCount: 1
|
|
maxReplicaCount: 20
|
|
cooldownPeriod: 120
|
|
pollingInterval: 15
|
|
triggers:
|
|
- type: prometheus
|
|
metadata:
|
|
serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
|
|
metricName: video_clipping_queue_pending
|
|
threshold: "5"
|
|
query: sum(video_clipping_queue_pending)
|
|
- type: prometheus
|
|
metadata:
|
|
serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
|
|
metricName: video_clipping_queue_oldest_pending_age_seconds
|
|
threshold: "30"
|
|
query: max(video_clipping_queue_oldest_pending_age_seconds)
|
|
```
|
|
|
|
Operational note:
|
|
|
|
- If production uses SQS instead of the database queue, keep the same policy shape but source queue depth and age from native cloud metrics instead of the local Micrometer gauges.
|