video_editing_poc/docs/autoscaling-policies.md

Autoscaling Policies

This document captures the implementation and rollout guidance for performance plan item Autoscaling policies.

Exposed metrics:

• /actuator/prometheus is enabled for scraping.
• video.clipping.queue.pending
• video.clipping.queue.processing
• video.clipping.queue.dlq
• video.clipping.queue.oldest.pending.age.seconds

Worker policy:

• Start with one FFmpeg process per worker pod.
• Use minReplicas: 1 and maxReplicas: 20.
• Scale out when either queue depth or queue age indicates backlog:
• video_clipping_queue_pending > 5 per worker target.
• video_clipping_queue_oldest_pending_age_seconds > 30.
• Keep CPU as a secondary safeguard with average utilization near 70%.

API policy:

• Use a separate HPA from workers.
• Scale API pods on CPU and request latency or concurrent requests if those metrics are available from the ingress/controller layer.
• Start with minReplicas: 2, maxReplicas: 10, and CPU target near 60%.

Sizing rationale:

• The worker CPU benchmark and FFmpeg preset benchmark show clip generation is CPU-bound once source staging is local.
• The object-storage bandwidth benchmark shows throughput degradation becomes material below 50 MiB/s, so queue-age should remain part of the worker policy even when CPU is not yet saturated.
• The visibility-timeout default is conservative, so scaling should respond to queue buildup before stale-claim recovery becomes common.

Suggested worker HPA or KEDA shape:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: video-clipping-worker
spec:
  scaleTargetRef:
    name: video-clipping-worker
  minReplicaCount: 1
  maxReplicaCount: 20
  cooldownPeriod: 120
  pollingInterval: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        metricName: video_clipping_queue_pending
        threshold: "5"
        query: sum(video_clipping_queue_pending)
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        metricName: video_clipping_queue_oldest_pending_age_seconds
        threshold: "30"
        query: max(video_clipping_queue_oldest_pending_age_seconds)

Operational note:

• If production uses SQS instead of the database queue, keep the same policy shape but source queue depth and age from native cloud metrics instead of the local Micrometer gauges.