video_editing_poc/docs/alerts.md

Alerts

This document captures the alerting artifact for operational readiness plan item Alerts.

Current metric surface:

• Prometheus scrapes /actuator/prometheus.
• Database-backed queue deployments expose:
• video_clipping_queue_pending
• video_clipping_queue_processing
• video_clipping_queue_dlq
• video_clipping_queue_oldest_pending_age_seconds
• Standard Spring Boot and JVM metrics include:
• http_server_requests_seconds_count
• process_cpu_usage
• system_cpu_usage
• jvm_memory_used_bytes
• up

Alert coverage implemented now:

• Queue age too high.
• DLQ messages present.
• API 5xx rate.
• Process unavailable.
• Sustained process CPU saturation.

Alert rules:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: video-clipping-service
  labels:
    app.kubernetes.io/name: video-clipping-service
spec:
  groups:
    - name: video-clipping-service
      rules:
        - alert: VideoClippingQueueOldestPendingHigh
          expr: max(video_clipping_queue_oldest_pending_age_seconds) > 120
          for: 10m
          labels:
            severity: warning
            service: video-clipping-service
          annotations:
            summary: Video clipping queue age is elevated
            description: Oldest pending queue message has been older than 120 seconds for 10 minutes.

        - alert: VideoClippingQueueOldestPendingCritical
          expr: max(video_clipping_queue_oldest_pending_age_seconds) > 300
          for: 15m
          labels:
            severity: critical
            service: video-clipping-service
          annotations:
            summary: Video clipping queue age is critically high
            description: Oldest pending queue message has been older than 300 seconds for 15 minutes.

        - alert: VideoClippingDlqPresent
          expr: sum(video_clipping_queue_dlq) > 0
          for: 5m
          labels:
            severity: critical
            service: video-clipping-service
          annotations:
            summary: Video clipping DLQ contains messages
            description: At least one clip job message has remained in the dead-letter queue for 5 minutes.

        - alert: VideoClippingApi5xxRateHigh
          expr: |
            (
              sum(rate(http_server_requests_seconds_count{uri!~"/actuator.*",status=~"5.."}[5m]))
              /
              clamp_min(sum(rate(http_server_requests_seconds_count{uri!~"/actuator.*"}[5m])), 0.001)
            ) > 0.05            
          for: 10m
          labels:
            severity: warning
            service: video-clipping-service
          annotations:
            summary: Video clipping API 5xx ratio is high
            description: More than 5 percent of non-Actuator API requests have returned 5xx for 10 minutes.

        - alert: VideoClippingDownloadUrl5xxRateHigh
          expr: |
            (
              sum(rate(http_server_requests_seconds_count{uri=~".*/download-url",status=~"5.."}[5m]))
              /
              clamp_min(sum(rate(http_server_requests_seconds_count{uri=~".*/download-url"}[5m])), 0.001)
            ) > 0.02            
          for: 10m
          labels:
            severity: warning
            service: video-clipping-service
          annotations:
            summary: Video clipping signed download URL failures are elevated
            description: More than 2 percent of signed clip download URL requests have returned 5xx for 10 minutes.

        - alert: VideoClippingInstanceDown
          expr: up{job=~".*video-clipping.*"} == 0
          for: 5m
          labels:
            severity: critical
            service: video-clipping-service
          annotations:
            summary: Video clipping target is down
            description: Prometheus has failed to scrape a video-clipping-service target for 5 minutes.

        - alert: VideoClippingProcessCpuHigh
          expr: max(process_cpu_usage{application="video-clipping-service"}) > 0.85
          for: 15m
          labels:
            severity: warning
            service: video-clipping-service
          annotations:
            summary: Video clipping process CPU is saturated
            description: Process CPU usage has remained above 85 percent for 15 minutes.

Threshold rationale:

• 120s queue age aligns with the red dashboard threshold and represents backlog outside normal steady-state behavior.
• 300s queue age is long enough to page because it risks breaching throughput expectations before the 15 minute visibility timeout becomes relevant.
• Any nonzero DLQ count is actionable because failed clip jobs require investigation or redrive.
• A sustained 5% API 5xx ratio is large enough to avoid paging on isolated errors while still catching broken releases or dependencies.
• A stricter 2% threshold is used for download-url because that endpoint should usually be lightweight and storage-signing failures directly block clip retrieval.
• 85% sustained process CPU is a warning signal for worker saturation; it should be correlated with queue backlog before scaling policy changes.

Operational notes:

• If production uses SQS instead of the database queue, replace the queue-depth and queue-age queries with native cloud queue metrics while keeping the same alert semantics.
• The up{job=~".*video-clipping.*"} matcher should be narrowed to the actual Prometheus scrape job names used in the target environment.
• The process_cpu_usage{application="video-clipping-service"} selector assumes the Micrometer application label is present from spring.application.name. If the label set differs in production, adjust the selector accordingly.
• The download-url matcher depends on the uri label shape exported by Spring Boot HTTP metrics. Verify that the label resolves to the route template used by the deployment.

Current instrumentation gaps:

• Job failure rate above threshold is not directly alertable yet because the service does not currently export a counter for failed clip jobs.
• Worker disk usage high is not directly exposed by this service; alert it from Kubernetes or node metrics such as container filesystem or ephemeral-volume usage.
• FFmpeg exit-code spike is not directly alertable yet because FFmpeg outcome counters by exit class are not exported.
• Object storage errors are only indirectly visible through API/worker failures today; a dedicated storage operation failure counter is still needed.
• Signed URL generation failures are only approximated via the download-url endpoint 5xx alert; a dedicated business metric would make this more precise.

Recommended next instrumentation when these alerts need to become first-class:

• Add counters for clip-job outcomes such as video.clipping.jobs.started, video.clipping.jobs.succeeded, and video.clipping.jobs.failed.
• Add counters for storage operation failures by operation type.
• Add counters for signed URL generation success and failure.
• Add counters for FFmpeg failure categories or exit-code classes.