# Alerts This document captures the alerting artifact for operational readiness plan item `Alerts`. Current metric surface: - Prometheus scrapes `/actuator/prometheus`. - Database-backed queue deployments expose: - `video_clipping_queue_pending` - `video_clipping_queue_processing` - `video_clipping_queue_dlq` - `video_clipping_queue_oldest_pending_age_seconds` - Standard Spring Boot and JVM metrics include: - `http_server_requests_seconds_count` - `process_cpu_usage` - `system_cpu_usage` - `jvm_memory_used_bytes` - `up` Alert coverage implemented now: - Queue age too high. - DLQ messages present. - API 5xx rate. - Process unavailable. - Sustained process CPU saturation. Alert rules: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: video-clipping-service labels: app.kubernetes.io/name: video-clipping-service spec: groups: - name: video-clipping-service rules: - alert: VideoClippingQueueOldestPendingHigh expr: max(video_clipping_queue_oldest_pending_age_seconds) > 120 for: 10m labels: severity: warning service: video-clipping-service annotations: summary: Video clipping queue age is elevated description: Oldest pending queue message has been older than 120 seconds for 10 minutes. - alert: VideoClippingQueueOldestPendingCritical expr: max(video_clipping_queue_oldest_pending_age_seconds) > 300 for: 15m labels: severity: critical service: video-clipping-service annotations: summary: Video clipping queue age is critically high description: Oldest pending queue message has been older than 300 seconds for 15 minutes. - alert: VideoClippingDlqPresent expr: sum(video_clipping_queue_dlq) > 0 for: 5m labels: severity: critical service: video-clipping-service annotations: summary: Video clipping DLQ contains messages description: At least one clip job message has remained in the dead-letter queue for 5 minutes. - alert: VideoClippingApi5xxRateHigh expr: | ( sum(rate(http_server_requests_seconds_count{uri!~"/actuator.*",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{uri!~"/actuator.*"}[5m])), 0.001) ) > 0.05 for: 10m labels: severity: warning service: video-clipping-service annotations: summary: Video clipping API 5xx ratio is high description: More than 5 percent of non-Actuator API requests have returned 5xx for 10 minutes. - alert: VideoClippingDownloadUrl5xxRateHigh expr: | ( sum(rate(http_server_requests_seconds_count{uri=~".*/download-url",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{uri=~".*/download-url"}[5m])), 0.001) ) > 0.02 for: 10m labels: severity: warning service: video-clipping-service annotations: summary: Video clipping signed download URL failures are elevated description: More than 2 percent of signed clip download URL requests have returned 5xx for 10 minutes. - alert: VideoClippingInstanceDown expr: up{job=~".*video-clipping.*"} == 0 for: 5m labels: severity: critical service: video-clipping-service annotations: summary: Video clipping target is down description: Prometheus has failed to scrape a video-clipping-service target for 5 minutes. - alert: VideoClippingProcessCpuHigh expr: max(process_cpu_usage{application="video-clipping-service"}) > 0.85 for: 15m labels: severity: warning service: video-clipping-service annotations: summary: Video clipping process CPU is saturated description: Process CPU usage has remained above 85 percent for 15 minutes. ``` Threshold rationale: - `120s` queue age aligns with the red dashboard threshold and represents backlog outside normal steady-state behavior. - `300s` queue age is long enough to page because it risks breaching throughput expectations before the `15` minute visibility timeout becomes relevant. - Any nonzero DLQ count is actionable because failed clip jobs require investigation or redrive. - A sustained `5%` API 5xx ratio is large enough to avoid paging on isolated errors while still catching broken releases or dependencies. - A stricter `2%` threshold is used for `download-url` because that endpoint should usually be lightweight and storage-signing failures directly block clip retrieval. - `85%` sustained process CPU is a warning signal for worker saturation; it should be correlated with queue backlog before scaling policy changes. Operational notes: - If production uses SQS instead of the database queue, replace the queue-depth and queue-age queries with native cloud queue metrics while keeping the same alert semantics. - The `up{job=~".*video-clipping.*"}` matcher should be narrowed to the actual Prometheus scrape job names used in the target environment. - The `process_cpu_usage{application="video-clipping-service"}` selector assumes the Micrometer `application` label is present from `spring.application.name`. If the label set differs in production, adjust the selector accordingly. - The `download-url` matcher depends on the `uri` label shape exported by Spring Boot HTTP metrics. Verify that the label resolves to the route template used by the deployment. Current instrumentation gaps: - Job failure rate above threshold is not directly alertable yet because the service does not currently export a counter for failed clip jobs. - Worker disk usage high is not directly exposed by this service; alert it from Kubernetes or node metrics such as container filesystem or ephemeral-volume usage. - FFmpeg exit-code spike is not directly alertable yet because FFmpeg outcome counters by exit class are not exported. - Object storage errors are only indirectly visible through API/worker failures today; a dedicated storage operation failure counter is still needed. - Signed URL generation failures are only approximated via the `download-url` endpoint 5xx alert; a dedicated business metric would make this more precise. Recommended next instrumentation when these alerts need to become first-class: - Add counters for clip-job outcomes such as `video.clipping.jobs.started`, `video.clipping.jobs.succeeded`, and `video.clipping.jobs.failed`. - Add counters for storage operation failures by operation type. - Add counters for signed URL generation success and failure. - Add counters for FFmpeg failure categories or exit-code classes.