3.6 KiB

Raw Blame History

Local CV Visual Analysis Guide

This service can use a local computer-vision worker for highlight visual analysis.

Default Behavior

By default the service uses the built-in heuristic provider:

video-clipping:
  editing:
    visual-analysis:
      provider: heuristic

This keeps the app runnable without Python, GPU drivers, model downloads, or a separate service.

Enable A Local CV Worker

Run a local worker that accepts JSON over HTTP, then start this Spring service with:

video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      endpoint: http://127.0.0.1:8091/v1/analyze-visuals
      timeout-ms: 30000
      fallback-to-heuristic: true

Equivalent environment variables:

VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true

This repository includes an optional starter worker:

tools/run_local_cv_worker.sh

The starter worker installs tools/local_cv_requirements.txt into .venv-local-cv. It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through ultralytics for object detection.

Default model behavior:

LOCAL_CV_YOLO_MODEL defaults to yolov8n.pt.
Ultralytics downloads/caches yolov8n.pt on first use if it is not already present.
Set LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt to use local weights.
Set LOCAL_CV_DISABLE_YOLO=true to run only OpenCV-based analysis.

Health check:

curl http://127.0.0.1:8091/health

Worker Request Contract

The Spring service sends:

{
  "source": {
    "clipId": "porsche-drive",
    "sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
    "durationSeconds": 42.0,
    "videoCodec": "h264",
    "audioCodec": "aac",
    "width": 1920,
    "height": 1080,
    "frameRate": 30.0
  },
  "thumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "shotSegments": [
    {
      "shotId": "shot_0001",
      "startSeconds": 0.0,
      "endSeconds": 8.0,
      "durationSeconds": 8.0,
      "representativeTimestampSeconds": 4.0
    }
  ]
}

Worker Response Contract

The worker must return:

{
  "clipId": "porsche-drive",
  "blurScore": 0.81,
  "exposureScore": 0.67,
  "motionScore": 0.73,
  "compositionScore": 0.79,
  "facePresence": "faces_detected",
  "objectLabels": [
    {
      "label": "car",
      "confidence": 0.91,
      "source": "yolo"
    },
    {
      "label": "luxury sports car",
      "confidence": 0.84,
      "source": "clip"
    }
  ],
  "representativeThumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "analysisMethod": "local_cv_yolo_clip_mediapipe"
}

Scores must be normalized from 0.0 to 1.0.

Recommended Local Model Split

Use YOLO for object detection such as car, person, food, dog, bottle, and scene objects.
Use CLIP or SigLIP for semantic labels such as luxury sports car, family birthday, or restaurant dish.
Use MediaPipe or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.

Runtime Behavior

When fallback-to-heuristic=true, the service logs event=local_cv_visual_analysis_fallback and writes heuristic visual analysis if the CV worker is down or returns an error.

When fallback-to-heuristic=false, local CV failures fail the source analysis and the source video is rejected by the scheduler.

3.6 KiB Raw Blame History