3.3 KiB

Raw Blame History

Local CV Visual Analysis Guide

This service can use a local computer-vision worker for highlight visual analysis.

Default Behavior

By default the service uses the built-in heuristic provider:

video-clipping:
  editing:
    visual-analysis:
      provider: heuristic

This keeps the app runnable without Python, GPU drivers, model downloads, or a separate service.

Enable A Local CV Worker

Run a local worker that accepts JSON over HTTP, then start this Spring service with:

video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      endpoint: http://127.0.0.1:8091/v1/analyze-visuals
      timeout-ms: 30000
      fallback-to-heuristic: true

Equivalent environment variables:

VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true

This repository includes an optional starter worker:

python3 -m venv .venv-local-cv
. .venv-local-cv/bin/activate
pip install fastapi uvicorn opencv-python

# Optional YOLO object detection:
pip install ultralytics
export LOCAL_CV_YOLO_MODEL=yolov8n.pt

uvicorn tools.local_cv_worker:app --host 127.0.0.1 --port 8091

Worker Request Contract

The Spring service sends:

{
  "source": {
    "clipId": "porsche-drive",
    "sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
    "durationSeconds": 42.0,
    "videoCodec": "h264",
    "audioCodec": "aac",
    "width": 1920,
    "height": 1080,
    "frameRate": 30.0
  },
  "thumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "shotSegments": [
    {
      "shotId": "shot_0001",
      "startSeconds": 0.0,
      "endSeconds": 8.0,
      "durationSeconds": 8.0,
      "representativeTimestampSeconds": 4.0
    }
  ]
}

Worker Response Contract

The worker must return:

{
  "clipId": "porsche-drive",
  "blurScore": 0.81,
  "exposureScore": 0.67,
  "motionScore": 0.73,
  "compositionScore": 0.79,
  "facePresence": "faces_detected",
  "objectLabels": [
    {
      "label": "car",
      "confidence": 0.91,
      "source": "yolo"
    },
    {
      "label": "luxury sports car",
      "confidence": 0.84,
      "source": "clip"
    }
  ],
  "representativeThumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "analysisMethod": "local_cv_yolo_clip_mediapipe"
}

Scores must be normalized from 0.0 to 1.0.

Recommended Local Model Split

Use YOLO for object detection such as car, person, food, dog, bottle, and scene objects.
Use CLIP or SigLIP for semantic labels such as luxury sports car, family birthday, or restaurant dish.
Use MediaPipe or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.

Runtime Behavior

When fallback-to-heuristic=true, the service logs event=local_cv_visual_analysis_fallback and writes heuristic visual analysis if the CV worker is down or returns an error.

When fallback-to-heuristic=false, local CV failures fail the source analysis and the source video is rejected by the scheduler.

3.3 KiB Raw Blame History