4.8 KiB

Raw Blame History

Local CV Visual Analysis Guide

This service can use a local computer-vision worker for highlight visual analysis.

Default Behavior

The packaged application.yml defaults to local-cv with heuristic fallback enabled:

video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      fallback-to-heuristic: true

This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.

Enable A Local CV Worker

Run a local worker that accepts JSON over HTTP, then start this Spring service with:

video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      endpoint: http://127.0.0.1:8091/v1/analyze-visuals
      timeout-ms: 30000
      fallback-to-heuristic: true
      local-cv-worker:
        auto-start: false
        script: ./tools/run_local_cv_worker.sh
        startup-wait-ms: 0

Equivalent environment variables:

VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=0

This repository includes an optional starter worker:

tools/run_local_cv_worker.sh

The starter worker installs tools/local_cv_requirements.txt into .venv-local-cv. It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through ultralytics for object detection.

Default model behavior:

LOCAL_CV_YOLO_MODEL defaults to yolov8n.pt.
Ultralytics downloads/caches yolov8n.pt on first use if it is not already present.
Set LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt to use local weights.
Set LOCAL_CV_DISABLE_YOLO=true to run only OpenCV-based analysis.

Health check:

curl http://127.0.0.1:8091/health

Spring-Managed Worker Startup

The Spring application can start the local CV worker for local runs:

VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
mvn spring-boot:run

When auto-start is enabled, Spring starts tools/run_local_cv_worker.sh, derives LOCAL_CV_HOST and LOCAL_CV_PORT from the configured visual-analysis endpoint, streams worker output into application logs, and stops the worker process during application shutdown.

Keep auto-start disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates .venv-local-cv, installs Python dependencies, and may download/cache the configured YOLO model.

Worker Request Contract

The Spring service sends:

{
  "source": {
    "clipId": "porsche-drive",
    "sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
    "durationSeconds": 42.0,
    "videoCodec": "h264",
    "audioCodec": "aac",
    "width": 1920,
    "height": 1080,
    "frameRate": 30.0
  },
  "thumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "shotSegments": [
    {
      "shotId": "shot_0001",
      "startSeconds": 0.0,
      "endSeconds": 8.0,
      "durationSeconds": 8.0,
      "representativeTimestampSeconds": 4.0
    }
  ]
}

Worker Response Contract

The worker must return:

{
  "clipId": "porsche-drive",
  "blurScore": 0.81,
  "exposureScore": 0.67,
  "motionScore": 0.73,
  "compositionScore": 0.79,
  "facePresence": "faces_detected",
  "objectLabels": [
    {
      "label": "car",
      "confidence": 0.91,
      "source": "yolo"
    },
    {
      "label": "luxury sports car",
      "confidence": 0.84,
      "source": "clip"
    }
  ],
  "representativeThumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "analysisMethod": "local_cv_yolo_clip_mediapipe"
}

Scores must be normalized from 0.0 to 1.0.

Recommended Local Model Split

Use YOLO for object detection such as car, person, food, dog, bottle, and scene objects.
Use CLIP or SigLIP for semantic labels such as luxury sports car, family birthday, or restaurant dish.
Use MediaPipe or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.

Runtime Behavior

When fallback-to-heuristic=true, the service logs event=local_cv_visual_analysis_fallback and writes heuristic visual analysis if the CV worker is down or returns an error.

When fallback-to-heuristic=false, local CV failures fail the source analysis and the source video is rejected by the scheduler.

4.8 KiB Raw Blame History