video_editing_poc/docs/local-cv-visual-analysis-guide.md

# Local CV Visual Analysis Guide

This service can use a local computer-vision worker for highlight visual analysis.

## Default Behavior

The packaged `application.yml` defaults to `local-cv` with heuristic fallback enabled:

```yaml
video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      fallback-to-heuristic: true
```

This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.

## Enable A Local CV Worker

Run a local worker that accepts JSON over HTTP, then start this Spring service with:

```yaml
video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      endpoint: http://127.0.0.1:8091/v1/analyze-visuals
      timeout-ms: 30000
      fallback-to-heuristic: true
      local-cv-worker:
        auto-start: true
        script: ./tools/run_local_cv_worker.sh
        startup-wait-ms: 120000
        health-path: /health
        health-check-interval-ms: 1000
```

Equivalent environment variables:

```bash
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=120000
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_PATH=/health
VIDEO_EDITING_LOCAL_CV_WORKER_HEALTH_CHECK_INTERVAL_MS=1000
```

This repository includes an optional starter worker:

```bash
tools/run_local_cv_worker.sh
```

The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.

Default model behavior:

- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.

Health check:

```bash
curl http://127.0.0.1:8091/health
```

## Preload Dependencies And Model

Do this once before starting the Spring service if you do not want dependency installation or YOLO model download to happen during application startup:

```bash
LOCAL_CV_PRELOAD_ONLY=true tools/run_local_cv_worker.sh
```

This creates `.venv-local-cv`, installs `tools/local_cv_requirements.txt`, and downloads/caches the configured YOLO model. After that, Spring-managed startup should be much faster.

Do not commit `.venv-local-cv`, Python wheels, Torch binaries, or YOLO weights into source control. They are large, platform-specific runtime artifacts. For production, prefer a prebuilt container image or a VM/bootstrap step that runs the preload command during deployment.

## Spring-Managed Worker Startup

The Spring application can start the local CV worker for local runs:

```bash
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
mvn spring-boot:run
```

When `auto-start` is enabled, Spring starts `tools/run_local_cv_worker.sh`, derives `LOCAL_CV_HOST` and `LOCAL_CV_PORT` from the configured visual-analysis endpoint, polls the configured health endpoint until it returns HTTP 2xx, streams worker output into application logs, and stops the worker process during application shutdown.

Keep `auto-start` disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates `.venv-local-cv`, installs Python dependencies, and may download/cache the configured YOLO model. Use the preload command or a prebuilt image to avoid that startup cost.

## Worker Request Contract

The Spring service sends:

```json
{
  "source": {
    "clipId": "porsche-drive",
    "sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
    "durationSeconds": 42.0,
    "videoCodec": "h264",
    "audioCodec": "aac",
    "width": 1920,
    "height": 1080,
    "frameRate": 30.0
  },
  "thumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "shotSegments": [
    {
      "shotId": "shot_0001",
      "startSeconds": 0.0,
      "endSeconds": 8.0,
      "durationSeconds": 8.0,
      "representativeTimestampSeconds": 4.0
    }
  ]
}
```

## Worker Response Contract

The worker must return:

```json
{
  "clipId": "porsche-drive",
  "blurScore": 0.81,
  "exposureScore": 0.67,
  "motionScore": 0.73,
  "compositionScore": 0.79,
  "facePresence": "faces_detected",
  "objectLabels": [
    {
      "label": "car",
      "confidence": 0.91,
      "source": "yolo"
    },
    {
      "label": "luxury sports car",
      "confidence": 0.84,
      "source": "clip"
    }
  ],
  "representativeThumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "analysisMethod": "local_cv_yolo_clip_mediapipe"
}
```

Scores must be normalized from `0.0` to `1.0`.

## Recommended Local Model Split

- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.

## Runtime Behavior

When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.

When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.