# Local CV Visual Analysis Guide

This service can use a local computer-vision worker for highlight visual analysis.

## Default Behavior

By default the service uses the built-in heuristic provider:

```yaml
video-clipping:
  editing:
    visual-analysis:
      provider: heuristic
```

This keeps the app runnable without Python, GPU drivers, model downloads, or a separate service.

## Enable A Local CV Worker

Run a local worker that accepts JSON over HTTP, then start this Spring service with:

```yaml
video-clipping:
  editing:
    visual-analysis:
      provider: local-cv
      endpoint: http://127.0.0.1:8091/v1/analyze-visuals
      timeout-ms: 30000
      fallback-to-heuristic: true
```

Equivalent environment variables:

```bash
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
```

This repository includes an optional starter worker:

```bash
tools/run_local_cv_worker.sh
```

The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.

Default model behavior:

- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.

Health check:

```bash
curl http://127.0.0.1:8091/health
```

## Worker Request Contract

The Spring service sends:

```json
{
  "source": {
    "clipId": "porsche-drive",
    "sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
    "durationSeconds": 42.0,
    "videoCodec": "h264",
    "audioCodec": "aac",
    "width": 1920,
    "height": 1080,
    "frameRate": 30.0
  },
  "thumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "shotSegments": [
    {
      "shotId": "shot_0001",
      "startSeconds": 0.0,
      "endSeconds": 8.0,
      "durationSeconds": 8.0,
      "representativeTimestampSeconds": 4.0
    }
  ]
}
```

## Worker Response Contract

The worker must return:

```json
{
  "clipId": "porsche-drive",
  "blurScore": 0.81,
  "exposureScore": 0.67,
  "motionScore": 0.73,
  "compositionScore": 0.79,
  "facePresence": "faces_detected",
  "objectLabels": [
    {
      "label": "car",
      "confidence": 0.91,
      "source": "yolo"
    },
    {
      "label": "luxury sports car",
      "confidence": 0.84,
      "source": "clip"
    }
  ],
  "representativeThumbnails": [
    "output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
  ],
  "analysisMethod": "local_cv_yolo_clip_mediapipe"
}
```

Scores must be normalized from `0.0` to `1.0`.

## Recommended Local Model Split

- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.

## Runtime Behavior

When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.

When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.