161 lines
4.8 KiB
Markdown
161 lines
4.8 KiB
Markdown
# Local CV Visual Analysis Guide
|
|
|
|
This service can use a local computer-vision worker for highlight visual analysis.
|
|
|
|
## Default Behavior
|
|
|
|
The packaged `application.yml` defaults to `local-cv` with heuristic fallback enabled:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
visual-analysis:
|
|
provider: local-cv
|
|
fallback-to-heuristic: true
|
|
```
|
|
|
|
This keeps the app runnable when the local worker is unavailable, but visual analysis is stronger when the worker is running.
|
|
|
|
## Enable A Local CV Worker
|
|
|
|
Run a local worker that accepts JSON over HTTP, then start this Spring service with:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
visual-analysis:
|
|
provider: local-cv
|
|
endpoint: http://127.0.0.1:8091/v1/analyze-visuals
|
|
timeout-ms: 30000
|
|
fallback-to-heuristic: true
|
|
local-cv-worker:
|
|
auto-start: false
|
|
script: ./tools/run_local_cv_worker.sh
|
|
startup-wait-ms: 0
|
|
```
|
|
|
|
Equivalent environment variables:
|
|
|
|
```bash
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
|
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=false
|
|
VIDEO_EDITING_LOCAL_CV_WORKER_SCRIPT=./tools/run_local_cv_worker.sh
|
|
VIDEO_EDITING_LOCAL_CV_WORKER_STARTUP_WAIT_MS=0
|
|
```
|
|
|
|
This repository includes an optional starter worker:
|
|
|
|
```bash
|
|
tools/run_local_cv_worker.sh
|
|
```
|
|
|
|
The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
|
|
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.
|
|
|
|
Default model behavior:
|
|
|
|
- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
|
|
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
|
|
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
|
|
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.
|
|
|
|
Health check:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8091/health
|
|
```
|
|
|
|
## Spring-Managed Worker Startup
|
|
|
|
The Spring application can start the local CV worker for local runs:
|
|
|
|
```bash
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv \
|
|
VIDEO_EDITING_LOCAL_CV_WORKER_AUTO_START=true \
|
|
mvn spring-boot:run
|
|
```
|
|
|
|
When `auto-start` is enabled, Spring starts `tools/run_local_cv_worker.sh`, derives `LOCAL_CV_HOST` and `LOCAL_CV_PORT` from the configured visual-analysis endpoint, streams worker output into application logs, and stops the worker process during application shutdown.
|
|
|
|
Keep `auto-start` disabled in production unless the app host is intended to own the Python worker lifecycle. The first run can take time because the script creates `.venv-local-cv`, installs Python dependencies, and may download/cache the configured YOLO model.
|
|
|
|
## Worker Request Contract
|
|
|
|
The Spring service sends:
|
|
|
|
```json
|
|
{
|
|
"source": {
|
|
"clipId": "porsche-drive",
|
|
"sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
|
|
"durationSeconds": 42.0,
|
|
"videoCodec": "h264",
|
|
"audioCodec": "aac",
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"frameRate": 30.0
|
|
},
|
|
"thumbnails": [
|
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
|
],
|
|
"shotSegments": [
|
|
{
|
|
"shotId": "shot_0001",
|
|
"startSeconds": 0.0,
|
|
"endSeconds": 8.0,
|
|
"durationSeconds": 8.0,
|
|
"representativeTimestampSeconds": 4.0
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Worker Response Contract
|
|
|
|
The worker must return:
|
|
|
|
```json
|
|
{
|
|
"clipId": "porsche-drive",
|
|
"blurScore": 0.81,
|
|
"exposureScore": 0.67,
|
|
"motionScore": 0.73,
|
|
"compositionScore": 0.79,
|
|
"facePresence": "faces_detected",
|
|
"objectLabels": [
|
|
{
|
|
"label": "car",
|
|
"confidence": 0.91,
|
|
"source": "yolo"
|
|
},
|
|
{
|
|
"label": "luxury sports car",
|
|
"confidence": 0.84,
|
|
"source": "clip"
|
|
}
|
|
],
|
|
"representativeThumbnails": [
|
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
|
],
|
|
"analysisMethod": "local_cv_yolo_clip_mediapipe"
|
|
}
|
|
```
|
|
|
|
Scores must be normalized from `0.0` to `1.0`.
|
|
|
|
## Recommended Local Model Split
|
|
|
|
- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
|
|
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
|
|
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
|
|
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.
|
|
|
|
## Runtime Behavior
|
|
|
|
When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.
|
|
|
|
When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.
|