forked from jsl/video_editing_poc
139 lines
3.6 KiB
Markdown
139 lines
3.6 KiB
Markdown
# Local CV Visual Analysis Guide
|
|
|
|
This service can use a local computer-vision worker for highlight visual analysis.
|
|
|
|
## Default Behavior
|
|
|
|
By default the service uses the built-in heuristic provider:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
visual-analysis:
|
|
provider: heuristic
|
|
```
|
|
|
|
This keeps the app runnable without Python, GPU drivers, model downloads, or a separate service.
|
|
|
|
## Enable A Local CV Worker
|
|
|
|
Run a local worker that accepts JSON over HTTP, then start this Spring service with:
|
|
|
|
```yaml
|
|
video-clipping:
|
|
editing:
|
|
visual-analysis:
|
|
provider: local-cv
|
|
endpoint: http://127.0.0.1:8091/v1/analyze-visuals
|
|
timeout-ms: 30000
|
|
fallback-to-heuristic: true
|
|
```
|
|
|
|
Equivalent environment variables:
|
|
|
|
```bash
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_PROVIDER=local-cv
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_ENDPOINT=http://127.0.0.1:8091/v1/analyze-visuals
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_TIMEOUT_MS=30000
|
|
VIDEO_EDITING_VISUAL_ANALYSIS_FALLBACK_TO_HEURISTIC=true
|
|
```
|
|
|
|
This repository includes an optional starter worker:
|
|
|
|
```bash
|
|
tools/run_local_cv_worker.sh
|
|
```
|
|
|
|
The starter worker installs `tools/local_cv_requirements.txt` into `.venv-local-cv`.
|
|
It uses OpenCV for blur, exposure, and face-presence signals. It uses YOLO through `ultralytics` for object detection.
|
|
|
|
Default model behavior:
|
|
|
|
- `LOCAL_CV_YOLO_MODEL` defaults to `yolov8n.pt`.
|
|
- Ultralytics downloads/caches `yolov8n.pt` on first use if it is not already present.
|
|
- Set `LOCAL_CV_YOLO_MODEL=/absolute/path/to/model.pt` to use local weights.
|
|
- Set `LOCAL_CV_DISABLE_YOLO=true` to run only OpenCV-based analysis.
|
|
|
|
Health check:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8091/health
|
|
```
|
|
|
|
## Worker Request Contract
|
|
|
|
The Spring service sends:
|
|
|
|
```json
|
|
{
|
|
"source": {
|
|
"clipId": "porsche-drive",
|
|
"sourcePath": "output/highlight-projects/porsche-drive/source/porsche.mp4",
|
|
"durationSeconds": 42.0,
|
|
"videoCodec": "h264",
|
|
"audioCodec": "aac",
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"frameRate": 30.0
|
|
},
|
|
"thumbnails": [
|
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
|
],
|
|
"shotSegments": [
|
|
{
|
|
"shotId": "shot_0001",
|
|
"startSeconds": 0.0,
|
|
"endSeconds": 8.0,
|
|
"durationSeconds": 8.0,
|
|
"representativeTimestampSeconds": 4.0
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Worker Response Contract
|
|
|
|
The worker must return:
|
|
|
|
```json
|
|
{
|
|
"clipId": "porsche-drive",
|
|
"blurScore": 0.81,
|
|
"exposureScore": 0.67,
|
|
"motionScore": 0.73,
|
|
"compositionScore": 0.79,
|
|
"facePresence": "faces_detected",
|
|
"objectLabels": [
|
|
{
|
|
"label": "car",
|
|
"confidence": 0.91,
|
|
"source": "yolo"
|
|
},
|
|
{
|
|
"label": "luxury sports car",
|
|
"confidence": 0.84,
|
|
"source": "clip"
|
|
}
|
|
],
|
|
"representativeThumbnails": [
|
|
"output/highlight-projects/porsche-drive/analysis/frames/porsche_0001.jpg"
|
|
],
|
|
"analysisMethod": "local_cv_yolo_clip_mediapipe"
|
|
}
|
|
```
|
|
|
|
Scores must be normalized from `0.0` to `1.0`.
|
|
|
|
## Recommended Local Model Split
|
|
|
|
- Use `YOLO` for object detection such as `car`, `person`, `food`, `dog`, `bottle`, and scene objects.
|
|
- Use `CLIP` or `SigLIP` for semantic labels such as `luxury sports car`, `family birthday`, or `restaurant dish`.
|
|
- Use `MediaPipe` or OpenCV Haar cascades for face presence when lightweight local face detection is enough.
|
|
- Use OpenCV/Laplacian variance and brightness histograms for blur and exposure scores.
|
|
|
|
## Runtime Behavior
|
|
|
|
When `fallback-to-heuristic=true`, the service logs `event=local_cv_visual_analysis_fallback` and writes heuristic visual analysis if the CV worker is down or returns an error.
|
|
|
|
When `fallback-to-heuristic=false`, local CV failures fail the source analysis and the source video is rejected by the scheduler.
|