video_editing_poc/.claude/skills/video-editing-debugging-playbook/SKILL.md

---
name: video-editing-debugging-playbook
description: "Load when a concrete video-editing symptom needs symptom-to-branch triage: startup/readiness failure, missing or stalled highlight output, inconsistent project state, missing category/candidate artifact, rejected director plan, FFmpeg/audio/codec failure, scheduler quarantine, API queue/DLQ/storage/JPA incident, profile drift, or clean-checkout failure. Use it to choose discriminating read-only experiments; move to the proof toolkit only after triage identifies competing causal mechanisms."
---

# Debug the Video Editing Service

Use this runbook from the repository root. Treat code, tests, runtime artifacts, and structured logs as evidence; do not judge media quality by eye alone.

Facts marked **current** were verified against `main` on 2026-07-21. **Target** means the requested Fortune 500 production architecture, not implemented behavior.

## Safety boundary

The no-waiver policy lives in `video-editing-change-control`. During triage, make no dependency/model download, external-AI call, model/media network call (including loopback), unlicensed-asset use, placeholder acceptance, unapproved render, or production-default change. Do not run `tools/run_local_cv_worker.sh` or `tools/run_local_asset_worker.sh`: both can install packages automatically, and the CV script can fetch a YOLO model.

Keep the first pass read-only. A local FFprobe or decode-to-null command measures an existing file; it does not render or publish a replacement.

Route every proposed code, dependency, configuration, asset, migration, redrive, or render change through `video-editing-change-control`. Use `video-editing-failure-archaeology` before revisiting a historical fix. Never mutate queue rows during triage.

## Know the terms

| Term | Meaning here |
|---|---|
| Highlight project | Single-source filesystem workflow rooted by default at `output/highlight-projects/<project-id>/`. |
| Edit project | Multi-clip filesystem workflow rooted by default at `output/edit-projects/<project-id>/`. It has a different plan scanner and validator. |
| Candidate | Ranked source time range proposed for a highlight. Its expected single-source file is `analysis/highlight-candidates.json`. |
| Director plan | Human/local-director-authored `director/edit-plan.json` consumed by the single-source renderer. |
| Strict runtime | `video-clipping.editing.local-asset-worker.strict-runtime`; when asset auto-start is enabled, incomplete resident model/runtime readiness throws during startup. |
| DLQ | Dead-letter queue; terminal database-queue state after the configured maximum attempts. |
| Marker | A failed folder input renamed in place to `<name>.failed` when movement to the rejected directory also fails. |
| Discriminating experiment | A check whose possible outcomes select different causes or next actions. |

## Start with a read-only evidence bundle

Set shell variables only to shorten commands; do not infer project IDs from display names.

```bash
project_id='<exact-project-directory-name>'
project="output/highlight-projects/$project_id"
logs='<path-to-captured-service-log>'
git status --short
git rev-parse --short HEAD
java -version
mvn -version
ffmpeg -version | sed -n '1,3p'
ffprobe -version | sed -n '1,3p'
find "$project" -maxdepth 4 -type f -print | sort
sed -n '1,220p' "$project/project.json"
rg 'event=(highlight|local_cv|local_asset|visual_analysis)' "$logs" | tail -n 250
```

If a command reports a missing path, preserve that as evidence. Do not create the path to make the check pass.

## Fast symptom map

| Symptom | First discriminator | If X, branch to | If not X, branch to |
|---|---|---|---|
| Startup hangs/fails | Search `local_cv_worker_*` and `local_asset_*` events | Bootstrap/readiness | Spring/profile/config |
| Source disappears; no final | Inventory source, working, processed, rejected, project | Scheduler claim/analysis | Wrong directory/profile |
| `CREATED` persists | Check `analysis/source-analysis.json` and failure logs | Partial analysis | Prompt/status persistence |
| `WAITING_FOR_DIRECTOR` persists | Check plan, approval, asset request files | Plan/approval/assets | Scanner disabled or wrong root |
| `RENDERING` persists | Check renderer logs and per-highlight `final.mp4` | FFmpeg/publish failure | Stale status/artifact mismatch |
| `FAILED` | Read `failureMessage` and correlate the same project | Empty highlight list | Unrecorded exception; do not invent cause |
| No category/candidates | Compare single-source artifacts with generator wiring | Known pipeline gap | Corrupt/mislocated JSON |
| Folder input is rejected/marked | Inspect validator and movement events | Invalid media/collision | FFmpeg failure |
| API jobs stall/fail | Confirm queue adapter and queue row states | DB queue/DLQ | Memory queue/process-local loss |
| Works locally, clean build fails | Run isolated offline archive build | Untracked runtime dependency | Toolchain/cache mismatch |

## Triage startup and local models

### Local CV worker

**Current:** `application.yml` selects `local-cv`, enables auto-start, sets `startup-wait-ms: 0`, and permits heuristic fallback. A zero wait logs `local_cv_worker_ready` without an HTTP health check. The launch script defaults `LOCAL_CV_BOOTSTRAP_MODE=auto`, installs packages when its virtualenv is incomplete, and loads `yolov8n.pt`, which may trigger a model fetch.

Run only local, non-starting checks:

```bash
test -x .venv-local-cv/bin/python && echo python-present || echo python-missing
.venv-local-cv/bin/python -c 'import fastapi,uvicorn,cv2,numpy,ultralytics; print("cv-imports-ready")'
test -f yolov8n.pt && stat -f '%N %z bytes' yolov8n.pt 2>/dev/null || stat -c '%n %s bytes' yolov8n.pt 2>/dev/null
rg 'local_cv_worker_(start|health|process)|local_cv_visual_analysis_(completed|fallback|failed)' "$logs"
```

- If imports or the model are missing, stop. The runtime image/environment is not preprovisioned; use `video-editing-build-and-env`. Do not invoke bootstrap.
- If `local_cv_worker_ready` appears with `local_cv_worker_health_wait_skipped`, readiness is unproved.
- If `local_cv_visual_analysis_fallback` appears, inspect `analysis/visual-analysis.json`. An `analysisMethod` beginning `local_cv_failed_fallback_` proves degraded heuristic output.
- If the worker exited unexpectedly, inspect preceding `local_cv_worker_output`; do not merely restart it.
- **Target:** immutable, licensed, checksummed model artifacts inside the approved runtime; startup must prove readiness without downloads.

### Local asset runtime

**Current working tree:** application startup no longer invokes asset bootstrap. Strict readiness requires existing Piper, MusicGen, and AudioGen paths and throws when incomplete. Java/Python generation returns failure and removes invalid output instead of using host speech, silence, or tones. The standalone launcher still runs `pip install` in `auto` mode; never use it for triage or certified operation.

```bash
test -x .venv-local-asset/bin/python && echo python-present || echo python-missing
.venv-local-asset/bin/python -c 'import torch,audiocraft,soundfile,numpy; print("audio-imports-ready")'
test -n "${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:-}" && test -f "$VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH" && echo piper-model-present || echo piper-model-missing
command -v piper || true
rg 'local_asset_(bootstrap|runtime|python_stack|worker|generation|voiceover)' "$logs"
```

- If `local_asset_runtime_check_failed` appears, do not render or promote. In strict mode the application must fail startup; in non-strict mode it logs and continues.
- The current asset verifier does not invoke its bootstrap script. Treat any bootstrap log as a historical revision or a separately invoked prohibited shell launcher.
- Successful Python imports do not prove model weights are present. Require the approved runtime inventory/checksum evidence; do not discover availability by invoking `get_pretrained`.
- If logs say `local_asset_worker_completed`, verify the waveform; exit zero is not proof of a model-generated asset.
- **Target:** missing or invalid required local models fail startup; every produced asset carries model/provenance evidence; placeholders are impossible.

## Detect prohibited silence and tones

Apply to every generated voiceover, music, and SFX file. First correlate generation events; legacy `strategy=silence`, `strategy=fallback-tone`, `model=silence`, or `model=ffmpeg-sine` in an older artifact is an immediate failure. The current worker must fail instead of emitting those strategies.

```bash
audio='<generated-audio-file>'
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels,duration -of default=noprint_wrappers=1 "$audio"
ffmpeg -hide_banner -nostats -i "$audio" -af 'astats=metadata=1:reset=0' -f null - 2>&1 | rg 'Peak level dB|RMS level dB|Zero crossings rate'
ffmpeg -hide_banner -nostats -i "$audio" -af 'silencedetect=noise=-50dB:d=0.5' -f null - 2>&1 | rg 'silence_(start|end|duration)'
```

- If peak/RMS is `-inf`, reject as silence.
- If generation failed and a narrow periodic signal appears, treat it as suspected fallback tone; confirm against the log strategy or compare its dominant behavior with the documented 110 Hz music / 880 Hz SFX implementation. Do not approve by listening.
- If speech is audible but came from macOS `say` or `espeak`, it is a fallback, not proof that the configured Piper model ran.
- Route deterministic asset certification to `video-editing-validation-and-qa`.

**Costly trap:** commit `97ba827` added strict runtime after placeholder-producing paths already existed. The 2026-07-21 working tree now fails strict startup and removes placeholder success, but file existence, a license sidecar, or exit code zero still does not prove production audio quality, provenance authenticity, or allowed use.

## Diagnose the single-source highlight flow

### Establish the artifact frontier

```bash
for f in project.json analysis/source-analysis.json analysis/category.json analysis/highlight-candidates.json director/director-prompt.md director/edit-plan.json final.mp4 render-manifest.json; do
  test -f "$project/$f" && printf 'present %s\n' "$f" || printf 'missing %s\n' "$f"
done
find "$project/highlights" -maxdepth 3 -type f -print 2>/dev/null | sort
```

Interpret the last present artifact as the frontier, then inspect the first missing producer. Do not use downstream absence as a root cause.

### State-specific branches

| State | Current meaning | Discriminating branch |
|---|---|---|
| `CREATED` | Project record was written. Source analysis may not have completed. | If `analysis/source-analysis.json` is missing and source moved to rejected, inspect `highlight_analysis_step_failed`. If analysis exists but state remains `CREATED`, prompt generation/status update did not complete. |
| `ANALYZING` | Enum exists, but the single-source scheduler does not currently write it. | Treat it as external/stale data unless code history proves otherwise. |
| `WAITING_FOR_DIRECTOR` | Prompt generation updated status; no validated plan has entered the flow. | If a plan exists, inspect approval and validator rejection in `failureMessage`/logs. |
| `PLANNED` | The director plan passed `HighlightDirectorPlanValidator`; required assets may still be pending. | Inspect asset request JSON, immutable local model inventory, and worker failure logs. Do not send it back to director unless the plan itself changes. |
| `RENDERING` | Flow set status immediately before renderer invocation. | Missing final plus FFmpeg exception means interrupted render; current flow does not catch all exceptions to mark `FAILED`. |
| `RENDERED` | Project-level final and manifest were written before status update. | Still probe both media and manifest; status alone is insufficient QA. |
| `FAILED` | Single-source flow explicitly writes this for an empty highlight list. | Read `failureMessage`; other exceptions may escape without this state. |

### No category or candidates

**Current behavior:** `HighlightSourceAnalyzer` writes FFprobe, scene, audio, visual, and source analysis. `HighlightSourceScheduler` then invokes `HighlightCandidateGenerator`, which writes `analysis/category.json` and `analysis/highlight-candidates.json` before prompt generation. Missing files on a newly created project are therefore a failed/incomplete candidate stage; older projects may predate this wiring.

```bash
rg -n 'HighlightCandidateGenerator|category\.json|highlight-candidates\.json' src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java src/main/java/org/example/videoclips/editing/HighlightCandidateGenerator.java src/main/java/org/example/videoclips/editing/HighlightDirectorPromptGenerator.java
rg -n 'writeJson.*(category|highlight-candidates)' src/main/java/org/example/videoclips/editing
```

- If files are absent on a new project, inspect `highlight_candidates_started`, `highlight_candidate_scored`, and `highlight_candidates_completed`; do not continue to director or render.
- If files exist, parse and validate their project ID, category confidence, nonempty candidate list, bounds, and scores against `analysis/source-analysis.json`.
- Do not copy multi-clip root files (`category.json`, `highlight-candidates.json`) into the single-source `analysis/` tree as a workaround.

### Director plan, approval, and pending assets

```bash
sed -n '1,260p' "$project/director/edit-plan.json"
rg 'highlight_flow_(waiting_for_approval|waiting_for_assets|started|completed)|highlight_render_scan' "$logs" | rg "$project_id"
find "$project" -path '*/requests/*.json' -type f -print -exec sed -n '1,180p' {} \;
```

- If plan is absent, remain `WAITING_FOR_DIRECTOR`; do not fabricate one during incident response.
- If `require-director-approval=true` and the configured approval file is absent, this is an intentional gate.
- If request JSON remains pending, inspect the matching target path and worker logs. Never satisfy it with placeholder media.
- If the plan has an empty `highlights` array, the flow marks `FAILED`.
- If JSON is malformed, deserialization throws. Semantic validation is performed by `HighlightDirectorPlanValidator`, which requires category/candidate artifacts, safe IDs, candidate-contained ranges, duration bounds, and the production creative directions. A rejection marks the project `FAILED`; read `failureMessage` before changing the plan.
- If `contentCategory` is missing or unknown, rendering silently maps it to `GENERIC_VLOG`; treat this as invalid production input, not successful classification.

**Costly trap:** history labeled commit `5d889b0` “working version but not cinematic.” A completed render proved mechanical execution, not creative correctness. Require candidate evidence, certified assets, media probes, and the QA rubric.

### Multi-clip manual plans are different

Use this branch only for `output/edit-projects/<project-id>/inbox/edit-plan.json`, not highlight projects.

```bash
edit_project='output/edit-projects/<project-id>'
find "$edit_project" -maxdepth 2 -type f -print | sort
rg 'event=edit_plan_(inbox_detected|rejected|imported|saved)|event=edit_render_waiting' "$logs" | tail -n 100
```

- Invalid JSON or validation failure moves the inbox plan to `edit-plan.json.rejected`; read the associated rejection message.
- A valid import writes root `edit-plan.json`, renames the inbox file `.accepted`, and writes `PLANNED` before any approved automatic render.
- The validator checks project/style identity, source and timeline bounds, playback speed, transitions, audio cues, voiceover, overlays, target duration, and minimum cinematic richness. It also requires referenced SFX files to exist.
- If auto-render is disabled, `PLANNED` is expected. If approval is required, the configured file belongs in the project `inbox/` directory.
- Do not move `.rejected` back or edit it in place during triage. Correct the producer through change control and submit a new reviewed plan.

## Diagnose FFmpeg, filters, audio, and codecs

Probe the source and output before changing commands:

```bash
media='<source-or-output-media>'
ffprobe -v error -show_entries format=duration,size:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$media"
ffmpeg -v error -i "$media" -f null -
```

- If FFprobe fails, isolate corrupt/unreadable media or an unsupported container before inspecting filter graphs.
- If FFprobe succeeds but decode-to-null fails, isolate codec/decoder or damaged packets.
- If decode succeeds and rendering fails, capture the exact logged FFmpeg command/output. Test that exact command only in an approved disposable workspace; do not simplify it until the failing filter is identified.
- If failure names `drawtext`, verify the selected font file and escaping. If it names `loudnorm`, `amix`, or `sidechaincompress`, inventory actual audio inputs and durations. If concat copy fails, compare codecs, time bases, resolution, and audio layouts across each per-highlight final.
- If per-highlight outputs exist but project `final.mp4` does not, inspect concat. The project concatenates with `-c copy`, so incompatible streams are a separate cause from segment rendering.
- A `qa-report.json` pass is not a creative certificate. Existing QA contains structural checks and warning-level probes; use `video-editing-validation-and-qa` for acceptance evidence.

## Diagnose folder scheduler quarantine

```bash
find input/source input/working input/processed input/rejected -maxdepth 1 -type f -print 2>/dev/null | sort
find output/clips -maxdepth 2 -type f -print 2>/dev/null | sort
rg 'event=(candidate|validation|ffprobe|ffmpeg|processing|failed_source|failure_)' "$logs" | tail -n 200
```

- Source only: candidate may be ignored because it is hidden, unsupported, `.tmp`, `.part`, `.download`, or `.failed`, or scheduler/profile is disabled.
- Working then rejected: validation or clipping failed; use event ordering to distinguish them.
- Rejected collision: the scheduler refuses overwrite. If quarantine then also fails, it renames the working file to `.failed`; an existing marker causes `failure_marker_skipped` and leaves the source unchanged.
- Processed collision after clips were created: source is moved to rejected. Inventory the output directory before retrying to avoid duplicate artifacts.
- `ffmpeg_completed exit_code=0` with zero clips is still failure.

Do not delete markers or move quarantined files back during diagnosis. Route recovery through `video-editing-run-and-operate` and change control.

## Diagnose API queue, DLQ, storage, and JPA

First identify adapters from resolved configuration/logs. **Current defaults:** memory repository, memory storage, memory queue, and stub processing. `application-jpa.properties` opts into JPA and DB queue but uses H2 in PostgreSQL compatibility mode; it is not a production PostgreSQL profile.

```bash
rg -n '^video-clipping\.(repository|storage|queue|processing)=|^spring\.datasource|^spring\.jpa|^spring\.flyway' src/main/resources/application*.properties
rg 'DatabaseBackedClipJobQueueAdapter|InMemoryClipJobQueueAdapter|JpaVideoClippingRepository|S3ObjectStorageAdapter' "$logs"
```

For an approved read-only PostgreSQL session, run:

```sql
select status, count(*) from queue_messages group by status order by status;
select id, job_id, status, attempt_count, available_at, claimed_at, last_error
from queue_messages where status in ('PENDING','PROCESSING','DLQ') order by available_at limit 50;
select id, asset_id, status, attempt_count, progress_percent, error_message
from clip_jobs where id = '<job-id>';
select event_type, message, created_at from job_events
where aggregate_id = '<job-id>' order by created_at;
```

- Rising `PENDING`: worker capacity/polling or a downstream bottleneck.
- Old `PROCESSING`: compare age with `visibility-timeout-ms` (current default 900000 ms); reclaim is expected only after expiry.
- Retry below max attempts: job should return to `QUEUED`, not terminal `FAILED`.
- `DLQ`: current code marks the job `FAILED` and records `DLQ` and `FAILED` events. Fix cause before any controlled redrive.
- Memory queue: state is process-local and disappears at restart; DB runbooks do not apply.
- Storage metadata without a retrievable object is failure. Verify the persisted `source_object_key`/clip `object_key` against the configured adapter using an approved provider procedure; do not infer an object key from an ID.
- Actuator custom health indicators currently report adapter identity/capability without exercising repository, queue, or storage. `UP` is not dependency reachability evidence.

**Historical traps:** commits `66e998e` and `7307082` fixed metadata-only clip artifacts and fabricated download paths; `1737d8b` fixed retryable DB failures being marked terminal on the first exception. Read `video-editing-failure-archaeology` before changing these contracts.

## Detect profile and documentation drift

Resolve behavior from code plus the active profile, never from a plan checkbox alone.

```bash
rg -n 'spring\.profiles\.active|application-[^.]+\.(yml|properties)' pom.xml src/main/resources src/test/resources
rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml
rg -n '@ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
git log -1 --format='%h %cI %s' -- src/main/resources/application.yml docs
```

- Tests force the `test` profile via Surefire; test resources disable schedulers and worker auto-start. A passing test does not prove default application startup safety.
- Base `application.yml` currently enables the folder scheduler, editing, local CV auto-start, local asset auto-start, local director, and highlight ingestion; highlight rendering defaults off and director approval defaults on.
- `application-cinematic-editing-local.yml` changes multi-clip local-director behavior, not the entire single-source highlight contract.
- When docs and executable code differ, record the discrepancy and route doc correction to `video-editing-docs-and-writing`; do not silently choose the more convenient claim.

## Reproduce clean-checkout build failures without network

**Current as of 2026-07-21:** the working tree passes 245 tests in 62 test classes with `mvn -q -o verify`. An earlier archived clean tree failed `LocalAssetGenerationStageTest` because it depended on untracked local synthesis/host fallback; the test is now deterministic and fail-closed, but a new clean-archive run is still required. The repository has no Maven wrapper or CI configuration.

Use an already populated Maven cache and forbid network:

```bash
tmp="$(mktemp -d)"
git archive HEAD | tar -x -C "$tmp"
(cd "$tmp" && mvn -o -q verify)
git ls-files mvnw .mvn .github .gitlab-ci.yml README.md
git check-ignore -v .venv-local-asset .venv-local-cv target
```

- Workspace passes, archive fails: find untracked runtime/tool/asset assumptions; do not copy them into the archive as a fix.
- Both fail identically: diagnose the first failing test, not the final Maven summary.
- Offline dependency resolution fails before tests: report the cache limitation separately; it does not prove source failure.
- Archive passes: compare JDK, Maven, FFmpeg, locale, and environment variables with the failing environment.
- Route reproducibility changes to `video-editing-build-and-env` and change control.

## Do not use this skill when

- Designing a fix or changing defaults: use `video-editing-change-control` and `video-editing-architecture-contract`.
- Cataloging configuration: use `video-editing-config-and-flags`.
- Running an approved production operation, redrive, or render: use `video-editing-run-and-operate`.
- Certifying cinematic/audio quality: use `video-editing-validation-and-qa`.
- Advancing candidate identification and production local-model rendering: use `video-editing-cinematic-highlights-campaign`.
- Researching whether an old approach was rejected: use `video-editing-failure-archaeology`.

## Exit checklist

- [ ] Record commit, active profile, resolved adapters, tool versions, project/job ID, and timestamps.
- [ ] Identify the first missing or invalid artifact, not merely the last visible symptom.
- [ ] Correlate state, artifact, and log evidence; note contradictions explicitly.
- [ ] Run at least one experiment that separates the leading hypotheses.
- [ ] Reject downloads, external calls, unlicensed assets, placeholders, unapproved renders, and default changes.
- [ ] State current behavior separately from the production target.
- [ ] Route any mutation through change control with a test and rollback plan.

## Provenance and maintenance

Re-verify state writers: `rg -n 'HighlightProjectStatus\.|markStatus|markFailed' src/main/java/org/example/videoclips/editing`

Re-verify scheduler/default guards: `rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml src/test/resources/application*.properties`

Re-verify prohibited bootstrap behavior: `rg -n 'pip install|YOLO\(|write_silence|write_fallback_tone|fallbackTone|writeSilence' tools src/main/java/org/example/videoclips/editing`

Re-verify single-source artifact producers: `rg -n 'writeJson.*(source-analysis|category|highlight-candidates)|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing`

Re-verify queue states and retry rules: `rg -n 'PENDING|PROCESSING|COMPLETED|DLQ|maxAttempts|visibilityTimeout|markRetry|markTerminal' src/main/java/org/example/videoclips/queue src/main/java/org/example/videoclips/processing`

Re-verify build/profile behavior: `rg -n 'spring.profiles.active|maven-surefire|jacoco|java.version|spring-boot-starter-parent' pom.xml && git ls-files mvnw .mvn .github .gitlab-ci.yml README.md`

Re-verify historical incidents: `git show --stat --oneline 1737d8b 7307082 66e998e bd620fc 5d889b0 97ba827`