25 KiB

Raw Blame History

name	description
video-editing-debugging-playbook	Load when a concrete video-editing symptom needs symptom-to-branch triage: startup/readiness failure, missing or stalled highlight output, inconsistent project state, missing category/candidate artifact, rejected director plan, FFmpeg/audio/codec failure, scheduler quarantine, API queue/DLQ/storage/JPA incident, profile drift, or clean-checkout failure. Use it to choose discriminating read-only experiments; move to the proof toolkit only after triage identifies competing causal mechanisms.

name

description

video-editing-debugging-playbook

Load when a concrete video-editing symptom needs symptom-to-branch triage: startup/readiness failure, missing or stalled highlight output, inconsistent project state, missing category/candidate artifact, rejected director plan, FFmpeg/audio/codec failure, scheduler quarantine, API queue/DLQ/storage/JPA incident, profile drift, or clean-checkout failure. Use it to choose discriminating read-only experiments; move to the proof toolkit only after triage identifies competing causal mechanisms.

Debug the Video Editing Service

Use this runbook from the repository root. Treat code, tests, runtime artifacts, and structured logs as evidence; do not judge media quality by eye alone.

Facts marked current were verified against main on 2026-07-21. Target means the requested Fortune 500 production architecture, not implemented behavior.

Safety boundary

The no-waiver policy lives in video-editing-change-control. During triage, make no dependency/model download, external-AI call, model/media network call (including loopback), unlicensed-asset use, placeholder acceptance, unapproved render, or production-default change. Do not run tools/run_local_cv_worker.sh or tools/run_local_asset_worker.sh: both can install packages automatically, and the CV script can fetch a YOLO model.

Keep the first pass read-only. A local FFprobe or decode-to-null command measures an existing file; it does not render or publish a replacement.

Route every proposed code, dependency, configuration, asset, migration, redrive, or render change through video-editing-change-control. Use video-editing-failure-archaeology before revisiting a historical fix. Never mutate queue rows during triage.

Know the terms

Term	Meaning here
Highlight project	Single-source filesystem workflow rooted by default at `output/highlight-projects/<project-id>/`.
Edit project	Multi-clip filesystem workflow rooted by default at `output/edit-projects/<project-id>/`. It has a different plan scanner and validator.
Candidate	Ranked source time range proposed for a highlight. Its expected single-source file is `analysis/highlight-candidates.json`.
Director plan	Human/local-director-authored `director/edit-plan.json` consumed by the single-source renderer.
Strict runtime	`video-clipping.editing.local-asset-worker.strict-runtime`; when asset auto-start is enabled, incomplete resident model/runtime readiness throws during startup.
DLQ	Dead-letter queue; terminal database-queue state after the configured maximum attempts.
Marker	A failed folder input renamed in place to `<name>.failed` when movement to the rejected directory also fails.
Discriminating experiment	A check whose possible outcomes select different causes or next actions.

Start with a read-only evidence bundle

Set shell variables only to shorten commands; do not infer project IDs from display names.

project_id='<exact-project-directory-name>'
project="output/highlight-projects/$project_id"
logs='<path-to-captured-service-log>'
git status --short
git rev-parse --short HEAD
java -version
mvn -version
ffmpeg -version | sed -n '1,3p'
ffprobe -version | sed -n '1,3p'
find "$project" -maxdepth 4 -type f -print | sort
sed -n '1,220p' "$project/project.json"
rg 'event=(highlight|local_cv|local_asset|visual_analysis)' "$logs" | tail -n 250

If a command reports a missing path, preserve that as evidence. Do not create the path to make the check pass.

Fast symptom map

Symptom	First discriminator	If X, branch to	If not X, branch to
Startup hangs/fails	Search `local_cv_worker_` and `local_asset_` events	Bootstrap/readiness	Spring/profile/config
Source disappears; no final	Inventory source, working, processed, rejected, project	Scheduler claim/analysis	Wrong directory/profile
`CREATED` persists	Check `analysis/source-analysis.json` and failure logs	Partial analysis	Prompt/status persistence
`WAITING_FOR_DIRECTOR` persists	Check plan, approval, asset request files	Plan/approval/assets	Scanner disabled or wrong root
`RENDERING` persists	Check renderer logs and per-highlight `final.mp4`	FFmpeg/publish failure	Stale status/artifact mismatch
`FAILED`	Read `failureMessage` and correlate the same project	Empty highlight list	Unrecorded exception; do not invent cause
No category/candidates	Compare single-source artifacts with generator wiring	Known pipeline gap	Corrupt/mislocated JSON
Folder input is rejected/marked	Inspect validator and movement events	Invalid media/collision	FFmpeg failure
API jobs stall/fail	Confirm queue adapter and queue row states	DB queue/DLQ	Memory queue/process-local loss
Works locally, clean build fails	Run isolated offline archive build	Untracked runtime dependency	Toolchain/cache mismatch

Triage startup and local models

Local CV worker

Current: application.yml selects local-cv, enables auto-start, sets startup-wait-ms: 0, and permits heuristic fallback. A zero wait logs local_cv_worker_ready without an HTTP health check. The launch script defaults LOCAL_CV_BOOTSTRAP_MODE=auto, installs packages when its virtualenv is incomplete, and loads yolov8n.pt, which may trigger a model fetch.

Run only local, non-starting checks:

test -x .venv-local-cv/bin/python && echo python-present || echo python-missing
.venv-local-cv/bin/python -c 'import fastapi,uvicorn,cv2,numpy,ultralytics; print("cv-imports-ready")'
test -f yolov8n.pt && stat -f '%N %z bytes' yolov8n.pt 2>/dev/null || stat -c '%n %s bytes' yolov8n.pt 2>/dev/null
rg 'local_cv_worker_(start|health|process)|local_cv_visual_analysis_(completed|fallback|failed)' "$logs"

If imports or the model are missing, stop. The runtime image/environment is not preprovisioned; use video-editing-build-and-env. Do not invoke bootstrap.
If local_cv_worker_ready appears with local_cv_worker_health_wait_skipped, readiness is unproved.
If local_cv_visual_analysis_fallback appears, inspect analysis/visual-analysis.json. An analysisMethod beginning local_cv_failed_fallback_ proves degraded heuristic output.
If the worker exited unexpectedly, inspect preceding local_cv_worker_output; do not merely restart it.
Target: immutable, licensed, checksummed model artifacts inside the approved runtime; startup must prove readiness without downloads.

Local asset runtime

Current working tree: application startup no longer invokes asset bootstrap. Strict readiness requires existing Piper, MusicGen, and AudioGen paths and throws when incomplete. Java/Python generation returns failure and removes invalid output instead of using host speech, silence, or tones. The standalone launcher still runs pip install in auto mode; never use it for triage or certified operation.

test -x .venv-local-asset/bin/python && echo python-present || echo python-missing
.venv-local-asset/bin/python -c 'import torch,audiocraft,soundfile,numpy; print("audio-imports-ready")'
test -n "${VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH:-}" && test -f "$VIDEO_EDITING_LOCAL_ASSET_PIPER_MODEL_PATH" && echo piper-model-present || echo piper-model-missing
command -v piper || true
rg 'local_asset_(bootstrap|runtime|python_stack|worker|generation|voiceover)' "$logs"

If local_asset_runtime_check_failed appears, do not render or promote. In strict mode the application must fail startup; in non-strict mode it logs and continues.
The current asset verifier does not invoke its bootstrap script. Treat any bootstrap log as a historical revision or a separately invoked prohibited shell launcher.
Successful Python imports do not prove model weights are present. Require the approved runtime inventory/checksum evidence; do not discover availability by invoking get_pretrained.
If logs say local_asset_worker_completed, verify the waveform; exit zero is not proof of a model-generated asset.
Target: missing or invalid required local models fail startup; every produced asset carries model/provenance evidence; placeholders are impossible.

Detect prohibited silence and tones

Apply to every generated voiceover, music, and SFX file. First correlate generation events; legacy strategy=silence, strategy=fallback-tone, model=silence, or model=ffmpeg-sine in an older artifact is an immediate failure. The current worker must fail instead of emitting those strategies.

audio='<generated-audio-file>'
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels,duration -of default=noprint_wrappers=1 "$audio"
ffmpeg -hide_banner -nostats -i "$audio" -af 'astats=metadata=1:reset=0' -f null - 2>&1 | rg 'Peak level dB|RMS level dB|Zero crossings rate'
ffmpeg -hide_banner -nostats -i "$audio" -af 'silencedetect=noise=-50dB:d=0.5' -f null - 2>&1 | rg 'silence_(start|end|duration)'

If peak/RMS is -inf, reject as silence.
If generation failed and a narrow periodic signal appears, treat it as suspected fallback tone; confirm against the log strategy or compare its dominant behavior with the documented 110 Hz music / 880 Hz SFX implementation. Do not approve by listening.
If speech is audible but came from macOS say or espeak, it is a fallback, not proof that the configured Piper model ran.
Route deterministic asset certification to video-editing-validation-and-qa.

Costly trap: commit 97ba827 added strict runtime after placeholder-producing paths already existed. The 2026-07-21 working tree now fails strict startup and removes placeholder success, but file existence, a license sidecar, or exit code zero still does not prove production audio quality, provenance authenticity, or allowed use.

Diagnose the single-source highlight flow

Establish the artifact frontier

for f in project.json analysis/source-analysis.json analysis/category.json analysis/highlight-candidates.json director/director-prompt.md director/edit-plan.json final.mp4 render-manifest.json; do
  test -f "$project/$f" && printf 'present %s\n' "$f" || printf 'missing %s\n' "$f"
done
find "$project/highlights" -maxdepth 3 -type f -print 2>/dev/null | sort

Interpret the last present artifact as the frontier, then inspect the first missing producer. Do not use downstream absence as a root cause.

State-specific branches

State	Current meaning	Discriminating branch
`CREATED`	Project record was written. Source analysis may not have completed.	If `analysis/source-analysis.json` is missing and source moved to rejected, inspect `highlight_analysis_step_failed`. If analysis exists but state remains `CREATED`, prompt generation/status update did not complete.
`ANALYZING`	Enum exists, but the single-source scheduler does not currently write it.	Treat it as external/stale data unless code history proves otherwise.
`WAITING_FOR_DIRECTOR`	Prompt generation updated status; no validated plan has entered the flow.	If a plan exists, inspect approval and validator rejection in `failureMessage`/logs.
`PLANNED`	The director plan passed `HighlightDirectorPlanValidator`; required assets may still be pending.	Inspect asset request JSON, immutable local model inventory, and worker failure logs. Do not send it back to director unless the plan itself changes.
`RENDERING`	Flow set status immediately before renderer invocation.	Missing final plus FFmpeg exception means interrupted render; current flow does not catch all exceptions to mark `FAILED`.
`RENDERED`	Project-level final and manifest were written before status update.	Still probe both media and manifest; status alone is insufficient QA.
`FAILED`	Single-source flow explicitly writes this for an empty highlight list.	Read `failureMessage`; other exceptions may escape without this state.

No category or candidates

Current behavior: HighlightSourceAnalyzer writes FFprobe, scene, audio, visual, and source analysis. HighlightSourceScheduler then invokes HighlightCandidateGenerator, which writes analysis/category.json and analysis/highlight-candidates.json before prompt generation. Missing files on a newly created project are therefore a failed/incomplete candidate stage; older projects may predate this wiring.

rg -n 'HighlightCandidateGenerator|category\.json|highlight-candidates\.json' src/main/java/org/example/videoclips/editing/HighlightSourceScheduler.java src/main/java/org/example/videoclips/editing/HighlightCandidateGenerator.java src/main/java/org/example/videoclips/editing/HighlightDirectorPromptGenerator.java
rg -n 'writeJson.*(category|highlight-candidates)' src/main/java/org/example/videoclips/editing

If files are absent on a new project, inspect highlight_candidates_started, highlight_candidate_scored, and highlight_candidates_completed; do not continue to director or render.
If files exist, parse and validate their project ID, category confidence, nonempty candidate list, bounds, and scores against analysis/source-analysis.json.
Do not copy multi-clip root files (category.json, highlight-candidates.json) into the single-source analysis/ tree as a workaround.

Director plan, approval, and pending assets

sed -n '1,260p' "$project/director/edit-plan.json"
rg 'highlight_flow_(waiting_for_approval|waiting_for_assets|started|completed)|highlight_render_scan' "$logs" | rg "$project_id"
find "$project" -path '*/requests/*.json' -type f -print -exec sed -n '1,180p' {} \;

If plan is absent, remain WAITING_FOR_DIRECTOR; do not fabricate one during incident response.
If require-director-approval=true and the configured approval file is absent, this is an intentional gate.
If request JSON remains pending, inspect the matching target path and worker logs. Never satisfy it with placeholder media.
If the plan has an empty highlights array, the flow marks FAILED.
If JSON is malformed, deserialization throws. Semantic validation is performed by HighlightDirectorPlanValidator, which requires category/candidate artifacts, safe IDs, candidate-contained ranges, duration bounds, and the production creative directions. A rejection marks the project FAILED; read failureMessage before changing the plan.
If contentCategory is missing or unknown, rendering silently maps it to GENERIC_VLOG; treat this as invalid production input, not successful classification.

Costly trap: history labeled commit 5d889b0 “working version but not cinematic.” A completed render proved mechanical execution, not creative correctness. Require candidate evidence, certified assets, media probes, and the QA rubric.

Multi-clip manual plans are different

Use this branch only for output/edit-projects/<project-id>/inbox/edit-plan.json, not highlight projects.

edit_project='output/edit-projects/<project-id>'
find "$edit_project" -maxdepth 2 -type f -print | sort
rg 'event=edit_plan_(inbox_detected|rejected|imported|saved)|event=edit_render_waiting' "$logs" | tail -n 100

Invalid JSON or validation failure moves the inbox plan to edit-plan.json.rejected; read the associated rejection message.
A valid import writes root edit-plan.json, renames the inbox file .accepted, and writes PLANNED before any approved automatic render.
The validator checks project/style identity, source and timeline bounds, playback speed, transitions, audio cues, voiceover, overlays, target duration, and minimum cinematic richness. It also requires referenced SFX files to exist.
If auto-render is disabled, PLANNED is expected. If approval is required, the configured file belongs in the project inbox/ directory.
Do not move .rejected back or edit it in place during triage. Correct the producer through change control and submit a new reviewed plan.

Diagnose FFmpeg, filters, audio, and codecs

Probe the source and output before changing commands:

media='<source-or-output-media>'
ffprobe -v error -show_entries format=duration,size:stream=index,codec_type,codec_name,width,height,r_frame_rate,sample_rate,channels -of json "$media"
ffmpeg -v error -i "$media" -f null -

If FFprobe fails, isolate corrupt/unreadable media or an unsupported container before inspecting filter graphs.
If FFprobe succeeds but decode-to-null fails, isolate codec/decoder or damaged packets.
If decode succeeds and rendering fails, capture the exact logged FFmpeg command/output. Test that exact command only in an approved disposable workspace; do not simplify it until the failing filter is identified.
If failure names drawtext, verify the selected font file and escaping. If it names loudnorm, amix, or sidechaincompress, inventory actual audio inputs and durations. If concat copy fails, compare codecs, time bases, resolution, and audio layouts across each per-highlight final.
If per-highlight outputs exist but project final.mp4 does not, inspect concat. The project concatenates with -c copy, so incompatible streams are a separate cause from segment rendering.
A qa-report.json pass is not a creative certificate. Existing QA contains structural checks and warning-level probes; use video-editing-validation-and-qa for acceptance evidence.

Diagnose folder scheduler quarantine

find input/source input/working input/processed input/rejected -maxdepth 1 -type f -print 2>/dev/null | sort
find output/clips -maxdepth 2 -type f -print 2>/dev/null | sort
rg 'event=(candidate|validation|ffprobe|ffmpeg|processing|failed_source|failure_)' "$logs" | tail -n 200

Source only: candidate may be ignored because it is hidden, unsupported, .tmp, .part, .download, or .failed, or scheduler/profile is disabled.
Working then rejected: validation or clipping failed; use event ordering to distinguish them.
Rejected collision: the scheduler refuses overwrite. If quarantine then also fails, it renames the working file to .failed; an existing marker causes failure_marker_skipped and leaves the source unchanged.
Processed collision after clips were created: source is moved to rejected. Inventory the output directory before retrying to avoid duplicate artifacts.
ffmpeg_completed exit_code=0 with zero clips is still failure.

Do not delete markers or move quarantined files back during diagnosis. Route recovery through video-editing-run-and-operate and change control.

Diagnose API queue, DLQ, storage, and JPA

First identify adapters from resolved configuration/logs. Current defaults: memory repository, memory storage, memory queue, and stub processing. application-jpa.properties opts into JPA and DB queue but uses H2 in PostgreSQL compatibility mode; it is not a production PostgreSQL profile.

rg -n '^video-clipping\.(repository|storage|queue|processing)=|^spring\.datasource|^spring\.jpa|^spring\.flyway' src/main/resources/application*.properties
rg 'DatabaseBackedClipJobQueueAdapter|InMemoryClipJobQueueAdapter|JpaVideoClippingRepository|S3ObjectStorageAdapter' "$logs"

For an approved read-only PostgreSQL session, run:

select status, count(*) from queue_messages group by status order by status;
select id, job_id, status, attempt_count, available_at, claimed_at, last_error
from queue_messages where status in ('PENDING','PROCESSING','DLQ') order by available_at limit 50;
select id, asset_id, status, attempt_count, progress_percent, error_message
from clip_jobs where id = '<job-id>';
select event_type, message, created_at from job_events
where aggregate_id = '<job-id>' order by created_at;

Rising PENDING: worker capacity/polling or a downstream bottleneck.
Old PROCESSING: compare age with visibility-timeout-ms (current default 900000 ms); reclaim is expected only after expiry.
Retry below max attempts: job should return to QUEUED, not terminal FAILED.
DLQ: current code marks the job FAILED and records DLQ and FAILED events. Fix cause before any controlled redrive.
Memory queue: state is process-local and disappears at restart; DB runbooks do not apply.
Storage metadata without a retrievable object is failure. Verify the persisted source_object_key/clip object_key against the configured adapter using an approved provider procedure; do not infer an object key from an ID.
Actuator custom health indicators currently report adapter identity/capability without exercising repository, queue, or storage. UP is not dependency reachability evidence.

Historical traps: commits 66e998e and 7307082 fixed metadata-only clip artifacts and fabricated download paths; 1737d8b fixed retryable DB failures being marked terminal on the first exception. Read video-editing-failure-archaeology before changing these contracts.

Detect profile and documentation drift

Resolve behavior from code plus the active profile, never from a plan checkbox alone.

rg -n 'spring\.profiles\.active|application-[^.]+\.(yml|properties)' pom.xml src/main/resources src/test/resources
rg -n 'enabled:|auto-start:|strict-runtime:|render-enabled:|require.*approval|fallback-to-heuristic' src/main/resources/application*.yml
rg -n '@ConditionalOn(Property|Expression)|@Scheduled' src/main/java/org/example/videoclips
git log -1 --format='%h %cI %s' -- src/main/resources/application.yml docs

Tests force the test profile via Surefire; test resources disable schedulers and worker auto-start. A passing test does not prove default application startup safety.
Base application.yml currently enables the folder scheduler, editing, local CV auto-start, local asset auto-start, local director, and highlight ingestion; highlight rendering defaults off and director approval defaults on.
application-cinematic-editing-local.yml changes multi-clip local-director behavior, not the entire single-source highlight contract.
When docs and executable code differ, record the discrepancy and route doc correction to video-editing-docs-and-writing; do not silently choose the more convenient claim.

Reproduce clean-checkout build failures without network

Current as of 2026-07-21: the working tree passes 245 tests in 62 test classes with mvn -q -o verify. An earlier archived clean tree failed LocalAssetGenerationStageTest because it depended on untracked local synthesis/host fallback; the test is now deterministic and fail-closed, but a new clean-archive run is still required. The repository has no Maven wrapper or CI configuration.

Use an already populated Maven cache and forbid network:

tmp="$(mktemp -d)"
git archive HEAD | tar -x -C "$tmp"
(cd "$tmp" && mvn -o -q verify)
git ls-files mvnw .mvn .github .gitlab-ci.yml README.md
git check-ignore -v .venv-local-asset .venv-local-cv target

Workspace passes, archive fails: find untracked runtime/tool/asset assumptions; do not copy them into the archive as a fix.
Both fail identically: diagnose the first failing test, not the final Maven summary.
Offline dependency resolution fails before tests: report the cache limitation separately; it does not prove source failure.
Archive passes: compare JDK, Maven, FFmpeg, locale, and environment variables with the failing environment.
Route reproducibility changes to video-editing-build-and-env and change control.

Do not use this skill when

Designing a fix or changing defaults: use video-editing-change-control and video-editing-architecture-contract.
Cataloging configuration: use video-editing-config-and-flags.
Running an approved production operation, redrive, or render: use video-editing-run-and-operate.
Certifying cinematic/audio quality: use video-editing-validation-and-qa.
Advancing candidate identification and production local-model rendering: use video-editing-cinematic-highlights-campaign.
Researching whether an old approach was rejected: use video-editing-failure-archaeology.

Exit checklist

Record commit, active profile, resolved adapters, tool versions, project/job ID, and timestamps.
Identify the first missing or invalid artifact, not merely the last visible symptom.
Correlate state, artifact, and log evidence; note contradictions explicitly.
Run at least one experiment that separates the leading hypotheses.
Reject downloads, external calls, unlicensed assets, placeholders, unapproved renders, and default changes.
State current behavior separately from the production target.
Route any mutation through change control with a test and rollback plan.

Provenance and maintenance

Re-verify state writers: rg -n 'HighlightProjectStatus\.|markStatus|markFailed' src/main/java/org/example/videoclips/editing

Re-verify single-source artifact producers: rg -n 'writeJson.*(source-analysis|category|highlight-candidates)|CinematicHighlightAnalyzer' src/main/java/org/example/videoclips/editing

Re-verify build/profile behavior: rg -n 'spring.profiles.active|maven-surefire|jacoco|java.version|spring-boot-starter-parent' pom.xml && git ls-files mvnw .mvn .github .gitlab-ci.yml README.md

Re-verify historical incidents: git show --stat --oneline 1737d8b 7307082 66e998e bd620fc 5d889b0 97ba827

25 KiB Raw Blame History