video_editing_poc/.claude/skills/video-editing-failure-archaeology/SKILL.md

name

description

video-editing-failure-archaeology

Load this skill when investigating whether a video-editing failure, tempting fix, stale generated artifact, completed checklist item, deleted test, or unusual Git object has already been encountered in this repository. Use it before reopening storage, queue/DLQ, local-file cleanup, FFmpeg segment timing, cinematic-quality, local-model asset, port/configuration, or single-source highlight incidents, and before treating historical output as proof.

Video Editing Failure Archaeology

Purpose

Reconstruct what the repository actually proves before changing behavior. Treat a root cause as the mechanism demonstrated by code, a patch, a test, or a reproducible observation. Treat a commit subject, plan checkbox, and generated media as leads rather than proof.

Facts and workspace observations in this skill were checked on 2026-07-21.

Use This Skill When

• A symptom resembles an earlier storage, queue, filesystem, FFmpeg, render, or local-model problem.
• A plan says an item is complete but production evidence is unclear.
• An ignored or committed runtime artifact appears to prove a workflow works.
• Git history contains a deletion, rewrite, unreachable commit, or apparently abandoned approach.
• A proposed fix repeats an earlier partial fix.

Do Not Use This Skill When

• Triaging a live symptom from scratch: use video-editing-debugging-playbook first, then return here after identifying the subsystem.
• Changing code, configuration, dependencies, defaults, or runtime behavior: use video-editing-change-control; this skill never authorizes a change.
• Deciding whether output meets acceptance thresholds: use video-editing-validation-and-qa.
• Looking up current defaults or environment wiring: use video-editing-config-and-flags.
• Operating or recovering a deployed instance: use video-editing-run-and-operate.

Non-Negotiable Investigation Rules

1. Do not run tools/run_local_cv_worker.sh or tools/run_local_asset_worker.sh during archaeology. They can create virtual environments, install packages, and download model weights.
2. Do not use network access, external AI services, unlicensed assets, placeholder silence or tones, or unapproved rendering to reproduce a result.
3. Do not change a production-facing default to test a theory.
4. Do not mutate Git history. Use git log, git show, git diff, and git fsck; never reset, rebase, clean, or delete artifacts.
5. Do not promote ignored media, a manifest, or a green structural test as cinematic-quality evidence.
6. Route every behavior change through video-editing-change-control, including a retry change, port change, fallback change, or FFmpeg flag change.

Evidence Labels

Use these labels in incident notes and pull requests:

Label	Meaning	Required evidence
FIXED	The demonstrated mechanism was changed and regression coverage exists.	Patch plus current code/test location.
ACTIVE	The mechanism remains in current code or reproducibly fails.	Current source or dated experiment.
PARTIAL	One failure mode was fixed, but adjacent obligations remain.	Explicit boundary and remaining gap.
HISTORICAL	The evidence describes an earlier tree, not necessarily current behavior.	Commit hash and date.
OPEN	Evidence shows a concern but not its cause or intent.	Known facts and the missing discriminator.
NOT EVIDENCE	The artifact cannot support the claimed conclusion.	Reason it is non-representative or unverified.

Fast History Triage

Run read-only checks from the repository root:

git status --short --ignored
git log --oneline --decorate --all --reverse
git log --all --name-status -- src/main src/test docs
git fsck --no-reflogs --unreachable --no-progress
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|open question|fallback|silence|tone' docs src tools

Before accepting a historical claim, inspect the exact patch:

export COMMIT="${COMMIT:?set COMMIT to a full commit ID}"
export FILE_PATH="${FILE_PATH:?set FILE_PATH to a repository-relative path}"
export OLDER_COMMIT="${OLDER_COMMIT:?set OLDER_COMMIT to a full commit ID}"
export NEWER_COMMIT="${NEWER_COMMIT:?set NEWER_COMMIT to a full commit ID}"
git show --format=fuller --stat "$COMMIT"
git show --format=fuller --no-ext-diff "$COMMIT" -- "$FILE_PATH"
git diff "$OLDER_COMMIT" "$NEWER_COMMIT" -- "$FILE_PATH"

Chronicle At A Glance

Date	Symptom or risk	Demonstrated mechanism	Status on 2026-07-21
2026-07-08/09	Upload completion and clips existed as state/metadata without a complete storage path.	Storage operations and persisted object keys were absent from earlier paths.	FIXED for the implemented adapters; production S3 proof remains separate.
2026-07-09	Internal object locations leaked through normal API responses and logs.	objectKey and staged local path were returned/recorded directly.	FIXED by 0d45f89; regression assertions remain.
2026-07-09	Worker files accumulated after success or failure.	Per-job staging/output lacked guaranteed deletion.	PARTIAL: finally cleanup and scheduled local cleanup exist; errors are swallowed and other editing trees are outside that job.
2026-07-09	Retryable DB-queue failure became terminal on the first exception.	ClipProcessor persisted FAILED before the queue decided retry versus DLQ.	FIXED by 1737d8b for the DB queue contract.
2026-07-10	Folder clips could preserve quality or hit exact boundaries, but not both.	Stream copy cuts on source keyframes; exact cuts force keyframes by transcoding.	ACTIVE design trade-off; default selects preservation.
2026-07-11	Renderer worked but was explicitly “not cinematic.”	Exact cause was not recorded; one patch added a uniform grade, followed by broader planning/render/QA work.	OPEN: no certified creative-quality evidence.
2026-07-12 onward	Single-source projects stopped at CREATED or WAITING_FOR_DIRECTOR.	Local artifacts lack a director plan/final output; current scanner requires director/edit-plan.json.	ACTIVE missing local-director capability; artifacts alone do not prove a regression.
2026-07-21	Workspace build passed while clean archived checkout failed local-asset tests.	Tests invoke the default untracked Python runtime and real host fallbacks instead of a hermetic fake.	ACTIVE; clean-checkout reproducibility is not established.

Case 1: Metadata-Only Storage Grew Into A Real Data Path

Symptom. API state could say upload or clip work completed while the service did not yet prove a durable source-to-generated-object path.

Demonstrated mechanism. The sequence is visible in the patches:

Commit	Added missing responsibility
7e8a214	uploads:complete called ObjectStoragePort.completeMultipartUpload instead of only changing metadata.
9df9736	VideoAsset.sourceObjectKey was persisted and passed to clipping.
bd620fc	Source objects were materialized into local staging before FFmpeg.
66e998e	Generated clips carried local paths, were uploaded, and persisted an objectKey.
7307082	Signed download URLs used the persisted clip key instead of deriving a fake key from clipId.

Rejected partial fixes. Do not reintroduce any of these:

• Marking upload complete without completing the provider upload.
• Saving clip rows before uploading the generated files.
• Deriving storage keys from public IDs when the persisted key is authoritative.
• Treating the in-memory adapter's no-op completion as proof that S3 completion works.

Current invariant. ClipProcessor must follow materialize source -> generate local clips -> upload each clip -> save each clip record -> mark the job successful. ObjectStoragePort owns materialization, upload, delete, and signed-download operations.

Discriminate metadata drift from storage failure. Read the job events and compare the clip record with backend existence. Do not expose the key in a client response merely to debug it. Inspect code with:

git show --stat 7e8a214 9df9736 bd620fc 66e998e 7307082
rg -n 'materializeSourceObject|uploadGeneratedClip|createClipDownloadUrl' src/main/java
rg -n 'source_object_key|object_key' src/main/resources/db/migration

Status. FIXED at the code-contract level. No checked-in production S3 end-to-end evidence makes the external system production-certified.

Case 2: Object-Key And Local-Path Disclosure

Symptom. Normal API payloads exposed sourceObjectKey/clip objectKey, and a processing event included the staged local path.

Root cause. Commit 66e998e initially added objectKey to toClipResponse; earlier asset mapping also exposed the source key. Internal routing identifiers crossed the API boundary. The materialization event also included a filesystem path.

Fix. Commit 0d45f89 removed those fields from normal responses and removed the path from the event message. Current assertions in VideoAssetControllerTest require sourceObjectKey and clip objectKey not to exist. Clients obtain a signed URL through the download endpoint.

Wrong path. Do not return a key “temporarily,” put it in Problem Details, or log the full local path. Diagnose through authorized backend tooling and redacted correlation identifiers.

git show --no-ext-diff 0d45f89 -- src/main/java src/test/java
rg -n 'sourceObjectKey.*doesNotExist|objectKey.*doesNotExist' src/test/java/org/example/videoclips/api
rg -n 'toAssetResponse|toClipResponse' src/main/java/org/example/videoclips/application/VideoAssetService.java

Status. FIXED for the tested response paths. Any new DTO, log, manifest, or error mapping must repeat the disclosure review through change control.

Case 3: Retry And DLQ State Split

Symptom. A transient DB-queue processing exception marked the clip job FAILED on the first attempt even though the queue intended to retry it.

Root cause. The processor owned terminal failure persistence while the queue adapter owned attempt policy. The lower layer finalized state before the policy layer knew whether the attempt was terminal.

Fix. Commit 1737d8b added processNow(jobId, false) for DB-queue attempts. The queue adapter now increments attempts, schedules a non-terminal PENDING retry with RETRY_SCHEDULED, and calls markTerminalFailure only when max-attempts is reached; terminal events are DLQ and FAILED.

Do not repeat. Do not mark a job failed in an adapter-independent catch block when a caller owns retry policy. Do not blindly redrive DLQ rows. docs/dlq-redrive-procedure.md requires root-cause correction, source existence, a small sample, and transactional row changes; the repository has no built-in redrive command.

Discriminating observations. Before the last attempt, expect message PENDING, job QUEUED, and a RETRY_SCHEDULED event. At exhaustion, expect message DLQ, job FAILED, then DLQ and FAILED events. A job already FAILED while its message is retryable indicates the old split-brain symptom or another writer.

git show --no-ext-diff 1737d8b -- src/main/java/org/example/videoclips/processing/ClipProcessor.java src/main/java/org/example/videoclips/queue/DatabaseBackedClipJobQueueAdapter.java
rg -n 'RETRY_SCHEDULED|markTerminalFailure|setStatus\("DLQ"\)' src/main/java src/test/java
sed -n '1,320p' docs/dlq-redrive-procedure.md

Status. FIXED for the database-backed queue. The memory queue and any future external queue require their own semantics and tests.

Case 4: Worker Temporary-File Accumulation

Symptom. Staged sources and generated local clips accumulated across successful and failed jobs.

Root cause. The processing path had no guaranteed cleanup boundary.

Fixes. Commit 8f4804c put job-local deletion in ClipProcessor.finally, gated by video-clipping.ffmpeg.cleanup-local-files. Commit 3d961f2 added scheduled age-based cleanup for FFmpeg input/output, in-memory storage, and stub output.

Limitations. LocalArtifactCleanupJob catches and ignores IOException; deletion failure can be silent. Its roots do not include the cinematic edit and highlight project trees. Retention of durable storage/database records belongs to RetentionCleanupJob, not this local sweep.

Wrong path. Do not disable cleanup to make a failing test inspectable in production. Preserve a controlled diagnostic copy outside the worker path under an approved procedure.

git show --stat 8f4804c 3d961f2
rg -n 'cleanupLocalFiles|cleanupLocalArtifacts|local-artifact-retention' src/main/java src/main/resources src/test/java
du -sh tmp input output 2>/dev/null
find tmp -type f -mtime +1 -print 2>/dev/null | sed -n '1,100p'

Status. PARTIAL: job-local cleanup and a safety sweep exist, but cleanup observability and all editing artifact retention remain open.

Case 5: Preserve Quality Versus Exact Segment Timing

Symptom. An “8-second clip” expectation conflicts with clips cut at nearby keyframes.

Mechanism. FolderFfmpegClipper has two mutually exclusive command shapes:

preserve-input-quality	FFmpeg mode	Consequence
true (packaged default)	-c copy plus segment muxer	No generation loss; boundaries can drift to source keyframes.
false	libx264, CRF 20, forced keyframes, AAC	More exact segment boundaries; re-encode cost and generation loss.

The trade-off is documented in docs/input-folder-scheduler-plan.md. The plan still asks whether exact timing or keyframe alignment is required, so “exact” is not a settled product requirement for this workflow.

Wrong paths. Do not call stream-copy output exact. Do not switch the default to transcoding to satisfy one fixture. Do not infer visual preservation solely from container/codec names.

rg -n 'preserve-input-quality|force_key_frames|segment_time|qualityMode' src/main docs/input-folder-scheduler-plan.md
git show --no-ext-diff aaacd7f -- src/main/java/org/example/videoclips/folder/FolderFfmpegClipper.java

Status. ACTIVE intentional trade-off. Any default or acceptance-threshold decision is a behavior change and must pass change control and measured validation.

Case 6: “Working Version But Not Cinematic”

Symptom. Commit 5d889b0 records the only direct historical judgment: the render worked but was not cinematic.

What the patch proves. That commit added a fixed contrast/saturation/sharpen/vignette filter for non-empty visual treatments, plus compatibility/test wiring. It does not record a controlled comparison, reviewer rubric, or measured root cause.

What followed. Later commits added category-aware planning (5006dfd), stricter plan validation (3348ae8), local asset selection (376bc41), overlays (43fccb7), audio mastering (bb23ac8), dynamic crops (2d0c331), QA reports (9b56e89), and approval gating (9536928). This sequence identifies areas the project invested in; it does not prove each was causally necessary or that current output is cinematic.

Rejected explanation. “It needed a LUT/grade” is not established. A uniform grade was the immediate change, yet the repository then required planning, sound, motion, QA, and approval work.

Current status. OPEN. Structural render success, qa-report.json, and a committed manifest do not establish creative quality. Use the cinematic campaign and video-editing-validation-and-qa; require a licensed asset inventory, local-model provenance, objective media probes, and scored human review. Never fill missing audio with silence or tones.

git show --no-ext-diff 5d889b0 -- docs src/main/java/org/example/videoclips/editing/FfmpegEditRenderer.java
git log --oneline 5d889b0..9536928
git show --stat 5006dfd 3348ae8 376bc41 43fccb7 bb23ac8 2d0c331 9b56e89 9536928

Case 7: Single-Source Highlight Projects Stalled

Observed on 2026-07-21. Ignored workspace projects under output/highlight-projects/ include one CREATED project and one WAITING_FOR_DIRECTOR project from 2026-07-12. Neither contains a final output. The later project contains analysis and a director prompt but no director/edit-plan.json.

Separate stale-artifact trap. Git tracks output/edit-projects/source-clips/project.json and render-manifest.json, which claim RENDERED and refer to final.mp4; Git does not track that media file. A clean checkout therefore contains success metadata without the claimed output. The /input/, /output/, and /tmp/ ignore rules live in clone-local .git/info/exclude, not a shared repository ignore file. Do not use either the tracked manifest or this clone's ignored media as certified evidence.

Current gate. HighlightDirectorPlanScanner only selects a project with director/edit-plan.json and without final.mp4. HighlightDirectorFlowService returns missing_director_plan without that file and may also return approval_missing or assets_pending.

Evidence boundary. These projects were created after 618ba9a and before 97ba827. Their state is consistent with the documented missing-plan gate, so it is not proof that the flow implementation regressed. Current code generates a prompt and watches for a plan file; no runtime local-director model consumes the prompt and writes that plan. Their manifest.json also declares analysis filenames that do not exactly match every generated filename. Treat them as diagnostic snapshots, not goldens.

Discriminate the gate without rendering. Inspect only metadata and file presence:

find output/highlight-projects -maxdepth 3 -type f -print 2>/dev/null | sort
find output/highlight-projects -name project.json -exec sed -n '1,100p' {} \; 2>/dev/null
find output/highlight-projects -path '*/director/edit-plan.json' -o -name final.mp4 2>/dev/null
rg -n 'missing_director_plan|approval_missing|assets_pending|findNextRenderableProject' src/main/java/org/example/videoclips/editing
git ls-files output input tmp

Status. Artifact state is HISTORICAL; the absence of an in-runtime local director is ACTIVE, and end-to-end production readiness is OPEN. Reproduce only with approved, licensed local inputs and pre-provisioned local models. Do not let startup download anything or render without approval.

Case 8: Local Asset Build Is Not Clean-Checkout Hermetic

Observed on 2026-07-21. The workspace suite passed, and its Surefire report shows LocalAssetGenerationStageTest invoked ./.venv-local-asset/bin/python. That virtual environment exists locally but is untracked. A valid run from inside a clean archived checkout ran 216 tests and failed one assertion, reusesExistingSharedAssetsAndSynthesizesMissingVoiceover. An earlier two-failure run used mvn -f from the workspace and is invalid clean-checkout evidence because it violated the test working-directory assumption.

Root cause. The archived tests constructed the production LocalAssetSynthesizer with default configuration. They expected missing voiceover/SFX to become render-ready but did not provide a fake process. The default Python path pointed into the untracked virtual environment, host say/espeak availability affected voiceover, and the former worker could return success after writing silence or tones. Status 2026-07-21: the working tree removed those success fallbacks, deletes failed/inaudible output, and makes the affected test deterministic and fail-closed.

Consequences. The historical workspace pass was environment-coupled. The 2026-07-21 working tree removes placeholder success and makes strict readiness fail, but the bootstrap script can still install dependencies automatically and no complete resident model bundle is present. Re-run the clean archive before declaring the incident closed.

Wrong paths. Do not commit the virtual environment, enable network during tests, weaken strict mode, accept placeholder audio, or skip the test. Make tests hermetic and make production readiness fail closed through change control.

git ls-files .venv-local-asset tools/local_asset_worker.py tools/run_local_asset_worker.sh
rg -n 'new LocalAssetSynthesizer|readyForRender' src/test/java/org/example/videoclips/editing/LocalAssetGenerationStageTest.java
rg -n 'write_silence|write_fallback_tone|strict_runtime|strict-runtime|pip install' tools src/main src/main/resources
sed -n '1,120p' target/surefire-reports/org.example.videoclips.editing.LocalAssetGenerationStageTest.txt 2>/dev/null

Status. ACTIVE, release-blocking under the clean-checkout and no-placeholder requirements. Do not re-run bootstrap as a “fix.”

Case 9: Port And Runtime-Shape Drift

Observed configuration. API examples assume Spring Boot's default port 8080. The local CV HTTP worker defaults to 127.0.0.1:8091, and the process manager derives host/port from its configured endpoint. tools/run_local_asset_worker.sh defines LOCAL_ASSET_HOST and LOCAL_ASSET_PORT defaulting to 8092, but it launches a one-shot CLI script, not an HTTP server; those variables are unused by local_asset_worker.py.

Interpretation. This is verified semantic drift, not a documented outage. Do not diagnose local asset generation by probing port 8092, and do not “fix” it by moving another service to that port. First identify whether the component is HTTP (local-cv) or process-invoked (local-asset).

rg -n '8080|8091|8092|LOCAL_CV_PORT|LOCAL_ASSET_PORT|server.port' docs src tools
lsof -nP -iTCP -sTCP:LISTEN | rg ':(8080|8091|8092)\b'
rg -n 'ProcessBuilder|local_asset_worker.py|uvicorn' src/main/java/org/example/videoclips/editing tools

Status. ACTIVE documentation/configuration debt. No causal history establishes that a port collision caused a project failure.

Case 10: Deleted Tenant Quota Test

Evidence. TenantQuotaControllerTest was added with tenant quota work and existed when 1737d8b reported its pass. Commit 7fd92a7 deleted it while converting folder-scheduler configuration from properties to YAML. The commit message does not explain the deletion. A file named input/source/TenantQuotaControllerTest.java.failed is ignored local runtime debris, not the deleted test and not proof of intent.

Conclusion. Label the deletion OPEN. Do not claim the behavior was obsolete, flaky, or intentionally replaced unless a maintainer or equivalent coverage proves it.

git log --all --follow --format='%h %ad %s' --date=iso-strict -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
git show 7fd92a7^:src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
git show --no-ext-diff 7fd92a7 -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java
rg -n 'quota|QuotaExceeded' src/test src/main

Status. OPEN test-coverage ambiguity. Restoring or replacing coverage requires change control; the historical file is evidence for intended behavior, not automatically the correct present test.

Case 11: Rewritten Commits, Not Dead Feature Branches

Observed on 2026-07-21. git fsck reports six unreachable commits around the signed-download and disclosure fixes: 19ef4d3, 5919210, 65a29a4, 84e6694, eb06021, and ba3f988. Several share subjects, trees, parents, or near-identical changes with reachable 7307082 and 0d45f89.

Conclusion. The evidence supports local amend/rewrite debris. It does not support a story about a reverted architecture or abandoned branch. The reachable history contains no commit whose subject is an explicit revert and only the main branch is present in the inspected repository.

git fsck --no-reflogs --unreachable --no-progress
git show -s --format='%H%nparents=%P%ntree=%T%nauthor=%ad%nsubject=%s%n' --date=iso-strict <unreachable-commit>
git branch -a -vv
git log --all --oneline --grep='^Revert'

Status. NOT EVIDENCE for a rejected solution. If unreachable objects disappear after Git maintenance, this conclusion should rely on the recorded hashes here and reachable patches, not object availability.

Case 12: Completed Plans Are Not Production Signoff

Symptom. docs/video-clipping-service-implementation-plan.md marks every repository-scoped milestone complete, including backup/restore validation and load-test signoff.

What the docs actually say. The same plan says real backup/restore and production-like load execution remain. docs/load-test-signoff.md says Documentation complete and Execution evidence pending; it explicitly says no checked-in end-to-end environment load evidence exists. The plan's recommended baseline also differs from current pom.xml and the current Fortune 500 modular-monolith objective.

Wrong path. Do not translate [x] into deployed, exercised, secure, scalable, or production-ready. A document, dashboard JSON, benchmark harness, or procedure proves only its own existence until execution evidence is attached.

sed -n '688,760p' docs/video-clipping-service-implementation-plan.md
sed -n '28,180p' docs/load-test-signoff.md
rg -n 'not prove|pending|not executed|still must|template' docs/backup-and-restore-validation.md docs/load-test-signoff.md docs/dlq-redrive-procedure.md
rg -n 'spring-boot-starter-parent|<java.version>' pom.xml

Status. ACTIVE evidence-labeling risk. Use current code/configuration and dated execution records as truth; route documentation corrections through change control.

Investigation Record Template

Use this structure in a new incident note or pull request:

Symptom:
First known good / first known bad:
Scope and environment:
Evidence label: FIXED | ACTIVE | PARTIAL | HISTORICAL | OPEN | NOT EVIDENCE
Candidate mechanism:
Prediction before experiment:
Read-only or approved experiment:
Observed result:
Negative observations the mechanism also explains:
Historical analog and commit:
Rejected paths and why:
Behavior/config/security consequences:
Change-control classification:
Validation required before promotion:
Remaining uncertainty:

Do not fill the root-cause field until one mechanism explains the positive and negative observations. Record a disconfirming result; it prevents the next engineer from repeating the dead end.

Provenance and maintenance

Primary evidence: reachable Git history through 97ba827, current source/tests/configuration, planning and operations documents, current .git/info/exclude, Surefire reports, and dated ignored workspace artifacts, all inspected on 2026-07-21. Workspace artifacts and unreachable Git objects are volatile and are never load-bearing proof.

Re-verify the reachable incident sequence:

git log --oneline --all --reverse | rg '7e8a214|9df9736|bd620fc|66e998e|7307082|0d45f89|8f4804c|3d961f2|1737d8b|5d889b0'

Re-verify current storage, queue, cleanup, and segmentation contracts:

rg -n 'materializeSourceObject|uploadGeneratedClip|RETRY_SCHEDULED|markTerminalFailure|cleanupLocalFiles|preserve-input-quality|force_key_frames' src/main src/test src/main/resources

Re-verify local-model prohibitions and hermeticity risks without starting workers:

rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|strict-runtime|auto-start|fallback-to-heuristic' tools src/main src/test src/main/resources

Re-verify volatile project artifacts and ignore rules:

git status --short --ignored
git ls-files input output tmp .venv-local-asset .venv-local-cv
sed -n '1,160p' .git/info/exclude

Re-verify ambiguity around rewrites and the deleted test:

git fsck --no-reflogs --unreachable --no-progress
git log --all --follow --name-status -- src/test/java/org/example/videoclips/api/TenantQuotaControllerTest.java

Re-verify documentation claims against their own caveats:

rg -n 'All repository-scoped|Execution evidence pending|no checked-in evidence|Remaining work' docs