video_editing_poc/.claude/skills/video-editing-research-methodology/SKILL.md

name

description

video-editing-research-methodology

Load when turning an observation or hunch in this video-editing repository into a controlled experiment, preregistering numeric predictions, designing baselines/counterfactuals/ablations, assigning adversarial review, managing a disabled experiment flag, replicating offline on macOS and Linux, deciding whether a result is a candidate for adoption, or retiring an inconclusive or disproven idea without losing the negative evidence.

Video Editing Research Methodology

Purpose

Turn a hunch into an auditable result without weakening production controls. Require one causal mechanism (the rules by which a cause produces an outcome) to explain positive observations, negative observations, boundary cases, and failures. Accept it only after an assigned reviewer tries to refute it with a prediction that differs from competing explanations.

Use repository facts verified on 2026-07-21. Treat numeric targets from planning documents or sibling skills as candidate thresholds until video-editing-change-control approves and freezes them.

Route The Work

Need	Use this skill?	Exact sibling
Form a hypothesis, preregister numbers, run a research lifecycle, or retire an idea	Yes	This skill
Select an open problem or make an external novelty claim	No	video-editing-research-frontier or video-editing-external-positioning
Derive a metric, invariant, causal discriminator, or uncertainty calculation	No	video-editing-proof-and-analysis-toolkit
Execute the current single-source cinematic-quality program	No	video-editing-cinematic-highlights-campaign
Define release evidence, goldens, or the acceptance portfolio	No	video-editing-validation-and-qa
Diagnose an unexplained live failure	No	video-editing-debugging-playbook, then video-editing-failure-archaeology
Change code, dependencies, configuration, defaults, runtime, or promotion status	No	video-editing-change-control
Look up an existing property or add a configuration axis	No	video-editing-config-and-flags

Do not use this skill for routine implementation, incident triage, release certification, or current-campaign execution; use the exact sibling named in the table.

Do not label missing authentication, offline packaging, fail-closed behavior, reproducible builds, measured QA, or deployment controls as research. Those are engineering obligations. Use research only when two or more plausible mechanisms need a controlled comparison.

Define The Terms Once

Term	Definition in this project
Hypothesis	A falsifiable claim that names a mechanism and predicts measured outcomes before data collection.
Baseline	The frozen current or simplest approved method against which a candidate is compared.
Counterfactual	An otherwise equivalent case in which the proposed cause is absent or changed.
Ablation	A counterfactual made by removing exactly one candidate component while holding the rest fixed.
Negative control	A case where the mechanism predicts no effect; it detects leakage, confounding, and false positives.
Slice	A preregistered subgroup such as source category, speech presence, lighting, codec, platform, or failure mode.
Holdout	Data frozen before tuning and opened only for the confirmatory run.
Uncertainty	A stated interval or error bound around an estimate, produced by a preregistered method.
Replication	A repeat on independently provisioned runtime or unseen sources, not a rerun of the same warm process.
Adversarial reviewer	An independent person assigned before results to construct the strongest counterexample and try to falsify the mechanism.
Candidate	A replicated research result eligible for change-control review; it is not production-ready.
Retirement	A documented decision to stop an idea while preserving its prediction, evidence, and revisit condition.
nDCG	Normalized discounted cumulative gain, a ranking metric that rewards relevant candidates more when they appear earlier.

Enforce The Research Perimeter

Stop before experimentation when any item fails:

• Run with network access denied. Do not call external AI services, model hubs, package indexes, remote telemetry, or loopback model services during certified experiments.
• Pre-provision every dependency and model. Record immutable version, SHA-256, source, license, approved use, platform, runtime loader, and resource envelope. Never bootstrap or download at runtime.
• Use only rights-cleared footage, voices, music, sound effects, fonts, LUTs, and model outputs. Record consent or voice rights where applicable. Generated output is not automatically licensed.
• Reject missing capabilities. Never replace requested voiceover, music, or SFX with silence, tones, host TTS, generic media, another model, or another analysis mechanism.
• Obtain explicit render approval before FFmpeg creates a candidate render. Analysis permission is not render permission.
• Leave every production-facing packaged default unchanged. Activate research only in an isolated, approved environment with a dedicated experiment control whose safe default is off.
• Keep research inputs and results out of production queues, tenant data, normal application logs, and uncontrolled metrics. Preserve privacy and retention classifications.
• Route any executable code, flag, schema, fixture, dependency, or behavior change through video-editing-change-control before use.

Do not run tools/run_local_cv_worker.sh, tools/run_local_asset_worker.sh, or ordinary application startup as an experiment. On the verified baseline, those paths can install packages, resolve model weights, scan inputs, or render. Inspect first:

rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
rg -n 'auto-start:|render-enabled:|require-director-approval:|fallback-to-heuristic:' src/main/resources/application.yml

Use The Research Lifecycle

Never skip a gate. A failed gate produces a negative result or retirement record, not an improvised change to the method.

Stage	Required artifact	Gate to advance
0. Observe	Observation record with evidence label	The symptom is reproducible or explicitly marked historical/open.
1. Hypothesize	Preregistered experiment record	Mechanism, alternatives, numeric predictions, controls, slices, and stop rules are frozen.
2. Isolate	Approved disabled experiment control and evidence location	Default is off; scope, owner, expiry, cleanup, and production isolation are verified.
3. Calibrate	Frozen baseline on development data	Metric definitions, denominators, uncertainty method, and data exclusions are fixed.
4. Discriminate	Positive, negative, boundary, counterfactual, and ablation results	Competing mechanisms make different predictions and the observations select among them.
5. Replicate	Unseen-source and platform replication packet	Result repeats offline on supported macOS development and target Linux/VPS runtime.
6. Refute	Signed adversarial review	Independent reviewer cannot produce an unexplained counterexample within the preregistered scope.
7. Decide	Adoption candidate or retirement record	Evidence is complete; no result is silently left behind an expiring flag.
8. Govern	Change-control decision	Adopt through normal gates or remove/disable and preserve the negative ledger.

Stage 0: Record The Observation

Distinguish an observation from an explanation. Record the exact source commit, input identity, effective configuration, environment, and command that exposed it. Label a generated file, plan checkbox, or hard-coded QA boolean NOT EVIDENCE when it cannot support the claim.

Before inventing a new explanation, search the established history:

git log --oneline --decorate --all
rg -n -i 'TODO|FIXME|not implemented|execution evidence pending|fallback|silence|tone' docs src tools

Use video-editing-failure-archaeology to determine whether the mechanism is already fixed, active, partial, historical, or open. Do not rerun a retired path unchanged.

Stage 1: Preregister A Numeric Hypothesis

Write and freeze the experiment record before inference, rendering, or opening the holdout. Make the hypothesis predict numbers, not adjectives. Include expected direction, minimum effect, allowed regression, denominator, interval method, and expected values for every negative control.

When no trustworthy baseline exists, run a labeled exploratory baseline study first. Use its data only to design a later confirmatory study. Do not set a threshold from a dataset and claim success on that same dataset.

For highlight work, name metrics such as nDCG@3, Recall@3 at a stated temporal IoU, false-positive rate on no-highlight footage, per-category effects, and blinded preference. For audio/render work, name probe-derived duration, loudness, true peak, silence/tone rejection, A/V synchronization, technical failure rate, and blinded rubric dimensions. Get formulas and current campaign floors from the proof toolkit and cinematic campaign; do not duplicate or silently revise them here.

Stage 2: Isolate With A Safe Experiment Control

The verified repository has no dedicated experiment registry. Current scheduler, fallback, model, and render properties are operational controls, not research assignment controls. Do not repurpose them and do not treat render-enabled or fallback-to-heuristic as experiment flags.

Introduce any experiment control only through change control. Require this contract:

Field	Required value
Identifier	Stable, descriptive, and unique; never a user or media identifier.
Owner	Named accountable team/person and independent reviewer.
Safe default	false in every packaged configuration and matchIfMissing; missing means disabled.
Scope	Approved fixtures/environments only; never implicit all-traffic assignment.
Start and expiry	UTC dates with a short expiry and automated or reviewed removal checkpoint.
Activation	Explicit test/experiment configuration outside production defaults.
Observability	Bounded, low-cardinality experiment ID; no source path, prompt, secret, or personal data.
Failure	Fail the experiment; never fall back to baseline and label the result candidate.
Removal	Delete or permanently disable after adoption/retirement; owner verifies no stale path remains.

Do not create a speculative flag without a funded experiment, owner, decision date, and deletion condition. A flag is not approval to render or bypass asset/model preflight.

Stage 3: Freeze Baselines, Inputs, And Analysis

Use a source-disjoint development/holdout split. Compute a corpus-manifest SHA-256 and split SHA-256. Freeze the baseline commit, candidate commit, toolchain, feature schema, random seeds, render plan, models, assets, licenses, and analysis script before the confirmatory run.

Design element	Required practice	Common invalid result
Baseline	Run the same inputs, candidate budget, render profile, and reviewer protocol.	Candidate gets more clips, assets, compute, or review time.
Counterfactual	Change only the claimed cause and verify output equivalence elsewhere.	Two renders differ in timing, encoding, or mix beyond the intended factor.
Ablation	Remove one component per variant; preregister multiple-comparison handling.	A kitchen-sink comparison cannot assign causality.
Negative control	Include shuffled timestamps, no-highlight footage, silence/tone, missing model, or unrelated audio as appropriate.	Positive-only examples hide leakage or fallback.
Slice	Report each preregistered category/platform/failure slice and its denominator.	Aggregate improvement hides a harmed category.
Uncertainty	Report confidence interval or error bound and method; include reviewer disagreement.	A point estimate or one clip is called a result.
Exclusion	Freeze objective exclusion rules before results; publish every exclusion count/reason.	Failed outputs disappear from the denominator.

Stage 4: Execute Once, Preserve Everything

Run only the exact approved commands in the record. Capture stdout/stderr, exit codes, timestamps, resource measurements, raw per-item results, failed outputs, and hashes. Do not edit the hypothesis, threshold, split, or exclusion rule after seeing results. Record deviations and classify the run invalid; preregister a replacement instead of repairing history.

Add every failed prediction, neutral ablation, negative control, and platform difference to the negative-result ledger. Preserve a false start even when a later variant works.

Stage 5: Replicate Offline On macOS And Linux

Use the same committed source and immutable corpus, split, dependency, model, and asset manifests. Provision them before execution through the approved artifact process. Deny egress for the whole process tree. Record OS version, architecture, Java, Maven, Python runtime if applicable, FFmpeg, ffprobe, CPU/GPU, memory, precision, thread/concurrency settings, seeds, and hashes.

Define replication before running as either:

• Byte replication: identical artifact SHA-256 is required; or
• Metric-bounded replication: deterministic bytes are not expected, so every preregistered metric must remain within an approved tolerance and every categorical gate must agree.

Do not average platform disagreement away. Mark the result platform-specific or inconclusive when one supported platform fails. A warm-cache repeat on one machine is not replication.

Stage 6: Assign Adversarial Refutation

Assign the reviewer before results. The author, experiment operator, fixture annotator, and promoter must not be the sole adversarial reviewer. Blind media variants and randomize order where practical.

Require the reviewer to:

1. State the strongest competing mechanism.
2. Identify one case where the proposed mechanism predicts an effect and the competitor does not.
3. Identify one negative case where the proposed mechanism predicts no effect.
4. Look for data leakage, source overlap, hidden fallbacks, changed budgets, reviewer unblinding, selective exclusions, uncontrolled randomness, and platform-specific behavior.
5. Recompute a sample from raw data and verify hashes/licenses for a sample of inputs/models/assets.
6. Explain every positive, negative, boundary, and failed observation with the proposed mechanism.
7. Return supported, refuted, or inconclusive; never return “looks good.”

Refutation is not an approval gate substitute. A supported result still goes through change control, validation, security, operations, licensing, and render approval.

Stage 7: Adopt Or Retire

Classify the idea exactly once at the decision date:

Status	Meaning	Required next action
candidate	First controlled result met its preregistered gate.	Replicate; do not promote.
replicated-candidate	Held-out and platform replication passed adversarial review.	Submit the eight-part decision packet to change control.
inconclusive	Evidence cannot distinguish mechanisms or uncertainty crosses the gate.	Retire temporarily or fund a newly preregistered discriminator.
refuted	A prediction failed or a counterexample broke the mechanism.	Retire; remove/disable the experiment path.
retired	Work stopped for evidence, cost, rights, safety, operability, or priority reasons.	Preserve the ledger, stop reason, and revisit condition.

For an adoption candidate, provide: requirement; selected approach; alternatives; benefits/trade-offs; operational consequences; security implications; verification; and revisit conditions. Then follow video-editing-change-control. Research status never changes a production default.

Use These Records

Keep each record in the approved review artifact or a repository docs-of-record location established through change control. Do not invent a new production path or store secrets, proprietary model weights, personal data, or unlicensed media in the record.

Experiment Record Template

Experiment ID / title / UTC date:
Owner / operator / independent adversarial reviewer:
Observation and evidence label:
Question:
Mechanism:
Strongest competing mechanisms:
Baseline commit and exact method:
Candidate commit and safe-disabled experiment control:
Flag owner / safe default=false / start / expiry / deletion condition:
Corpus-manifest hash / split hash / annotation version / licenses:
Dependency, binary, model, asset hashes and approved-use records:
macOS and Linux runtime/hardware/resource envelope:
Primary metric, denominator, formula, predicted baseline and candidate numbers:
Minimum effect / allowed regression / uncertainty method / alpha or interval:
Slice predictions and minimum sample counts:
Positive / boundary / negative controls and predicted numbers:
Counterfactuals / one-factor ablations / multiple-comparison method:
Randomization / blinding / seeds / exclusion rules:
Exact offline commands and egress-denial evidence:
Render approval reference, if rendering is required:
Stop rules / retirement rule / revisit condition:
Raw evidence location and retention/classification:
Observed results, deviations, failed predictions, and uncertainty (after run):
Replication outcome by platform (after run):
Disposition and change-control reference (after review):

Adversarial Review Template

Reviewer / UTC date / independence conflicts:
Mechanism reviewed:
Strongest alternative:
Differentiating positive prediction:
Required negative prediction:
Counterexample attempted and exact command/data:
Leakage/fallback/blinding/exclusion/platform checks:
Raw result recomputation and hash/license sample:
Unexplained positive, negative, boundary, or failure observations:
Verdict: supported | refuted | inconclusive
Required follow-up and change-control blockers:

Negative-Result Ledger Template

Experiment ID / UTC date / commit:
Hypothesis and preregistered numeric prediction:
Observed number and uncertainty:
Failed slice/control/ablation/platform:
Evidence hashes and exact command:
Why the mechanism failed or remains unresolved:
Decision: refuted | inconclusive | retired
Experiment control removed/disabled by / date:
Do not retry unchanged because:
Revisit only when this falsifiable condition changes:

Stop And Revisit Discipline

Stop immediately for a rights gap, checksum mismatch, unexpected network attempt, automatic download, external AI call, missing model, fallback, placeholder audio, unapproved render, path escape, sensitive data leak, production-default drift, invalid blinding, split contamination, or corrupted evidence.

Stop the idea at the preregistered resource cap, sample size, failure-rate limit, effect threshold, or expiry date. Do not extend until significance appears. Revisit a retired idea only when its recorded condition changes, such as a newly licensed local model, a larger source-disjoint corpus, a corrected measurement defect, a new discriminating mechanism, or a documented resource-envelope change.

Learn From The Repository's Actual Origins

Historical origin	Methodological rule it established
Real storage behavior arrived across commits beginning 7e8a214; earlier state/metadata was not a durable media path.	Measure the real side effect and artifact existence. A successful state transition is not delivery evidence.
Commit 1737d8b moved terminal failure ownership to the DB queue's retry/DLQ policy.	Put the decision at the layer that owns the mechanism; test transient, terminal, duplicate, and boundary attempts.
Commit 5d889b0 said “working version but not cinematic.”	Separate renderability from semantic selection and creative quality; require blinded baselines, rubrics, and ablations.
Category-aware planning and richer rendering followed in commits such as 5006dfd, but no controlled causal comparison is recorded.	Treat richer analysis, effects, and assets as candidates until each contribution survives counterfactual testing.
Commit 9b56e89 added structural render QA; the current highlight path adds selected duration/black/silence/peak probes while other checks remain structural.	Diagnostics must distinguish measured outputs from plan/command assertions and predict reviewer-relevant failures.
Commit 97ba827 added local asset generation with runtime install/model resolution and silence/tone paths.	“Local” does not mean offline, licensed, immutable, or fail-closed; test missing and corrupted capabilities as negative controls.

Use video-editing-failure-archaeology for the full chronicle. History suggests research questions; it does not prove a current mechanism.

Completion Checklist

• Observation, mechanism, alternatives, and numeric predictions were frozen before the run.
• Baseline, counterfactuals, one-factor ablations, negative controls, boundary cases, slices, denominators, exclusions, and uncertainty were specified.
• Dependencies, models, assets, inputs, splits, licenses, approvals, and checksums are complete.
• The experiment was isolated behind an approved safe-disabled control with owner and expiry.
• No download, network, external AI, unlicensed asset, placeholder, unapproved render, fallback, or production-default change occurred.
• The result replicated offline on supported macOS and target Linux/VPS, or is labeled otherwise.
• An independent reviewer attempted adversarial refutation and explained positive and negative cases.
• Failed predictions and neutral results are in the negative-result ledger.
• The idea became a replicated candidate routed through change control, or was documented and retired.

Provenance and maintenance

Facts and volatile repository observations were rechecked on 2026-07-21 against application.yml, current source/tests, the six cited commits, and the completed sibling skills.

Re-verify the research perimeter and operational-control defaults:

rg -n 'pip install|get_pretrained|write_silence|write_fallback_tone|fallbackTone' tools src/main
rg -n -C 3 'fallback-to-heuristic:|strict-runtime:|local-director:|highlight-scheduler:|render-enabled:|require-director-approval:' src/main/resources/application.yml

Re-verify historical claims and measurement implementations:

git show -s --format='%h %cs %s' 7e8a214 1737d8b 5d889b0 5006dfd 9b56e89 97ba827
rg -n 'RETRY_SCHEDULED|markTerminalFailure|duration_matches_timeline' src/main src/test

Re-verify the absence or presence of a dedicated experiment registry before documenting a flag:

rg -n -i 'experiment|feature.?flag' pom.xml src/main src/test src/main/resources docs

If any command changes its result, update this skill and the authoritative sibling in the same knowledge-only change. Do not reinterpret drift as approval to alter behavior.