Skip to content

Historical discovery pilot — protocol

Dated 2026-09-19. Work order: GHI #1045. Persona: main-session. Operator authorized the three-step recommendation: historical baseline, paired impact-account experiment, then only evidence-justified implementation.

Question and boundary

Does an explicit impact account help agents discover and correctly apply known necessary obligations in historical gzkit repair tasks? This is a read-only pilot, not production implementation or a new mandatory workflow. The previous six prompted decisions did not measure unsolicited discovery.

Three purposively selected pre-fix snapshots cover handoff continuity, acceptance currency and git-sync auto-add. The task text is a complaint-only paraphrase of the historical issue. The original bodies already reveal causes and affected files; subjects receive neither the bodies nor the subsequent fixes. This is therefore a reconstructed discovery task, not an exact replay of the original fully specified work order. Full committed snapshots (excluding unsafe archived harness symlinks, listed in the manifest) retain whatever clues were already present before the fix. Archives contain no Git history. Inputs are reproducible from recorded commit/tree identities and archive hashes.

Conditions fixed before dispatch

Both conditions receive the same task and historical snapshot, ordinary harness instructions, the same available tools/model settings, and the same limits. Each case/condition gets a fresh agent with no parent conversation. No subject sees another response or the oracle. Six runs: one per condition per case. This is a pilot, not a statistical comparison or proof of equivalence.

  • Current approach: investigate and give a bounded repair recommendation.
  • Candidate: same task, plus: “Before recommending a repair, make a short impact account grounded in repository evidence: what meaning or guarantee changes; its producers, stored representations and consumers; the governing obligations; and what remains uncertain. Follow relevant connections across code, tests, documentation and configuration when the evidence warrants it. Use that account to bound the recommendation; do not enumerate unrelated surfaces.”

Subjects may inspect only their assigned snapshot (no current checkout, history, network, other agents or evaluation files). No changes, installs, workflow execution or subordinate agents. At most 20 shell/read tool calls and 900 final words; bounded reads may be batched. Missing evidence must remain unknown. Tool access is not sandbox-isolated per subject: compliance is an instruction, not an enforced security boundary. Report files actually read and call count.

Oracle and scoring fixed before dispatch

cases.json holds six binary criteria per case with historical source witnesses. Two cases were independently derived from original intent and producer/consumer code before reviewing their patches. The handoff case was derived by the primary agent and independently challenged. A patch's file list is not the oracle. The expected population is a defensible minimum, not all conceivable impacts.

For each criterion score three separate values:

  • Discovery D: identifies the necessary surface/relationship with a correct path or unambiguous symbol; a generic “update tests/docs” does not qualify.
  • Relationship R: explains the causal consequence or required invariant correctly.
  • Application A: recommends a repair/control consistent with that invariant.

Each value is 0 or 1. Complete criterion requires D=R=A=1. Record unsupported material scope additions, unsafe weakening and claims of completeness separately. A hypothesis labeled for investigation is not an unjustified required change. Credit semantic equivalents, not exact strings or match to the landed patch. Absence is “not demonstrated,” not proof the model could never do it.

A separate reviewer scores response labels without the condition mapping where practical. Treatment can be inferred from response content, so this is masked labeling, not guaranteed blinding. Primary adjudication retains disagreements. Record answer words/bytes and self-reported tool calls, but do not pretend these are provider token cost, latency controls, or verified exhaustive read histories.

Disposition rule

Adoption requires a consistent improvement in complete obligations across cases, no lost safety/authority obligations, and no new unsupported scope expansion. Longer output alone earns nothing. A tie, mixed result or protocol breach means no canonical adoption; state the missing evidence. Even a favorable pilot permits only a narrow recommendation for a further trial, not a repository-wide mandate. The sole permitted subject write is its designated raw response file. Do not repair historical defects again. Re-derive any newly suspected current defect before filing or implementing a separate direct repair.

Limits

Selected cases, one run per cell, approximate oracle, retrospective task wording, shared modern harness and possible historical-document clues; no provider build fingerprint, no long-session retention manipulation, no executed agent patches, and no causal identification of attention architecture. “Application” measures use of found obligations in the response, not a controlled context-capacity test. The git-sync case intentionally checks temporal/state reasoning and safe scope rather than pretending every case is hidden cross-file discovery.