Historical discovery pilot — results¶
Dated 2026-09-19. Persona: main-session. Work order:
GHI #1045.
Current-code reference for this investigation: 7bc931575.
Disposition¶
Do not adopt the impact-account instruction as a canonical rule. Six fresh historical investigations do not establish a reliable improvement. Both arms found important producer/consumer relationships, but both proposed an unbooked reversal of the sync guard's explicit inspection-error policy. One candidate run also breached the snapshot-only boundary by listing the current checkout's persona directory. The frozen disposition rule rejects adoption on a protocol breach independently of the criterion totals.
This completes the commissioned pilot; it does not complete general semantic dependency discovery. No runtime code, canonical instruction, proof-currency policy, ADR or OBPI was changed by this experiment. A null result is a reason to withhold an unsupported remedy, not to manufacture another implementation.
What was measured¶
Three complaint-only historical tasks, two conditions each, one fresh agent per cell. Both conditions had the same case snapshot and resource limits. The candidate alone received a request to account for changed meaning, producers, representations, consumers, obligations and uncertainty before recommending a repair. The protocol, cases and exact inputs were frozen before dispatch. Curation records the independent oracle challenge and pre-dispatch corrections.
All six answers are retained verbatim as R01–R06. A separate reviewer received response labels and case identities without the condition mapping; treatment could still be inferred from the writing. Scores concern expressed discovery, reasoning and proposed application, not executed patches or actual regression rates. Criteria describe a minimum expected population, not exhaustive scope.
Criterion results and adjudication¶
The primary agent reviewed the raw answers, frozen criteria and cited historical witnesses and accepts the independent scores without changing any bit. D = discovery, R = correct relationship, A = consistent application; complete requires all three. Each cell has six criteria.
| Case | Baseline D / R / A / complete | Candidate D / R / A / complete |
|---|---|---|
| H: handoff | 6 / 6 / 5 / 5 | 6 / 6 / 5 / 5 |
| C: acceptance | 5 / 6 / 5 / 5 | 6 / 6 / 6 / 6 |
| S: sync | 6 / 6 / 6 / 6 | 6 / 6 / 6 / 6 |
| Total, out of 18 each | 17 / 18 / 16 / 16 | 18 / 18 / 17 / 17 |
The candidate's one additional complete criterion is C5: naming the contrary environment-invalidation test and specifically targeting the governance freshness explanation for reconciliation. R05 lists that document among reads, but does not demonstrate finding and reconciling its particular claim. Reading a file is not evidence that its governing obligation reached the recommendation.
That one-point difference is borderline, not robust comparative evidence. R02 does not quote the governance sentence either; the reviewer credits its named document, explicit test and targeted freshness amendment together. A stricter reading removes the candidate point and yields 16–16. The primary adjudication retains the frozen semantic-equivalence rule and credits it, while reporting this sensitivity. R05's C4 diagnostic repair also receives credit from its explicit false-blame diagnosis plus conditional diagnostic proposal; it is less explicit than neutral mismatch wording. Neither ambiguity changes the no-adoption disposition.
Both H answers discover the budget and advise protecting lineage under overflow, but omit explicit preservation of freshness warnings in that overflow control (H5 A=0). This is missing demonstrated coverage, not an observed regression. Both S answers achieve six complete minimum criteria and carry two material scope flags: reversing fail-open inspection policy and expanding admission to the no-auto-add/final-recheck path. A perfect minimum-coverage score therefore does not certify a safe bounded recommendation. These flags were recorded separately, as the protocol required, rather than retroactively adding criteria.
Both S answers also propose verified generated-package exceptions. The reviewer checked the historical bundled surface-sync attribution rule and accepted that as a defensible design alternative, while retaining its added implementation burden and unresolved edge cases. Both H answers propose richer ancestor-body projection and orientation coverage; these plausibly address the complaint but are not mandatory oracle items. No score demands matching the historical patch. The reviewer found no claim that a proposed repair had actually been executed.
Execution integrity and burden¶
The verification record confirms all six prompt hashes and the frozen oracle/protocol hashes, response equality with the agents' saved answers, and snapshot integrity: 7,516 H files, 7,401 C files and 6,860 S files compared with their committed archives, with zero content differences. Unsafe archived harness symlinks were omitted before dispatch and are listed in the case manifest; no ordinary source or document was omitted.
| Case | Baseline run / words / calls | Candidate run / words / calls |
|---|---|---|
| H: handoff | R01 / 733 / 12 | R04 / 673 / 15 |
| C: acceptance | R05 / 650 / 10 | R02 / 703 / 12 |
| S: sync | R03 / 654 / 8 | R06 / 667 / 10 |
| Total | 2,037 / 30 | 2,043 / 37 |
Word counts use whitespace splitting over the entire final answer. Call counts come from the command traces and match self-reports. They are neither provider token cost nor controlled latency measurements. Both arms respected the 900-word and 20-call per-run ceilings. Several reads returned truncated output; whole-file coverage is not certified. R05 also had a failed search against a nonexistent historical directory, retained in its trace. No retry erased either that failure or R04's directory-listing deviation.
Implications for the three pillars¶
- Usable context: within these short investigations, a named budget and accessible files did not ensure every constraint was applied. This does not test long-session retention, attention scaling or context-window capacity.
- Relationship reasoning: both conditions traced real producers and consumers, including the distinction between valid Task syntax and semantic ownership. Finding those connections did not reliably bound the remedy.
- Discovery: the tasks withheld the later patches and affected-file lists. This improves on the previous explicitly cued decision exercise, but the selected snapshots still contain clues. Neither result measures recall over all unknown obligations, and an impact account is not a dependency oracle.
The actionable adjustment is to retain separate evidence for finding a surface, understanding its obligation and honoring it in the proposed change. This is an interpretation of the evaluation, not a new mandatory workflow. In particular, do not narrow proof currency on an agent's self-selected impact list.
Current-code reconciliation and remaining boundaries¶
These are historical failures, not newly discovered current defects. The
acceptance case's old repair omitted its contrary governance prose, but reading
current docs/governance/acceptance-obligations.md and the identity producer
confirmed that the current documentation already distinguishes ambient
environment from artifact identity. No duplicate repair was opened.
Earlier handoff and skill-audit repairs remain described in the recovered assessment. This pilot neither replaces their regression evidence nor expands their completeness claim. GHI #1028 remains production observation, #1029 the proof-currency design, and
943 adjacent model-profile calibration; all three were verified open on¶
2026-09-19. No synthetic OBPI was used to manufacture production evidence.
Before any broader claim, a further authorized study would need a held-out case set, repeated runs, stronger access isolation and actual patch verification. That is a limit on inference, not a newly initiated study or a prerequisite invented for closing this bounded investigation.