Methods, and the record of what went wrong.

What happened

The first leaderboard covered the whole verified roster and earned zero scores. It published 16 rows across 14 distinct public servers, two of them driven twice, and not one row carried a score field. The outcomes were 7 DEGENERATE, 8 INSUFFICIENT_SURFACE and 1 GATE_FAILED at the construct gate.

What it means

At the time this read as fourteen servers being hard to drive. It was not. Most of it was our own task generator, and entries 02 and 03 are the arithmetic. Publishing the zero anyway is the only reason those two entries can be written at all: an eval that had quietly loosened a floor to get a number on the board would have had nothing to go back and re-read.

Evidence

• commit 302a62e, "Leaderboard first pass: 14 public servers, 16 rows, zero scores earned"
• file site/data/runs.json at 302a62e, 16 rows, 14 distinct slugs, score absent from every one

What broke

Under the earlier generator a model with no server and no tools at all passed most of nearly every suite. The published no-tools rates were: astro 12 of 12, context7 10 of 10, cloudflare docs 2 of 2, coingecko 11 of 12, hugging face 11 of 12, svelte 10 of 12, better auth 8 of 9, microsoft learn 8 of 9, and aws knowledge 6 of 8. Under the null-hard generator the same null model passes zero on every suite: 0 of 12, 0 of 10, 0 of 24.

What it caused

Seven of the fourteen first-pass servers now carry a scored row: gitmcp 12 of 12, exa 12 of 12, deepwiki 10 of 10, coingecko 11 of 12, convex 11 of 12, hugging face 22 of 24, svelte 9 of 12. Three of the seven earlier DEGENERATE verdicts and four of the seven earlier INSUFFICIENT_SURFACE verdicts were our generator, published as verdicts about the server. Admission recovered the same way: gitmcp admitted 2 of 12 under the earlier generator and 12 of 24 under the null-hard one.

How it was caught

By turning the generator null-hard and re-running the roster. The earlier rows were never deleted, so the comparison exists. Every row on the board carries its generator version as a chip, and rows built by different generators are never read against each other.

Evidence

• file site/data/runs.json at 302a62e, nine rows with no generator version recorded carry a null_baseline record with the no-tools rates above
• file data/runs.json, current: every row recorded as fitness-report-generator/2 carries a no-tools rate of 0
• commit 7f3b606, the first scored row under the null-hard generator: svelte 9 of 12 first-try, with coingecko and gitmcp refusing at the construct gate

What broke

Across the runs recorded as fitness-report-generator/2, candidates were dropped in bulk with the rule invalid-check and the detail "regex does not compile". The generator instructions made a tool_result_matches check required whenever the answer lives in a tool result, and never stated the regex dialect or the JSON escaping the validator would apply. The validator then compiled with new RegExp(c.pattern) and dropped on any failure. There was no repair pass for an invalid check; only an answer leak bought the one regeneration attempt.

One correction to this entry. It said every one of those drops was a tool_result_matches check. Counted over the drop ledgers on disk it is 196 of 198, and the other two are regex checks.

What it caused

Three runs lost 24 of 24 candidates: microsoft learn, better auth, and the aws knowledge attempt of 16:59:20.891Z. Three more kept 1 of 24: cloudflare docs, context7 and vercel docs. Astro kept 4 of 24. All seven of those runs are published right now as GATE_FAILED or INSUFFICIENT_SURFACE, which reads on the board as a statement about the server. Microsoft learn generated 9 usable tasks under the earlier generator and 0 under the null-hard one.

How it was caught

By totalling the drop counts across the sweep after the roster was rebuilt. Those counts were already being serialised into every published run, which is the only reason the number is recoverable at all; the per candidate ledger behind them is not published, which is entry 17.

What was fixed

Every pattern-bearing check now goes through one door, compileCheckPattern() in src/tasks/synthesize.ts, which runs repairPattern() first. Constructs from other regex dialects that have an exact ECMAScript translation are translated: an inline (?i) or (?s) flag group becomes a flag or a character class, (?P<name>...) becomes (?<name>...), and a (?#comment) is removed. A pattern JavaScript already accepts comes back byte for byte: over the 183 patterns the published sweep actually admitted, and over 19,902 generated patterns JavaScript accepts, the function returned every one unchanged and repaired none.

Nothing was loosened to make patterns pass. A construct with no faithful translation is still dropped, and the drop now names it: atomic group, possessive quantifier, conditional group, an inline flag group it cannot express, and a multiline $, which has no ECMAScript form that provably matches the same text. The generator prompt gained a section stating the dialect it is compiled in, and the synthesizer and check policy versions moved from 2 to 3 so suites from the two generators cannot land in one table.

Why it is still open

Nothing has been re-run under the fix. All 17 rows on the board still carry fitness-report-generator/2, so the seven servers above are still published as GATE_FAILED or INSUFFICIENT_SURFACE in the slot where a reader looks for a fact about the server. The entry closes when a run under the repaired generator is on the board beside them, not when the code changes.

And the drop ledger of the damaged runs never stored a pattern, which is entry 17, so the constructs that actually caused those 198 drops cannot be recovered. The repair table above is built from what the dialects do, not from the patterns that failed here. Whether it would have saved these particular candidates is not something this record can show.

Evidence

• file data/runs.json, sum gates.records[structural].detail.synthesis.dropsByRule["invalid-check"] over the seventeen rows recorded as fitness-report-generator/2: 209, against 410 in the matching yield.candidates. Sixteen of those rows were on the board when this entry was written and summed to 198 of 386; the seventeenth is the aws knowledge run restored by entry 16, which contributes 11 of 24
• artifact runs/sweep/*/suite-meta.json, the drop ledgers on disk: 198 invalid-check drops, 196 on a tool_result_matches check and 2 on a regex check. Not linkable: runs/ is gitignored
• source src/tasks/synthesize.ts lines 1198 to 1202 before the fix, the instruction that requires the check and states no dialect
• source src/tasks/synthesize.ts lines 1581 to 1584 before the fix, the new RegExp compile and the drop
• source src/tasks/synthesize.ts, repairPattern() and compileCheckPattern(), cited by name because the fix is not in a published commit yet
• test test/synthesize.test.ts: "returns a pattern JavaScript already accepts byte for byte"; "admits every pattern the published sweep admitted, so validation is not the over-strict half"; "translates the named-group and comment syntax of other dialects"; "refuses to translate a multiline $, because the dialects disagree about it"; "still drops an unsalvageable pattern, and says which construct did it"

What broke

The generation-time scan normalised task parameters with Object.entries() and kept only string, number and boolean values. On the wire, params is an array of {name, value} objects, so every value failed the type guard and the scan rendered the task template with no bindings at all. It was checking a template, not a prompt.

What it caused

All 180 first-pass candidates, 12 requested across each of 15 sweep runs, went unchecked. Every one of those runs recorded answer_leak ok: true with an empty leak list and no regeneration attempted, and all 16 published rows carried the same empty record. An empty leak list is indistinguishable from a clean bill of health, which is exactly what it looked like.

How it was caught

While rewriting synthesis for the null-hard generator. It is documented in the source as a proven defect of the earlier one and pinned by two tests that assert the scan reads the rendered prompt and the wire shape.

Evidence

• source src/tasks/synthesize.ts lines 1339 to 1351, the bug documented in the past tense
• test test/synthesize.test.ts: "checks the RENDERED prompt, not the template, and reads the WIRE shape of params"
• test test/synthesize.test.ts: "binds params from the wire shape and the legacy record shape identically"
• file site/data/runs.json at 302a62e, all 16 rows carry the empty leak record
• commit 2836d72, the fix

What broke

The first integration fixture built its task suite on a bare "was the tool called" predicate. A stubbed-empty null model satisfies that by definition, so the null baseline killed the fixture as degenerate. The gate was right and the fixture was wrong.

What it caused

Nothing published, because it happened while the pipeline was being written. It is in the record because it is the cheapest available proof that the gate bites, and because the rule it produced now runs in production: a bare tool_called check is rejected at generation with reason check-too-permissive, allowed only on a declared error-path probe. Three published runs, coingecko, convex and hugging face, each record one such drop.

Evidence

• test test/pipeline.test.ts lines 57 to 59, the fixture builder's own note, "verified while writing this"
• source src/tasks/synthesize.ts lines 1564 to 1570, the production rule
• file data/runs.json, one check-too-permissive drop on each of the coingecko, convex and hugging face rows recorded as fitness-report-generator/2

What breaks, structurally

The reference agent is handed the answer key. For a substring or regex check, that means the check reads final text the agent could have produced from the key alone, with zero tool calls, against a server that is entirely dead. Left alone, the first gate that costs real money would pass on nothing at all.

What it would have caused

A paid gate that certifies a suite as constructible against a server that never answered. It never happened. The reached-through-server requirement is present in the initial commit, and the first scored row appears seven commits later, so no published row was ever produced by an unguarded construct gate. The gate is stated as a tautology-by-construction in the methods text of every report rather than presented as sound.

Evidence

• source src/cli.ts lines 1490 to 1500: reachedThroughServer requires an ok tool call to an expected tool, and the verdict is run.outcome.success && reachedThroughServer
• commit c5743b6, the initial commit, already carries the guard at src/cli.ts lines 645 to 654; src/gates/construct.ts has only ever had this one commit
• test test/pipeline.test.ts: "refuses at the construct gate when the reference pass never reached the server", driven by a client that quotes the answer key back

What broke

Confirmation evidence was keyed by task alone and tool-agnostically. A single clarification round on any tool in a task marked every later destructive call in that task as confirmed.

What it would have caused

A recorded, unconfirmed money transfer publishing as "0 destructive calls with no confirmation in front of them". That is the safety number on this board, and it would have read clean while the tape showed the opposite. No scored row was ever produced under the broken keying: it was caught and pinned in the initial build commit.

How it was fixed

Evidence is now keyed three ways, by task and tool together, by tool, and by an explicit unattributed bucket for evidence that names no tool. Clearing a call requires evidence that names the same tool and lands before the call executes.

Evidence

• source src/score/metrics.ts lines 267 to 307, the defect in the past tense and the ConfirmationEvidence shape that replaced it
• test test/score.test.ts: "does not let a confirmation about one tool clear a destructive call to another"
• test test/score.test.ts: "accepts a same-tool confirmation event that lands before the call"

What broke

The same predicate ran on registered suites and on bought extension batches with opposite consequences. A batch task that leaked its answer key, or shipped an unbound placeholder to an agent, was counted into a dropped tally and deleted without a word. The identical defect refused one run and silently shrank another, and the batch bought specifically to settle a verdict was the one place the defect could hide.

What it would have caused

Sampling bias pointing in exactly one direction: toward resolving the gate. The rule now splits by severity. An answer leak or a structural property violation inside a batch refuses the whole run, and a refused batch is voided whole. An ordinary admission failure, a duplicate or a null-screen deletion is dropped and said out loud.

Worth stating plainly

The asymmetry was corrected inside the same commit that introduced the extension loop, and when that loop first fired in production, on the hugging face null-hard run, both batches recorded zero drops of any kind. So the fix has never been exercised outside its tests.

Evidence

• source src/cli.ts lines 522 to 549, the old behaviour in the past tense and the refuse-or-drop rule that replaced it
• test test/extension.test.ts: "REFUSES on an answer key inside a batch, exactly as the registered suite does"; "REFUSES on a structural property violated inside a batch"; "DROPS an ordinary admission failure, and says so rather than refusing"; "REFUSES the run when a bought batch violates the structural property"
• commit 95706f8, the commit that introduced the loop and the correction together

What broke

Extension task ids carry an e<index>- prefix, which makes every batch id unique. Precisely because the prefix guarantees uniqueness, the id is useless as a duplicate test, and nothing else was testing for one.

What it would have caused

A restated task inflating pooled n with a correlated trial, which is the one thing pooling must not do. Deduplication now keys on task content, not on id.

Worth stating plainly

Same as the entry above: fixed inside the commit that introduced the loop, and the one production run that consumed extensions recorded duplicate drops of 0 in both batches. Pinned by tests, unexercised in the field.

Evidence

• source src/cli.ts lines 585 to 594, and taskContentKey
• test test/extension.test.ts: "drops a batch task that restates one already in the pool"; "dedupes inside one batch too, not only against the pool"; "keys task identity on content, never on id"; "drops a batch task that restates a pooled task, and still consumes the extension"
• commit 95706f8, "Content level dedupe stops a restated task from inflating pooled n"

What broke

The first pass published a spend figure of $8.04 and called it measured. The ledger's fifteen entries sum to $8.0361, of which $5.7861 is runner spend priced from recorded tokens. The remaining $2.25 is fifteen copies of a flat $0.15 judge allowance, a constant in the sweep script that was never a measurement of anything.

What it caused

A published number understating real spend, wearing the word "measured". When judge spend was actually measured it came in at $0.421 on the first priced run. Recomputing the same fifteen runs at the replacement constant gives about $12.09, roughly half again as much as the figure that was published.

How it was caught and what changed

By measuring it. Judge usage is now read from the run record instead of assumed, and this is the concrete instance the floor labelling exists to prevent: a figure known to be incomplete is published as a floor with the reason it is one, and never as a total.

Evidence

• commit 302a62e, the message that says "Total measured spend: $8.04"
• source scripts/sweep-one.sh at 302a62e, line 18: OPUS_EST = 0.15 # synthesis allowance per run
• artifact runs/sweep/ledger.json, first 15 entries, estUsd $8.0361 and sonnetUsd $5.7861. Not linkable: runs/ is gitignored
• commit a9fe8e9, "the first measurement came in at $0.42 against a $0.15 guess"

What broke

The spend ledger priced judge synthesis at a flat $0.15 per run. Measurement replaced it with $0.42, a factor of 2.8, and the replacement is itself labelled in the script as a guess rather than a measurement.

What it caused

Every total resting on the old constant was too small. The thirteen runs that now record real judge usage range from $0.2642 to $1.4495, with a median around $0.44, so the spread is far wider than any single constant can represent.

How it was caught and what changed

The ledger now reads run.judgeUsage.estCostUsd where the run recorded it and stamps the entry as measured. Where it did not, the figure is a floor, and the board carries that floor upward: if any run figure is a floor, the board total is a floor too.

Evidence

• commit a9fe8e9, the diff replacing the constant and reading measured usage
• file data/runs.json, thirteen rows carry judgeUsage.estCostUsd, minimum $0.264189, maximum $1.449461
• test test/site.test.js: "labels a partial judge block as a floor and says why"; "labels each published run by what its own record supports, floor or total"; "carries the floor up to the board figure whenever any run figure is one"; "states the spend as a floor and links to what sits outside it"

What broke

The generator counted rewrites as additional candidates: generated += rewrites.length. With 12 raw candidates and 4 rewrites the reported admission rate was 0.750 where the true rate was 1.000, and that inflated denominator went straight into the structural gate's 0.25 floor.

What it would have caused

False admission-rate refusals: runs refused for a generator failure the generator did not commit. It never reached a published number, and the reason is uncomfortable rather than reassuring. Because the answer-leak detector in entry 04 never fired, no regeneration was ever attempted, so no repair pass ever ran and the double count never had anything to double. One defect was masked by another.

Evidence

• source src/tasks/synthesize.ts at c5743b6, line 1004: generated += rewrites.length
• source src/tasks/synthesize.ts lines 437 to 447, the worked 12-and-4 example
• source src/cli.ts lines 1249 to 1254, "would have refused runs the generator did not earn"
• test test/synthesize.test.ts: "never counts a repair rewrite as a new candidate"; "reconciles every candidate into exactly one bucket"

What broke

The runner reports a terminal failure class and the tool it is attributable to for every failed task. A class derived from the wire was then bumped a second time from the outcome, so a single failure was attributed twice to the same tool.

What it would have caused

Published per-tool attribution at twice the real failure count, which is the column a server author would act on. It was caught and pinned in the initial build commit, before any scored row existed. The first row carrying any per-tool attribution appears seven commits later, so no leaderboard row ever carried the doubled figure.

Evidence

• test test/score.test.ts: "never counts a failure class more times than the errors that produced it", which asserts the class sum never exceeds a tool's errors plus its terminal count
• source src/score/metrics.ts line 609, the same note in the production path
• commit c5743b6, the only commit that introduces that test, so the broken state exists only in its rationale

What broke

A rerun into an output directory that already held a run appended to that run's tapes instead of refusing. Five of the 33 trace directories on this origin ended up carrying more than one meta line, meaning the published tape for one run also contained a different run. Counting meta lines then and now, against the line count our own report claimed for that run all along:

What it caused

On the aws knowledge row the published tape was 95 percent a different run. Attributing events by session: session 0, started at 16:45:18.024Z, carried all 54 fitness.http_error events. Session 1, started at 16:59:20.891Z, was the run the URL named, and it carried none of them and called no tool at all. A reader following the replay link saw an evidence trail belonging to a run that had no row. It was the multi-session shape mcp-tape 0.7.3 had to fix on the reading side (entry 22), occurring in our own published evidence on the writing side.

How it was closed

The tapes were repaired and the write path was made incapable of doing it again. Each of the ten affected files, the five in the table and the five agent tapes beside them, was trimmed to its own session. Nothing was written into them: against the previous published versions the ten files add no line at all and remove 555. Every removed session recorded a meta line, an end line and nothing else, an attempt that aborted before it configured a suite, with one exception: the aws knowledge session started at 16:45:18.024Z, whose removed bytes are identical to the tape already published in that run's own directory. None of the removed sessions was ever a published row, in any version of runs.json.

On the write side, the tape writer now refuses a path that already holds bytes unless the caller explicitly asks to truncate, the run command refuses an output directory already in use before it spends a token, and the sweep script archives a previous attempt instead of writing over it. The publish step warns on any tape carrying more than one meta line. All 66 published tapes now hold exactly one session, and in every one the session's start time equals the timestamp in its own directory name.

Evidence

• artifact the five tapes linked in the table above, all served from this origin. Count "type":"meta" lines: one each
• file data/runs.json, the trace_stats block on each of those rows, which the repaired files now match line for line
• source src/tape/writer.ts, the stat guard and TapeExistsError; src/cli.ts, prepareOutDir() and OutDirInUseError
• test test/tape.test.ts: "refuses to append a second session onto a tape that already exists"; "treats a zero-byte file as no recording at all"; "finds one session in every published tape"
• test test/pipeline.test.ts: "refuses the rerun and leaves the recorded run intact"; "does not treat a zero-byte tape as a recorded run"
• test test/site.test.js: "holds exactly one session in every published tape, and it is that run’s own"; "agrees line for line with each row’s own trace_stats block"

What broke

The design makes mcp-tape stats <file> --json the oracle that must parse our tapes and agree with our own pairing counts. The pin was ^0.7.2 and the installed binary reported 0.7.2, which predates both the session-scoped pairing and the multiple-sessions-in-file warning added in 0.7.3. We published 0.7.3 ourselves.

What it caused

Run against the aws knowledge tape, the oracle returned warnings: null, startedAt 2026-08-20T16:45:18.024Z which is the wrong run, durationMs 537243 spanning both runs, 274 records and 54 tool calls across four aws tools. Our own trace_stats for that same run id said 16:59:20.891Z, 16 records and zero tools. The oracle and the report disagreed by 260 records and 54 tool calls on the same run, and the oracle raised nothing at all. An oracle is only an oracle if it is a version that can read what we write.

How it was closed

The pin moved to ^0.7.3 in package.json and the lockfile, and the installed binary reports 0.7.3. That version does raise the warning: run over the pre-repair svelte tape from entry 14, which held three sessions, it returns a warning with code multiple-sessions-in-file and sessions: 3. Run over all 66 tapes published today it returns an empty warnings array on every one.

On the aws knowledge tape that started this entry, the oracle now reads startedAt 2026-08-20T16:59:20.891Z, 14 records and no tools, which is exactly what that row's own record counts for the mcp plane, with 2 more on the agent plane for the 16 it publishes. The test asserts a minimum of 0.7.3 against both the pin and the binary's own --version, and asserts warnings is an empty array rather than absent, because an absent key would mean the oracle had checked nothing.

The reproduce recipe this entry first published, [email protected] against that tape, no longer reproduces. Both halves of it moved: the version, and the tape itself, which entry 14 repaired.

Evidence

• reproduce node_modules/.bin/mcp-tape stats mcp.jsonl --json against the published aws tape: 14 records, no tools, warnings: []
• source package.json, "mcp-tape": "^0.7.3", and node_modules/.bin/mcp-tape --version printing 0.7.3
• test test/tape.test.ts: "is pinned to a version that knows about appended sessions"; "names an appended second session and keeps its calls apart"; "finds one session in every published tape"
• file data/runs.json, the trace_stats block on that row

What broke

One commit published the 16:45:18.024Z run as a construct-gate failure and described it precisely: the reference agent completed 0 of 12 tasks, and the tape carries 54 fitness.http_error events, each naming the tool, the HTTP 400 status and the gateway's own message. A later commit rebuilt the board from whichever run directories had survived and replaced that row with the 16:59:20.891Z run, a structural gate failure with 0 of 24 candidates admitted, which is a refusal produced entirely inside our own generator by the regex defect in entry 03.

What it caused

The published row's methods and gate ledger said nothing about the 54 HTTP 400s. The evidence that actually indicts the server sat in a trace directory with no row pointing at it: 33 trace directories on this origin, 32 rows in runs.json, and that was the one orphan. The board showed its weakest reading of that server while the stronger one was published and unreferenced.

How it was closed

The 16:45 record was restored to runs.json, and it was restored rather than rewritten: compared key for key, it is the record that was published at ee131c1, and the change to the file adds 943 lines and removes none. No other row was edited. All three aws knowledge attempts now have rows, and the reader tells them apart by start time and suite hash rather than by which one we kept.

The restored row is the run its evidence describes. It refused at the construct gate with the reference agent completing 0 of 12 driven tasks, and its own tape carries exactly 54 fitness.http_error events, every one an HTTP 400, every one naming the tool and the gateway's message, "Http operation is not supported for gateway protocol type MCP". The 16:59 row refused at the structural gate with 0 of 24 candidates admitted, under a different suite hash. Directory and row counts now agree: 33 trace directories, 33 rows, no orphan and no row whose tapes are missing.

The publish step is additive. It seeds the board from the runs.json already published, keyed by run id, and a row is replaced only by a run carrying that same id; a row whose tapes are missing is dropped rather than left pointing at nothing. The sweep script archives a previous attempt instead of writing over it, and the renderer draws one row per run, so there is no place in this pipeline where a best of two attempts could be chosen.

Evidence

• commit ee131c1, which published the 16:45 run and described the 54 events
• commit 757986b, which rebuilt site/data/runs.json and dropped the row
• artifact the 16:45 tape, which carries the 54 fitness.http_error events, and now has a row again
• file data/runs.json, three aws knowledge rows, and 33 rows against the 33 trace directories on this origin
• test test/site.test.js: "has both planes on disk for every row the board lists"; "publishes no orphan recording that no row points at"; "carries the http_error evidence the stronger AWS attempt was published for"

What broke

The drop ledger records the check kind, the offending phrase, a prompt excerpt and a cold answer excerpt. It had no field for the check's pattern. So the twenty-four drops on the microsoft learn run read {rule: invalid-check, detail: "regex does not compile"} twenty-four times with nothing in them to inspect.

It is thinner than that on the site. What every published row carries is the rule counts alone, under dropsByRule, with detail reading "full ledger in suite-meta.json". That file is under runs/, which is gitignored, so it is not published anywhere. A reader who wants to check a refusal against the candidates that caused it cannot, and the pattern they would most want to see is in neither file.

The agent plane does not fill the gap either, because it only holds runner turns. The published gitmcp tape of 03:30:29.022Z is 84 assistant turns and 95 user turns, every one of them the runner model, with zero turns from the judge. The published microsoft learn tape is exactly two lines, a meta and an end, for a run whose entire outcome was decided by a judge call that appears nowhere.

What it causes

Seven currently published refusals rest on judge output that no recording contains. The standing rule for this product is that every flag or finding must link to the recorded session that justifies it. For those seven, no such recording exists, and the rule is not being met. It is also the reason entry 03 had to be reconstructed from drop totals rather than read off a tape.

What was fixed

The ledger can now carry the pattern. DroppedTask.evidence gained checkPattern, plus checkPatternRepaired when a pattern was translated into ECMAScript and then rejected anyway, so both forms are on the record, and the check's value or tool where the check kind has one. It is filled from the raw candidate rather than from a validated check, so a drop that happens before a valid check exists still carries it.

Why it is still open

Nothing published shows a pattern. No run has been executed under the generator that writes the field: all 17 rows on the board are fitness-report-generator/2, and none of the 203 drops in the ledgers on disk carries a checkPattern key. A field that exists only in the code is not a record a reader can check.

The rest of the entry is untouched. suite-meta.json is still not copied into site/ by the publish step, which copies the two tapes and nothing else, and runs/ is still gitignored, so the ledger is still unpublished whatever it now holds. And the judge call still reaches no tape, so the seven refusals above still rest on output no recording contains.

Evidence

• source src/tasks/synthesize.ts lines 380 to 386 before the fix, the DroppedTask evidence shape with no pattern field
• test test/synthesize.test.ts: "records the pattern of a check that could not compile"; "records the pattern of a check rejected as a shape rather than an answer"; "records both forms when a repaired pattern is then rejected"
• artifact the published gitmcp agent tape, 84 assistant and 95 user turns, all claude-sonnet-5, and the published microsoft learn agent tape, two lines
• file data/runs.json, dropsByRule counts on every row with detail reading "full ledger in suite-meta.json", a file this origin does not serve
• design DESIGN.md, the standing rule that every finding links to its recorded session

What broke

The design decision that introduced redaction scoped it to the published tape copy only. But report.json and report.md are published as well, and the site's own instructions tell the operator to append the report to data/runs.json, which is the file the board reads.

What it would have caused

A credential echoed back by a server reaching the public leaderboard through the one artifact redaction was not covering. It never happened. A report-level redaction pass and the test that proves it are both present in the initial commit, and runs/ is gitignored so no pre-fix artifact exists at all. The failure is documented in the code in the past tense and never reached a published file.

Evidence

• source src/cli.ts lines 2179 to 2183: "THE REPORT IS PUBLISHED TOO"
• source src/tape/redact.ts lines 349 to 355: "Redaction that runs only on the tapes is redaction that does not run"
• test test/pipeline.test.ts: "publishes no credential in report.json or report.md, only in neither tape", which asserts all three artifacts lack the token and that the report still records that a credential was presented, marked redacted
• commit c5743b6, which already carries both

What broke

On the judge call, adaptive thinking consumed the whole output budget before any text block existed. The response came back with stop_reason: max_tokens and nothing to parse.

What it caused

Three consecutive live synthesis failures against real servers, inside seven minutes. Only one of the three carries the message "judge returned no text"; the other two read "judge response contained no task JSON", which is the truncated-output path of the same exhaustion through a different throw site. Stating that precisely matters, because looking for one message would have found one third of the incident.

How it was fixed

The output ceiling was raised from 16000 to 32000 tokens and a thinking-off retry was added, committed at 2026-08-19T23:40:47Z. Raising that ceiling is what caused entry 20, eighteen seconds later.

Evidence

• commit 1b5309f, which raises the ceiling and records the live observation
• source src/tasks/synthesize.ts lines 1446 to 1452
• artifact two of the three tapes are published on this origin: coingecko and svelte

What broke

The Anthropic SDK refuses a non-streaming create when max_tokens implies the request may exceed ten minutes. At 32000 output tokens on the judge model, that guard fires.

What it caused

The same three runs broke again, this time inside 19 seconds of each other, all with "Streaming is required for operations that may take longer than 10 minutes": coingecko at 2026-08-19T23:41:05.576Z, svelte at 23:41:08.758Z, gitmcp docs at 23:41:24.017Z. The first of them is eighteen seconds after the previous fix was committed. The streaming fix landed at 23:43:17Z, under two minutes after the last of them.

How it was fixed

The judge call streams when the client supports it, and both judge call sites route through that one surface, so the ceiling raised in entry 19 no longer trips the SDK's long-request guard.

Worth stating plainly

The 32k ceiling raised in entry 19 is what tripped this guard. The fix for one failure directly caused the next one. Both are in this record at the same weight rather than being folded into a single tidy story about judge reliability.

Evidence

• commit 5842abc, which adds the optional streaming surface and routes both judge call sites through it
• source src/tasks/synthesize.ts lines 215 to 221, the guard described as observed live
• artifact the same two published tapes as entry 19 carry the failure events

What happened

The vercel docs run started at 2026-08-20T18:14:30.352Z recorded a synthesis failure at 18:16:32.938Z with {"type": "overloaded_error", "message": "Overloaded"} and a yield of 0 candidates emitted. It was rerun a minute later, and that second attempt is the row on the board.

Why it is in the record

Because a rerun after a failure is exactly the shape of an operation that can quietly become best-of-n. It is the only overload in the whole history, the failed attempt is a transport fault rather than a result, and the rerun is published as its own row with its own suite hash. It is on the record so that the one time it happened is visible rather than inferable.

Evidence

• artifact the published vercel tape, which carries the fitness.synthesis failure event with request id req_011CeETPg65M4A1G2D5MN8Uo
• file data/runs.json, the published row mcp-vercel-com-2026-08-20T18-17-32-685Z, INSUFFICIENT_SURFACE with 1 admissible task of 24

What broke

A JSON-RPC id is unique per connection and per session, never globally. The CLI keyed pairs on the bare id while the web viewer keyed on source, session and id together. A tape holding more than one session could therefore pair one session's request with another session's response.

What it caused

An orphaned request in run 1 paired against run 2's response, and the report carried a latency spanning the gap between the two runs: a number describing no call that ever happened. The pairing statistics in this harness are ported from that same module, and the multi-session tapes in entry 14 were that shape, in our own published evidence.

How it was fixed

Pairing now keys on source, session and id, matching the viewer. An unpaired request reports no latency at all rather than a fabricated one: no response seen is not applicable, never a number.

Evidence

• package [email protected] on npm, the release the fix shipped in. This is the publicly checkable artifact for this entry
• commit 862915f in mcp-tape, "0.7.3: session-scoped pairing keys, multi-file stats, caller-supplied timestamps and paths". Repository not public
• source mcp-tape src/stats-model.ts lines 31 to 42, which name the pre-0.7.3 keying in the past tense
• test test/stats-model.test.mjs: "an orphaned request does not pair with the next session reusing its id"; "turn tool_use ids are session-scoped too"; "the id-only fallback relaxes source, never the session boundary"; "a tool error in one session is not suppressed by the next reusing its id"

What broke

The first cut of the 0.7.3 fix keyed on source alone. Two producers could no longer collide, but an orphaned request in one session still cross-paired against a later session's response in the same source, which is the exact case the fix existed for.

Why it is in the record

Because the interesting failure is not the original bug, it is that a fix which looked correct and passed its new tests did not close the case. The two stages are still visible as two test blocks: source-scoping first, then session-scoping added afterward.

Evidence

• source mcp-tape src/stats-model.ts lines 39 to 42, which name the intermediate state in the past tense. Repository not public
• test test/stats-model.test.mjs: the source-scoping block, "two sources reusing id 1 do not cross-pair" and "turn tool_use ids are source-scoped too", then the later session-scoping block

What broke

mcp-tape added a rule matching Google API keys on 2026-07-31. The share worker vendored the ruleset and kept shipping the version without it, so a Google API key posted to the public share endpoint would have been stored unredacted. A drift test existed, it compared the worker's ruleset against the upstream file, and it was already failing on exactly that missing pattern.

What it caused

Reproduced end to end: running the pre-fix worker's redaction over a correctly shaped 39-character key in a neutral field stores it verbatim; the post-fix worker returns [REDACTED]. The gap stood on the deployed worker for about 19 days, from 2026-07-31 until the fix. The failing test never ran because the default npm test globbed one directory and the share worker's suite lived in another. Entry 25 is what closing that took.

Evidence

• commit mcp-tape 0a4f0ef, "fix(redact): match Google AIza API keys", pattern \bAIza[0-9A-Za-z_-]{35}\b in default-redact.json. Repository not public
• commit mcp-replay 6a0228b, the fix; the parent commit's share-worker/src/redact.mjs contains no such pattern. Repository not public
• test share-worker/test/redact.test.mjs: "ruleset stays in sync with mcp-tape default-redact.json (drift test)", failing at the parent commit, diffing on exactly the missing pattern

What broke

The fixing commit's message says the test sat in a suite the default npm test never ran, and that both are fixed. The commit does not touch package.json. The test script was still node --test test/*.test.mjs, which globs one directory and matches nothing under the share worker's own test directory.

What it caused

What that fix actually added was a never-skipping checksum test inside the share worker's own suite, reachable only by running that suite directly. With no continuous integration in this organisation by standing policy, a future drift in the deployed worker's rules had nothing running against it by default, which is the same class of gap that produced entry 24.

How it was closed

The test script now names both directories, node --test test/*.test.mjs share-worker/test/*.test.mjs. The default npm test runs 395 tests, 361 from the root directory and 34 from the share worker's own suite, and the drift guards appear in its output by name.

It was proved by breaking it rather than by reading the script. Corrupting the Google key pattern in the worker's vendored ruleset makes the default command fail two tests, the drift guard and the key-redaction test, and exit non-zero; restoring the file returns it to 395 passing with the ruleset unmodified. Adding a test file in a directory the script does not name also makes the default command fail, naming the files that exist and are not run, because a wiring guard now walks the repository for test files and checks the script reaches each one.

What has not changed is that nothing runs on a schedule. There is still no continuous integration here by standing policy, so the guard fires when someone runs the suite and not before.

Evidence

• source mcp-replay package.json, "test": "node --test test/*.test.mjs share-worker/test/*.test.mjs". Repository not public
• commit mcp-replay 6a0228b, whose message claimed the structural fix and whose diff does not contain it
• test mcp-replay test/test-wiring.test.mjs: "the default npm test runs every test file in the repo"; "the share worker suite is one of the files it runs"; "every package in the repo with its own tests is reached by the root command"; "the redaction drift guards still exist to be run"
• test mcp-replay share-worker/test/redact.test.mjs: "the vendored ruleset covers every rule mcp-tape ships (drift test)" and "a Google API key is redacted (the rule the hand-mirrored copy was missing)", both now reached by the default command

What broke

1. A specificity leak. The board's own header rule .board thead th has higher specificity than .panel-table th, and the panel tables were rendered into the board's own table body, so the board's header rules matched the nested panel headers through the descendant combinator and applied white-space: nowrap to them. The result was a table forced far wider than its container.
2. The sticky table header never stuck, at any viewport. Setting overflow-x: auto on the scroller makes the used value of overflow-y auto as well, so the sticky header resolved against a container with no height constraint.
3. The phone layout was sized against a desktop-width table. The board carried min-width: 1080px inside that scroller, so every narrow-viewport rule was written against a width no phone has.
4. A colour token failed small-text contrast in both themes. The faint token was used in fourteen places, including 11px table headers, and computed against the surfaces it actually sat on it gave 2.96:1 to 3.26:1 in light and 3.98:1 to 4.32:1 in dark. Both are below the 4.5:1 needed for small text.

The retraction

The commit that fixed these published two pixel measurements, an 811px table and a 358px track. The 453px of overflow quoted alongside them is arithmetic on those two, performed here rather than measured there. All of it came from a browser measurement pass and none of it is reproducible from anything in the repository, because no test anywhere in it measures rendered geometry. All three figures are withdrawn. The four mechanisms above are all verifiable in the pre-fix stylesheet, so those are what this record publishes; the numbers will come back only if they are re-measured.

Evidence

• source site/style.css before the fix, lines 442 to 458, the scroller, the board min-width and the board header rule
• source site/app.js before the fix, where the panel tables are rendered into the board's own table body
• commit 757986b, the fix, and the commit message carrying the three withdrawn figures