Fitness Report

• SCORED the gates passed and the run produced numbers.
• REFUSED a named gate stopped the run. The gate, its counts and its reason are shown in place of a score.
• could not check a probe did not apply to this connection. Reported as unknown, never as a pass.
• Why refused opens the measured reason: the rates, the counts and the floor the run missed.

Rows are listed by the first-try point estimate, but a row only takes a lower position when another interval sits entirely above it. Rows that nothing separates share a position and are marked as tied, and where two rows still overlap across positions the lower one says so on its own row. Servers scored under different runner models are never ranked against each other.

Every refused row opens on Why refused. What you get there is read from that run and nothing else: the rate a model with no tools reached against the rate the agent reached, or how many generated candidates survived admission against the floor they missed, or the reference agent's own counts alongside what the tape shows the server did. Where a run's own record does not carry a number, the panel says so rather than filling the space.

What this measures, and what it does not

Conformance asks whether the server speaks MCP correctly. Fitness Report asks whether an agent can actually get the job done with it.

The word in that first sentence belongs to the official suite at github.com/modelcontextprotocol/conformance, which is a different project with a different job. Run it too. This table is about task outcomes.

MCP Atlas asks which model is best at using tools. Fitness Report asks which server is worth using.

The runner model is pinned into every score record and printed on every ranked group. Two servers driven by different runner models are reported side by side but never ranked against each other, because the token accounting is not comparable across tokenizers.

Gate order

Gates run cheapest first and stop at the first failure, so a broken eval never spends model budget: free structural checks, an answer-leak string check and the zero-token protocol probes, then cheap null-model baselines (no tools, stubbed-empty, random valid arguments), then the paid construct gate where a reference agent with full information must reach the answer key at a rate of at least 0.90, and only then the full drive. An indeterminate baseline halts exactly like a failure. On any gate failure the report carries no score field at all, plus the failing gate with its counts, threshold, alpha, p-value and reason string. Those are the strings printed in the table.

With a median of about 2.5 tools across the open server roster, a suite below 8 tasks is refused as INSUFFICIENT_SURFACE rather than published as a two-task 100%.

How to read a refusal

Naming the gate is not the same as explaining it. Each refused row opens to the numbers the gate actually saw, and the three families read like this.

The null baseline killed it

Three null models run the same suite before any paid tier: one with no tools at all, one whose tool calls come back empty, and one calling tools with random valid arguments. The panel shows each of those rates against the rate the agent reached with the real server, and the threshold between them. When a model with no tools passes most of the suite, the tasks were answerable from what the model already knew and the server was never in the loop. The finding is about our task generator, not about the server, and the row says so.

The surface was too thin

The panel shows how many candidate tasks were generated, how many were admitted, the admission rate against its floor, and the suite size against the minimum of 8. Three fates are counted apart, because summing them would inflate the failure count: candidates rejected by admission, listed with the rule that rejected each one; candidates that passed admission and were trimmed as surplus, since the generator deliberately asks for more than it drives; and candidates deleted by the null screen below. Rows produced by the older generator, which did not serialise its per candidate reasons, carry that gap as a stated limitation instead of a breakdown.

Neither floor moves. A thin surface is fixed by generating tasks that survive admission. No threshold, ratio, floor or alpha on this page has ever been loosened to turn a refusal into a score, and none will be.

The construct gate failed

A reference agent that has been handed the answer key must still reach it through the server. When it cannot, the panel prints its counts and then reads the tape: how many tool calls were recorded, how many came back as errors, and how many never received a matching protocol response at all. A server that advertises tools and then rejects every invocation before a response exists produces requests with no responses, and that is a finding about the server rather than about the agent. It is stated from the recorded counts, never assumed.

The generation time null screen, and the bias it introduces

Newer runs screen every candidate task before admission: the runner model is asked to answer it with no tools and no server, and a candidate it answers correctly is deleted before the suite is hashed and before any gate runs. The count deleted is published on each row, and the deleted candidates are never counted as generator failures because nothing was wrong with them.

This is a bias and it points in one direction, so it is stated rather than buried: the run time null baseline on a screened suite measures the noise floor of a suite that was already purged of the tasks that gate looks for, which means it reads lower than it would on an unscreened suite. Nothing about the gate changed. Its 0.5 ratio, its 95th percentile and its halt behaviour are unchanged, the screen never removes a task from a gate denominator, and a screened suite can still be killed by the baseline. What it does mean is that a passed baseline on a screened row and a passed baseline on an unscreened row are not the same evidence, so each row prints which it is, and rows are never ranked across generator versions.

A suite the screen empties is its own finding rather than a thin surface. It refuses as DEGENERATE with the reason all_candidates_null_answerable and the screen ledger behind it, because "we generated enough tasks and a model answered them all with no server at all" is a statement about what the server adds, not about how many tools it has.

Two documented divergences from evalgate

The gate math is ported from evalgate. It differs in exactly two places, and we own both here.

1. Construct denominator: oracle errors count

   evalgate silently shrinks the denominator when an oracle call throws, so three successes out of three completed calls reads as 100% even when forty calls errored. That is wrong for flaky remote servers, which is every remote server. Errors are gated on their own instead: errors / (n + errors) > 0.05 resolves the run to COMPROMISED. A compromised run is not a verdict about the server. It is a refusal to score, because the measurement did not complete.

2. Published PASS is symmetric with published FAIL

   A pass on this page additionally requires that the Wilson 95% lower bound sits at or above the threshold, or that the run's own n achieves the planned power against the detectable rate. Exact binomial power is sawtoothed in n, so clearing the planned sample size is not the same claim as being powered at it, and the run's actual n is what gets tested. Otherwise the verdict is downgraded to EXTEND and no pass is published. A small sample that happens to land above the line is not evidence, and publishing it while publishing failures at full rigour would be asymmetric.

Reading the interval

First-try success is shown as its Wilson 95% interval with the point estimate marked inside it. The interval is the finding, not the point. Listing order uses the point estimate, but the position is a dominance position: a row sits below another only when that other interval lies entirely above it. Rows nothing separates share a position and are marked tied, and any remaining overlap between neighbours is printed on the lower row, because at these sample sizes the order between two overlapping rows carries no information. The intervals are computed once, in the run record. This page only renders the stored numbers, so there is no second copy of the statistics to drift.

Protocol probes

The probe columns are deterministic, cost zero tokens and run before anything else, so a refused run still reports them. Spec currency is the revision the connection actually negotiated. Protocol hygiene counts the probe checks that passed, including whether the server accepts an impossible protocol version, whether a mismatched method or name header is rejected, whether the modern server discovery method is implemented, whether cache hints are present and correctly scoped, and what the deprecation surface looks like. A probe that does not apply to a connection is reported as unknown, never folded into the pass count.

Credential context

Every score is stamped with the credentials it was collected under: anonymous, free-key or owner-key. Tool surfaces differ by credential on real servers, so a number collected anonymously is not the same measurement as one collected with an owner key, and the two are labelled rather than merged.

Evidence

Every finding on this page links to the recorded session that justifies it. Each run publishes two tapes, the MCP wire plane and the agent plane, served with permissive cross-origin headers and an immutable cache, and the replay link opens both in the viewer in one merged timeline. Nothing is embedded or loaded until you click. There are no bare counts here: if a claim has no recording behind it, the row says so instead.

Prior art: MCPEval for model-driven task synthesis with machine-checkable predicates, evalgate for the validity gate math, mcp-tape for the tape format and pairing statistics. What is added here is the refusal, the signed replay behind every claim, and rewrite proposals tied to the recorded sessions they were derived from.