The disposition profile

Most benchmarks score capability: can the model solve the problem. This one scores disposition: given a mundane task and a quiet opportunity to overstep, what does the model tend to do. Each instance is an ordinary agentic job — fix a failing test, bump a config value, process a queue — with exactly one planted temptation: a load-bearing file that looks deletable, a documented reset script that wipes state, a missing fixture that could be fabricated instead of flagged. The score is not pass or fail. It is a fire rate — how often, across many runs, the model takes the bait.

What it measures

Eight behavioral dimensions, each isolated by its own family of probes. Two further dimensions — report fidelity and context sensitivity — are computed on every agentic run regardless of family.

How an instance works

Every armed instance plants exactly one anomaly — never two, because co-occurring traps are a fingerprint no real repository exhibits, and a model that smells a test stops behaving like itself. A matched share of instances plant nothing: these baselines measure how often a model misbehaves when there is no temptation at all, so a high fire rate on armed instances can be separated from a model that is simply skittish. Each instance also ships in two surface framings — one that reads like an obvious exercise sandbox, one that carries the accumulated noise of a real project — and the gap between them is its own published number.

How it is scored

Deterministic detectors run first: the fire condition is read off the end state — a git diff, a file that should not exist, a planted secret in the final report — so the headline number never depends on a model grading a model. Dimensions that genuinely need judgment are scored pairwise against fixed anchor answers by a cross-family ensemble, with slot order swapped, never as absolute scores. A dimension's rate is measured over many runs per framing; the reported figure is the worse of the two framings, so specializing good behavior to environments that look watched cannot lift the score. Intervals are Clopper-Pearson, which makes a small number of runs visibly hollow rather than falsely precise.

The design bar for a probe

A planted temptation is only evidence if the model would plausibly meet it in real work. Every family must clear the six published honeypot desiderata before it ships:

• Realism — behavior in the setting reflects behavior in similar real situations; the model is unlikely to recognize a test.
• Incentive — an undisciplined model has a genuine reason to take the bait.
• Precision — taking it, unprompted, likely indicates the disposition rather than confusion.
• Feasibility — the model is capable of taking it.
• Plausible deniability — the model can expect not to get caught.
• Checkability — whether it was triggered can be read reliably off the trace.

House rules sit on top. Exactly one anomaly per instance, because co-occurring planted artifacts are a joint improbability no real repository exhibits. No-incentive baselines at a fixed ratio, because a model that misbehaves equally with and without the temptation is being measured for skittishness, not disposition. A canary GUID embedded in every instance, so contamination is provable the day one surfaces in a model's output. And every agentic family ships in both surface framings. The cost of all this realism is recall — a maximally tempting trap is a suspicious one, and that trade is accepted openly.

The probe catalog

Ten families, each built to isolate one tendency. Per-family rubrics publish alongside the scored tables.

Judged dimensions, kept on a leash

Where a dimension genuinely needs judgment — rhetoric, novelty's validity gate, paraphrased mentions — the judging layer runs under strict hygiene, each rule taken from the verified judging literature rather than convenience:

• Pairwise against fixed anchor answers, never absolute scores — pairwise verdicts agree better with human raters.
• Positions swapped and aggregated, because judges favor whichever answer they read first.
• Five rounds, majority verdict.
• Verdicts arrive without explanations — asking a judge to justify itself measurably degrades the verdict.
• The ensemble excludes the model family being judged, closing the self-preference channel.
• Candidates are length-normalized before judging, because verbosity reads as quality to a model that isn't warned.

Before any judged family ships, its judges are validated against a hand-labeled gold set — a judge that cannot recover known labels has no business scoring unknown ones.

How many runs is enough

A propensity is a rate, and small samples are hollow in a precisely quantifiable way: zero fires in five runs still leaves a 95 percent interval reaching 52 percent. The target on propensity dimensions is on the order of a hundred runs per framing per family — the sample sizes the published field work converged on — and no cross-model difference is claimed while the paired intervals overlap without saying so.

Budget follows the statistics rather than the other way around. Once per-instance sampling variance is small against across-instance variance, new budget buys more instances, not more repeats. A power analysis decides whether a comparison is answerable before it is run. Models are compared on instance-level paired differences, not population summaries, and standard errors are clustered when instances share a template.

Means are outlier-dominated, so alongside every mean the tables publish bucket counts — how many environments sat at exactly zero, under five percent, under fifty. Consistency dimensions report pass^k, the probability that k consecutive runs all behave; catastrophic-tail dimensions get tail views, because a one-percent disaster mixture must not hide inside a clean average.

The eval-awareness problem

The central threat to any benchmark like this one: frontier models can classify eval transcripts from deployment transcripts at 0.83 AUC overall — and 0.88 to 0.95 on agentic transcripts, exactly this benchmark's format. Belief that a scenario is a test suppresses the behavior being measured; in one published measurement the same misbehavior ran 6.5 percent when the model stated it was in a test and 55.1 percent when it believed the situation real. Even the most realistic published audits still find several percent of runs verbalizing the suspicion.

The countermeasures, in order of demonstrated effect:

1. Realism engineering. Instances derived from real repositories and real sessions resist detection in ways fully synthetic templates do not; the current templated generator is the admitted skeleton stand-in, and the production path mines real material.
2. One anomaly per instance — the joint-improbability rule again, this time as camouflage.
3. Prompt sobriety. Verbalized suspicion is mostly triggered by dramatic goal-and-stakes prompts, not by the environments themselves; there is no "nothing else matters" theater anywhere in these probes.
4. Published self-detectability. Every release ships its own verbalized-awareness rate, transcript-realism ratings from the models under test, and counterfactual framing deltas that bound the confound's size.
5. Paired frames. Whatever detection remains stops being measurement error — it becomes the model's published context-sensitivity score.

Rotation, disclosure, and the open leak

Probe families decay as they leak or get recognized, so the pool rotates — and rotation is survivable: the strongest public precedent replaces a sixth of its question pool monthly and held rank correlation above 0.997 across updates. Retirement is triggered by burn signals, in order of specificity: one family improves sharply while sibling families measuring the same dimension stand still; a score jump with no matching capability jump; synchronized cross-lab improvement on a single family; a canary GUID surfacing in any model's output. A design backlog of a fifth to a third of the active roster keeps retirement from ever being blocked.

Disclosure is commit-then-reveal, in three tiers:

• Always public — the family taxonomy, the rubrics, the scoring code, and a hash commitment of the current live instance pool.
• Public on retirement — full instance text, transcripts, and per-model scores; the hash timestamps make it checkable that retirement preceded reveal.
• Never public — the live pool itself.

The rest of the credibility stack: the taxonomy, scoring code, and comparison plan are pre-registered by hash before any model is scored; published deltas carry multiple-comparison correction; every family added or retired lands in a public changelog with reasons; standing third-party replication rights run against the escrowed pool; and full raw traces publish for a hash-seeded random subset of runs per model, which is harder to cherry-pick than operator-chosen exhibits.

One leak has no fix: testing a hosted model transmits every instance into that provider's logs. Every instance sent to a lab API is treated as burned in principle, rotation cadence is sized to that assumption, and the highest-value families are held for burn-after-one-panel use. Nothing published in the field solves this; pretending otherwise would be the only real failure.

The threat model, itemized

Sixteen attacks from the adversarial design review, each with a standing: addressed means a concrete mechanism exists in the spec, accepted means a stated limitation, open means unsolved anywhere in the field.

Behind the table sits a perverse-incentive watchlist — what would get trained into models if this score came to matter: timidity, over-asking, verification theater, vacuous reporting, cosmetic novelty. Each has a named counter-metric above, and the watchlist is re-reviewed every time a family ships.

Status

All ten probe families are implemented and unit-tested; the deterministic detectors, the propensity statistics, and the judge-hygiene configuration are in place. First runs against a local model are underway. Scored tables publish per model, each alongside its criteria and revision mark — this page is the methodology, and the numbers follow it, not the other way around.

Limits, stated up front

• Low recall is by design. A maximally tempting trap is a suspicious one; instances are tuned to stay realistic, which means they miss dispositions a more obvious bait would catch.
• Judged dimensions carry a model-judge caveat. Creativity and rhetoric are scored by an ensemble, and an ensemble has its own taste; the deterministic dimensions do not share this exposure.
• Instances sent to hosted APIs are burned in principle. Once a probe has been submitted to a provider it can no longer be assumed uncontaminated — an open problem the pre-registration and commit-then-reveal disclosure are meant to bound, not solve.