Generated by python3 taxonomy.py from the workload files, so it describes the bytes that are sent. Each probe is placed on six dimensions. The point is coverage: which corners of the space of real traffic do we measure, and which do we not.
| Dimension | What it is | Why it matters for measurement |
|---|---|---|
| Context | prompt tokens per request (log scale, 50 to 200k) | Prefill cost and time scale with it; caches usually need about a thousand tokens before they engage; the latency signature of a cache hit only shows on long contexts. |
| Reuse | share of prompt tokens that repeat an earlier request of the same session: what a cache could recognize | This, not the task, decides how much a working cache can save. Zero for a one-shot question, about half for a cache pair, above 90% for an agent session. |
| Depth | requests that share context in one session (log scale, 1 to 50) | Deep sessions are where a router moving the conversation to another provider costs the most. |
| Output share | completion tokens over all tokens: where output price and throughput matter | Input-heavy probes test input price and caching; output-heavy ones test output price, tokens per second and truncation. |
| Prompt entropy | compressed size over raw size of the prompt text: 0 is one word repeated, about 0.3 is a code digest or a ledger; short prompts read high because there is little to compress | A property of the exact text, not of the task. It is a confounder: a cache that matches blocks by content, without chaining them to the start of the prompt, can report a hit inside one low-entropy prompt it has never seen. The canary sits at the extreme on purpose; the ledger and the coding digest are the contrast. |
| Answer entropy | how open the answer is: low = one token, number or choice; medium = a constrained artifact; high = free prose | A property of the task. Low-entropy answers can be checked exactly, so any provider that gets them wrong is serving a degraded model; high-entropy answers need constraints or a judge. It is also the capability-routing question (does this need a frontier model?), after the low / medium / high buckets Alex Atallah of OpenRouter describes: "who should review this PR?", "review this PR", "write a PR". |
| Task type | Shape | Probes | Used by | Answer entropy |
|---|---|---|---|---|
| Coding agent | multi-turn | 3 | default-session, select, session | medium |
| Cache pair | same bytes twice | 3 | event, pair-a, pair-b | low |
| One-shot, exact answer | one-shot | 3 | select, tasks, time-of-day | low |
| Content generation | one-shot, long output | 1 | tasks | high |
| Coding task, graded by running | one-shot, one repair turn | 1 | task-cost | medium |
| Decision | one-shot, one token | 1 | none yet | low |
| Image, audio, video | future work | none yet | future work |
Five turns over a repository digest with one real bug. The shape that dominates router traffic: a long shared prefix, short new instructions, a growing transcript.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| Pinned coding session | session | 5,679 | 93% | 15 | 1.0% | 0.315 | medium |
| Default-routed coding session | default-session | 5,679 | 79% | 5 | 1.0% | 0.315 | medium |
| Selection sample | select | 5,496 | 0% | 1 | 1.3% | 0.312 | medium |
One request, then the identical request again, a one-word answer. Isolates the cache: request 1 can recognize nothing, request 2 can recognize all of it. Three text families.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| Cache pair · coding agent session | pair-a, pair-b, event | 5,479 | 50% | 2 | 0.0% | 0.311 | low |
| Cache pair · ledger lookup | pair-a, pair-b, event | 5,441 | 50% | 2 | 0.0% | 0.266 | low |
| Cache pair · canary passage | pair-a, pair-b, event | 5,544 | 50% | 2 | 0.0% | 0.006 | low |
A single question whose answer a script can check exactly. No shared context in the short question (the control: bill should equal quote); a long unique ledger in the lookup.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| One-shot question | tasks, select | 132 | 0% | 1 | 4.3% | 0.679 | low |
| Ledger lookup | tasks | 5,454 | 0% | 1 | 0.1% | 0.267 | low |
| Time-of-day puzzle | time-of-day | 1,186 | 0% | 1 | 45.8% | 0.179 | low |
A short brief in, about 300 words out. The only output-heavy probe: output price, tokens per second, instruction following.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| Release announcement | tasks | 197 | 0% | 1 | 68.1% | 0.608 | high |
A small, fully specified function. The reply’s code is run against hidden tests; if any fail, one repair turn shows the failing cases. The task-cost study sends it to the same arms for months: the cost of one fixed piece of work, over time.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| Coding task: parse_duration | task-cost | 1,799 | 13% | 2 | 52.6% | 0.515 | medium |
Pick one of a few options. For decision-only models such as TypeSafe Jev: latency and accuracy, not price or cache. Designed, not run.
| Probe | Job | Context (tokens) | Reuse | Requests | Output share | Prompt entropy | Answer entropy |
|---|---|---|---|---|---|---|---|
| Decision item | none yet | 70 | 0% | 1 | 1.4% | 0.800 | low |
Not built. On OpenRouter today 11 models output images and 4 output audio, all from Google or OpenAI with at most two providers each, and none output video, so there is little provider competition to measure here yet. Pricing is per image or per second, there is no prompt cache to test, and an answer cannot be checked by a script beyond format and size.
Token counts here are estimates from character length; the records carry what each provider reported. Where records exist, taxonomy.py also measures answer disagreement: for the same bytes at temperature 0, one minus the share of providers giving the most common answer.
The long-horizon literature measures capability over time: METR's time horizon is the human-time length of a task an agent completes half the time, doubling every few months and now measured in many hours for frontier models; the "half-life" reading of the same data is a roughly constant failure rate per minute of task, so success decays exponentially with length. Our probes are not on that axis at all. They are one to fifteen requests, under 6,000 tokens of context, a few seconds to a minute of work, and they measure the market (price, cache, routing, serving quality), not what a model can do.
Horizon still matters here, as a property of the traffic. A long agent run is hundreds of requests over one growing context, so almost every prompt token is a repeat: reuse goes to 95% and above, cache behaviour decides nearly the whole bill, one reassignment to a provider without the prefix costs a full re-read of a 100k-token context, and per-request failure compounds the way per-minute failure does in the capability papers. The routing analogue of a time horizon is a session half-life: how many consecutive requests a session survives before a failed request, a change of provider, or a dropped cache. Nothing here measures that yet. A fixed, replayed 60-turn trace growing to about 100k tokens would, at roughly $0.5 to $15 per endpoint per run depending on the model, which means a few endpoints weekly, not the whole panel.
tools, so tool-call formatting by provider is unmeasured.