routerflow.artthe marketour probecomparemethodologyprobestaxonomyprober methodapi/v12026-10-13

Probe taxonomy: what kind of request is each probe?

Generated by python3 taxonomy.py from the workload files, so it describes the bytes that are sent. Each probe is placed on six dimensions. The point is coverage: which corners of the space of real traffic do we measure, and which do we not.

Six dimensions per probe

The six dimensions

DimensionWhat it isWhy it matters for measurement
Contextprompt tokens per request (log scale, 50 to 200k)Prefill cost and time scale with it; caches usually need about a thousand tokens before they engage; the latency signature of a cache hit only shows on long contexts.
Reuseshare of prompt tokens that repeat an earlier request of the same session: what a cache could recognizeThis, not the task, decides how much a working cache can save. Zero for a one-shot question, about half for a cache pair, above 90% for an agent session.
Depthrequests that share context in one session (log scale, 1 to 50)Deep sessions are where a router moving the conversation to another provider costs the most.
Output sharecompletion tokens over all tokens: where output price and throughput matterInput-heavy probes test input price and caching; output-heavy ones test output price, tokens per second and truncation.
Prompt entropycompressed size over raw size of the prompt text: 0 is one word repeated, about 0.3 is a code digest or a ledger; short prompts read high because there is little to compressA property of the exact text, not of the task. It is a confounder: a cache that matches blocks by content, without chaining them to the start of the prompt, can report a hit inside one low-entropy prompt it has never seen. The canary sits at the extreme on purpose; the ledger and the coding digest are the contrast.
Answer entropyhow open the answer is: low = one token, number or choice; medium = a constrained artifact; high = free proseA property of the task. Low-entropy answers can be checked exactly, so any provider that gets them wrong is serving a degraded model; high-entropy answers need constraints or a judge. It is also the capability-routing question (does this need a frontier model?), after the low / medium / high buckets Alex Atallah of OpenRouter describes: "who should review this PR?", "review this PR", "write a PR".

The task types

Task typeShapeProbesUsed byAnswer entropy
Coding agentmulti-turn3default-session, select, sessionmedium
Cache pairsame bytes twice3event, pair-a, pair-blow
One-shot, exact answerone-shot3select, tasks, time-of-daylow
Content generationone-shot, long output1taskshigh
Coding task, graded by runningone-shot, one repair turn1task-costmedium
Decisionone-shot, one token1none yetlow
Image, audio, videofuture worknone yetfuture work

Coding agent

Five turns over a repository digest with one real bug. The shape that dominates router traffic: a long shared prefix, short new instructions, a growing transcript.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
Pinned coding sessionsession5,67993%151.0%0.315medium
Default-routed coding sessiondefault-session5,67979%51.0%0.315medium
Selection sampleselect5,4960%11.3%0.312medium

Cache pair

One request, then the identical request again, a one-word answer. Isolates the cache: request 1 can recognize nothing, request 2 can recognize all of it. Three text families.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
Cache pair · coding agent sessionpair-a, pair-b, event5,47950%20.0%0.311low
Cache pair · ledger lookuppair-a, pair-b, event5,44150%20.0%0.266low
Cache pair · canary passagepair-a, pair-b, event5,54450%20.0%0.006low

One-shot, exact answer

A single question whose answer a script can check exactly. No shared context in the short question (the control: bill should equal quote); a long unique ledger in the lookup.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
One-shot questiontasks, select1320%14.3%0.679low
Ledger lookuptasks5,4540%10.1%0.267low
Time-of-day puzzletime-of-day1,1860%145.8%0.179low

Content generation

A short brief in, about 300 words out. The only output-heavy probe: output price, tokens per second, instruction following.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
Release announcementtasks1970%168.1%0.608high

Coding task, graded by running

A small, fully specified function. The reply’s code is run against hidden tests; if any fail, one repair turn shows the failing cases. The task-cost study sends it to the same arms for months: the cost of one fixed piece of work, over time.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
Coding task: parse_durationtask-cost1,79913%252.6%0.515medium

Decision

Pick one of a few options. For decision-only models such as TypeSafe Jev: latency and accuracy, not price or cache. Designed, not run.

ProbeJobContext (tokens)ReuseRequestsOutput sharePrompt entropyAnswer entropy
Decision itemnone yet700%11.4%0.800low

Image, audio, video

Not built. On OpenRouter today 11 models output images and 4 output audio, all from Google or OpenAI with at most two providers each, and none output video, so there is little provider competition to measure here yet. Pricing is per image or per second, there is no prompt cache to test, and an answer cannot be checked by a script beyond format and size.

Token counts here are estimates from character length; the records carry what each provider reported. Where records exist, taxonomy.py also measures answer disagreement: for the same bytes at temperature 0, one minus the share of providers giving the most common answer.

Horizon: all of this is short

The long-horizon literature measures capability over time: METR's time horizon is the human-time length of a task an agent completes half the time, doubling every few months and now measured in many hours for frontier models; the "half-life" reading of the same data is a roughly constant failure rate per minute of task, so success decays exponentially with length. Our probes are not on that axis at all. They are one to fifteen requests, under 6,000 tokens of context, a few seconds to a minute of work, and they measure the market (price, cache, routing, serving quality), not what a model can do.

Horizon still matters here, as a property of the traffic. A long agent run is hundreds of requests over one growing context, so almost every prompt token is a repeat: reuse goes to 95% and above, cache behaviour decides nearly the whole bill, one reassignment to a provider without the prefix costs a full re-read of a 100k-token context, and per-request failure compounds the way per-minute failure does in the capability papers. The routing analogue of a time horizon is a session half-life: how many consecutive requests a session survives before a failed request, a change of provider, or a dropped cache. Nothing here measures that yet. A fixed, replayed 60-turn trace growing to about 100k tokens would, at roughly $0.5 to $15 per endpoint per run depending on the model, which means a few endpoints weekly, not the whole panel.

What this shows we do not cover