routerflow.artthe marketour probecomparemethodologyprobestaxonomyprober methodapi/v12026-10-13

The prober: plan, prompts, and how results are shown

The prober is an active measurement service. It sends fixed, versioned workloads through inference routes and records what actually came back: realized cost, asserted cache counts, timing, the provider that served the request, and (for confidential routes) the attestation receipt. It turns the site's provider-asserted numbers into measured ones and produces an open "quotes vs realized outcomes" dataset, with cache-honesty evidence that has so far only been measured once (Chitra, "The Souq", Aug 2026; see section 9), made continuous.

Code: this repository (probe.py, monitor.py, config.json, schema.json, workloads/). Running it: ops/ (launchd on macOS, sandboxed systemd units on Linux), or the agent-devops role routerflow for the project's own box.

1. What we measure

InstrumentQuestionMeasured quantity
I-1 execution qualityWhat did a realistic agent conversation cost per provider, versus the quote?effective_price = realized_usd / (prompt + completion tokens); slippage = effective / quoted_blended − 1
I-2 cache honestyDid the provider recognize a prefix it just served?cache_recognition = cached_tokens_reported / prefix_tokens_repeated; corroborated by TTFT drop on repeat
I-3 selection functionWhom does the router pick when nobody pins?share of unpinned requests served per provider vs quoted input price; fitted exponent k
I-5 cross-gatewaySame model on OpenRouter vs Vercel vs direct: price, cache, latencyper-route effective price, cache recognition, TTFT, tok/s
I-8 identityIs provider X serving the model it says?agreement with the model builder's endpoint on a fixed logprob/exact-match battery
I-6 attested routesDo confidential providers verify, and what is the penalty?attestation validity, TTFT and price vs plain routes

2. Workloads (the prompts)

The prompts as sent, in full and generated from the files the prober reads: PROBES.md. Files: workloads/<name>/workload.json (plus prefix.md for the coding digest); the sha256 of those bytes is on every record.

Workloads are split by task shape, because the shape decides what can be measured, and every one carries an answer a script can check (no judge model), so the same requests also give a per-provider quality signal (I-8) at no extra cost.

WorkloadTaskShapeWhat it is for
agent-tracecodingmulti-turn, 5 turns over a ~5,400-token repository digest with one real bugrealized cost against the quote, cache recognition on a growing transcript, the router's pick, per-turn checks
record-setlookupone-shot over ~340 generated ledger rowsa cache-pair family with no fixed text to recognize; exact-match answer
canarycanaryone-shot over one fresh word repeated ~1,500 times (after The Souq)a cache-pair family; the cheapest unique-per-session text
generationcontent generationone-shot, ~300 words outoutput price realized, tokens per second, instruction following
oneshotsingle questionone-shot, no shared contextthe control: no cache possible, so bill = quote; cold time to first token; exact answer
decisiondecision10 choice itemsdesigned, not run: decision-only models such as TypeSafe Jev (latency and accuracy, not price or cache) against generative comparators

Rules that hold for all of them:

3. Jobs, panel and budget

Panel. Models with at least three competing providers, excluding OpenAI and Anthropic models. Panel A = the ten with the most one-month tokens (203 endpoints, about 61% of mapped OpenRouter tokens as of 2026-09-19), panel B = the other 59 (404 endpoints). python3 probe.py panel rebuilds panel.json from the collector's data; rebuilt monthly.

JobWhat runsCadenceWorst case per day
pair-acache pair (cold, then the same bytes again, a one-word answer, streamed) on every panel A endpoint; the family rotates agent-trace, record-set, canary3x daily at jittered hours$5.42
pair-bthe same on a third of panel Bdaily, so each model every third day$0.77
eventthe same pair, fired by the 5-minute quote poll when an endpoint cuts its input price by 1% or more, sent to the mover and to the model builder at the same momenton events, at most 4 per endpoint per daycapped $2.00
selectunpinned agent-trace turn 1: 4 default, 1 sort: price, 1 sort: throughput, plus 1 unpinned oneshot, per model per slice10 slices daily$1.99
default-sessionthe five-turn session unpinned, one arm with session_id (the sticky-routing key) and one without3x daily$0.99
sessionthe pinned five-turn session, 3 repeats (15 requests) per endpointa seventh of panel A daily, so each model weekly$2.06
tasksgeneration, oneshot, record-set pinned on every panel A endpointdaily$1.23

Worst case (every repeat billed at list) is $14.47 a day, $440 a month; python3 probe.py plan recomputes it from live quotes. Endpoint order is shuffled per run (seeded by the recorded run id), and timer jitter moves each sweep across the hours, so no endpoint is always measured at the same time of day.

Budget policy: continuity first. A time series with holes is worth less than one that ran over budget, so the monthly figure ($500) is a target, not a wall (budget in config.json, python3 probe.py budget):

  1. A sweep runs completely or not at all. Budget is checked before a sweep starts, never in the middle of one, so no day is half-measured and no endpoint is dropped because it came late in the shuffle.
  2. Over target means a warning, not a stop. The run-rate is the mean of the last seven complete days times 30.4. Above target (with 5% tolerance) the state becomes over_pace: a notice goes out once, everything keeps running, and a 7-day grace period starts. A reminder goes out with two days left.
  3. Three ways to settle it: raise the target, change the schedule, or accept the overspend until a date (probe.py budget --ack 2026-10-31 --note ...), which is recorded.
  4. If the grace period lapses, the least valuable data gives way first, in a fixed order and only as far as needed to get back to target: one-shot tasks to every third day, the weekly pinned session paused, panel B pairs to every sixth day, then the selection sample and default-routed sessions to alternate days. Panel A cache pairs and reprice-event pairs are never thinned. Thinning lowers the cadence evenly; it does not leave ragged gaps.
  5. Every skipped sweep is recorded in runs.jsonl as skipped with the policy that caused it, so a declared gap is always distinguishable from a failure or an outage.
  6. The only hard stops are runaway guards, there for bugs and absurd reprices, not for budgeting: $750 a month, $35 a day, $0.25 for any single request (that request is skipped, the sweep goes on), and a job spending three times its expected daily amount.
  7. The other way to get a hole is running out of funds, which makes every request fail. The monitor turns to low funds when the key limit or account balance covers fewer than ten days at the current run-rate. The OpenRouter key's own credit limit should be set at the hard ceiling, not at the target, or it becomes the wall this policy avoids.

Why this allocation: under a stable pinned session the published measurement found caches mostly work, so the weekly pinned session is a baseline, not the main instrument. The money goes to the conditions where the effect was found: frequent cheap pairs at varying hours, price-cut events, and default routing.

4. Records (what we publish)

One JSON record per request, append-only, data/probes/<date>.jsonl, with two side files keyed to it: <date>.quotes.jsonl (every endpoint's list prices as captured at the start of each run) and <date>.generations.jsonl (the GET /generation re-fetch per generation_id: total cost, cache discount, provider-native token counts, latency), plus runs.jsonl (one start and one end line per invocation) and data/quotes/<date>.jsonl (every quote change seen by the 5-minute poll). probe.py summarize derives <date>.summary.json. After review these are published as api/v1/probes/<date>.jsonl and api/v1/probes/<date>.json. Schema in schema.json:

run_id, job, session_id, ts, route (openrouter|vercel|direct:<provider>), model, provider_requested, provider_served,
workload, workload_hash, nonce, family, arm, event_id, turn, repeat, pinned, repeated_prompt_tokens, repeat_gap_s, check {passed, failed}, generated, quote {prompt_usd_per_m, completion_usd_per_m, cache_read_usd_per_m, captured_at},
usage {prompt_tokens, completion_tokens, cached_tokens, reasoning_tokens}, realized_usd (from /generation), generation_id,
timing {ttft_ms, total_ms, tokens_per_s}, attestation {present, verified, verifier}, response_sha256, error

The summary reports realized cost against two references computed from the same token counts: the list cost (no cache discount at all) and the ideal cached cost (every repeated token billed at the quoted cache-read price). slippage_vs_list and slippage_vs_ideal_cached bracket each provider between "never discounts" and "discounts everything it could".

generation_id is the tx hash of this dataset: anyone can call OpenRouter's /api/v1/generation?id= and re-fetch the realized cost we claim.

5. How we show it

Modeled on InferenceX: every number is traceable to a run, runs are labeled precisely, and the methodology page is a glossary, not an essay.

6. Publication

Results are public from the first sweep, on the "Our probe" and "Compare" tabs of routerflow.art, labelled early and unreviewed, with n on every figure. (The first version of this document promised outside review of a week of records before anything measured went public. That was dropped on 2026-09-20 by the project owner in favour of publishing as the data arrives; review happens in the open instead.) What that obliges:

7. What is needed to start

Until the keys exist, python3 probe.py plan emits the exact request plan, the workload hashes, live quotes for every target endpoint and a cost estimate at list prices (data/probes/plan.json), so the method can be reviewed without spending. The first paid run is a smoke test on two endpoints with --budget 1.

8. Build status

Built and tested (21 unit tests, including every job end to end against a local stand-in router): the six workloads with checks, the seven jobs, the quote poller with the reprice-event trigger, the continuity-first budget policy with runaway guards, plan, panel, summarize (cache recognition with bill and latency ratios per pair, dollars per session, slippage against list and against an ideal cache, task pass rates, selection shares with a maximum-likelihood Tullock fit and its profile-likelihood interval, reassignment counts for default-routed sessions), the admin monitor, the generated probes page. A doctor command checks readiness without spending; state files are written atomically; one run per job at a time; every run is stamped with the prober version.

Not built: prefix-mutation (25/75/100%) and stretched-wait treatments, the decision route, the identity battery with logprobs, direct-provider and second-router control arms, attested routes, public results panels. No paid request has been sent yet.

Known limit: the prompts are published, so a provider could recognize prober traffic. Two of the three cache-pair families are generated fresh per session; rotating paraphrases of the coding digest is the next mitigation if review asks for it.

9. Prior measurement: "The Souq" (Chitra, Robot Ventures, Aug 2026)

Essay with appendices: https://robvc.com/research/the-souq. Short version: "Caching Cheaters on OpenRouter". No data or code is published with it, which is the gap this prober fills. What it measured and how, as far as the text states:

The SouqThis prober (as built)
ModelsGLM-5.2 (main), Kimi K3, one Mimo v2.5 Pro exampleGLM 5.3 Flash, DeepSeek V4 Flash, Kimi K3
Pinningone provider, fallbacks disabled, session identifier held fixedthe same: provider.order + allow_fallbacks: false + session_id
Workloadscanary (a freshly drawn word repeated several thousand times, sent twice), record set (random ledger rows per session), conversation (own answers appended), agent trace (fixed tool-call log, only the last instruction changes)canary, record set, fixed coding trace, plus output-heavy and no-context one-shot tasks
Cold startopening context unique to the session, repeated part byte-identicalthe same, through the session header line
Treatmentsprefix mutated after 25%, 75% or all of its content; waits between requests stretched to test TTL expirynone yet
Selection1,648 default-routed choices; maximum-likelihood Tullock fit r = 1.968, 95% profile-likelihood interval [1.82, 2.11], block bootstrap over 411 price-event blocks; held-out multinomial logit on log input, output and cached pricesabout 700 unpinned picks a day; the same maximum-likelihood fit and interval; no bootstrap or logit yet
Price datapublic quotes captured every five minutesthe same, for the 69 panel models, plus quotes at the start of each run
Cache fieldusage.prompt_tokens_details.cached_tokensthe same, plus native_tokens_cached from GET /generation

Findings that change what this prober should measure:

  1. Under a stable pinned session the repricers' caches largely work. "With every request sent to one provider and the session identifier held fixed, the aggressive repricers recognize roughly ninety percent of repeated tokens, against ninety-nine percent at Z.ai." W1 as built measures exactly this condition, so on its own it would mostly report the benign case.
  2. The overpayment appears in two other conditions: at repricing events (159 price-cut events, each paired against the model builder on the same prompt at the same instant: the moving provider recognizes 71.3 points less repeated input, 95% interval [-77.1, -64.8], and passes on 51.7 points less of the saving), and under default routing, when the router reassigns a conversation to a provider that does not hold the prefix.
  3. The router ranks on quoted input price only. Output price carries at most about 4% weight and the cached-token quote none, so a provider can win flow with a low input quote and recover it on uncached repeats. About five repeated queries to the worst repricer cost more than the builder's own endpoint.
  4. Stated limits: an economic effect, not intent ("cannot distinguish deliberate eviction from a poor implementation or an operational failure"); a clean window of only a few weeks from 2026-07-21; roughly a tenth of OpenRouter's flow observed; the $3.5M and $14M annualized figures rest on assumed parameters (45% of an agentic prompt repeated, cached price = 10% of input).

What followed from it: the reprice-event pair, the default-routed session with and without session_id, the canary and record-set families, and the maximum-likelihood fit are built (section 3). Still open: the prefix-mutation and stretched-wait treatments, the block bootstrap and the multinomial logit, with r = 2 (OpenRouter's documented rule) as the pre-registered null.

10. Proposed probes (not built)

The working thesis these test: as agent sessions get longer and more repetitive, the routing advantage moves from picking the cheapest quote on each request to finding and keeping the right workload–provider match, and the quantity to optimize is total cost per successful session: cache reuse, latency, failures and router fees included. The reported data already points this way (on 2026-09-18, 8 of 19 model boards had a different provider cheapest on listed input price than on reported effective input price), but reported effective prices come from different customers with different workloads, so they do not say what the same customer would save by switching. The probes below are that same-customer measurement.

Pre-registered predictions, each with the probe that tests it:

  1. Cache-aware placement beats per-request cheapest-price selection, and the gap widens with session length (P-1, P-2).
  2. Different workload shapes converge on different best providers, not one universal cheapest (P-3).
  3. Changes in provider share are better predicted by effective cost and service quality than by listed price (P-6).
ProbeWhat is sentQuestionHeadline number
P-1 three-arm sessionthe same multi-turn session, byte-identical, under three policies: (a) cheapest-price, unpinned with sort: price and no session_id, so the router re-picks every turn; (b) fixed qualified provider, pinned to the cheapest provider that passes the workload checks; (c) cache-aware, pinned to the provider with the lowest expected session cost from the last 7 days of cache-pair records (recognition × cache price + miss × list price)Does cache-aware placement save money over cheapest-price routing at equal quality?dollars per successful session per arm, with the difference and its interval; share of sessions where (c) wins
P-2 session-length ladderP-1 at 5, 10, 20 and 40 turns, extending the agent-trace fixed trace (the same canonical replies, more tool rounds over the same repository digest)Does the cache-aware advantage grow with length?savings of (c) over (a) against turn count; the turn at which (c) pays for itself
P-3 workload–provider fitP-1 across three shapes: prefix-heavy coding session, lookup over a generated record set, output-heavy generationIs the best provider the same for every workload?per model, the winning provider per shape; count of models where the winner differs across shapes
P-4 mid-session switcha pinned session on provider A for k turns, then the remaining turns on provider B (k = 1, 3, 5), against staying on A and starting on BWhat does one switch cost? This is the price of the router's reassignment, measured directlyextra dollars and extra time to first token per switch, per provider pair
P-5 route overheadthe pinned session through OpenRouter, Vercel AI Gateway and the provider's own API, same model and providerAfter fees, is an independent router worth its margin over a free bundled gateway or going direct?dollars per successful session per route, fee included (credit-purchase and BYOK fees applied from the published schedule); cache recognition per route
P-6 share predictionno new requests: an analysis over the collector's daily provider tokens joined to listed price, reported effective price, and the prober's measured recognition, uptime and TTFTWhich predicts next week's change in provider share best?out-of-sample fit of three nested models (listed price; + effective price; + measured quality)

Definitions for all of them:

Cost at list prices, rough: one five-turn pinned session is about $0.024 on panel A (from the session job's $2.06 for about 435 requests a day), and session cost grows roughly with total prompt tokens. P-1 + P-2 on one panel A model a day (3 arms × 5/10/20 turns, 40 turns weekly) is about $0.60 a day worst case; P-3 and P-4 on the same rotation add about as much again. P-5 needs the Vercel and direct keys listed in section 7. probe.py plan will price the exact schedule once the jobs exist.

Limits to state with the results: the panel is open models with at least three providers, so results say nothing about single-provider models; the policies in P-1 are ours, not any router's product, so (c) is an upper bound on what a router could capture using public data plus our pairs; router margins and customer retention stay unobserved.

11. Studies (built 2026-10-14)

Two jobs that are not about the cache. Records go in the same daily files, with job set to the study name; the routed arm of each has arm: "routed" and is kept out of the selection statistics.

time-of-day: does what is served change with the hour? Every hour at :05 UTC (10 min jitter), four arms get the same four freshly generated puzzles (workloads/puzzles, generated from the hour stamp, so every arm sees the same items in a given hour and nothing can be memorized): anthropic/claude-sonnet-5.5 pinned to Anthropic, openai/gpt-6.1-sol pinned to OpenAI, deepseek/deepseek-v4.1-flash pinned to DeepInfra (control), and the same model routed by OpenRouter. Reasoning effort is fixed at low. The model must reply with the answer line only. Items are the hard end of two families (60-90 list operations, 100-150 trades between five people), calibrated on 2026-10-14 so each arm has items it sometimes fails: Claude about 19%, GPT about 88%, DeepSeek about 31% on 16 items each. Read per arm by hour of day: pass rate, reasoning and output tokens per item (a cut in thinking at peak shows here even where the pass rate does not move), time to first token and tokens per second (load), and for the routed arm who served it. The question as posed: is quality worst during the San Francisco working day (about 16:00-01:00 UTC) and best around 05:00-06:00 UTC (1 am Eastern)? About $3.1 a day. With four items an hour per arm, two weeks gives about 500 items in each nine-hour block, enough to see a gap of about 8 points; the token counts resolve much smaller changes.

task-cost: what does one fixed piece of work cost, over months? Six times a day (every four hours, 3 h jitter, so over days it covers every hour), eight arms get the same task (workloads/code-duration): write parse_duration to a full spec. The reply's code is run against 33 hidden tests in a separate interpreter (empty environment, CPU, memory and file limits, imports limited to a short safe list, inside the unit's sandbox); if any test fails the model gets one repair turn with the failing cases. Reasoning effort is fixed at high for every arm, and the model's catalog default that day is recorded beside it. Arms: Claude Sonnet 5.5 at Anthropic, GPT-6.1 Sol at OpenAI, GLM-5.3 at Z.AI, DeepInfra and routed, DeepSeek V4.1 Flash at Together, DeepInfra and routed. (DeepSeek's own endpoint is excluded by this account's training-data setting.) The series: dollars, tokens and wall-clock per task, and pass rate on the first try and after repair. About $0.13 a run, $0.8 a day. More tasks come later; this one is deliberately basic.

Published as two pages on the site, studies/time-of-day.html and studies/task-cost.html, drawn from api/v1/probe/time-of-day.json and api/v1/probe/task-cost.json. studies.py builds both from the records; monitor.py --export writes them beside latest.json (which keeps study records out of every Our probe figure and carries a one-line headline per study). Both studies are also probes in taxonomy.py: the puzzle under One-shot, exact answer, the coding task as its own type, Coding task, graded by running.