The prober is an active measurement service. It sends fixed, versioned workloads through inference routes and records what actually came back: realized cost, asserted cache counts, timing, the provider that served the request, and (for confidential routes) the attestation receipt. It turns the site's provider-asserted numbers into measured ones and produces an open "quotes vs realized outcomes" dataset, with cache-honesty evidence that has so far only been measured once (Chitra, "The Souq", Aug 2026; see section 9), made continuous.
Code: this repository (probe.py, monitor.py, config.json, schema.json, workloads/). Running it: ops/ (launchd on macOS, sandboxed systemd units on Linux), or the agent-devops role routerflow for the project's own box.
| Instrument | Question | Measured quantity |
|---|---|---|
| I-1 execution quality | What did a realistic agent conversation cost per provider, versus the quote? | effective_price = realized_usd / (prompt + completion tokens); slippage = effective / quoted_blended − 1 |
| I-2 cache honesty | Did the provider recognize a prefix it just served? | cache_recognition = cached_tokens_reported / prefix_tokens_repeated; corroborated by TTFT drop on repeat |
| I-3 selection function | Whom does the router pick when nobody pins? | share of unpinned requests served per provider vs quoted input price; fitted exponent k |
| I-5 cross-gateway | Same model on OpenRouter vs Vercel vs direct: price, cache, latency | per-route effective price, cache recognition, TTFT, tok/s |
| I-8 identity | Is provider X serving the model it says? | agreement with the model builder's endpoint on a fixed logprob/exact-match battery |
| I-6 attested routes | Do confidential providers verify, and what is the penalty? | attestation validity, TTFT and price vs plain routes |
The prompts as sent, in full and generated from the files the prober reads: PROBES.md. Files: workloads/<name>/workload.json (plus prefix.md for the coding digest); the sha256 of those bytes is on every record.
Workloads are split by task shape, because the shape decides what can be measured, and every one carries an answer a script can check (no judge model), so the same requests also give a per-provider quality signal (I-8) at no extra cost.
| Workload | Task | Shape | What it is for |
|---|---|---|---|
agent-trace | coding | multi-turn, 5 turns over a ~5,400-token repository digest with one real bug | realized cost against the quote, cache recognition on a growing transcript, the router's pick, per-turn checks |
record-set | lookup | one-shot over ~340 generated ledger rows | a cache-pair family with no fixed text to recognize; exact-match answer |
canary | canary | one-shot over one fresh word repeated ~1,500 times (after The Souq) | a cache-pair family; the cheapest unique-per-session text |
generation | content generation | one-shot, ~300 words out | output price realized, tokens per second, instruction following |
oneshot | single question | one-shot, no shared context | the control: no cache possible, so bill = quote; cold time to first token; exact answer |
decision | decision | 10 choice items | designed, not run: decision-only models such as TypeSafe Jev (latency and accuracy, not price or cache) against generative comparators |
Rules that hold for all of them:
Session <id> · workspace /work/ledgerline, fresh per session and identical inside it. Request 1 of a session is therefore a guaranteed cold prefix (a provider that reports cached tokens on it is reporting a hit on bytes it has never seen), and nothing another session warmed can leak in. For unpinned requests it also defeats OpenRouter's warm-cache stickiness, which would otherwise turn the selection sample into a measurement of stickiness.pair_gap_s in config.json; the realized gap is on every record.temperature 0, seed 7, the least reasoning the model allows, read from the catalog per model (off where optional, the lowest supported effort where mandatory, as on GLM 5.3 and GLM 5.3 Flash). Cache pairs and one-shot tasks are streamed, so time to first token is measured client-side; the pinned session is not, and takes it from GET /generation.provider.order = [endpoint tag] with allow_fallbacks: false and a session_id held fixed for the session, as in The Souq.Panel. Models with at least three competing providers, excluding OpenAI and Anthropic models. Panel A = the ten with the most one-month tokens (203 endpoints, about 61% of mapped OpenRouter tokens as of 2026-09-19), panel B = the other 59 (404 endpoints). python3 probe.py panel rebuilds panel.json from the collector's data; rebuilt monthly.
| Job | What runs | Cadence | Worst case per day |
|---|---|---|---|
pair-a | cache pair (cold, then the same bytes again, a one-word answer, streamed) on every panel A endpoint; the family rotates agent-trace, record-set, canary | 3x daily at jittered hours | $5.42 |
pair-b | the same on a third of panel B | daily, so each model every third day | $0.77 |
event | the same pair, fired by the 5-minute quote poll when an endpoint cuts its input price by 1% or more, sent to the mover and to the model builder at the same moment | on events, at most 4 per endpoint per day | capped $2.00 |
select | unpinned agent-trace turn 1: 4 default, 1 sort: price, 1 sort: throughput, plus 1 unpinned oneshot, per model per slice | 10 slices daily | $1.99 |
default-session | the five-turn session unpinned, one arm with session_id (the sticky-routing key) and one without | 3x daily | $0.99 |
session | the pinned five-turn session, 3 repeats (15 requests) per endpoint | a seventh of panel A daily, so each model weekly | $2.06 |
tasks | generation, oneshot, record-set pinned on every panel A endpoint | daily | $1.23 |
Worst case (every repeat billed at list) is $14.47 a day, $440 a month; python3 probe.py plan recomputes it from live quotes. Endpoint order is shuffled per run (seeded by the recorded run id), and timer jitter moves each sweep across the hours, so no endpoint is always measured at the same time of day.
Budget policy: continuity first. A time series with holes is worth less than one that ran over budget, so the monthly figure ($500) is a target, not a wall (budget in config.json, python3 probe.py budget):
over_pace: a notice goes out once, everything keeps running, and a 7-day grace period starts. A reminder goes out with two days left.probe.py budget --ack 2026-10-31 --note ...), which is recorded.runs.jsonl as skipped with the policy that caused it, so a declared gap is always distinguishable from a failure or an outage.low funds when the key limit or account balance covers fewer than ten days at the current run-rate. The OpenRouter key's own credit limit should be set at the hard ceiling, not at the target, or it becomes the wall this policy avoids.Why this allocation: under a stable pinned session the published measurement found caches mostly work, so the weekly pinned session is a baseline, not the main instrument. The money goes to the conditions where the effect was found: frequent cheap pairs at varying hours, price-cut events, and default routing.
One JSON record per request, append-only, data/probes/<date>.jsonl, with two side files keyed to it: <date>.quotes.jsonl (every endpoint's list prices as captured at the start of each run) and <date>.generations.jsonl (the GET /generation re-fetch per generation_id: total cost, cache discount, provider-native token counts, latency), plus runs.jsonl (one start and one end line per invocation) and data/quotes/<date>.jsonl (every quote change seen by the 5-minute poll). probe.py summarize derives <date>.summary.json. After review these are published as api/v1/probes/<date>.jsonl and api/v1/probes/<date>.json. Schema in schema.json:
run_id, job, session_id, ts, route (openrouter|vercel|direct:<provider>), model, provider_requested, provider_served,
workload, workload_hash, nonce, family, arm, event_id, turn, repeat, pinned, repeated_prompt_tokens, repeat_gap_s, check {passed, failed}, generated, quote {prompt_usd_per_m, completion_usd_per_m, cache_read_usd_per_m, captured_at},
usage {prompt_tokens, completion_tokens, cached_tokens, reasoning_tokens}, realized_usd (from /generation), generation_id,
timing {ttft_ms, total_ms, tokens_per_s}, attestation {present, verified, verifier}, response_sha256, error
The summary reports realized cost against two references computed from the same token counts: the list cost (no cache discount at all) and the ideal cached cost (every repeated token billed at the quoted cache-read price). slippage_vs_list and slippage_vs_ideal_cached bracket each provider between "never discounts" and "discounts everything it could".
generation_id is the tx hash of this dataset: anyone can call OpenRouter's /api/v1/generation?id= and re-fetch the realized cost we claim.
Modeled on InferenceX: every number is traceable to a run, runs are labeled precisely, and the methodology page is a glossary, not an essay.
model · route · workload@hash · date · n requests. Nothing is shown without the label.k with its confidence interval, refreshed weekly.Results are public from the first sweep, on the "Our probe" and "Compare" tabs of routerflow.art, labelled early and unreviewed, with n on every figure. (The first version of this document promised outside review of a week of records before anything measured went public. That was dropped on 2026-09-20 by the project owner in favour of publishing as the data arrives; review happens in the open instead.) What that obliges:
python3 monitor.py --export DIR) carries aggregates per endpoint only: no response text, no generation ids, no keys.vault_openrouter_api_key in the host's SOPS vars, next to vault_routerflow_admin_key for the monitor).routerflow-01 (the Tofu blueprint and Ansible role are already in agent-devops PR #12; the prober timer is part of the same role).Until the keys exist, python3 probe.py plan emits the exact request plan, the workload hashes, live quotes for every target endpoint and a cost estimate at list prices (data/probes/plan.json), so the method can be reviewed without spending. The first paid run is a smoke test on two endpoints with --budget 1.
Built and tested (21 unit tests, including every job end to end against a local stand-in router): the six workloads with checks, the seven jobs, the quote poller with the reprice-event trigger, the continuity-first budget policy with runaway guards, plan, panel, summarize (cache recognition with bill and latency ratios per pair, dollars per session, slippage against list and against an ideal cache, task pass rates, selection shares with a maximum-likelihood Tullock fit and its profile-likelihood interval, reassignment counts for default-routed sessions), the admin monitor, the generated probes page. A doctor command checks readiness without spending; state files are written atomically; one run per job at a time; every run is stamped with the prober version.
Not built: prefix-mutation (25/75/100%) and stretched-wait treatments, the decision route, the identity battery with logprobs, direct-provider and second-router control arms, attested routes, public results panels. No paid request has been sent yet.
Known limit: the prompts are published, so a provider could recognize prober traffic. Two of the three cache-pair families are generated fresh per session; rotating paraphrases of the coding digest is the next mitigation if review asks for it.
Essay with appendices: https://robvc.com/research/the-souq. Short version: "Caching Cheaters on OpenRouter". No data or code is published with it, which is the gap this prober fills. What it measured and how, as far as the text states:
| The Souq | This prober (as built) | |
|---|---|---|
| Models | GLM-5.2 (main), Kimi K3, one Mimo v2.5 Pro example | GLM 5.3 Flash, DeepSeek V4 Flash, Kimi K3 |
| Pinning | one provider, fallbacks disabled, session identifier held fixed | the same: provider.order + allow_fallbacks: false + session_id |
| Workloads | canary (a freshly drawn word repeated several thousand times, sent twice), record set (random ledger rows per session), conversation (own answers appended), agent trace (fixed tool-call log, only the last instruction changes) | canary, record set, fixed coding trace, plus output-heavy and no-context one-shot tasks |
| Cold start | opening context unique to the session, repeated part byte-identical | the same, through the session header line |
| Treatments | prefix mutated after 25%, 75% or all of its content; waits between requests stretched to test TTL expiry | none yet |
| Selection | 1,648 default-routed choices; maximum-likelihood Tullock fit r = 1.968, 95% profile-likelihood interval [1.82, 2.11], block bootstrap over 411 price-event blocks; held-out multinomial logit on log input, output and cached prices | about 700 unpinned picks a day; the same maximum-likelihood fit and interval; no bootstrap or logit yet |
| Price data | public quotes captured every five minutes | the same, for the 69 panel models, plus quotes at the start of each run |
| Cache field | usage.prompt_tokens_details.cached_tokens | the same, plus native_tokens_cached from GET /generation |
Findings that change what this prober should measure:
What followed from it: the reprice-event pair, the default-routed session with and without session_id, the canary and record-set families, and the maximum-likelihood fit are built (section 3). Still open: the prefix-mutation and stretched-wait treatments, the block bootstrap and the multinomial logit, with r = 2 (OpenRouter's documented rule) as the pre-registered null.
The working thesis these test: as agent sessions get longer and more repetitive, the routing advantage moves from picking the cheapest quote on each request to finding and keeping the right workload–provider match, and the quantity to optimize is total cost per successful session: cache reuse, latency, failures and router fees included. The reported data already points this way (on 2026-09-18, 8 of 19 model boards had a different provider cheapest on listed input price than on reported effective input price), but reported effective prices come from different customers with different workloads, so they do not say what the same customer would save by switching. The probes below are that same-customer measurement.
Pre-registered predictions, each with the probe that tests it:
| Probe | What is sent | Question | Headline number |
|---|---|---|---|
| P-1 three-arm session | the same multi-turn session, byte-identical, under three policies: (a) cheapest-price, unpinned with sort: price and no session_id, so the router re-picks every turn; (b) fixed qualified provider, pinned to the cheapest provider that passes the workload checks; (c) cache-aware, pinned to the provider with the lowest expected session cost from the last 7 days of cache-pair records (recognition × cache price + miss × list price) | Does cache-aware placement save money over cheapest-price routing at equal quality? | dollars per successful session per arm, with the difference and its interval; share of sessions where (c) wins |
| P-2 session-length ladder | P-1 at 5, 10, 20 and 40 turns, extending the agent-trace fixed trace (the same canonical replies, more tool rounds over the same repository digest) | Does the cache-aware advantage grow with length? | savings of (c) over (a) against turn count; the turn at which (c) pays for itself |
| P-3 workload–provider fit | P-1 across three shapes: prefix-heavy coding session, lookup over a generated record set, output-heavy generation | Is the best provider the same for every workload? | per model, the winning provider per shape; count of models where the winner differs across shapes |
| P-4 mid-session switch | a pinned session on provider A for k turns, then the remaining turns on provider B (k = 1, 3, 5), against staying on A and starting on B | What does one switch cost? This is the price of the router's reassignment, measured directly | extra dollars and extra time to first token per switch, per provider pair |
| P-5 route overhead | the pinned session through OpenRouter, Vercel AI Gateway and the provider's own API, same model and provider | After fees, is an independent router worth its margin over a free bundled gateway or going direct? | dollars per successful session per route, fee included (credit-purchase and BYOK fees applied from the published schedule); cache recognition per route |
| P-6 share prediction | no new requests: an analysis over the collector's daily provider tokens joined to listed price, reported effective price, and the prober's measured recognition, uptime and TTFT | Which predicts next week's change in provider share best? | out-of-sample fit of three nested models (listed price; + effective price; + measured quality) |
Definitions for all of them:
realized_usd from GET /generation over all turns, including retries, plus the route's fee.Cost at list prices, rough: one five-turn pinned session is about $0.024 on panel A (from the session job's $2.06 for about 435 requests a day), and session cost grows roughly with total prompt tokens. P-1 + P-2 on one panel A model a day (3 arms × 5/10/20 turns, 40 turns weekly) is about $0.60 a day worst case; P-3 and P-4 on the same rotation add about as much again. P-5 needs the Vercel and direct keys listed in section 7. probe.py plan will price the exact schedule once the jobs exist.
Limits to state with the results: the panel is open models with at least three providers, so results say nothing about single-provider models; the policies in P-1 are ours, not any router's product, so (c) is an upper bound on what a router could capture using public data plus our pairs; router margins and customer retention stay unobserved.
Two jobs that are not about the cache. Records go in the same daily files, with job set to the study name; the routed arm of each has arm: "routed" and is kept out of the selection statistics.
time-of-day: does what is served change with the hour? Every hour at :05 UTC (10 min jitter), four arms get the same four freshly generated puzzles (workloads/puzzles, generated from the hour stamp, so every arm sees the same items in a given hour and nothing can be memorized): anthropic/claude-sonnet-5.5 pinned to Anthropic, openai/gpt-6.1-sol pinned to OpenAI, deepseek/deepseek-v4.1-flash pinned to DeepInfra (control), and the same model routed by OpenRouter. Reasoning effort is fixed at low. The model must reply with the answer line only. Items are the hard end of two families (60-90 list operations, 100-150 trades between five people), calibrated on 2026-10-14 so each arm has items it sometimes fails: Claude about 19%, GPT about 88%, DeepSeek about 31% on 16 items each. Read per arm by hour of day: pass rate, reasoning and output tokens per item (a cut in thinking at peak shows here even where the pass rate does not move), time to first token and tokens per second (load), and for the routed arm who served it. The question as posed: is quality worst during the San Francisco working day (about 16:00-01:00 UTC) and best around 05:00-06:00 UTC (1 am Eastern)? About $3.1 a day. With four items an hour per arm, two weeks gives about 500 items in each nine-hour block, enough to see a gap of about 8 points; the token counts resolve much smaller changes.
task-cost: what does one fixed piece of work cost, over months? Six times a day (every four hours, 3 h jitter, so over days it covers every hour), eight arms get the same task (workloads/code-duration): write parse_duration to a full spec. The reply's code is run against 33 hidden tests in a separate interpreter (empty environment, CPU, memory and file limits, imports limited to a short safe list, inside the unit's sandbox); if any test fails the model gets one repair turn with the failing cases. Reasoning effort is fixed at high for every arm, and the model's catalog default that day is recorded beside it. Arms: Claude Sonnet 5.5 at Anthropic, GPT-6.1 Sol at OpenAI, GLM-5.3 at Z.AI, DeepInfra and routed, DeepSeek V4.1 Flash at Together, DeepInfra and routed. (DeepSeek's own endpoint is excluded by this account's training-data setting.) The series: dollars, tokens and wall-clock per task, and pass rate on the first try and after repair. About $0.13 a run, $0.8 a day. More tasks come later; this one is deliberately basic.
Published as two pages on the site, studies/time-of-day.html and studies/task-cost.html, drawn from api/v1/probe/time-of-day.json and api/v1/probe/task-cost.json. studies.py builds both from the records; monitor.py --export writes them beside latest.json (which keeps study records out of every Our probe figure and carries a one-line headline per study). Both studies are also probes in taxonomy.py: the puzzle under One-shot, exact answer, the coding task as its own type, Coding task, graded by running.