Canary-testing an AI API relay: a method you can run yourself

A canary test answers one question: when you ask a relay for a model, does that model answer? You cannot read the relay’s routing table and its marketing page is not evidence, so the only way to find out is to send a request whose answer differs depending on which model handled it. This is the method, with a script you can run in about five minutes.

It matters because the failure it detects is common. A study of 28 relays found that 45.83% of endpoints returned a model that did not match the one requested. Substitution is invisible in normal use — the response is fluent, the shape is right, the bill is lower — and it only shows up when you go looking for it.

Why you cannot simply ask the model

The obvious test is to ask "which model are you?" and it is close to worthless. Three reasons:

• Models are unreliable narrators about their own identity. They answer from training data, which was written before they existed, and they are often confidently wrong.
• A relay can inject a system prompt. If the answer to that question is worth money, it can be made to say whatever the operator wants.
• A substituted model has usually been fine-tuned or prompted to behave like the expensive one, because that is the whole point of the substitution.

A useful canary does not ask the model to describe itself. It asks something where the correct behaviour is a property of the model rather than a claim about it.

What makes a good canary prompt

Four properties, and a prompt missing any one of them produces results you cannot act on.

1. 01Deterministic. The same prompt at temperature 0 gives the same answer from the same model. If the answer wanders, you are measuring sampling noise rather than identity.
2. 02Discriminating. Different models give measurably different answers. A question every model answers identically tells you nothing.
3. 03Cheap. A few tokens in, a few out. You will run it many times.
4. 04Stable over the version you care about. The answer should depend on the model, not on the date, the weather, or a web search the model might make.

Prompt families that work

The method

Five steps. The whole thing hinges on step two, which is the part people skip.

1. 01Pick three to five canary prompts and fix them. Write them down; do not improvise per run.
2. 02Build the answer key by sending them to the provider directly, at temperature 0, with no system prompt. This is your reference. Without it you are comparing the relay against your expectations rather than against reality.
3. 03Send the identical prompts to the relay, same model name, same temperature, same absence of a system prompt.
4. 04Compare. Exact match on short deterministic answers; for longer ones compare structure and the specific discriminating detail rather than the whole string.
5. 05Repeat across several requests and, ideally, across the day. Routing is per-request, so a single run only tells you about one route.

A script

This sends the same prompt to two base URLs and prints both answers side by side. It has no dependencies beyond a shell and curl.

#!/usr/bin/env bash
# Usage: MODEL=gpt-5.6-sol ./canary.sh
# Set both keys in your environment first. Never paste keys into a file.
set -euo pipefail

MODEL="${MODEL:?set MODEL}"
PROMPT='Reply with only the number of characters in the string "strawberry-jam-2026". No words.'

ask() {  # ask <base-url> <api-key>
  curl -sS "$1/chat/completions" \
    -H "Authorization: Bearer $2" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"$MODEL\",\"temperature\":0,
         \"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}" \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); \
      print(d["choices"][0]["message"]["content"].strip(), "|", \
            d.get("usage",{}).get("completion_tokens"), "tokens")'
}

echo "provider: $(ask "$PROVIDER_BASE" "$PROVIDER_KEY")"
for i in 1 2 3 4 5; do
  echo "relay $i : $(ask "$RELAY_BASE" "$RELAY_KEY")"
done

Run it five times against the relay and once against the provider. Identical answers across all six are a pass for that model, that prompt, that moment. Divergence is not proof of substitution on its own — check the token counts too, since a different tokenizer usually shows up there before the text does.

Reading the result honestly

Canary testing is evidence, not proof, and overclaiming what it shows is the fastest way to reach a wrong conclusion.

The most common false positive is a system prompt you did not send. Some relays inject one for safety or for branding, which changes formatting and refusal behaviour without changing the model. If everything differs in tone but the discriminating facts match, suspect a prompt before suspecting substitution.

Run it against us

We have not published a canary run of our own, and we are not going to present one as independent evidence — a relay grading its own homework is worth roughly what you would expect. What we can do is make the test easy and tell you what to expect.

The base URL is https://proxystation.co/v1, it is OpenAI-compatible, and GET /v1/models will list what your key can reach. The model names, per-token prices and route counts are all in the public catalogue without an account. If a canary run against us diverges from the provider, we want to know — that is a routing bug and it is the kind we would rather hear about from a reader than not hear about at all.

Every relay you evaluate this way, including ours, is one you understand better than the marketing page allows. That is the entire argument for doing it.

What to do when a canary fails

A divergence is a beginning, not a verdict. Work through it in this order, because the cheap explanations are also the common ones.

1. 01Re-run at temperature 0. If the answer moves between runs against the *provider* too, the prompt is not deterministic and the result is noise. Replace the prompt.
2. 02Check for an injected system prompt. Ask for a deliberately odd format — all lowercase, no punctuation. If the relay overrides it while the provider obeys, something is being added to your context.
3. 03Compare token counts. Matching text with different completion_tokens points at a different serving stack; different text with matching counts points at sampling.
4. 04Vary the model. If every model on the relay diverges identically, suspect the relay’s middleware. If one model diverges, suspect that model’s routing.
5. 05Ask the operator. A relay that can name the upstream channel and explain the difference is behaving well. One that cannot has told you something either way.

Only after those four does substitution become the best explanation — and even then it is worth repeating the run a few hours later, because a route that misbehaves is often one route of several rather than the whole model.

Turning it into a standing check

A canary run is most useful when it is boring: the same prompts, on a schedule, with the answers recorded. A one-off test tells you about one moment; a series tells you whether anything changed.

The minimum useful version is a cron job that runs the script daily, appends the answer and token count to a file, and alerts you when a line differs from the previous one. Three things make it worth the effort:

• Routing changes are invisible otherwise. Operators add and remove upstream channels routinely, and nothing announces it.
• It dates your evidence. "It matched on 2026-08-30" is a fact; "it seemed fine" is not.
• It costs almost nothing — a handful of tokens a day for a check on the thing you are actually buying.

This method is one of four checks in the guide to how relays work and how to audit one. The companion piece explains why a single passing run is not enough: one model name can have several routes. And if a run diverges in tone rather than in fact, the likely cause is an injected system prompt — what OpenAI-compatible has to mean covers that.