What “OpenAI-compatible” actually has to mean

"OpenAI-compatible" is the phrase that sells relays and self-hosted servers alike, and it covers a wide range. At one end is a faithful reimplementation where existing code moves unchanged. At the other is a URL that accepts a similar JSON body and returns a similar one. Both get described the same way.

Here is the surface that actually has to match, roughly in the order things break when it does not.

The part everyone gets right

The request body. model, messages with roles, temperature, max_tokens, and a response with choices[0].message.content plus a usage block. If this did not work the endpoint would not be usable at all, so it always works.

from openai import OpenAI

client = OpenAI(
    base_url="https://proxystation.co/v1",   # the only line that changes
    api_key="sk-...",
)

resp = client.chat.completions.create(
    model="gpt-5.6-sol",
    messages=[{"role": "user", "content": "hi"}],
)

That the base URL is the only change is the entire commercial proposition. It is also why leaving is cheap, which is the strongest protection a customer of any relay has.

Streaming, where implementations diverge

The first real fault line. A faithful implementation forwards server-sent events as they arrive, preserving chunk boundaries and the data: [DONE] terminator. A lazy one waits for the whole upstream response, then replays it as a burst of chunks.

To code that only reads the final assembled text, these are indistinguishable. To a user watching tokens appear, they are completely different products: one starts answering in a few hundred milliseconds, the other sits silent and then dumps everything at once. If your interface streams, test this specifically — it is the compatibility gap most likely to be quietly missing.

Two related details worth checking: whether usage is returned on the final streaming chunk, since some implementations drop it and your accounting depends on it; and whether the connection is closed cleanly on client abort, since a relay that keeps generating upstream after you disconnect is billing you for tokens nobody will read.

Errors, which almost nobody gets right

The OpenAI error contract is specific: an HTTP status, and a body with error.type, error.code and error.message. Client libraries branch on those fields — retrying a rate limit, failing fast on an invalid request.

The common failure is a server that answers HTTP 200 with a business failure in the body. Every SDK treats that as success. Your retry logic never fires, your error monitoring stays quiet, and the failure surfaces later as a null where a string should be.

Tools, structured output and the long tail

Beyond text completion the surface widens and support gets patchy. In rough order of how often it is missing:

The pattern to watch for is the same throughout: a parameter accepted and ignored. Rejecting an unsupported parameter is honest and easy to handle. Accepting it and doing nothing produces a system that appears to work and quietly does not, and that is much more expensive to discover.

The system prompt question

Some relays inject a system prompt of their own — for safety, for branding, or to shape behaviour. It changes tone, formatting and refusal boundaries without changing the model.

This matters twice. It makes your prompt engineering less predictable, since you are no longer the only author of the context. And it is the most common false positive when canary-testing a relay: responses that differ from the provider’s in style while the discriminating facts still match usually mean an injected prompt rather than a substituted model.

A ten-minute compatibility check

1. 01GET /v1/models — does it list what you expect, and does your key reach it?
2. 02A plain completion, non-streaming. Read usage and reconcile it against what you were charged.
3. 03The same request streaming. Time the first chunk; confirm usage arrives at the end.
4. 04A nonexistent model name. Expect a 404 with a typed error, not a 200.
5. 05A malformed body. Expect a 400 with a typed error.
6. 06A tool call, if you use them. Check the arguments parse as JSON.
7. 07Abort a streaming request halfway and confirm the charge stops.

Seven requests, and they tell you more about an endpoint than any amount of documentation. Our own base URL is https://proxystation.co/v1 and the API docs show the same setup for the OpenAI SDK, the Anthropic SDK and plain cURL — but run the seven anyway, here and anywhere else you are evaluating.

Two protocols, one endpoint

Some relays speak more than one dialect. This one accepts both the OpenAI chat-completions shape and Anthropic’s messages shape on the same base URL, which is why an Anthropic SDK — or a tool built on it — can be pointed here without rewriting the call site.

The distinction is worth understanding because the two protocols disagree about real things. Anthropic takes the system prompt as a top-level parameter rather than a message with a role; the content block structure differs for multimodal input; and stop reasons are named differently. A relay claiming both has to translate, and translation is where behaviour goes missing.

What to check if you rely on it: that the system prompt survives in whichever form you send it, that stop reasons map to something your code can branch on, and that streaming works in both dialects rather than only the one the operator uses themselves.

Rate limits and what the headers tell you

The providers return rate-limit state in response headers — remaining requests, remaining tokens, and when the window resets. Well-behaved clients read them and pace themselves instead of retrying into a wall.

Relays vary in what they do with these. Three behaviours, in descending order of usefulness:

The third case is common and survivable, but it means your retry policy is the only pacing you have — so it needs to be an exponential backoff with jitter rather than a fixed delay. A 429 with no reset hint retried on a fixed timer produces a thundering herd against the exact endpoint that just asked you to slow down.

Related, and easy to miss: whether a 429 from upstream reaches you as a 429 or gets translated into a 500. The two demand different client behaviour — one says wait, the other says something is broken — and a relay that collapses them removes your ability to tell.

Compatibility is what makes a relay cheap to adopt and cheap to leave. The wider guide to relays covers the rest, and canary testing covers the one check compatibility cannot give you: whether the model answering is the one you named.