The API every engine speaks

Why there is a standard at all

Every serious local engine exposes an HTTP interface modelled on OpenAI’s. This was not coordinated; it happened because client libraries, editor plugins and frameworks were already written against that shape, and speaking it made an engine usable on day one.

The practical consequence is large. Code written against a local server runs unchanged against a hosted provider, and back again. That is what makes the audition from the first block possible, and it makes an API-versus-local comparison a one-line change.

The two endpoints

# chat: the server applies the chat template
curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "local",
  "messages": [{"role": "user", "content": "Two sentences on tides."}],
  "temperature": 0.7, "max_tokens": 200
}'

# completions: you send the exact string, template and all
curl http://localhost:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "The three states of matter are", "max_tokens": 40}'

Use chat for normal work and completions when you need to control the literal prompt: debugging a template, doing fill-in-the-middle code completion, or running a base model.

From code

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-used")

resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Name three tides."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)
print(resp.usage)

api_key must be a non-empty string even when the server ignores it, because the client library requires one. model is often ignored by single-model servers and required by multi-model ones; send the real name and it works in both cases.

The usage field is worth reading every time. prompt_tokens against your configured window is the diagnostic from the previous lesson, available for free on every call.

Streaming

stream = client.chat.completions.create(
    model="local", messages=msgs, stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Streaming does not make generation faster. It makes the wait visible, which matters enormously on local hardware where the first token can be seconds away. For anything a person watches, stream by default.

Note that many servers omit usage from a streamed response unless you ask; there is usually an option such as stream_options: {"include_usage": true}.

Where compatibility runs out

“OpenAI-compatible” is a claim about shape, not about completeness. Expect the following to vary:

• Tool calling. Widely supported and inconsistently formatted, especially for streamed tool calls. Test against your specific engine and model rather than assuming.
• Structured output. response_format with a JSON schema is supported by some engines and not others; several offer their own grammar parameter instead.
• Logprobs. Available in some, absent in others, and shaped differently.
• Vision and audio inputs. Depend entirely on the model and the engine build.
• Sampler parameters. min_p, repeat_penalty, dry_multiplier and similar are not in the original specification. Engines accept them as extra fields, and the OpenAI client library needs them passed through an extra_body argument rather than as normal keyword arguments.

A useful habit: on first contact with any server, request GET /v1/models and read the engine’s own documentation for its extensions. Fifteen minutes there saves a day of guessing which parameter was silently ignored.

Errors, and which of them are worth retrying

A local endpoint fails differently from a hosted one, and a client written against a hosted API usually retries the wrong things.

400 with a context message means the request was longer than the configured window. Retrying is pointless; the fix is to shorten the input or raise the server’s context. This is by far the most common error in local deployments and it is the one blind retry loops hammer forever.

422 or 400 on a field means a parameter the engine does not accept. Also not retryable.

429 comes from your gateway rather than the engine, and carries a wait. Retry with backoff, honouring Retry-After if present.

500 or 503 usually means the engine is loading, restarting after a crash, or out of memory. Retry two or three times with increasing delays, then surface it, because an engine that is down will not recover inside a request’s lifetime.

Streaming adds one case worth handling explicitly: a connection that ends mid-stream leaves you with a partial answer and no final chunk. Detect it by checking for the terminating event rather than by the connection closing, and decide deliberately whether to show the partial text or discard it. Silently treating a truncated stream as a complete answer is a bug users will report as the model being wrong.

Silent ignoring is the real hazard

Most servers accept unknown fields without complaint. If you send top_k to an engine that does not implement it, you get a normal response computed without it, and you may spend an afternoon tuning a parameter that never left your client. When a setting appears to do nothing, verify it is implemented before concluding it does not matter: a controlled test at temperature 0 with an extreme value will tell you in one call.