{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"the-openai-compatible-api","title":"The API every engine speaks","url":"https://addaly.com/learn/running-models-yourself/the-openai-compatible-api","position":34,"of":82,"minutes":8,"answer":"Every local engine speaks the same HTTP dialect, so code moves between local and hosted unchanged, but compatibility is a claim about shape, and unknown parameters are usually accepted and silently ignored.","keyIdea":"Every local engine speaks the same HTTP dialect, so code moves between local and hosted unchanged, but compatibility is a claim about shape, and unknown parameters are usually accepted and silently ignored."},"body":{"format":"markdown","text":"## Why there is a standard at all\n\nEvery serious local engine exposes an HTTP interface modelled on OpenAI's. This was not coordinated; it happened because client libraries, editor plugins and frameworks were already written against that shape, and speaking it made an engine usable on day one.\n\nThe practical consequence is large. Code written against a local server runs unchanged against a hosted provider, and back again. That is what makes the audition from the first block possible, and it makes an API-versus-local comparison a one-line change.\n\n## The two endpoints\n\n```bash\n# chat: the server applies the chat template\ncurl http://localhost:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' -d '{\n  \"model\": \"local\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Two sentences on tides.\"}],\n  \"temperature\": 0.7, \"max_tokens\": 200\n}'\n\n# completions: you send the exact string, template and all\ncurl http://localhost:8080/v1/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"prompt\": \"The three states of matter are\", \"max_tokens\": 40}'\n```\n\nUse chat for normal work and completions when you need to control the literal prompt: debugging a template, doing fill-in-the-middle code completion, or running a base model.\n\n## From code\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"not-used\")\n\nresp = client.chat.completions.create(\n    model=\"local\",\n    messages=[{\"role\": \"user\", \"content\": \"Name three tides.\"}],\n    temperature=0.2,\n)\nprint(resp.choices[0].message.content)\nprint(resp.usage)\n```\n\n`api_key` must be a non-empty string even when the server ignores it, because the client library requires one. `model` is often ignored by single-model servers and required by multi-model ones; send the real name and it works in both cases.\n\nThe `usage` field is worth reading every time. `prompt_tokens` against your configured window is the diagnostic from the previous lesson, available for free on every call.\n\n## Streaming\n\n```python\nstream = client.chat.completions.create(\n    model=\"local\", messages=msgs, stream=True,\n)\nfor chunk in stream:\n    delta = chunk.choices[0].delta.content\n    if delta:\n        print(delta, end=\"\", flush=True)\n```\n\nStreaming does not make generation faster. It makes the wait visible, which matters enormously on local hardware where the first token can be seconds away. For anything a person watches, stream by default.\n\nNote that many servers omit `usage` from a streamed response unless you ask; there is usually an option such as `stream_options: {\"include_usage\": true}`.\n\n## Where compatibility runs out\n\n\"OpenAI-compatible\" is a claim about shape, not about completeness. Expect the following to vary:\n\n- **Tool calling.** Widely supported and inconsistently formatted, especially for streamed tool calls. Test against your specific engine and model rather than assuming.\n- **Structured output.** `response_format` with a JSON schema is supported by some engines and not others; several offer their own grammar parameter instead.\n- **Logprobs.** Available in some, absent in others, and shaped differently.\n- **Vision and audio inputs.** Depend entirely on the model and the engine build.\n- **Sampler parameters.** `min_p`, `repeat_penalty`, `dry_multiplier` and similar are not in the original specification. Engines accept them as extra fields, and the OpenAI client library needs them passed through an `extra_body` argument rather than as normal keyword arguments.\n\nA useful habit: on first contact with any server, request `GET /v1/models` and read the engine's own documentation for its extensions. Fifteen minutes there saves a day of guessing which parameter was silently ignored.\n\n## Errors, and which of them are worth retrying\n\nA local endpoint fails differently from a hosted one, and a client written against a hosted API usually retries the wrong things.\n\n**400 with a context message** means the request was longer than the configured window. Retrying is pointless; the fix is to shorten the input or raise the server's context. This is by far the most common error in local deployments and it is the one blind retry loops hammer forever.\n\n**422 or 400 on a field** means a parameter the engine does not accept. Also not retryable.\n\n**429** comes from your gateway rather than the engine, and carries a wait. Retry with backoff, honouring `Retry-After` if present.\n\n**500 or 503** usually means the engine is loading, restarting after a crash, or out of memory. Retry two or three times with increasing delays, then surface it, because an engine that is down will not recover inside a request's lifetime.\n\nStreaming adds one case worth handling explicitly: a connection that ends mid-stream leaves you with a partial answer and no final chunk. Detect it by checking for the terminating event rather than by the connection closing, and decide deliberately whether to show the partial text or discard it. Silently treating a truncated stream as a complete answer is a bug users will report as the model being wrong.\n\n## Silent ignoring is the real hazard\n\nMost servers accept unknown fields without complaint. If you send `top_k` to an engine that does not implement it, you get a normal response computed without it, and you may spend an afternoon tuning a parameter that never left your client. When a setting appears to do nothing, verify it is implemented before concluding it does not matter: a controlled test at temperature 0 with an extreme value will tell you in one call."},"check":{"question":"A developer sets `min_p` to 0.1 through the OpenAI Python client and sees no change in output diversity at any value. What should they check first?","options":[{"text":"Whether temperature is set to 0, which would make any truncation sampler irrelevant.","correct":false,"tempting":"A real interaction worth knowing, but it would also make the output visibly deterministic, which the developer would have noticed."},{"text":"Whether the model's own generation config overrides client-supplied sampler values.","correct":false,"tempting":"Inverts the precedence in every mainstream engine, where request parameters override defaults rather than the reverse."},{"text":"Whether the parameter reached the server at all: it is not part of the standard schema, so the client library must pass it through an extra body field, and servers accept unknown fields silently.","correct":true},{"text":"Whether the engine applies min_p before temperature, since the order would neutralise the effect.","correct":false,"tempting":"Names a genuine ordering subtlety that changes how strong the effect is, not whether it exists at all."}]},"prev":{"slug":"context-settings-that-truncate-in-silence","title":"Context settings that truncate in silence","url":"https://addaly.com/learn/running-models-yourself/context-settings-that-truncate-in-silence"},"next":{"slug":"structured-output-locally","title":"Output that parses every time","url":"https://addaly.com/learn/running-models-yourself/structured-output-locally"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The API every engine speaks', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/the-openai-compatible-api (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The API every engine speaks","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"Every local engine speaks the same HTTP dialect, so code moves between local and hosted unchanged, but compatibility is a claim about shape, and unknown parameters are usually accepted and silently ignored.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-there-is-a-standard-at-all","runs":[{"text":"Why there is a standard at all"}]},{"t":"p","runs":[{"text":"Every serious local engine exposes an HTTP interface modelled on OpenAI’s. This was not coordinated; it happened because client libraries, editor plugins and frameworks were already written against that shape, and speaking it made an engine usable on day one."}]},{"t":"p","runs":[{"text":"The practical consequence is large. Code written against a local server runs unchanged against a hosted provider, and back again. That is what makes the audition from the first block possible, and it makes an API-versus-local comparison a one-line change."}]},{"t":"h2","id":"the-two-endpoints","runs":[{"text":"The two endpoints"}]},{"t":"code","lang":"bash","label":"Bash","text":"# chat: the server applies the chat template\ncurl http://localhost:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' -d '{\n  \"model\": \"local\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Two sentences on tides.\"}],\n  \"temperature\": 0.7, \"max_tokens\": 200\n}'\n\n# completions: you send the exact string, template and all\ncurl http://localhost:8080/v1/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"prompt\": \"The three states of matter are\", \"max_tokens\": 40}'"},{"t":"p","runs":[{"text":"Use chat for normal work and completions when you need to control the literal prompt: debugging a template, doing fill-in-the-middle code completion, or running a base model."}]},{"t":"h2","id":"from-code","runs":[{"text":"From code"}]},{"t":"code","lang":"python","label":"Python","text":"from openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"not-used\")\n\nresp = client.chat.completions.create(\n    model=\"local\",\n    messages=[{\"role\": \"user\", \"content\": \"Name three tides.\"}],\n    temperature=0.2,\n)\nprint(resp.choices[0].message.content)\nprint(resp.usage)"},{"t":"p","runs":[{"text":"api_key","code":true},{"text":" must be a non-empty string even when the server ignores it, because the client library requires one. "},{"text":"model","code":true},{"text":" is often ignored by single-model servers and required by multi-model ones; send the real name and it works in both cases."}]},{"t":"p","runs":[{"text":"The "},{"text":"usage","code":true},{"text":" field is worth reading every time. "},{"text":"prompt_tokens","code":true},{"text":" against your configured window is the diagnostic from the previous lesson, available for free on every call."}]},{"t":"h2","id":"streaming","runs":[{"text":"Streaming"}]},{"t":"code","lang":"python","label":"Python","text":"stream = client.chat.completions.create(\n    model=\"local\", messages=msgs, stream=True,\n)\nfor chunk in stream:\n    delta = chunk.choices[0].delta.content\n    if delta:\n        print(delta, end=\"\", flush=True)"},{"t":"p","runs":[{"text":"Streaming does not make generation faster. It makes the wait visible, which matters enormously on local hardware where the first token can be seconds away. For anything a person watches, stream by default."}]},{"t":"p","runs":[{"text":"Note that many servers omit "},{"text":"usage","code":true},{"text":" from a streamed response unless you ask; there is usually an option such as "},{"text":"stream_options: {\"include_usage\": true}","code":true},{"text":"."}]},{"t":"h2","id":"where-compatibility-runs-out","runs":[{"text":"Where compatibility runs out"}]},{"t":"p","runs":[{"text":"“OpenAI-compatible” is a claim about shape, not about completeness. Expect the following to vary:"}]},{"t":"ul","items":[[{"text":"Tool calling.","b":true},{"text":" Widely supported and inconsistently formatted, especially for streamed tool calls. Test against your specific engine and model rather than assuming."}],[{"text":"Structured output.","b":true},{"text":" "},{"text":"response_format","code":true},{"text":" with a JSON schema is supported by some engines and not others; several offer their own grammar parameter instead."}],[{"text":"Logprobs.","b":true},{"text":" Available in some, absent in others, and shaped differently."}],[{"text":"Vision and audio inputs.","b":true},{"text":" Depend entirely on the model and the engine build."}],[{"text":"Sampler parameters.","b":true},{"text":" "},{"text":"min_p","code":true},{"text":", "},{"text":"repeat_penalty","code":true},{"text":", "},{"text":"dry_multiplier","code":true},{"text":" and similar are not in the original specification. Engines accept them as extra fields, and the OpenAI client library needs them passed through an "},{"text":"extra_body","code":true},{"text":" argument rather than as normal keyword arguments."}]]},{"t":"p","runs":[{"text":"A useful habit: on first contact with any server, request "},{"text":"GET /v1/models","code":true},{"text":" and read the engine’s own documentation for its extensions. Fifteen minutes there saves a day of guessing which parameter was silently ignored."}]},{"t":"h2","id":"errors-and-which-of-them-are-worth-retrying","runs":[{"text":"Errors, and which of them are worth retrying"}]},{"t":"p","runs":[{"text":"A local endpoint fails differently from a hosted one, and a client written against a hosted API usually retries the wrong things."}]},{"t":"p","runs":[{"text":"400 with a context message","b":true},{"text":" means the request was longer than the configured window. Retrying is pointless; the fix is to shorten the input or raise the server’s context. This is by far the most common error in local deployments and it is the one blind retry loops hammer forever."}]},{"t":"p","runs":[{"text":"422 or 400 on a field","b":true},{"text":" means a parameter the engine does not accept. Also not retryable."}]},{"t":"p","runs":[{"text":"429","b":true},{"text":" comes from your gateway rather than the engine, and carries a wait. Retry with backoff, honouring "},{"text":"Retry-After","code":true},{"text":" if present."}]},{"t":"p","runs":[{"text":"500 or 503","b":true},{"text":" usually means the engine is loading, restarting after a crash, or out of memory. Retry two or three times with increasing delays, then surface it, because an engine that is down will not recover inside a request’s lifetime."}]},{"t":"p","runs":[{"text":"Streaming adds one case worth handling explicitly: a connection that ends mid-stream leaves you with a partial answer and no final chunk. Detect it by checking for the terminating event rather than by the connection closing, and decide deliberately whether to show the partial text or discard it. Silently treating a truncated stream as a complete answer is a bug users will report as the model being wrong."}]},{"t":"h2","id":"silent-ignoring-is-the-real-hazard","runs":[{"text":"Silent ignoring is the real hazard"}]},{"t":"p","runs":[{"text":"Most servers accept unknown fields without complaint. If you send "},{"text":"top_k","code":true},{"text":" to an engine that does not implement it, you get a normal response computed without it, and you may spend an afternoon tuning a parameter that never left your client. When a setting appears to do nothing, verify it is implemented before concluding it does not matter: a controlled test at temperature 0 with an extreme value will tell you in one call."}]}],"check":{"question":[{"text":"A developer sets "},{"text":"min_p","code":true},{"text":" to 0.1 through the OpenAI Python client and sees no change in output diversity at any value. What should they check first?"}],"options":[{"text":[{"text":"Whether temperature is set to 0, which would make any truncation sampler irrelevant."}],"correct":false,"tempting":[{"text":"A real interaction worth knowing, but it would also make the output visibly deterministic, which the developer would have noticed."}]},{"text":[{"text":"Whether the model’s own generation config overrides client-supplied sampler values."}],"correct":false,"tempting":[{"text":"Inverts the precedence in every mainstream engine, where request parameters override defaults rather than the reverse."}]},{"text":[{"text":"Whether the parameter reached the server at all: it is not part of the standard schema, so the client library must pass it through an extra body field, and servers accept unknown fields silently."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the engine applies min_p before temperature, since the order would neutralise the effect."}],"correct":false,"tempting":[{"text":"Names a genuine ordering subtlety that changes how strong the effect is, not whether it exists at all."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"context-settings-that-truncate-in-silence","title":"Context settings that truncate in silence","minutes":9},"next":{"slug":"structured-output-locally","title":"Output that parses every time","minutes":9}}}