The first call from your own server

What a call actually is

Every major provider exposes roughly the same shape: an HTTP POST with a list of messages, returning a message. Most also accept the format OpenAI defined, which Groq, OpenRouter, Ollama and Gemini’s compatibility endpoint all speak, so one piece of code reaches all of them by changing a URL and a key.

// app/api/summarise/route.ts
export async function POST(request: Request) {
  const session = await getSession(request);
  if (!session) return new Response("Unauthorized", { status: 401 });

  const { text } = await request.json();
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 30_000);

  const res = await fetch(`${process.env.LLM_BASE_URL}/chat/completions`, {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      model: process.env.LLM_MODEL,
      messages: [
        { role: "system", content: SUMMARISE_PROMPT },
        { role: "user", content: text },
      ],
      max_tokens: 400,
    }),
    signal: controller.signal,
  });
  clearTimeout(timer);

  if (!res.ok) return new Response("Model unavailable", { status: 502 });
  const data = await res.json();
  return Response.json({ summary: data.choices[0].message.content ?? "" });
}

Four things in that code are not in the generated version, and each is there for a reason.

The session check

The route is reachable by anyone who knows its address, which is anyone who opens the Network tab. Without the check, the world has a free model endpoint paid for by you. The check is the first line for the same reason it is the first line of every route in the data block.

The timeout

A model call takes between one and thirty seconds, depending on the length of the answer. fetch has no default timeout; a stalled connection waits forever, and on a serverless host “forever” means until the function is killed, which costs you the function’s running time and gives the user nothing. Thirty seconds is a reasonable ceiling for a non-streaming call; set it, and handle the abort as an error.

Reading the response

data.choices[0].message.content is where the text lives in this format, and it can be null when the model produced nothing or stopped for a safety reason. Code that assumes a string crashes; the ?? "" avoids that, and a later lesson covers what to show the user instead. data.usage reports the tokens consumed, which the cost lesson will want you to record.

The status codes that matter

401: the key is wrong or missing. Check the variable name on the host.

429: too many requests, or out of credit. On a free tier this is the per-minute limit and it is routine; the resilience lesson handles it.

400: the request was malformed; often a max_tokens above the model’s limit or a message with an empty content.

500 to 503: the provider is having a bad minute. Not your bug.

200 with an empty answer: the one that looks like success. Treat empty content as a failure, and say so in the log.

Why it feels broken without streaming

A 400-token answer takes a model ten to twenty seconds to generate. With the code above, the user sees a spinner for all of it and then a wall of text. Most people give up at five seconds, and hosts kill long functions: ten seconds is a common default for the free tier.

Streaming sends each token to the browser as it is generated. Add stream: true to the body, and the provider returns a sequence of small events instead of one document; your route passes them through to the browser, which appends them to the page:

const upstream = await fetch(url, { ...options, body: JSON.stringify({ ...body, stream: true }) });
return new Response(upstream.body, { headers: { "Content-Type": "text/event-stream" } });

The first word appears in under a second, the function is sending rather than waiting, and a twenty-second answer feels like reading rather than waiting. Libraries such as the Vercel AI SDK wrap the parsing on both ends in a few lines, for any provider, and are the sensible default once the plain version above has been understood.

Test it with a real length

The agent’s test will send “Hello” and get “Hello” back. Send a 3,000-word document, then an empty string, then a document in Hindi, then one with a table. Watch the time, the tokens, and the status code for each. Those four inputs find most of what the first version got wrong, and none of them is the one the agent tried.