Testing code that calls an API you pay for

Tests must not touch the real service

A test suite that calls a live API is slow, costs money, fails when the network does, and can be rate-limited into failing at exactly the wrong moment. Worse, it can write real data. The rule is absolute: no test makes a real network call.

That leaves the question of how you test the code that does.

The cheapest technique: hand the call in

From the pure-functions lesson:

def answer(question, call_model):
    prompt = build_prompt(question)
    reply = call_model(prompt)
    return parse_reply(reply)

The test supplies a fake:

def test_answer_extracts_text():
    def fake_model(prompt):
        assert "question" in prompt
        return {"choices": [{"message": {"content": "42"}}]}

    assert answer("what?", fake_model) == "42"

No mocking library, no patching, no network. It runs in a millisecond and it tests the two things you actually wrote: that the prompt contains what it should, and that the response is parsed correctly.

Designing for this is the technique. Everything below is for code you did not get to design.

Patching, when the call is buried

def test_answer(monkeypatch):
    def fake_post(url, **kwargs):
        class R:
            status_code = 200
            def json(self): return {"choices": [{"message": {"content": "42"}}]}
            def raise_for_status(self): pass
        return R()

    monkeypatch.setattr("billing.ai.requests.post", fake_post)
    assert answer("what?") == "42"

monkeypatch is a pytest fixture that replaces an attribute for the duration of one test and puts it back afterwards. The critical detail is where you patch: "billing.ai.requests.post", the name as used inside the module under test, not "requests.post". Patching the wrong path is the most common reason a mock silently does nothing and the test hits the real network.

For HTTP specifically, responses (for requests) and respx (for httpx) are free libraries that intercept at the transport layer and let you declare what each URL returns. They are less brittle than patching a function name.

Test the failures, because that is where the bugs are

The happy path is the easy 20 per cent. Write tests for:

• a 429 rate limit, and confirm your retry logic waits and retries
• a 500, and confirm it gives up after the right number of attempts
• a timeout
• a 200 with malformed content — the body that is not JSON, or is JSON with a missing key
• an empty response

That last group matters most with model APIs, because a 200 does not mean the content is usable. The status code tells you the request and reply worked; only the body tells you what the answer says.

Recorded responses

vcrpy records real HTTP traffic once and replays it in later runs. You get realistic payloads without repeated cost.

Two honest cautions. The recording goes stale, so a test suite full of cassettes can pass for a year after the API changed. And a recording captures your API key in the request headers unless you configure filter_headers, at which point committing the cassette leaks the key. If you use it, set the filter first and check the file before committing it.

The special problem: the answer changes every time

A language model given identical input returns different text on different runs. assert reply == "Delhi is the capital" will pass today and fail on Thursday. Setting temperature to 0 reduces the variation and does not remove it — batching, hardware and model updates all move the output.

So test the contract, not the words:

def test_reply_is_usable():
    result = summarise(document, call_model=fake_or_real)
    assert isinstance(result, dict)
    assert set(result) == {"summary", "topics"}
    assert 20 <= len(result["summary"].split()) <= 80
    assert all(isinstance(t, str) for t in result["topics"])

Shape, length, types, required fields, absence of forbidden content. These hold across runs. Whether the summary is any good is a different question with different machinery — a labelled set, a metric and a judge — and that is a whole course of its own on this site.

Two layers, run at different times

• Unit tests with fakes: run on every commit, take seconds, never touch the network.
• Integration tests against the real service: marked with @pytest.mark.integration, skipped by default, run deliberately before a release with pytest -m integration. They need a real key from the environment and they should be few.

pytestmark = pytest.mark.skipif(
    not os.getenv("OPENAI_API_KEY"), reason="no API key set"
)

That guard means the suite still passes for a contributor who has no key, which keeps the project open to people who cannot pay for one.