Python, from Zero, for AI · Chapter 06

Paying for the handshake once

2:28 · Narrated with a synthetic voice. Captions are available in the player.

Read the transcript

Module six is the one the course is named for: talking to a service properly. A single call looks like one action and is really four. The name is resolved, a connection is opened, encryption is negotiated, and only then does your request move. From a laptop in Delhi to a server in Virginia, those first three steps can be four fifths of the time. A session keeps the connection open between calls, so two hundred requests stop costing two hundred handshakes. Every serious client is a session, and every provider’s own library is a session with the authentication header and the retries already written into it. Then the decision that costs real money. Retrying is safe when the failed attempt provably did nothing: a connection that never opened, a rate limit, a service telling you it is busy. It is not safe when the request reached the server and only the reply was lost, because the model may already have generated the answer and billed for it. A firm in Indore had its bill nearly triple for exactly this reason, invisibly, because the duplicate answers overwrote each other in the database and only the provider’s usage export showed the truth. Separate your timeouts, retry what is safe, and use an idempotency key where the provider offers one. The module finishes with money and language. Every response carries the token counts you were billed for, so multiply them by the price table, keep a running total, and raise an error before the cap is crossed rather than after. The rule that four characters make one token is an English rule. The same message in Devanagari can cost two and a half times as much, because the tokeniser has long cheap pieces for the language it saw most of and short expensive ones for everything else. That is not a curiosity. It is your bill, your context window, and the point at which a prompt gets cut in half.

Sources and further reading (1)