Most AI features should not be a chat
The user is writing an email, filling in a form, reading a thread, fixing a bug. A chat box in the corner asks them to leave that context, describe it in words, and carry the answer back. The features people actually adopt at scale are the ones that appear inside the thing they were doing: grey text ahead of the cursor, a proposed reply under a message, a draft that lands in the field, a summary at the top of a long thread, a “fix this” beside a validation error. The model proposes; the person accepts, edits or ignores. That division is the whole design.
Ghost text, mechanically
The pattern behind code and writing completion: wait until typing pauses (a debounce of around 300 milliseconds) send the text before the cursor and a little after, ask for a short completion of thirty to sixty tokens, and render it in grey. Tab accepts; any other key dismisses and, on the next pause, asks again.
Three details decide whether it feels good or infuriating. Cancel the in-flight request on every keystroke, with an AbortController, and discard any response whose sequence number is older than the latest request, or stale suggestions will flicker in after the user has moved on. Use a small, fast model: a suggestion that arrives after 500 milliseconds arrives after the user has typed the word themselves, and is worse than nothing. And cache the prefix: the document before the cursor is stable between keystrokes, and prompt caching from the earlier module turns most of each request into a cache read.
Drafts and suggested replies
A draft is generated into the field, editable, and never sent on its own. The person is the author and the model is the ghost-writer; the moment a system sends a message on someone’s behalf without their reading it, every wrong draft becomes a sent mistake with their name on it. Show the draft as clearly provisional (a distinct background, a “suggested” label) and log whether it was sent unchanged, edited or discarded, because that log is the evaluation.
Summaries in place
A thread summary at the top, regenerated when new messages arrive, cached until then. The cost is one call per thread per change, which is small, and the failure is a summary that is wrong about the one thing the reader needed. Link each sentence of the summary to the message it came from, the way the retrieval module linked answers to chunks, so a reader can check in one click.
The measure is acceptance, not thumbs
Nobody clicks a thumbs-up on a completion, and the next lesson explains why those buttons are thin evidence anyway. Inline features come with better instruments built in. The acceptance rate: what share of suggestions were taken. The edit distance after acceptance: how much of an accepted draft survived. Time to complete the task against a control group without the feature. The undo rate, which catches suggestions that looked right for two seconds. Every one of these is recorded for every suggestion, automatically, which gives complete coverage where feedback buttons give two per cent.
Published acceptance rates for code completion sit around a fifth to a third of suggestions. If yours is far below that, the suggestions are wrong or slow; if far above, check that people are not accepting without reading.
The field where a wrong value becomes a fact
A suggestion in a free-text field is reviewed by the act of reading it. A suggestion in a field the user is not looking at (an account number, an amount, a date of birth, a delivery address) can be accepted by habit and become a record. Do not suggest into fields where a wrong value silently becomes a fact, or require an explicit confirmation there rather than a Tab. The convenience is not worth a transfer to the wrong account.
What it costs per user
The economics of inline AI are unlike a chat’s. A ghost-text feature might issue two thousand requests per active user per day, each small. At 500 tokens a request that is a million tokens a day per user; with a cheap model at around $0.10 per million input tokens, ten cents a day, three dollars a month, before caching cuts it. That is a pricing floor, and it is why these features run on the smallest capable model and why the debounce and cancellation matter as much as the prompt. Call count drives the bill, not call size.
Start with one field
Pick the field where people spend the most time and the suggestion is easiest to judge, ship the feature there, and read the acceptance rate for two weeks before adding a second. All of this works with any provider’s API and no framework; the free path is a debounce, a fetch, and a grey span.
Every lesson here is open. Your progress is recorded once you pass the Python, from Zero, for AI exam.
Check your understanding
A ghost-text feature costs far more per user than the team’s chat assistant, although each request is tiny. What explains the difference?
Pick the one you would defend. Your answer stays private.
Stuck on something?Ask Sayali, your AI tutor