{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"streaming-and-stopping","title":"Showing tokens as they arrive, and making them stop","url":"https://addaly.com/learn/hugging-face/streaming-and-stopping","position":17,"of":75,"minutes":8,"answer":"Streaming changes perceived latency rather than compute, and it works by running generate on a second thread feeding a queue.","keyIdea":"Streaming changes perceived latency rather than compute, and it works by running `generate` on a second thread feeding a queue, while a model that never stops usually has a mismatch between the end token its template emits and the `eos_token_id` its generation config is watching for."},"body":{"format":"markdown","text":"## Why streaming changes the product\n\nA 300-token answer at 25 tokens per second takes twelve seconds. Delivered at the end, that is a long silence and people reload the page. Delivered as it arrives, the first words appear in under a second and the wait is tolerable. Nothing about the computation changed; the perceived speed did.\n\nThis is why time to first token is the latency number worth tracking for anything interactive, and total time is the one worth tracking for anything batch.\n\n## The simple streamer\n\n```python\nfrom transformers import TextStreamer\n\nstreamer = TextStreamer(tok, skip_prompt=True, skip_special_tokens=True)\nmodel.generate(**enc, max_new_tokens=200, streamer=streamer)\n```\n\nThis prints to stdout as tokens decode. It is perfect for a terminal script and useless for a web application, because it writes rather than yields.\n\n## The one you use in an app\n\n```python\nfrom transformers import TextIteratorStreamer\nfrom threading import Thread\n\nstreamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)\nkwargs = dict(**enc, max_new_tokens=512, streamer=streamer)\n\nThread(target=model.generate, kwargs=kwargs).start()\n\nfor chunk in streamer:\n    yield chunk          # into a Gradio generator, or an SSE response\n```\n\n`generate` blocks, so it runs on a second thread while the main thread consumes the queue. This is the shape behind every streaming chat UI you have used, including the Gradio Spaces in the next module.\n\nChunks are not tokens and not words. The decoder buffers until it can emit valid text, so you may get `\" un\"` then `\"forgiving\"`. Never assume a chunk boundary is a word boundary, and never split on it for parsing.\n\n## Why models do not stop\n\nGeneration ends when one of three things happens: the model emits its end-of-sequence token, `max_new_tokens` is reached, or a stopping criterion fires. When a model rambles past its answer into an invented next turn, the usual cause is that the EOS it was trained to emit is not the EOS configured at generation time: a common mismatch in fine-tunes and conversions, where the template's end marker and `eos_token_id` disagree.\n\nCheck them against each other:\n\n```python\nprint(tok.eos_token, tok.eos_token_id)\nprint(model.generation_config.eos_token_id)\n```\n\nSome models legitimately have several stop tokens, and `eos_token_id` accepts a list:\n\n```python\nmodel.generate(**enc, eos_token_id=[tok.eos_token_id,\n                                    tok.convert_tokens_to_ids(\"<|eot_id|>\")])\n```\n\n## `max_new_tokens`, not `max_length`\n\n`max_length` counts prompt plus output, so a 900-token prompt with `max_length=1024` leaves room for 124 tokens and the answer gets cut off mid-sentence. `max_new_tokens` counts only the output and is what you almost always want. Both exist; only one behaves the way people expect.\n\nAlways set it. A model with no limit and a broken stop token will generate until it exhausts the context window, which on a 128k model is a long and expensive silence.\n\n## Stop strings\n\nWhen the model reliably emits a marker you want to cut at:\n\n```python\nmodel.generate(**enc, max_new_tokens=256, stop_strings=[\"\\nUser:\", \"```end\"],\n               tokenizer=tok)\n```\n\nStop strings are checked on decoded text, so the tokenizer must be passed. They are a patch over a prompt or template problem rather than a fix, but they are an effective patch and there is no shame in it.\n\n## What streaming costs you\n\nStreaming is not free of trade-offs, and two of them bite in production.\n\nYou cannot validate output you have already shown. If the answer must be checked (against a schema, a moderation pass, a groundedness test) you either buffer it and lose the benefit, or you display text you may have to retract. Most teams compromise: stream the prose, buffer anything structured.\n\nAnd a streamed response is harder to cache. A complete answer is one object you can store against a prompt hash. A stream is a sequence of events, so the usual pattern is to assemble the full text as it goes and write that to the cache at the end, serving later identical requests instantly and non-streamed.\n\n## Cancellation, which people forget\n\nA user closes the tab and your GPU keeps generating 500 tokens for nobody. In a threaded setup, a custom `StoppingCriteria` that checks a flag is the mechanism:\n\n```python\nfrom transformers import StoppingCriteria\n\nclass Cancelled(StoppingCriteria):\n    def __init__(self, flag): self.flag = flag\n    def __call__(self, input_ids, scores, **kw): return self.flag.is_set()\n```\n\nOn a free Space with one worker, one abandoned long generation blocks everybody in the queue. This is a small piece of code with a large effect on how a shared demo feels.\n\n## Do this now\n\nStream a 300-token answer and time two things: seconds to the first visible character, and seconds to the last. Then run the same prompt without streaming and note only the second number. The gap between those two experiences is the whole argument, and it costs six lines."},"check":{"question":"A fine-tuned chat model answers correctly and then continues into an invented next user turn until it hits the token limit. What is the first thing to check?","options":[{"text":"Whether `max_new_tokens` is set too high, since an unbounded limit encourages the model to keep producing text.","correct":false,"tempting":"Confuses a cap with a stop condition; the limit truncates rambling but does not cause it."},{"text":"Whether the template's end marker matches the `eos_token_id` generation watches for.","correct":true},{"text":"Whether the temperature is high enough for the model to escape the loop that keeps it generating.","correct":false,"tempting":"Treats continuation past the answer as a repetition problem, which sampling would address."},{"text":"Whether the streamer is buffering chunks, so the extra turn appears only because decoded text was released late.","correct":false,"tempting":"Blames display buffering for content the model actually generated."}]},"prev":{"slug":"batching-and-throughput","title":"Throughput and latency are different problems","url":"https://addaly.com/learn/hugging-face/batching-and-throughput"},"next":{"slug":"loads-but-outputs-rubbish","title":"It loaded, it ran, the output is nonsense","url":"https://addaly.com/learn/hugging-face/loads-but-outputs-rubbish"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Showing tokens as they arrive, and making them stop', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/streaming-and-stopping (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Showing tokens as they arrive, and making them stop","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"Streaming changes perceived latency rather than compute, and it works by running `generate` on a second thread feeding a queue, while a model that never stops usually has a mismatch between the end token its template emits and the `eos_token_id` its generation config is watching for.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-streaming-changes-the-product","runs":[{"text":"Why streaming changes the product"}]},{"t":"p","runs":[{"text":"A 300-token answer at 25 tokens per second takes twelve seconds. Delivered at the end, that is a long silence and people reload the page. Delivered as it arrives, the first words appear in under a second and the wait is tolerable. Nothing about the computation changed; the perceived speed did."}]},{"t":"p","runs":[{"text":"This is why time to first token is the latency number worth tracking for anything interactive, and total time is the one worth tracking for anything batch."}]},{"t":"h2","id":"the-simple-streamer","runs":[{"text":"The simple streamer"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import TextStreamer\n\nstreamer = TextStreamer(tok, skip_prompt=True, skip_special_tokens=True)\nmodel.generate(**enc, max_new_tokens=200, streamer=streamer)"},{"t":"p","runs":[{"text":"This prints to stdout as tokens decode. It is perfect for a terminal script and useless for a web application, because it writes rather than yields."}]},{"t":"h2","id":"the-one-you-use-in-an-app","runs":[{"text":"The one you use in an app"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import TextIteratorStreamer\nfrom threading import Thread\n\nstreamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)\nkwargs = dict(**enc, max_new_tokens=512, streamer=streamer)\n\nThread(target=model.generate, kwargs=kwargs).start()\n\nfor chunk in streamer:\n    yield chunk          # into a Gradio generator, or an SSE response"},{"t":"p","runs":[{"text":"generate","code":true},{"text":" blocks, so it runs on a second thread while the main thread consumes the queue. This is the shape behind every streaming chat UI you have used, including the Gradio Spaces in the next module."}]},{"t":"p","runs":[{"text":"Chunks are not tokens and not words. The decoder buffers until it can emit valid text, so you may get "},{"text":"\" un\"","code":true},{"text":" then "},{"text":"\"forgiving\"","code":true},{"text":". Never assume a chunk boundary is a word boundary, and never split on it for parsing."}]},{"t":"h2","id":"why-models-do-not-stop","runs":[{"text":"Why models do not stop"}]},{"t":"p","runs":[{"text":"Generation ends when one of three things happens: the model emits its end-of-sequence token, "},{"text":"max_new_tokens","code":true},{"text":" is reached, or a stopping criterion fires. When a model rambles past its answer into an invented next turn, the usual cause is that the EOS it was trained to emit is not the EOS configured at generation time: a common mismatch in fine-tunes and conversions, where the template’s end marker and "},{"text":"eos_token_id","code":true},{"text":" disagree."}]},{"t":"p","runs":[{"text":"Check them against each other:"}]},{"t":"code","lang":"python","label":"Python","text":"print(tok.eos_token, tok.eos_token_id)\nprint(model.generation_config.eos_token_id)"},{"t":"p","runs":[{"text":"Some models legitimately have several stop tokens, and "},{"text":"eos_token_id","code":true},{"text":" accepts a list:"}]},{"t":"code","lang":"python","label":"Python","text":"model.generate(**enc, eos_token_id=[tok.eos_token_id,\n                                    tok.convert_tokens_to_ids(\"<|eot_id|>\")])"},{"t":"h2","id":"maxnewtokens-not-maxlength","runs":[{"text":"max_new_tokens","code":true},{"text":", not "},{"text":"max_length","code":true}]},{"t":"p","runs":[{"text":"max_length","code":true},{"text":" counts prompt plus output, so a 900-token prompt with "},{"text":"max_length=1024","code":true},{"text":" leaves room for 124 tokens and the answer gets cut off mid-sentence. "},{"text":"max_new_tokens","code":true},{"text":" counts only the output and is what you almost always want. Both exist; only one behaves the way people expect."}]},{"t":"p","runs":[{"text":"Always set it. A model with no limit and a broken stop token will generate until it exhausts the context window, which on a 128k model is a long and expensive silence."}]},{"t":"h2","id":"stop-strings","runs":[{"text":"Stop strings"}]},{"t":"p","runs":[{"text":"When the model reliably emits a marker you want to cut at:"}]},{"t":"code","lang":"python","label":"Python","text":"model.generate(**enc, max_new_tokens=256, stop_strings=[\"\\nUser:\", \"```end\"],\n               tokenizer=tok)"},{"t":"p","runs":[{"text":"Stop strings are checked on decoded text, so the tokenizer must be passed. They are a patch over a prompt or template problem rather than a fix, but they are an effective patch and there is no shame in it."}]},{"t":"h2","id":"what-streaming-costs-you","runs":[{"text":"What streaming costs you"}]},{"t":"p","runs":[{"text":"Streaming is not free of trade-offs, and two of them bite in production."}]},{"t":"p","runs":[{"text":"You cannot validate output you have already shown. If the answer must be checked (against a schema, a moderation pass, a groundedness test) you either buffer it and lose the benefit, or you display text you may have to retract. Most teams compromise: stream the prose, buffer anything structured."}]},{"t":"p","runs":[{"text":"And a streamed response is harder to cache. A complete answer is one object you can store against a prompt hash. A stream is a sequence of events, so the usual pattern is to assemble the full text as it goes and write that to the cache at the end, serving later identical requests instantly and non-streamed."}]},{"t":"h2","id":"cancellation-which-people-forget","runs":[{"text":"Cancellation, which people forget"}]},{"t":"p","runs":[{"text":"A user closes the tab and your GPU keeps generating 500 tokens for nobody. In a threaded setup, a custom "},{"text":"StoppingCriteria","code":true},{"text":" that checks a flag is the mechanism:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import StoppingCriteria\n\nclass Cancelled(StoppingCriteria):\n    def __init__(self, flag): self.flag = flag\n    def __call__(self, input_ids, scores, **kw): return self.flag.is_set()"},{"t":"p","runs":[{"text":"On a free Space with one worker, one abandoned long generation blocks everybody in the queue. This is a small piece of code with a large effect on how a shared demo feels."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Stream a 300-token answer and time two things: seconds to the first visible character, and seconds to the last. Then run the same prompt without streaming and note only the second number. The gap between those two experiences is the whole argument, and it costs six lines."}]}],"check":{"question":[{"text":"A fine-tuned chat model answers correctly and then continues into an invented next user turn until it hits the token limit. What is the first thing to check?"}],"options":[{"text":[{"text":"Whether "},{"text":"max_new_tokens","code":true},{"text":" is set too high, since an unbounded limit encourages the model to keep producing text."}],"correct":false,"tempting":[{"text":"Confuses a cap with a stop condition; the limit truncates rambling but does not cause it."}]},{"text":[{"text":"Whether the template’s end marker matches the "},{"text":"eos_token_id","code":true},{"text":" generation watches for."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the temperature is high enough for the model to escape the loop that keeps it generating."}],"correct":false,"tempting":[{"text":"Treats continuation past the answer as a repetition problem, which sampling would address."}]},{"text":[{"text":"Whether the streamer is buffering chunks, so the extra turn appears only because decoded text was released late."}],"correct":false,"tempting":[{"text":"Blames display buffering for content the model actually generated."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"batching-and-throughput","title":"Throughput and latency are different problems","minutes":9},"next":{"slug":"loads-but-outputs-rubbish","title":"It loaded, it ran, the output is nonsense","minutes":9}}}