{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"speech-and-vision-locally","title":"Speech, vision and the rest of the local stack","url":"https://addaly.com/learn/running-models-yourself/speech-and-vision-locally","position":37,"of":82,"minutes":9,"answer":"Transcription, embedding, reranking and speech synthesis all run on an ordinary CPU.","keyIdea":"Transcription, embedding, reranking and speech synthesis all run on an ordinary CPU, so a complete private stack needs a GPU only for the language model. And a vision model needs its projector file or it loads without being able to see."},"body":{"format":"markdown","text":"## A language model is one component\n\nMost useful systems need more than text in and text out. All of the following run locally, several of them comfortably on a laptop CPU, and each has a free path.\n\n## Speech to text\n\nTranscription is the strongest local story in this whole subject. The models are small, the quality is good, and it runs on hardware that cannot run a language model at all.\n\n```bash\n# whisper.cpp\n./main -m models/ggml-base.en.bin -f interview.wav -l en -otxt\n\n# faster-whisper, a Python route using CTranslate2\npip install faster-whisper\n```\n\n```python\nfrom faster_whisper import WhisperModel\nmodel = WhisperModel(\"base\", device=\"cpu\", compute_type=\"int8\")\nsegments, info = model.transcribe(\"interview.wav\", language=\"en\")\nfor s in segments:\n    print(f\"[{s.start:.1f}s] {s.text}\")\n```\n\nSize guidance: `tiny` and `base` are fine for clear English speech and run faster than real time on any laptop. `small` and `medium` are noticeably better on accents and noise. `large` variants are best and want a GPU.\n\nThe honest limitations: accuracy falls sharply with background noise, crosstalk and heavy accents in languages with less training data; timestamps drift on long files; and speaker separation is a separate model entirely, usually `pyannote`, which needs its own setup. Transcription of a clean recording is close to solved. Transcription of four people in a café is not.\n\n## Text to speech\n\nTwo tiers worth knowing. Lightweight synthesisers such as Piper run on a Raspberry Pi and sound clearly synthetic but perfectly intelligible: right for notifications, accessibility and anything embedded. Neural systems in the few-hundred-megabyte class sound close to natural and still run on a CPU at faster than real time.\n\nVoice cloning from a short sample is technically easy now and ethically loaded. Cloning a voice without the speaker's consent is impersonation, whatever the local law says about it, and the law in several jurisdictions is moving quickly and inconsistently. Treat consent as the requirement rather than the licence terms.\n\n## Vision language models\n\nA VLM takes an image alongside text. Locally these run through llama.cpp with a separate projector file that maps image features into the language model's embedding space:\n\n```bash\nllama-server -m qwen2-vl-7b-Q4_K_M.gguf \\\n  --mmproj mmproj-qwen2-vl-7b-f16.gguf -c 8192\n```\n\nThe `--mmproj` file is not optional and downloading only the main GGUF is a common mistake: the model loads and cannot see images.\n\nWhat local VLMs do well: describing an image, reading clear printed text, answering questions about a chart's general shape, classifying photographs. What they do badly: precise counting, reading dense tables accurately, small handwriting, and any task where a plausible wrong reading is worse than none. Images are also expensive in tokens: a single image can consume the equivalent of hundreds or low thousands of text tokens, so a conversation with several images fills a context window quickly.\n\n## Document understanding\n\nFor scanned documents, a dedicated OCR pipeline usually beats a VLM: Tesseract is the long-standing free option, and newer layout-aware models handle tables and columns considerably better. The reliable pattern is OCR first, then a language model over the extracted text, rather than asking a VLM to do both, because when OCR fails you can see the failure, whereas a VLM misreading a figure produces a confident wrong answer with nothing to inspect.\n\n## Image generation\n\nStable Diffusion and its successors run locally through ComfyUI or similar front-ends, on 6 GB of VRAM upwards. This is a large subject with its own course; the relevant fact here is that it is entirely separate from the language model: different architecture, different tooling, and it competes for the same VRAM, so running both at once needs planning.\n\n## Putting a stack together\n\nA realistic private assistant on one machine:\n\n1. **Transcription** with whisper.cpp, CPU, always available.\n2. **Embeddings** with a small retrieval model, CPU, a few hundred megabytes.\n3. **Reranking** with a cross-encoder, CPU.\n4. **Generation** with an 8B at 4-bit, GPU if there is one.\n5. **Speech output** with a small neural voice, CPU.\n\nOnly step 4 needs real hardware, and the total VRAM requirement is set by it alone. That is the shape of most working local systems, and it is considerably more achievable than the discussion of graphics cards suggests."},"check":{"question":"A team building a scanned-invoice pipeline replaces their OCR-then-language-model design with a single vision language model. Accuracy on totals gets worse and the failures are harder to investigate. What explains both effects?","options":[{"text":"The VLM was loaded without its projector file, so it is answering from the text prompt alone.","correct":false,"tempting":"Would produce catastrophic, obvious failure rather than a modest accuracy decline on specific fields."},{"text":"Images consume far more context than text, so invoices are being truncated before the totals are reached.","correct":false,"tempting":"Names a real cost with the wrong consequence; truncation would lose whole regions rather than degrade numeric reading."},{"text":"The VLM reads numerals less reliably than dedicated OCR and produces a confident single answer, removing the intermediate text where a misreading would have been visible.","correct":true},{"text":"Vision models cannot process numeric characters, which are handled by a separate OCR head in most architectures.","correct":false,"tempting":"Invents an architectural split; VLMs read digits, just less reliably than purpose-built recognisers."}]},"prev":{"slug":"embeddings-and-rerankers-locally","title":"Embeddings and rerankers on your own machine","url":"https://addaly.com/learn/running-models-yourself/embeddings-and-rerankers-locally"},"next":{"slug":"engines-and-what-faster-means","title":"Serving engines, and what \"faster\" means","url":"https://addaly.com/learn/running-models-yourself/engines-and-what-faster-means"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Speech, vision and the rest of the local stack', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/speech-and-vision-locally (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Speech, vision and the rest of the local stack","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"Transcription, embedding, reranking and speech synthesis all run on an ordinary CPU, so a complete private stack needs a GPU only for the language model. And a vision model needs its projector file or it loads without being able to see.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-language-model-is-one-component","runs":[{"text":"A language model is one component"}]},{"t":"p","runs":[{"text":"Most useful systems need more than text in and text out. All of the following run locally, several of them comfortably on a laptop CPU, and each has a free path."}]},{"t":"h2","id":"speech-to-text","runs":[{"text":"Speech to text"}]},{"t":"p","runs":[{"text":"Transcription is the strongest local story in this whole subject. The models are small, the quality is good, and it runs on hardware that cannot run a language model at all."}]},{"t":"code","lang":"bash","label":"Bash","text":"# whisper.cpp\n./main -m models/ggml-base.en.bin -f interview.wav -l en -otxt\n\n# faster-whisper, a Python route using CTranslate2\npip install faster-whisper"},{"t":"code","lang":"python","label":"Python","text":"from faster_whisper import WhisperModel\nmodel = WhisperModel(\"base\", device=\"cpu\", compute_type=\"int8\")\nsegments, info = model.transcribe(\"interview.wav\", language=\"en\")\nfor s in segments:\n    print(f\"[{s.start:.1f}s] {s.text}\")"},{"t":"p","runs":[{"text":"Size guidance: "},{"text":"tiny","code":true},{"text":" and "},{"text":"base","code":true},{"text":" are fine for clear English speech and run faster than real time on any laptop. "},{"text":"small","code":true},{"text":" and "},{"text":"medium","code":true},{"text":" are noticeably better on accents and noise. "},{"text":"large","code":true},{"text":" variants are best and want a GPU."}]},{"t":"p","runs":[{"text":"The honest limitations: accuracy falls sharply with background noise, crosstalk and heavy accents in languages with less training data; timestamps drift on long files; and speaker separation is a separate model entirely, usually "},{"text":"pyannote","code":true},{"text":", which needs its own setup. Transcription of a clean recording is close to solved. Transcription of four people in a café is not."}]},{"t":"h2","id":"text-to-speech","runs":[{"text":"Text to speech"}]},{"t":"p","runs":[{"text":"Two tiers worth knowing. Lightweight synthesisers such as Piper run on a Raspberry Pi and sound clearly synthetic but perfectly intelligible: right for notifications, accessibility and anything embedded. Neural systems in the few-hundred-megabyte class sound close to natural and still run on a CPU at faster than real time."}]},{"t":"p","runs":[{"text":"Voice cloning from a short sample is technically easy now and ethically loaded. Cloning a voice without the speaker’s consent is impersonation, whatever the local law says about it, and the law in several jurisdictions is moving quickly and inconsistently. Treat consent as the requirement rather than the licence terms."}]},{"t":"h2","id":"vision-language-models","runs":[{"text":"Vision language models"}]},{"t":"p","runs":[{"text":"A VLM takes an image alongside text. Locally these run through llama.cpp with a separate projector file that maps image features into the language model’s embedding space:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m qwen2-vl-7b-Q4_K_M.gguf \\\n  --mmproj mmproj-qwen2-vl-7b-f16.gguf -c 8192"},{"t":"p","runs":[{"text":"The "},{"text":"--mmproj","code":true},{"text":" file is not optional and downloading only the main GGUF is a common mistake: the model loads and cannot see images."}]},{"t":"p","runs":[{"text":"What local VLMs do well: describing an image, reading clear printed text, answering questions about a chart’s general shape, classifying photographs. What they do badly: precise counting, reading dense tables accurately, small handwriting, and any task where a plausible wrong reading is worse than none. Images are also expensive in tokens: a single image can consume the equivalent of hundreds or low thousands of text tokens, so a conversation with several images fills a context window quickly."}]},{"t":"h2","id":"document-understanding","runs":[{"text":"Document understanding"}]},{"t":"p","runs":[{"text":"For scanned documents, a dedicated OCR pipeline usually beats a VLM: Tesseract is the long-standing free option, and newer layout-aware models handle tables and columns considerably better. The reliable pattern is OCR first, then a language model over the extracted text, rather than asking a VLM to do both, because when OCR fails you can see the failure, whereas a VLM misreading a figure produces a confident wrong answer with nothing to inspect."}]},{"t":"h2","id":"image-generation","runs":[{"text":"Image generation"}]},{"t":"p","runs":[{"text":"Stable Diffusion and its successors run locally through ComfyUI or similar front-ends, on 6 GB of VRAM upwards. This is a large subject with its own course; the relevant fact here is that it is entirely separate from the language model: different architecture, different tooling, and it competes for the same VRAM, so running both at once needs planning."}]},{"t":"h2","id":"putting-a-stack-together","runs":[{"text":"Putting a stack together"}]},{"t":"p","runs":[{"text":"A realistic private assistant on one machine:"}]},{"t":"ol","start":1,"items":[[{"text":"Transcription","b":true},{"text":" with whisper.cpp, CPU, always available."}],[{"text":"Embeddings","b":true},{"text":" with a small retrieval model, CPU, a few hundred megabytes."}],[{"text":"Reranking","b":true},{"text":" with a cross-encoder, CPU."}],[{"text":"Generation","b":true},{"text":" with an 8B at 4-bit, GPU if there is one."}],[{"text":"Speech output","b":true},{"text":" with a small neural voice, CPU."}]]},{"t":"p","runs":[{"text":"Only step 4 needs real hardware, and the total VRAM requirement is set by it alone. That is the shape of most working local systems, and it is considerably more achievable than the discussion of graphics cards suggests."}]}],"check":{"question":[{"text":"A team building a scanned-invoice pipeline replaces their OCR-then-language-model design with a single vision language model. Accuracy on totals gets worse and the failures are harder to investigate. What explains both effects?"}],"options":[{"text":[{"text":"The VLM was loaded without its projector file, so it is answering from the text prompt alone."}],"correct":false,"tempting":[{"text":"Would produce catastrophic, obvious failure rather than a modest accuracy decline on specific fields."}]},{"text":[{"text":"Images consume far more context than text, so invoices are being truncated before the totals are reached."}],"correct":false,"tempting":[{"text":"Names a real cost with the wrong consequence; truncation would lose whole regions rather than degrade numeric reading."}]},{"text":[{"text":"The VLM reads numerals less reliably than dedicated OCR and produces a confident single answer, removing the intermediate text where a misreading would have been visible."}],"correct":true,"tempting":null},{"text":[{"text":"Vision models cannot process numeric characters, which are handled by a separate OCR head in most architectures."}],"correct":false,"tempting":[{"text":"Invents an architectural split; VLMs read digits, just less reliably than purpose-built recognisers."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"embeddings-and-rerankers-locally","title":"Embeddings and rerankers on your own machine","minutes":10},"next":{"slug":"engines-and-what-faster-means","title":"Serving engines, and what “faster” means","minutes":11}}}