Speech, vision and the rest of the local stack

A language model is one component

Most useful systems need more than text in and text out. All of the following run locally, several of them comfortably on a laptop CPU, and each has a free path.

Speech to text

Transcription is the strongest local story in this whole subject. The models are small, the quality is good, and it runs on hardware that cannot run a language model at all.

# whisper.cpp
./main -m models/ggml-base.en.bin -f interview.wav -l en -otxt

# faster-whisper, a Python route using CTranslate2
pip install faster-whisper

from faster_whisper import WhisperModel
model = WhisperModel("base", device="cpu", compute_type="int8")
segments, info = model.transcribe("interview.wav", language="en")
for s in segments:
    print(f"[{s.start:.1f}s] {s.text}")

Size guidance: tiny and base are fine for clear English speech and run faster than real time on any laptop. small and medium are noticeably better on accents and noise. large variants are best and want a GPU.

The honest limitations: accuracy falls sharply with background noise, crosstalk and heavy accents in languages with less training data; timestamps drift on long files; and speaker separation is a separate model entirely, usually pyannote, which needs its own setup. Transcription of a clean recording is close to solved. Transcription of four people in a café is not.

Text to speech

Two tiers worth knowing. Lightweight synthesisers such as Piper run on a Raspberry Pi and sound clearly synthetic but perfectly intelligible: right for notifications, accessibility and anything embedded. Neural systems in the few-hundred-megabyte class sound close to natural and still run on a CPU at faster than real time.

Voice cloning from a short sample is technically easy now and ethically loaded. Cloning a voice without the speaker’s consent is impersonation, whatever the local law says about it, and the law in several jurisdictions is moving quickly and inconsistently. Treat consent as the requirement rather than the licence terms.

Vision language models

A VLM takes an image alongside text. Locally these run through llama.cpp with a separate projector file that maps image features into the language model’s embedding space:

llama-server -m qwen2-vl-7b-Q4_K_M.gguf \
  --mmproj mmproj-qwen2-vl-7b-f16.gguf -c 8192

The --mmproj file is not optional and downloading only the main GGUF is a common mistake: the model loads and cannot see images.

What local VLMs do well: describing an image, reading clear printed text, answering questions about a chart’s general shape, classifying photographs. What they do badly: precise counting, reading dense tables accurately, small handwriting, and any task where a plausible wrong reading is worse than none. Images are also expensive in tokens: a single image can consume the equivalent of hundreds or low thousands of text tokens, so a conversation with several images fills a context window quickly.

Document understanding

For scanned documents, a dedicated OCR pipeline usually beats a VLM: Tesseract is the long-standing free option, and newer layout-aware models handle tables and columns considerably better. The reliable pattern is OCR first, then a language model over the extracted text, rather than asking a VLM to do both, because when OCR fails you can see the failure, whereas a VLM misreading a figure produces a confident wrong answer with nothing to inspect.

Image generation

Stable Diffusion and its successors run locally through ComfyUI or similar front-ends, on 6 GB of VRAM upwards. This is a large subject with its own course; the relevant fact here is that it is entirely separate from the language model: different architecture, different tooling, and it competes for the same VRAM, so running both at once needs planning.

Putting a stack together

A realistic private assistant on one machine:

1. Transcription with whisper.cpp, CPU, always available.
2. Embeddings with a small retrieval model, CPU, a few hundred megabytes.
3. Reranking with a cross-encoder, CPU.
4. Generation with an 8B at 4-bit, GPU if there is one.
5. Speech output with a small neural voice, CPU.

Only step 4 needs real hardware, and the total VRAM requirement is set by it alone. That is the shape of most working local systems, and it is considerably more achievable than the discussion of graphics cards suggests.