{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"a-local-model-from-python","title":"A free model on your own machine, called from Python","url":"https://addaly.com/learn/python-for-ai/a-local-model-from-python","position":85,"of":89,"minutes":9,"answer":"Ollama, llama.cpp and transformers each run an open model on a CPU with no account; memory is parameters times bytes per weight, tokens per second is what a CPU gives you.","keyIdea":"Ollama, llama.cpp and transformers each run an open model on a CPU with no account; memory is parameters times bytes per weight, tokens per second is what a CPU gives you, and the same provider class from module 7 makes the local model a drop-in for the paid one."},"body":{"format":"markdown","text":"## Why a local model\n\nThree reasons, and a learner on a phone or an old laptop will feel all of them. It costs nothing per call, so you can run your script two hundred times while learning. It needs no account, no card and no key, so nothing can leak. And it works offline. What you give up is quality (a 1-billion-parameter model on a CPU is not a hosted frontier model) and speed. For learning, testing and many real tasks, that trade is good.\n\n## The arithmetic first\n\nA model's memory is its parameters multiplied by the bytes per weight:\n\n- 0.5B parameters × 2 bytes (float16) ≈ 1 GB\n- 3B × 2 ≈ 6 GB; 3B × 0.5 (4-bit quantised) ≈ 1.5 to 2 GB\n- 7B × 2 ≈ 14 GB; 7B × 0.5 ≈ 4 GB\n- 70B × 0.5 ≈ 35 to 40 GB\n\nAdd a gigabyte or so of working memory. A laptop with 8 GB runs a 3B model quantised comfortably, a 7B model quantised at a squeeze, and a 7B model in float16 not at all: the process is killed by the operating system with no Python traceback, because the failure happens below Python. Do the multiplication before the download.\n\nSpeed on a CPU is a few to a few dozen tokens per second depending on size and machine. A 0.5B model answers in a couple of seconds; a 7B model quantised writes at reading speed on a recent laptop and slower on an old one. A phone under Termux runs 0.5B to 1.5B models.\n\n## Ollama: the easy path\n\nOllama is a free program that downloads models, quantises them and serves them over HTTP on your machine.\n\n```bash\nollama pull qwen2.5:1.5b        # ~1 GB download\nollama run qwen2.5:1.5b         # chat in the terminal to check it works\n```\n\nFrom Python, you already know how to talk to it: module 6 did, through the OpenAI-compatible endpoint:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\nr = client.chat.completions.create(model=\"qwen2.5:1.5b\",\n        messages=[{\"role\": \"user\", \"content\": \"Explain a dict in one sentence.\"}])\nprint(r.choices[0].message.content)\n```\n\nOr with plain `requests` against Ollama's own API, which also streams:\n\n```python\nr = requests.post(\"http://localhost:11434/api/chat\", json={\n    \"model\": \"qwen2.5:1.5b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n    \"stream\": False,\n}, timeout=120)\nprint(r.json()[\"message\"][\"content\"])\n```\n\n`ollama list` shows what you have; models are stored under `~/.ollama`. Embedding models (`nomic-embed-text`, `bge-m3` for multilingual) run the same way, via `/api/embeddings`, so the notes search from the previous lesson can be entirely local.\n\n## llama.cpp: the engine underneath\n\nOllama wraps `llama.cpp`, a C++ inference engine that runs quantised models in the GGUF format on almost anything. Its Python binding gives you the model as an object with no server:\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(model_path=\"qwen2.5-1.5b-instruct-q4_k_m.gguf\", n_ctx=4096, verbose=False)\nout = llm.create_chat_completion(messages=[{\"role\": \"user\", \"content\": \"Hi\"}], max_tokens=200)\nprint(out[\"choices\"][0][\"message\"][\"content\"])\n```\n\nGGUF files come from Hugging Face; the `q4_k_m` in the name is the quantisation: 4-bit, medium, the usual choice. `n_ctx` is the context window you allocate, and memory grows with it. Use this when you want a single self-contained script with no background process, or on a machine where you cannot install a service.\n\n## transformers: the research path\n\nHugging Face's `transformers` loads the original weights, unquantised, through PyTorch:\n\n```python\nfrom transformers import pipeline\n\npipe = pipeline(\"text-generation\", model=\"Qwen/Qwen2.5-0.5B-Instruct\")\nout = pipe([{\"role\": \"user\", \"content\": \"Hi\"}], max_new_tokens=100)\nprint(out[0][\"generated_text\"][-1][\"content\"])\n```\n\nThis is slowest on a CPU and uses the most memory, and it is what you need when you want to inspect the model (its tokeniser, its attention, its logits) or fine-tune it, which is its own course. For plain generation, prefer Ollama or llama.cpp.\n\n## The provider class, again\n\nModule 7's `ChatProvider` is why none of this disturbs the rest of a program:\n\n```python\nclass OllamaProvider(ChatProvider):\n    def __init__(self, model=\"qwen2.5:1.5b\"):\n        self.client = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\n        self.model = model\n    def complete(self, messages, max_tokens=500):\n        r = self.client.chat.completions.create(model=self.model, messages=messages, max_tokens=max_tokens)\n        return r.choices[0].message.content\n```\n\nDevelop against this, run the tests against this, and switch to the hosted provider for the final run or when quality matters. A budget guard from module 6 that falls back to the local model when the cap is near is composition doing exactly what it was for.\n\n## What to expect from small models\n\nThey follow simple instructions well and complicated ones badly. They produce valid JSON less reliably, so module 6's validation and retry matter more. They know less, and they invent more confidently. They handle English best and Hindi variably; the Qwen and Gemma families are stronger on Indian languages than most at the same size. None of this is a reason not to use them; it is a reason to keep the validation layer and to test the prompt on the model you will actually run.\n\n## Try this now\n\nDo the memory arithmetic for your machine, pull the largest model that fits with room to spare, and time a 200-token reply. Then write `OllamaProvider`, run your chat loop against it, and swap in the fake provider from module 7 to confirm nothing else changed."},"check":{"question":"A learner with an 8 GB laptop tries to load a 7-billion-parameter model in float16 through `transformers` and the process is killed with no traceback. What is the arithmetic that predicted this?","options":[{"text":"7 billion parameters at 2 bytes each is 14 GB of weights before any working memory, which cannot fit in 8 GB.","correct":true},{"text":"`transformers` loads every model in float64 by default, so 7 billion parameters is 56 GB; passing `torch_dtype=torch.float16` would have fitted it in 8 GB.","correct":false,"tempting":"The default is float32 for most models, not float64, and even float16 needs 14 GB for the weights alone, so the dtype change does not rescue it."},{"text":"A CPU can address at most 4 GB per process, so any model above 2 billion parameters fails regardless of the machine's memory.","correct":false,"tempting":"A 64-bit process addresses far more than 4 GB; the limit is the physical memory, not the address space."},{"text":"The tokeniser for a 7B model is itself several gigabytes and loaded first, leaving no room for the weights.","correct":false,"tempting":"A tokeniser is a few megabytes; the weights are what consume memory."}]},"prev":{"slug":"search-over-your-own-notes","title":"Search over your own notes: chunks, embeddings, cosine, and sixty lines","url":"https://addaly.com/learn/python-for-ai/search-over-your-own-notes"},"next":{"slug":"a-web-interface-with-gradio","title":"A web interface in twenty lines: Gradio, and what launch() actually starts","url":"https://addaly.com/learn/python-for-ai/a-web-interface-with-gradio"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A free model on your own machine, called from Python', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/a-local-model-from-python (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A free model on your own machine, called from Python","courseTitle":"Python, from Zero, for AI","module":{"index":9,"slug":"building-the-thing","title":"Building the thing: a small AI program, end to end"},"answer":null,"keyIdea":"Ollama, llama.cpp and transformers each run an open model on a CPU with no account; memory is parameters times bytes per weight, tokens per second is what a CPU gives you, and the same provider class from module 7 makes the local model a drop-in for the paid one.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-a-local-model","runs":[{"text":"Why a local model"}]},{"t":"p","runs":[{"text":"Three reasons, and a learner on a phone or an old laptop will feel all of them. It costs nothing per call, so you can run your script two hundred times while learning. It needs no account, no card and no key, so nothing can leak. And it works offline. What you give up is quality (a 1-billion-parameter model on a CPU is not a hosted frontier model) and speed. For learning, testing and many real tasks, that trade is good."}]},{"t":"h2","id":"the-arithmetic-first","runs":[{"text":"The arithmetic first"}]},{"t":"p","runs":[{"text":"A model’s memory is its parameters multiplied by the bytes per weight:"}]},{"t":"ul","items":[[{"text":"0.5B parameters × 2 bytes (float16) ≈ 1 GB"}],[{"text":"3B × 2 ≈ 6 GB; 3B × 0.5 (4-bit quantised) ≈ 1.5 to 2 GB"}],[{"text":"7B × 2 ≈ 14 GB; 7B × 0.5 ≈ 4 GB"}],[{"text":"70B × 0.5 ≈ 35 to 40 GB"}]]},{"t":"p","runs":[{"text":"Add a gigabyte or so of working memory. A laptop with 8 GB runs a 3B model quantised comfortably, a 7B model quantised at a squeeze, and a 7B model in float16 not at all: the process is killed by the operating system with no Python traceback, because the failure happens below Python. Do the multiplication before the download."}]},{"t":"p","runs":[{"text":"Speed on a CPU is a few to a few dozen tokens per second depending on size and machine. A 0.5B model answers in a couple of seconds; a 7B model quantised writes at reading speed on a recent laptop and slower on an old one. A phone under Termux runs 0.5B to 1.5B models."}]},{"t":"h2","id":"ollama-the-easy-path","runs":[{"text":"Ollama: the easy path"}]},{"t":"p","runs":[{"text":"Ollama is a free program that downloads models, quantises them and serves them over HTTP on your machine."}]},{"t":"code","lang":"bash","label":"Bash","text":"ollama pull qwen2.5:1.5b        # ~1 GB download\nollama run qwen2.5:1.5b         # chat in the terminal to check it works"},{"t":"p","runs":[{"text":"From Python, you already know how to talk to it: module 6 did, through the OpenAI-compatible endpoint:"}]},{"t":"code","lang":"python","label":"Python","text":"from openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\nr = client.chat.completions.create(model=\"qwen2.5:1.5b\",\n        messages=[{\"role\": \"user\", \"content\": \"Explain a dict in one sentence.\"}])\nprint(r.choices[0].message.content)"},{"t":"p","runs":[{"text":"Or with plain "},{"text":"requests","code":true},{"text":" against Ollama’s own API, which also streams:"}]},{"t":"code","lang":"python","label":"Python","text":"r = requests.post(\"http://localhost:11434/api/chat\", json={\n    \"model\": \"qwen2.5:1.5b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}],\n    \"stream\": False,\n}, timeout=120)\nprint(r.json()[\"message\"][\"content\"])"},{"t":"p","runs":[{"text":"ollama list","code":true},{"text":" shows what you have; models are stored under "},{"text":"~/.ollama","code":true},{"text":". Embedding models ("},{"text":"nomic-embed-text","code":true},{"text":", "},{"text":"bge-m3","code":true},{"text":" for multilingual) run the same way, via "},{"text":"/api/embeddings","code":true},{"text":", so the notes search from the previous lesson can be entirely local."}]},{"t":"h2","id":"llama-cpp-the-engine-underneath","runs":[{"text":"llama.cpp: the engine underneath"}]},{"t":"p","runs":[{"text":"Ollama wraps "},{"text":"llama.cpp","code":true},{"text":", a C++ inference engine that runs quantised models in the GGUF format on almost anything. Its Python binding gives you the model as an object with no server:"}]},{"t":"code","lang":"python","label":"Python","text":"from llama_cpp import Llama\n\nllm = Llama(model_path=\"qwen2.5-1.5b-instruct-q4_k_m.gguf\", n_ctx=4096, verbose=False)\nout = llm.create_chat_completion(messages=[{\"role\": \"user\", \"content\": \"Hi\"}], max_tokens=200)\nprint(out[\"choices\"][0][\"message\"][\"content\"])"},{"t":"p","runs":[{"text":"GGUF files come from Hugging Face; the "},{"text":"q4_k_m","code":true},{"text":" in the name is the quantisation: 4-bit, medium, the usual choice. "},{"text":"n_ctx","code":true},{"text":" is the context window you allocate, and memory grows with it. Use this when you want a single self-contained script with no background process, or on a machine where you cannot install a service."}]},{"t":"h2","id":"transformers-the-research-path","runs":[{"text":"transformers: the research path"}]},{"t":"p","runs":[{"text":"Hugging Face’s "},{"text":"transformers","code":true},{"text":" loads the original weights, unquantised, through PyTorch:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import pipeline\n\npipe = pipeline(\"text-generation\", model=\"Qwen/Qwen2.5-0.5B-Instruct\")\nout = pipe([{\"role\": \"user\", \"content\": \"Hi\"}], max_new_tokens=100)\nprint(out[0][\"generated_text\"][-1][\"content\"])"},{"t":"p","runs":[{"text":"This is slowest on a CPU and uses the most memory, and it is what you need when you want to inspect the model (its tokeniser, its attention, its logits) or fine-tune it, which is its own course. For plain generation, prefer Ollama or llama.cpp."}]},{"t":"h2","id":"the-provider-class-again","runs":[{"text":"The provider class, again"}]},{"t":"p","runs":[{"text":"Module 7’s "},{"text":"ChatProvider","code":true},{"text":" is why none of this disturbs the rest of a program:"}]},{"t":"code","lang":"python","label":"Python","text":"class OllamaProvider(ChatProvider):\n    def __init__(self, model=\"qwen2.5:1.5b\"):\n        self.client = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\n        self.model = model\n    def complete(self, messages, max_tokens=500):\n        r = self.client.chat.completions.create(model=self.model, messages=messages, max_tokens=max_tokens)\n        return r.choices[0].message.content"},{"t":"p","runs":[{"text":"Develop against this, run the tests against this, and switch to the hosted provider for the final run or when quality matters. A budget guard from module 6 that falls back to the local model when the cap is near is composition doing exactly what it was for."}]},{"t":"h2","id":"what-to-expect-from-small-models","runs":[{"text":"What to expect from small models"}]},{"t":"p","runs":[{"text":"They follow simple instructions well and complicated ones badly. They produce valid JSON less reliably, so module 6’s validation and retry matter more. They know less, and they invent more confidently. They handle English best and Hindi variably; the Qwen and Gemma families are stronger on Indian languages than most at the same size. None of this is a reason not to use them; it is a reason to keep the validation layer and to test the prompt on the model you will actually run."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Do the memory arithmetic for your machine, pull the largest model that fits with room to spare, and time a 200-token reply. Then write "},{"text":"OllamaProvider","code":true},{"text":", run your chat loop against it, and swap in the fake provider from module 7 to confirm nothing else changed."}]}],"check":{"question":[{"text":"A learner with an 8 GB laptop tries to load a 7-billion-parameter model in float16 through "},{"text":"transformers","code":true},{"text":" and the process is killed with no traceback. What is the arithmetic that predicted this?"}],"options":[{"text":[{"text":"7 billion parameters at 2 bytes each is 14 GB of weights before any working memory, which cannot fit in 8 GB."}],"correct":true,"tempting":null},{"text":[{"text":"transformers","code":true},{"text":" loads every model in float64 by default, so 7 billion parameters is 56 GB; passing "},{"text":"torch_dtype=torch.float16","code":true},{"text":" would have fitted it in 8 GB."}],"correct":false,"tempting":[{"text":"The default is float32 for most models, not float64, and even float16 needs 14 GB for the weights alone, so the dtype change does not rescue it."}]},{"text":[{"text":"A CPU can address at most 4 GB per process, so any model above 2 billion parameters fails regardless of the machine’s memory."}],"correct":false,"tempting":[{"text":"A 64-bit process addresses far more than 4 GB; the limit is the physical memory, not the address space."}]},{"text":[{"text":"The tokeniser for a 7B model is itself several gigabytes and loaded first, leaving no room for the weights."}],"correct":false,"tempting":[{"text":"A tokeniser is a few megabytes; the weights are what consume memory."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"search-over-your-own-notes","title":"Search over your own notes: chunks, embeddings, cosine, and sixty lines","minutes":10},"next":{"slug":"a-web-interface-with-gradio","title":"A web interface in twenty lines: Gradio, and what launch() actually starts","minutes":8}}}