{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"models-on-phones","title":"What actually runs on a phone","url":"https://addaly.com/learn/open-models/models-on-phones","position":40,"of":71,"minutes":8,"answer":"Mobile operating systems cap per-application memory, which limits on-device models to roughly 1 to 3 GB, and sustained generation is halved by thermal throttling, so a phone model is a private text-transformation engine.","keyIdea":"Mobile operating systems cap per-application memory, which limits on-device models to roughly 1 to 3 GB, and sustained generation is halved by thermal throttling, so a phone model is a private text-transformation engine, not a knowledgeable assistant."},"body":{"format":"markdown","text":"## The constraint is not the chip\n\nA recent phone has a capable processor and 8 to 16 GB of RAM. What it does not have is permission to use it. Mobile operating systems cap how much memory a single application may hold (on iOS the ordinary limit is a fraction of physical RAM, raised only with a specific entitlement) and a process that exceeds it is terminated without ceremony.\n\nSo the practical budget for a model on a phone is roughly **1 to 3 GB**, which means a 1B to 3B model at 4-bit. Everything else about on-device inference follows from that number.\n\n## What that size can do, on a phone\n\nRealistically good: rewriting a message, summarising a notification or a short email, classifying and tagging, extracting a date and an amount from a text, offline translation of short passages with a language-tuned model, autocomplete, and simple structured extraction.\n\nRealistically bad: anything needing world knowledge, long documents, code, or multi-step reasoning. A 3B model on a phone is a text-transformation engine, not an assistant that knows things.\n\nThat is not a small category. Most of the useful private computing on a phone is text transformation, and doing it on the device rather than on a server is exactly the case where privacy is the point.\n\n## The routes\n\n**llama.cpp under Termux** on Android is the direct path: install Termux, install the package, run `llama-cli` with a GGUF from your download folder. It works, it is fiddly, and it is the fastest way to see real numbers on your own hardware.\n\n**MLC LLM** compiles models for mobile GPUs and ships a chat application for both platforms.\n\n**Google AI Edge and the MediaPipe LLM inference API** give an Android-native path with models packaged for it.\n\n**Apple's on-device foundation model**, exposed to applications through a framework, is the sanctioned route on iOS and avoids the memory limit entirely because the model belongs to the system rather than your app. You do not choose the model; that is the trade.\n\n**ExecuTorch** is the PyTorch path to a compiled on-device artefact, and is the most work.\n\n**Prebuilt applications** (PocketPal, ChatterUI and others) let you load a GGUF from the phone's storage and chat with it. This is the right first step: measure before you build.\n\n## Thermal throttling, which nobody mentions\n\nA phone has no fan. Sustained generation heats the chip, the operating system reduces clock speeds, and speed falls. The characteristic pattern is 15 to 20 tokens per second for the first thirty seconds, settling to half that after a minute or two of continuous work, on a device that is also getting uncomfortably warm.\n\nDesign around it. Short bursts are fine. A background job that summarises fifty documents is not a phone task, or if it is, it belongs on the charger overnight with expectations set accordingly. Battery drain is proportionate: continuous generation is comparable to gaming.\n\n## The NPU question\n\nPhones have neural accelerators, and the marketing numbers for them are large. Most general-purpose local inference does not use them. Reaching an NPU means going through a vendor-specific stack (Qualcomm's on Android, Apple's Neural Engine through Core ML) with model conversion, operator support limits and fixed shapes. Portable engines such as llama.cpp use the CPU and sometimes the GPU instead.\n\nThis is improving, but if you read a claim of very high on-device throughput, check whether it required a vendor toolchain and a specific converted model. It usually did.\n\n## Deciding whether to bother\n\nAsk what the device path buys you.\n\n- **Privacy is the requirement**: messages, health notes, photographs. Strong reason; the data never leaves.\n- **It must work with no network**: travel, field work, poor coverage. Strong reason.\n- **Latency for a tiny task**: a 200-millisecond rewrite with no round trip feels different from a 900-millisecond one. Real, and often underrated.\n- **Cost**: weak. A 3B model called a few times per user per day is very cheap to serve centrally, and you avoid shipping a 2 GB download in your application.\n\nThe honest summary: on-device inference is a genuine capability with a narrow, well-defined scope. Within that scope it is excellent, and it is the only place where you can promise a user that their words never left the phone and mean it."},"check":{"question":"A benchmark shows a phone generating 18 tokens per second with a 3B model, but users report it feeling much slower in a long session. What is the most likely explanation?","options":[{"text":"Users are running other applications, so the model is being paged out of memory between requests.","correct":false,"tempting":"Memory pressure causes termination or reload stalls rather than a steady halving of throughput."},{"text":"The benchmark measured prompt processing, which is much faster than generation.","correct":false,"tempting":"A real distinction, but the figure quoted is already generation speed, and the reported slowdown develops over time rather than being constant."},{"text":"The phone has no active cooling, so sustained generation heats the chip, the system reduces clock speed, and throughput falls to roughly half after a minute.","correct":true},{"text":"Battery saver mode reduces the model's precision, degrading both speed and quality.","correct":false,"tempting":"Power modes reduce clocks, not model precision; the weights on disk do not change."}]},"prev":{"slug":"rent-or-own","title":"Rent, or own: the arithmetic","url":"https://addaly.com/learn/open-models/rent-or-own"},"next":{"slug":"sampling-parameters","title":"The sampler, and the defaults that ship broken","url":"https://addaly.com/learn/open-models/sampling-parameters"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What actually runs on a phone', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/models-on-phones (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What actually runs on a phone","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Mobile operating systems cap per-application memory, which limits on-device models to roughly 1 to 3 GB, and sustained generation is halved by thermal throttling, so a phone model is a private text-transformation engine, not a knowledgeable assistant.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-constraint-is-not-the-chip","runs":[{"text":"The constraint is not the chip"}]},{"t":"p","runs":[{"text":"A recent phone has a capable processor and 8 to 16 GB of RAM. What it does not have is permission to use it. Mobile operating systems cap how much memory a single application may hold (on iOS the ordinary limit is a fraction of physical RAM, raised only with a specific entitlement) and a process that exceeds it is terminated without ceremony."}]},{"t":"p","runs":[{"text":"So the practical budget for a model on a phone is roughly "},{"text":"1 to 3 GB","b":true},{"text":", which means a 1B to 3B model at 4-bit. Everything else about on-device inference follows from that number."}]},{"t":"h2","id":"what-that-size-can-do-on-a-phone","runs":[{"text":"What that size can do, on a phone"}]},{"t":"p","runs":[{"text":"Realistically good: rewriting a message, summarising a notification or a short email, classifying and tagging, extracting a date and an amount from a text, offline translation of short passages with a language-tuned model, autocomplete, and simple structured extraction."}]},{"t":"p","runs":[{"text":"Realistically bad: anything needing world knowledge, long documents, code, or multi-step reasoning. A 3B model on a phone is a text-transformation engine, not an assistant that knows things."}]},{"t":"p","runs":[{"text":"That is not a small category. Most of the useful private computing on a phone is text transformation, and doing it on the device rather than on a server is exactly the case where privacy is the point."}]},{"t":"h2","id":"the-routes","runs":[{"text":"The routes"}]},{"t":"p","runs":[{"text":"llama.cpp under Termux","b":true},{"text":" on Android is the direct path: install Termux, install the package, run "},{"text":"llama-cli","code":true},{"text":" with a GGUF from your download folder. It works, it is fiddly, and it is the fastest way to see real numbers on your own hardware."}]},{"t":"p","runs":[{"text":"MLC LLM","b":true},{"text":" compiles models for mobile GPUs and ships a chat application for both platforms."}]},{"t":"p","runs":[{"text":"Google AI Edge and the MediaPipe LLM inference API","b":true},{"text":" give an Android-native path with models packaged for it."}]},{"t":"p","runs":[{"text":"Apple’s on-device foundation model","b":true},{"text":", exposed to applications through a framework, is the sanctioned route on iOS and avoids the memory limit entirely because the model belongs to the system rather than your app. You do not choose the model; that is the trade."}]},{"t":"p","runs":[{"text":"ExecuTorch","b":true},{"text":" is the PyTorch path to a compiled on-device artefact, and is the most work."}]},{"t":"p","runs":[{"text":"Prebuilt applications","b":true},{"text":" (PocketPal, ChatterUI and others) let you load a GGUF from the phone’s storage and chat with it. This is the right first step: measure before you build."}]},{"t":"h2","id":"thermal-throttling-which-nobody-mentions","runs":[{"text":"Thermal throttling, which nobody mentions"}]},{"t":"p","runs":[{"text":"A phone has no fan. Sustained generation heats the chip, the operating system reduces clock speeds, and speed falls. The characteristic pattern is 15 to 20 tokens per second for the first thirty seconds, settling to half that after a minute or two of continuous work, on a device that is also getting uncomfortably warm."}]},{"t":"p","runs":[{"text":"Design around it. Short bursts are fine. A background job that summarises fifty documents is not a phone task, or if it is, it belongs on the charger overnight with expectations set accordingly. Battery drain is proportionate: continuous generation is comparable to gaming."}]},{"t":"h2","id":"the-npu-question","runs":[{"text":"The NPU question"}]},{"t":"p","runs":[{"text":"Phones have neural accelerators, and the marketing numbers for them are large. Most general-purpose local inference does not use them. Reaching an NPU means going through a vendor-specific stack (Qualcomm’s on Android, Apple’s Neural Engine through Core ML) with model conversion, operator support limits and fixed shapes. Portable engines such as llama.cpp use the CPU and sometimes the GPU instead."}]},{"t":"p","runs":[{"text":"This is improving, but if you read a claim of very high on-device throughput, check whether it required a vendor toolchain and a specific converted model. It usually did."}]},{"t":"h2","id":"deciding-whether-to-bother","runs":[{"text":"Deciding whether to bother"}]},{"t":"p","runs":[{"text":"Ask what the device path buys you."}]},{"t":"ul","items":[[{"text":"Privacy is the requirement","b":true},{"text":": messages, health notes, photographs. Strong reason; the data never leaves."}],[{"text":"It must work with no network","b":true},{"text":": travel, field work, poor coverage. Strong reason."}],[{"text":"Latency for a tiny task","b":true},{"text":": a 200-millisecond rewrite with no round trip feels different from a 900-millisecond one. Real, and often underrated."}],[{"text":"Cost","b":true},{"text":": weak. A 3B model called a few times per user per day is very cheap to serve centrally, and you avoid shipping a 2 GB download in your application."}]]},{"t":"p","runs":[{"text":"The honest summary: on-device inference is a genuine capability with a narrow, well-defined scope. Within that scope it is excellent, and it is the only place where you can promise a user that their words never left the phone and mean it."}]}],"check":{"question":[{"text":"A benchmark shows a phone generating 18 tokens per second with a 3B model, but users report it feeling much slower in a long session. What is the most likely explanation?"}],"options":[{"text":[{"text":"Users are running other applications, so the model is being paged out of memory between requests."}],"correct":false,"tempting":[{"text":"Memory pressure causes termination or reload stalls rather than a steady halving of throughput."}]},{"text":[{"text":"The benchmark measured prompt processing, which is much faster than generation."}],"correct":false,"tempting":[{"text":"A real distinction, but the figure quoted is already generation speed, and the reported slowdown develops over time rather than being constant."}]},{"text":[{"text":"The phone has no active cooling, so sustained generation heats the chip, the system reduces clock speed, and throughput falls to roughly half after a minute."}],"correct":true,"tempting":null},{"text":[{"text":"Battery saver mode reduces the model’s precision, degrading both speed and quality."}],"correct":false,"tempting":[{"text":"Power modes reduce clocks, not model precision; the weights on disk do not change."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"rent-or-own","title":"Rent, or own: the arithmetic","minutes":9},"next":{"slug":"sampling-parameters","title":"The sampler, and the defaults that ship broken","minutes":9}}}