What actually runs on a phone

The constraint is not the chip

A recent phone has a capable processor and 8 to 16 GB of RAM. What it does not have is permission to use it. Mobile operating systems cap how much memory a single application may hold (on iOS the ordinary limit is a fraction of physical RAM, raised only with a specific entitlement) and a process that exceeds it is terminated without ceremony.

So the practical budget for a model on a phone is roughly 1 to 3 GB, which means a 1B to 3B model at 4-bit. Everything else about on-device inference follows from that number.

What that size can do, on a phone

Realistically good: rewriting a message, summarising a notification or a short email, classifying and tagging, extracting a date and an amount from a text, offline translation of short passages with a language-tuned model, autocomplete, and simple structured extraction.

Realistically bad: anything needing world knowledge, long documents, code, or multi-step reasoning. A 3B model on a phone is a text-transformation engine, not an assistant that knows things.

That is not a small category. Most of the useful private computing on a phone is text transformation, and doing it on the device rather than on a server is exactly the case where privacy is the point.

The routes

llama.cpp under Termux on Android is the direct path: install Termux, install the package, run llama-cli with a GGUF from your download folder. It works, it is fiddly, and it is the fastest way to see real numbers on your own hardware.

MLC LLM compiles models for mobile GPUs and ships a chat application for both platforms.

Google AI Edge and the MediaPipe LLM inference API give an Android-native path with models packaged for it.

Apple’s on-device foundation model, exposed to applications through a framework, is the sanctioned route on iOS and avoids the memory limit entirely because the model belongs to the system rather than your app. You do not choose the model; that is the trade.

ExecuTorch is the PyTorch path to a compiled on-device artefact, and is the most work.

Prebuilt applications (PocketPal, ChatterUI and others) let you load a GGUF from the phone’s storage and chat with it. This is the right first step: measure before you build.

Thermal throttling, which nobody mentions

A phone has no fan. Sustained generation heats the chip, the operating system reduces clock speeds, and speed falls. The characteristic pattern is 15 to 20 tokens per second for the first thirty seconds, settling to half that after a minute or two of continuous work, on a device that is also getting uncomfortably warm.

Design around it. Short bursts are fine. A background job that summarises fifty documents is not a phone task, or if it is, it belongs on the charger overnight with expectations set accordingly. Battery drain is proportionate: continuous generation is comparable to gaming.

The NPU question

Phones have neural accelerators, and the marketing numbers for them are large. Most general-purpose local inference does not use them. Reaching an NPU means going through a vendor-specific stack (Qualcomm’s on Android, Apple’s Neural Engine through Core ML) with model conversion, operator support limits and fixed shapes. Portable engines such as llama.cpp use the CPU and sometimes the GPU instead.

This is improving, but if you read a claim of very high on-device throughput, check whether it required a vendor toolchain and a specific converted model. It usually did.

Deciding whether to bother

Ask what the device path buys you.

• Privacy is the requirement: messages, health notes, photographs. Strong reason; the data never leaves.
• It must work with no network: travel, field work, poor coverage. Strong reason.
• Latency for a tiny task: a 200-millisecond rewrite with no round trip feels different from a 900-millisecond one. Real, and often underrated.
• Cost: weak. A 3B model called a few times per user per day is very cheap to serve centrally, and you avoid shipping a 2 GB download in your application.

The honest summary: on-device inference is a genuine capability with a narrow, well-defined scope. Within that scope it is excellent, and it is the only place where you can promise a user that their words never left the phone and mean it.