ONNX, Optimum, and models that run in a browser tab

A different kind of portability

PyTorch is a research framework that happens to run in production. ONNX is an interchange format: a frozen computation graph plus weights, executed by runtimes written for speed on specific hardware. Exporting a model to ONNX gives you something that runs without PyTorch installed: on a CPU-only server, inside a mobile app, or in a web page.

For small models on CPU, ONNX Runtime is commonly two to four times faster than PyTorch eager execution, because the graph is known ahead of time and can be fused and optimised. For large generative models the gap narrows and the tooling gets harder, so the sweet spot is embeddings, classifiers, token taggers and small speech models, which is most of what an ordinary application actually needs.

Optimum is the bridge

pip install "optimum[onnxruntime]"
optimum-cli export onnx --model distilbert-base-uncased-finetuned-sst-2-english out/

Then the API is deliberately the same shape as transformers:

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

model = ORTModelForSequenceClassification.from_pretrained("out/")
tok = AutoTokenizer.from_pretrained("out/")
pipe = pipeline("text-classification", model=model, tokenizer=tok)

Swapping AutoModelForSequenceClassification for ORTModelForSequenceClassification is the entire migration. Many Hub repositories already carry an onnx/ subfolder, in which case from_pretrained(name, subfolder="onnx") skips the export step.

Optimum also wraps quantization for CPU: dynamic int8 quantization typically halves model size and speeds up CPU inference by 2 to 3× on encoder models, with a small accuracy cost you should measure on your own set.

Transformers.js: the model in the page

<script type="module">
import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers";

const pipe = await pipeline("sentiment-analysis");
console.log(await pipe("The train was delayed but the staff were kind."));
</script>

This downloads an ONNX model into the browser and runs it in WebAssembly, or on the GPU through WebGPU where the browser supports it. No server, no API key, no request leaving the device.

The consequences are worth stating plainly, because they change what is possible for people without budget:

• Hosting is a static file. A GitHub Pages site or a static Hugging Face Space can host a working AI demo for nothing, forever, with no server to pay for or maintain.
• The data never leaves. For a tool handling medical notes, diaries or anything a user would not paste into a website, this is a genuine privacy property rather than a promise.
• It scales for free, because the user’s device does the work.

The limits are equally plain. The model downloads on first visit (30 MB for a small embedding model is fine, 2 GB for a chat model is not) though the browser caches it afterwards. WebAssembly is slower than native, roughly by a factor of two to five. WebGPU is much faster and is not available everywhere. And a phone on a weak connection will not finish a large download.

The practical ceiling today is models in the tens to low hundreds of megabytes: embeddings, classifiers, speech recognition for short clips, background removal, translation of short text. Within that ceiling it works well.

Other runtimes worth knowing exist

• OpenVINO, for Intel CPUs and integrated GPUs, also wrapped by Optimum. On an ordinary office laptop it is often the fastest CPU option.
• Core ML, for Apple devices, exported through coremltools or Optimum’s exporters.
• ExecuTorch and TensorFlow Lite, for on-device mobile deployment.

All four solve the same problem (run this graph fast on this hardware) and all four trade flexibility for it. Once exported, you cannot easily change the model’s shape, and dynamic control flow may not survive the conversion at all.

The caveat about export

Exports fail. A model with unusual control flow, custom operators or a trust_remote_code implementation may export with warnings and then produce subtly different numbers. Always verify:

import numpy as np
np.testing.assert_allclose(torch_logits, onnx_logits, rtol=1e-3, atol=1e-4)

If that fails, the export is not equivalent, and the difference will show up as an accuracy drop nobody can explain later.

Do this now

Put the eight-line transformers.js example in an HTML file and open it locally. You are running a neural network with no install, no key and no server; and you can host that file for free. For a lot of projects, that is the whole deployment story.