{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"onnx-optimum-and-the-browser","title":"ONNX, Optimum, and models that run in a browser tab","url":"https://addaly.com/learn/hugging-face/onnx-optimum-and-the-browser","position":24,"of":75,"minutes":9,"answer":"ONNX freezes a model into a portable graph that runs without PyTorch, and through transformers.js that graph runs inside a browser tab, which makes a static file a free, private.","keyIdea":"ONNX freezes a model into a portable graph that runs without PyTorch, and through transformers.js that graph runs inside a browser tab, which makes a static file a free, private, infinitely scalable deployment for any model small enough to download."},"body":{"format":"markdown","text":"## A different kind of portability\n\nPyTorch is a research framework that happens to run in production. ONNX is an interchange format: a frozen computation graph plus weights, executed by runtimes written for speed on specific hardware. Exporting a model to ONNX gives you something that runs without PyTorch installed: on a CPU-only server, inside a mobile app, or in a web page.\n\nFor small models on CPU, ONNX Runtime is commonly two to four times faster than PyTorch eager execution, because the graph is known ahead of time and can be fused and optimised. For large generative models the gap narrows and the tooling gets harder, so the sweet spot is embeddings, classifiers, token taggers and small speech models, which is most of what an ordinary application actually needs.\n\n## Optimum is the bridge\n\n```bash\npip install \"optimum[onnxruntime]\"\noptimum-cli export onnx --model distilbert-base-uncased-finetuned-sst-2-english out/\n```\n\nThen the API is deliberately the same shape as transformers:\n\n```python\nfrom optimum.onnxruntime import ORTModelForSequenceClassification\nfrom transformers import AutoTokenizer, pipeline\n\nmodel = ORTModelForSequenceClassification.from_pretrained(\"out/\")\ntok = AutoTokenizer.from_pretrained(\"out/\")\npipe = pipeline(\"text-classification\", model=model, tokenizer=tok)\n```\n\nSwapping `AutoModelForSequenceClassification` for `ORTModelForSequenceClassification` is the entire migration. Many Hub repositories already carry an `onnx/` subfolder, in which case `from_pretrained(name, subfolder=\"onnx\")` skips the export step.\n\nOptimum also wraps quantization for CPU: dynamic int8 quantization typically halves model size and speeds up CPU inference by 2 to 3× on encoder models, with a small accuracy cost you should measure on your own set.\n\n## Transformers.js: the model in the page\n\n```html\n<script type=\"module\">\nimport { pipeline } from \"https://cdn.jsdelivr.net/npm/@huggingface/transformers\";\n\nconst pipe = await pipeline(\"sentiment-analysis\");\nconsole.log(await pipe(\"The train was delayed but the staff were kind.\"));\n</script>\n```\n\nThis downloads an ONNX model into the browser and runs it in WebAssembly, or on the GPU through WebGPU where the browser supports it. No server, no API key, no request leaving the device.\n\nThe consequences are worth stating plainly, because they change what is possible for people without budget:\n\n- **Hosting is a static file.** A GitHub Pages site or a static Hugging Face Space can host a working AI demo for nothing, forever, with no server to pay for or maintain.\n- **The data never leaves.** For a tool handling medical notes, diaries or anything a user would not paste into a website, this is a genuine privacy property rather than a promise.\n- **It scales for free**, because the user's device does the work.\n\nThe limits are equally plain. The model downloads on first visit (30 MB for a small embedding model is fine, 2 GB for a chat model is not) though the browser caches it afterwards. WebAssembly is slower than native, roughly by a factor of two to five. WebGPU is much faster and is not available everywhere. And a phone on a weak connection will not finish a large download.\n\nThe practical ceiling today is models in the tens to low hundreds of megabytes: embeddings, classifiers, speech recognition for short clips, background removal, translation of short text. Within that ceiling it works well.\n\n## Other runtimes worth knowing exist\n\n- **OpenVINO**, for Intel CPUs and integrated GPUs, also wrapped by Optimum. On an ordinary office laptop it is often the fastest CPU option.\n- **Core ML**, for Apple devices, exported through `coremltools` or Optimum's exporters.\n- **ExecuTorch** and **TensorFlow Lite**, for on-device mobile deployment.\n\nAll four solve the same problem (run this graph fast on this hardware) and all four trade flexibility for it. Once exported, you cannot easily change the model's shape, and dynamic control flow may not survive the conversion at all.\n\n## The caveat about export\n\nExports fail. A model with unusual control flow, custom operators or a `trust_remote_code` implementation may export with warnings and then produce subtly different numbers. Always verify:\n\n```python\nimport numpy as np\nnp.testing.assert_allclose(torch_logits, onnx_logits, rtol=1e-3, atol=1e-4)\n```\n\nIf that fails, the export is not equivalent, and the difference will show up as an accuracy drop nobody can explain later.\n\n## Do this now\n\nPut the eight-line transformers.js example in an HTML file and open it locally. You are running a neural network with no install, no key and no server; and you can host that file for free. For a lot of projects, that is the whole deployment story."},"check":{"question":"A team wants a free, private tool that classifies short notes typed by users, with no server costs. Which constraint decides whether transformers.js is viable?","options":[{"text":"Whether the browser supports WebGPU, since WebAssembly cannot execute transformer attention operations.","correct":false,"tempting":"WebGPU improves speed but WebAssembly runs the same models, more slowly."},{"text":"Whether the model is small enough that users will tolerate downloading it once on first visit.","correct":true},{"text":"Whether the model has an ONNX export available, since transformers.js can only load repositories published by Hugging Face itself.","correct":false,"tempting":"Export availability matters and can be produced with Optimum; the ownership restriction is invented."},{"text":"Whether the notes can be batched, because per-request browser inference has fixed overhead that only batching amortises.","correct":false,"tempting":"Batching helps throughput but is not the gate on feasibility for short interactive inputs."}]},"prev":{"slug":"gguf-and-llama-cpp","title":"GGUF, and the laptop that has no GPU","url":"https://addaly.com/learn/hugging-face/gguf-and-llama-cpp"},"next":{"slug":"free-gpus-and-their-limits","title":"Where the free GPUs actually are","url":"https://addaly.com/learn/hugging-face/free-gpus-and-their-limits"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'ONNX, Optimum, and models that run in a browser tab', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/onnx-optimum-and-the-browser (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"ONNX, Optimum, and models that run in a browser tab","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"ONNX freezes a model into a portable graph that runs without PyTorch, and through transformers.js that graph runs inside a browser tab, which makes a static file a free, private, infinitely scalable deployment for any model small enough to download.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-different-kind-of-portability","runs":[{"text":"A different kind of portability"}]},{"t":"p","runs":[{"text":"PyTorch is a research framework that happens to run in production. ONNX is an interchange format: a frozen computation graph plus weights, executed by runtimes written for speed on specific hardware. Exporting a model to ONNX gives you something that runs without PyTorch installed: on a CPU-only server, inside a mobile app, or in a web page."}]},{"t":"p","runs":[{"text":"For small models on CPU, ONNX Runtime is commonly two to four times faster than PyTorch eager execution, because the graph is known ahead of time and can be fused and optimised. For large generative models the gap narrows and the tooling gets harder, so the sweet spot is embeddings, classifiers, token taggers and small speech models, which is most of what an ordinary application actually needs."}]},{"t":"h2","id":"optimum-is-the-bridge","runs":[{"text":"Optimum is the bridge"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install \"optimum[onnxruntime]\"\noptimum-cli export onnx --model distilbert-base-uncased-finetuned-sst-2-english out/"},{"t":"p","runs":[{"text":"Then the API is deliberately the same shape as transformers:"}]},{"t":"code","lang":"python","label":"Python","text":"from optimum.onnxruntime import ORTModelForSequenceClassification\nfrom transformers import AutoTokenizer, pipeline\n\nmodel = ORTModelForSequenceClassification.from_pretrained(\"out/\")\ntok = AutoTokenizer.from_pretrained(\"out/\")\npipe = pipeline(\"text-classification\", model=model, tokenizer=tok)"},{"t":"p","runs":[{"text":"Swapping "},{"text":"AutoModelForSequenceClassification","code":true},{"text":" for "},{"text":"ORTModelForSequenceClassification","code":true},{"text":" is the entire migration. Many Hub repositories already carry an "},{"text":"onnx/","code":true},{"text":" subfolder, in which case "},{"text":"from_pretrained(name, subfolder=\"onnx\")","code":true},{"text":" skips the export step."}]},{"t":"p","runs":[{"text":"Optimum also wraps quantization for CPU: dynamic int8 quantization typically halves model size and speeds up CPU inference by 2 to 3× on encoder models, with a small accuracy cost you should measure on your own set."}]},{"t":"h2","id":"transformers-js-the-model-in-the-page","runs":[{"text":"Transformers.js: the model in the page"}]},{"t":"code","lang":"html","label":"HTML","text":"<script type=\"module\">\nimport { pipeline } from \"https://cdn.jsdelivr.net/npm/@huggingface/transformers\";\n\nconst pipe = await pipeline(\"sentiment-analysis\");\nconsole.log(await pipe(\"The train was delayed but the staff were kind.\"));\n</script>"},{"t":"p","runs":[{"text":"This downloads an ONNX model into the browser and runs it in WebAssembly, or on the GPU through WebGPU where the browser supports it. No server, no API key, no request leaving the device."}]},{"t":"p","runs":[{"text":"The consequences are worth stating plainly, because they change what is possible for people without budget:"}]},{"t":"ul","items":[[{"text":"Hosting is a static file.","b":true},{"text":" A GitHub Pages site or a static Hugging Face Space can host a working AI demo for nothing, forever, with no server to pay for or maintain."}],[{"text":"The data never leaves.","b":true},{"text":" For a tool handling medical notes, diaries or anything a user would not paste into a website, this is a genuine privacy property rather than a promise."}],[{"text":"It scales for free","b":true},{"text":", because the user’s device does the work."}]]},{"t":"p","runs":[{"text":"The limits are equally plain. The model downloads on first visit (30 MB for a small embedding model is fine, 2 GB for a chat model is not) though the browser caches it afterwards. WebAssembly is slower than native, roughly by a factor of two to five. WebGPU is much faster and is not available everywhere. And a phone on a weak connection will not finish a large download."}]},{"t":"p","runs":[{"text":"The practical ceiling today is models in the tens to low hundreds of megabytes: embeddings, classifiers, speech recognition for short clips, background removal, translation of short text. Within that ceiling it works well."}]},{"t":"h2","id":"other-runtimes-worth-knowing-exist","runs":[{"text":"Other runtimes worth knowing exist"}]},{"t":"ul","items":[[{"text":"OpenVINO","b":true},{"text":", for Intel CPUs and integrated GPUs, also wrapped by Optimum. On an ordinary office laptop it is often the fastest CPU option."}],[{"text":"Core ML","b":true},{"text":", for Apple devices, exported through "},{"text":"coremltools","code":true},{"text":" or Optimum’s exporters."}],[{"text":"ExecuTorch","b":true},{"text":" and "},{"text":"TensorFlow Lite","b":true},{"text":", for on-device mobile deployment."}]]},{"t":"p","runs":[{"text":"All four solve the same problem (run this graph fast on this hardware) and all four trade flexibility for it. Once exported, you cannot easily change the model’s shape, and dynamic control flow may not survive the conversion at all."}]},{"t":"h2","id":"the-caveat-about-export","runs":[{"text":"The caveat about export"}]},{"t":"p","runs":[{"text":"Exports fail. A model with unusual control flow, custom operators or a "},{"text":"trust_remote_code","code":true},{"text":" implementation may export with warnings and then produce subtly different numbers. Always verify:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nnp.testing.assert_allclose(torch_logits, onnx_logits, rtol=1e-3, atol=1e-4)"},{"t":"p","runs":[{"text":"If that fails, the export is not equivalent, and the difference will show up as an accuracy drop nobody can explain later."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Put the eight-line transformers.js example in an HTML file and open it locally. You are running a neural network with no install, no key and no server; and you can host that file for free. For a lot of projects, that is the whole deployment story."}]}],"check":{"question":[{"text":"A team wants a free, private tool that classifies short notes typed by users, with no server costs. Which constraint decides whether transformers.js is viable?"}],"options":[{"text":[{"text":"Whether the browser supports WebGPU, since WebAssembly cannot execute transformer attention operations."}],"correct":false,"tempting":[{"text":"WebGPU improves speed but WebAssembly runs the same models, more slowly."}]},{"text":[{"text":"Whether the model is small enough that users will tolerate downloading it once on first visit."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the model has an ONNX export available, since transformers.js can only load repositories published by Hugging Face itself."}],"correct":false,"tempting":[{"text":"Export availability matters and can be produced with Optimum; the ownership restriction is invented."}]},{"text":[{"text":"Whether the notes can be batched, because per-request browser inference has fixed overhead that only batching amortises."}],"correct":false,"tempting":[{"text":"Batching helps throughput but is not the gate on feasibility for short interactive inputs."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"gguf-and-llama-cpp","title":"GGUF, and the laptop that has no GPU","minutes":9},"next":{"slug":"free-gpus-and-their-limits","title":"Where the free GPUs actually are","minutes":9}}}