{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"buying-the-card","title":"Buying hardware, in the right order","url":"https://addaly.com/learn/running-models-yourself/buying-the-card","position":68,"of":82,"minutes":9,"answer":"Buy VRAM first, bandwidth second and compute third, because a model that does not fit cannot be made to fit by being fast; and rent for a month before buying.","keyIdea":"Buy VRAM first, bandwidth second and compute third, because a model that does not fit cannot be made to fit by being fast; and rent for a month before buying, since the workload people predict is reliably larger than the one they measure."},"body":{"format":"markdown","text":"## The order of the constraints\n\nBuy in this order, because each constraint only matters once the one above it is satisfied:\n\n1. **Enough VRAM to hold the model and its KV cache.** Below this the model does not run, or runs partly on the CPU at a small fraction of the speed. There is no tuning that recovers it.\n2. **Memory bandwidth**, which sets single-stream generation speed almost by itself, because decode reads every weight for every token.\n3. **Compute**, which sets prefill speed and how well large batches scale.\n4. Everything else.\n\nThis ordering is why the advice \"buy the newest card you can afford\" is often wrong. A previous-generation card with more VRAM beats a newer one with less, for the simple reason that a model which does not fit cannot be made to fit by being fast.\n\nThe number worth computing for any candidate is gigabytes of VRAM per unit of currency, and then gigabytes per second of bandwidth per unit of currency. Both figures are published, both are comparable across generations, and together they explain most of what people discover the expensive way.\n\n## The used market, and what to check\n\nPrevious-generation flagship consumer cards are usually the best VRAM-per-unit-money available, and the used market is where this work is affordable at all for most people. The risks are real and manageable.\n\nWhat actually goes wrong on a used card is the cooling, not the silicon: dried thermal paste, degraded pads on the memory modules, and fan bearings. Memory errors from a card that ran hot for years are the failure that matters for inference, because a flipped bit in a weight produces subtly wrong output rather than a crash.\n\nTest before the return window closes, with free tools. `gpu-burn` loads the card and checks results; `memtestG80`-style and Vulkan-based memory testers exercise VRAM specifically. Run for at least twenty minutes and watch temperatures and throttle reasons. A card that holds its clocks for twenty minutes under full load is almost certainly fine.\n\nAccept that there is no warranty and price that in. A used card at 60% of the new price with no warranty is a different proposition from the same card at 90%.\n\n## Two cards or one\n\nTwo 16GB cards are not one 32GB card, and the difference depends on what you do with them.\n\nFor **two separate models** (a chat model and an embedding model, or two people's separate work) two cards are excellent, and they avoid the contention problems of sharing one.\n\nFor **one model split across both**, tensor parallelism sends activations between the cards on every layer. On consumer motherboards the link is PCIe, frequently at reduced lane counts when two cards are installed, and without NVLink on recent consumer generations. It works, and it costs real throughput. Pipeline parallelism (whole layers on each card) moves far less data and is the better choice on a slow link, at the price of leaving one card idle while the other works unless you have several requests in flight.\n\nCheck the practical details before buying the second card: physical slot spacing, whether the second slot runs at x4, and the power supply. Transient spikes from a large GPU can trip a supply's protection even when the average draw is well inside its rating, which presents as the whole machine restarting under load rather than as any kind of GPU error.\n\n## Apple Silicon\n\nUnified memory makes a Mac unusually good at one specific thing: holding a large model. A machine with 64GB or more of unified memory runs models that no consumer GPU at similar cost can hold at all, because the memory is shared rather than partitioned.\n\nThe limits are equally specific. Bandwidth spans a wide range across the chip tiers, from roughly a hundred gigabytes per second at the base to several hundred at the top, and that range decides generation speed. Prefill on long prompts is comparatively slow because it is compute-bound. Concurrency is weak, so a Mac serving several people at once degrades faster than a discrete GPU would. And there is no container GPU path, so the deployment shape is native or nothing.\n\nThe summary that holds: a Mac is excellent for one person running a large model at a moderate speed, and a poor choice as a shared server.\n\n## The option that is usually right\n\nBuy nothing yet. Rent a machine by the hour for a month, run your real workload on it, and record the tokens you actually consumed and the sizes you actually needed. Then compute the break-even from measurements rather than from an intuition about future usage that is, reliably, an overestimate.\n\nThe VRAM figure you need is also the one that ages worst. Model sizes have not stopped growing, and the card that comfortably fits today's preferred model is the card that just misses next year's. Buy some headroom, or accept that the machine has a defined useful life, which is exactly what the next lesson makes you write down."},"check":{"question":"Someone must choose between a newer card with 12GB of VRAM and a previous-generation card with 24GB at a similar price, for running a 14B model at long context. Which reasoning is sound?","options":[{"text":"The newer card, because its higher compute throughput will offset the smaller memory by processing layers faster as they are swapped in.","correct":false,"tempting":"Treats offloading as a speed problem; moving weights across PCIe for every token is orders of magnitude slower than reading them from VRAM, and no amount of compute compensates."},{"text":"The 24GB card, because the model plus its KV cache must fit in VRAM before any other property of the card can help.","correct":true},{"text":"The newer card, because a more aggressive quantisation will bring the model within 12GB with only a small quality loss.","correct":false,"tempting":"Plausible for the weights alone, but it ignores the KV cache at long context, which is the part of the budget this choice is really about."},{"text":"They are equivalent, since generation speed is set by memory bandwidth and both cards have comparable bandwidth figures.","correct":false,"tempting":"Correct about what governs generation speed, but the comparison only applies once both candidates can hold the model at all."}]},"prev":{"slug":"electricity-and-heat","title":"Electricity, heat and the power limit knob","url":"https://addaly.com/learn/running-models-yourself/electricity-and-heat"},"next":{"slug":"renting-by-the-hour","title":"Renting a GPU, and the meter that runs while you think","url":"https://addaly.com/learn/running-models-yourself/renting-by-the-hour"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Buying hardware, in the right order', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/buying-the-card (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Buying hardware, in the right order","courseTitle":"Running Models Yourself","module":{"index":8,"slug":"what-it-actually-costs","title":"What it actually costs"},"answer":null,"keyIdea":"Buy VRAM first, bandwidth second and compute third, because a model that does not fit cannot be made to fit by being fast; and rent for a month before buying, since the workload people predict is reliably larger than the one they measure.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-order-of-the-constraints","runs":[{"text":"The order of the constraints"}]},{"t":"p","runs":[{"text":"Buy in this order, because each constraint only matters once the one above it is satisfied:"}]},{"t":"ol","start":1,"items":[[{"text":"Enough VRAM to hold the model and its KV cache.","b":true},{"text":" Below this the model does not run, or runs partly on the CPU at a small fraction of the speed. There is no tuning that recovers it."}],[{"text":"Memory bandwidth","b":true},{"text":", which sets single-stream generation speed almost by itself, because decode reads every weight for every token."}],[{"text":"Compute","b":true},{"text":", which sets prefill speed and how well large batches scale."}],[{"text":"Everything else."}]]},{"t":"p","runs":[{"text":"This ordering is why the advice “buy the newest card you can afford” is often wrong. A previous-generation card with more VRAM beats a newer one with less, for the simple reason that a model which does not fit cannot be made to fit by being fast."}]},{"t":"p","runs":[{"text":"The number worth computing for any candidate is gigabytes of VRAM per unit of currency, and then gigabytes per second of bandwidth per unit of currency. Both figures are published, both are comparable across generations, and together they explain most of what people discover the expensive way."}]},{"t":"h2","id":"the-used-market-and-what-to-check","runs":[{"text":"The used market, and what to check"}]},{"t":"p","runs":[{"text":"Previous-generation flagship consumer cards are usually the best VRAM-per-unit-money available, and the used market is where this work is affordable at all for most people. The risks are real and manageable."}]},{"t":"p","runs":[{"text":"What actually goes wrong on a used card is the cooling, not the silicon: dried thermal paste, degraded pads on the memory modules, and fan bearings. Memory errors from a card that ran hot for years are the failure that matters for inference, because a flipped bit in a weight produces subtly wrong output rather than a crash."}]},{"t":"p","runs":[{"text":"Test before the return window closes, with free tools. "},{"text":"gpu-burn","code":true},{"text":" loads the card and checks results; "},{"text":"memtestG80","code":true},{"text":"-style and Vulkan-based memory testers exercise VRAM specifically. Run for at least twenty minutes and watch temperatures and throttle reasons. A card that holds its clocks for twenty minutes under full load is almost certainly fine."}]},{"t":"p","runs":[{"text":"Accept that there is no warranty and price that in. A used card at 60% of the new price with no warranty is a different proposition from the same card at 90%."}]},{"t":"h2","id":"two-cards-or-one","runs":[{"text":"Two cards or one"}]},{"t":"p","runs":[{"text":"Two 16GB cards are not one 32GB card, and the difference depends on what you do with them."}]},{"t":"p","runs":[{"text":"For "},{"text":"two separate models","b":true},{"text":" (a chat model and an embedding model, or two people’s separate work) two cards are excellent, and they avoid the contention problems of sharing one."}]},{"t":"p","runs":[{"text":"For "},{"text":"one model split across both","b":true},{"text":", tensor parallelism sends activations between the cards on every layer. On consumer motherboards the link is PCIe, frequently at reduced lane counts when two cards are installed, and without NVLink on recent consumer generations. It works, and it costs real throughput. Pipeline parallelism (whole layers on each card) moves far less data and is the better choice on a slow link, at the price of leaving one card idle while the other works unless you have several requests in flight."}]},{"t":"p","runs":[{"text":"Check the practical details before buying the second card: physical slot spacing, whether the second slot runs at x4, and the power supply. Transient spikes from a large GPU can trip a supply’s protection even when the average draw is well inside its rating, which presents as the whole machine restarting under load rather than as any kind of GPU error."}]},{"t":"h2","id":"apple-silicon","runs":[{"text":"Apple Silicon"}]},{"t":"p","runs":[{"text":"Unified memory makes a Mac unusually good at one specific thing: holding a large model. A machine with 64GB or more of unified memory runs models that no consumer GPU at similar cost can hold at all, because the memory is shared rather than partitioned."}]},{"t":"p","runs":[{"text":"The limits are equally specific. Bandwidth spans a wide range across the chip tiers, from roughly a hundred gigabytes per second at the base to several hundred at the top, and that range decides generation speed. Prefill on long prompts is comparatively slow because it is compute-bound. Concurrency is weak, so a Mac serving several people at once degrades faster than a discrete GPU would. And there is no container GPU path, so the deployment shape is native or nothing."}]},{"t":"p","runs":[{"text":"The summary that holds: a Mac is excellent for one person running a large model at a moderate speed, and a poor choice as a shared server."}]},{"t":"h2","id":"the-option-that-is-usually-right","runs":[{"text":"The option that is usually right"}]},{"t":"p","runs":[{"text":"Buy nothing yet. Rent a machine by the hour for a month, run your real workload on it, and record the tokens you actually consumed and the sizes you actually needed. Then compute the break-even from measurements rather than from an intuition about future usage that is, reliably, an overestimate."}]},{"t":"p","runs":[{"text":"The VRAM figure you need is also the one that ages worst. Model sizes have not stopped growing, and the card that comfortably fits today’s preferred model is the card that just misses next year’s. Buy some headroom, or accept that the machine has a defined useful life, which is exactly what the next lesson makes you write down."}]}],"check":{"question":[{"text":"Someone must choose between a newer card with 12GB of VRAM and a previous-generation card with 24GB at a similar price, for running a 14B model at long context. Which reasoning is sound?"}],"options":[{"text":[{"text":"The newer card, because its higher compute throughput will offset the smaller memory by processing layers faster as they are swapped in."}],"correct":false,"tempting":[{"text":"Treats offloading as a speed problem; moving weights across PCIe for every token is orders of magnitude slower than reading them from VRAM, and no amount of compute compensates."}]},{"text":[{"text":"The 24GB card, because the model plus its KV cache must fit in VRAM before any other property of the card can help."}],"correct":true,"tempting":null},{"text":[{"text":"The newer card, because a more aggressive quantisation will bring the model within 12GB with only a small quality loss."}],"correct":false,"tempting":[{"text":"Plausible for the weights alone, but it ignores the KV cache at long context, which is the part of the budget this choice is really about."}]},{"text":[{"text":"They are equivalent, since generation speed is set by memory bandwidth and both cards have comparable bandwidth figures."}],"correct":false,"tempting":[{"text":"Correct about what governs generation speed, but the comparison only applies once both candidates can hold the model at all."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"electricity-and-heat","title":"Electricity, heat and the power limit knob","minutes":8},"next":{"slug":"renting-by-the-hour","title":"Renting a GPU, and the meter that runs while you think","minutes":9}}}