Buying hardware, in the right order

The order of the constraints

Buy in this order, because each constraint only matters once the one above it is satisfied:

1. Enough VRAM to hold the model and its KV cache. Below this the model does not run, or runs partly on the CPU at a small fraction of the speed. There is no tuning that recovers it.
2. Memory bandwidth, which sets single-stream generation speed almost by itself, because decode reads every weight for every token.
3. Compute, which sets prefill speed and how well large batches scale.
4. Everything else.

This ordering is why the advice “buy the newest card you can afford” is often wrong. A previous-generation card with more VRAM beats a newer one with less, for the simple reason that a model which does not fit cannot be made to fit by being fast.

The number worth computing for any candidate is gigabytes of VRAM per unit of currency, and then gigabytes per second of bandwidth per unit of currency. Both figures are published, both are comparable across generations, and together they explain most of what people discover the expensive way.

The used market, and what to check

Previous-generation flagship consumer cards are usually the best VRAM-per-unit-money available, and the used market is where this work is affordable at all for most people. The risks are real and manageable.

What actually goes wrong on a used card is the cooling, not the silicon: dried thermal paste, degraded pads on the memory modules, and fan bearings. Memory errors from a card that ran hot for years are the failure that matters for inference, because a flipped bit in a weight produces subtly wrong output rather than a crash.

Test before the return window closes, with free tools. gpu-burn loads the card and checks results; memtestG80-style and Vulkan-based memory testers exercise VRAM specifically. Run for at least twenty minutes and watch temperatures and throttle reasons. A card that holds its clocks for twenty minutes under full load is almost certainly fine.

Accept that there is no warranty and price that in. A used card at 60% of the new price with no warranty is a different proposition from the same card at 90%.

Two cards or one

Two 16GB cards are not one 32GB card, and the difference depends on what you do with them.

For two separate models (a chat model and an embedding model, or two people’s separate work) two cards are excellent, and they avoid the contention problems of sharing one.

For one model split across both, tensor parallelism sends activations between the cards on every layer. On consumer motherboards the link is PCIe, frequently at reduced lane counts when two cards are installed, and without NVLink on recent consumer generations. It works, and it costs real throughput. Pipeline parallelism (whole layers on each card) moves far less data and is the better choice on a slow link, at the price of leaving one card idle while the other works unless you have several requests in flight.

Check the practical details before buying the second card: physical slot spacing, whether the second slot runs at x4, and the power supply. Transient spikes from a large GPU can trip a supply’s protection even when the average draw is well inside its rating, which presents as the whole machine restarting under load rather than as any kind of GPU error.

Apple Silicon

Unified memory makes a Mac unusually good at one specific thing: holding a large model. A machine with 64GB or more of unified memory runs models that no consumer GPU at similar cost can hold at all, because the memory is shared rather than partitioned.

The limits are equally specific. Bandwidth spans a wide range across the chip tiers, from roughly a hundred gigabytes per second at the base to several hundred at the top, and that range decides generation speed. Prefill on long prompts is comparatively slow because it is compute-bound. Concurrency is weak, so a Mac serving several people at once degrades faster than a discrete GPU would. And there is no container GPU path, so the deployment shape is native or nothing.

The summary that holds: a Mac is excellent for one person running a large model at a moderate speed, and a poor choice as a shared server.

The option that is usually right

Buy nothing yet. Rent a machine by the hour for a month, run your real workload on it, and record the tokens you actually consumed and the sizes you actually needed. Then compute the break-even from measurements rather than from an intuition about future usage that is, reliably, an overestimate.

The VRAM figure you need is also the one that ages worst. Model sizes have not stopped growing, and the card that comfortably fits today’s preferred model is the card that just misses next year’s. Buy some headroom, or accept that the machine has a defined useful life, which is exactly what the next lesson makes you write down.