AI
LocalAI Calc
VRAM + cost per token
Runs in your browser · no signup · no data sent

Why 70B Won't Fit in 24GB VRAM (Despite the Tutorials)

A 70B model at Q4 is 42GB of weights. Here is the arithmetic behind the tutorials that claim it runs on a 24GB card.

Updated 2026-11-10 6 min read

The arithmetic, in four lines

The claim that a 70B model runs on a 24 GB card is not a lie, exactly. It runs. What the tutorials leave out is the speed, and the fact that most of the model is not on the card at all.

Weights = 70.6 B params × 0.60 bytes = 42.4 GB KV cache = 2 × 80 layers × 8 kv heads × 128 dim × 2 bytes = 320 KiB/token at 8K context = 2.5 GB Overhead = 1.5 GB Total = 46.4 GB Usable on a 24 GB card (90 %) = 23.2 GB Shortfall = 23.2 GB

There is no quantization trick that closes a 25 GB gap. Even Q2_K, which degrades quality noticeably, leaves you around 25 GB of weights before any cache at all.

Why the tutorials say it works

Because llama.cpp and Ollama will happily offload the layers that do not fit to system RAM, and report that the model loaded. It did load. It is just running at 1–3 tokens per second.

The reason is bandwidth. Each generated token requires reading the full set of active weights. The 25 GB that lives in system RAM has to cross PCIe, which delivers roughly 20–25 GB/s in practice. Twenty-five gigabytes per token at 25 GB/s is about one token per second, before any compute. On a card with 1,008 GB/s of memory bandwidth, the same model would be limited by the card, not the bus.

The tell

If a tutorial shows a 70B running on 24 GB and does not mention tokens per second, it is showing you a loading screen, not a usable setup.

What actually runs a 70B

  • 48 GB on one card — an RTX A6000, or a professional card. Comfortable at 8K, tight at longer context.
  • Two 24 GB cards — two used 3090s. Weights split across both, KV cache included. The cheapest path to a usable 70B, if your case and power supply can take it.
  • 64 GB or more of unified memory — a Mac Studio. Slower per token, but silent and low power.
  • Renting — if you need 70B a few times a month, a cloud GPU costs less than the hardware.

What 24 GB is actually good for

A 24 GB card is an excellent choice — for the right models. It runs 8B and 14B models with room to spare, 24B models comfortably, and 32B models at Q4 with about 8K of context. That covers most chat, coding assistance and document summarisation work.

The mistake is not buying a 24 GB card. It is buying one expecting a 70B, then discovering the gap after the return window closes. Run the numbers on the calculator first — it takes about ten seconds and shows the shortfall directly.

Read next
Local LLM Hardware Guide: What to Buy in 2026
Read →
Local LLM vs Cloud API: Real Cost Per Million Tokens
Read →
Questions
Why does Q4 need 0.60 bytes per parameter, not 0.50?
Q4_K_M stores most weights at 4 bits but keeps scales, zero-points and some layers at higher precision. Measured files land around 0.60 bytes per parameter — Qwen3-32B is 19.8GB for 32.8B parameters. Using 0.50 underestimates memory by about 20% and is the main reason people buy a card that turns out too small.
Why calculate KV cache separately?
KV cache scales with context length, not with model size alone. Qwen3-32B needs 256 KiB per token at FP16 — 8GB at 32K context. A 24GB card holding 19.5GB of weights has roughly 3GB left, which caps you near 8K. That is why a model can load fine and then fail the moment you paste in a long document.
Are the tokens/sec numbers measured?
No. They are estimates derived from memory bandwidth, using a calibration factor of 0.29 taken from published RTX 4090 measurements on Llama 3 8B Q4 (about 62 tok/s against a 214 tok/s bandwidth ceiling). Real throughput varies with batch size, backend and quantization. Treat them as a planning number, not a benchmark.
Does a 70B model really not run on 24GB?
At Q4_K_M, 70B weights alone are about 42GB. On a 24GB card the remainder has to sit in system RAM and stream over PCIe, which drops generation to roughly 1–3 tok/s. It technically runs; it is not usable for interactive work. See the dedicated guide for the numbers.

All throughput figures are estimates derived from memory bandwidth and published measurements. VRAM figures are computed from model architecture and quantization assumptions. Verify against your own runtime before buying hardware.