Models
The gigabytes are not measured GGUF files. They follow the formula on the methodology page: parameters × bytes, plus KV cache for 8,192 tokens of context.
| Model | Parameters | Max context | Q4estimate | Q5estimate | Q8estimate | FP16estimate | Architecture source |
|---|---|---|---|---|---|---|---|
Small enough for a Mac mini or a modest GPU. A good stack to test your setup. | 3.21Bestimate | 131,072 | 3.5 | 4.0 | 5.2 | 8.2 | confighuggingface.co/unsloth/Llama-3.2-3B-Instruct/raw/main/config.json |
The default local model. Q4 fits in 8–12 GB, Q8 wants closer to 12–16 GB. | 8.03Bestimate | 131,072 | 6.4 | 7.4 | 10.5 | 17.9 | confighuggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct/raw/main/config.json |
Architecture is an estimate of the familiar 7B shape. No config was fetched for this dataset. | 7.25Bestimate | 32,768 | 5.9 | 6.9 | 9.6 | 16.4 | estimate no URL |
Gemma 2 alternates local and global attention. The KV cache is an approximation, not a measured file. | 9.24Bestimate | 8,192 | 8.1 | 9.3 | 12.8 | 21.4 | estimate no URL |
A strong mid-size model. Q4 needs about 10 GB plus context. Sliding window is off in the config. | 14.7Bestimate | 131,072 | 10.6 | 12.6 | 18.1 | 31.8 | confighuggingface.co/Qwen/Qwen2.5-14B/raw/main/config.json |
Architecture figures are an estimate. No config was fetched, so the VRAM line can be slightly off. | 14.7Bestimate | 16,384 | 10.7 | 12.6 | 18.2 | 31.9 | estimate no URL |
Same caveat as Gemma 2 9B: sliding window makes the KV estimate rougher. | 27.2Bestimate | 8,192 | 18.3 | 21.9 | 32.2 | 57.5 | estimate no URL |
Q4 wants a bit over 20 GB. A 16 GB card drops out; a 24 GB card is tight. | 32.8Bestimate | 131,072 | 21.3 | 25.6 | 38.0 | 68.5 | confighuggingface.co/Qwen/Qwen2.5-32B/raw/main/config.json |
Q4 is about 40 GB of weights. A single 24 GB or 32 GB card is not enough. | 70.6Bestimate | 131,072 | 43.0 | 52.2 | 79.0 | 144.7 | confighuggingface.co/unsloth/Llama-3.3-70B-Instruct/raw/main/config.json |
Same class as Llama 70B. Same conclusion: you need a large memory pool. | 72.7Bestimate | 131,072 | 44.2 | 53.6 | 81.3 | 148.9 | confighuggingface.co/Qwen/Qwen2.5-72B/raw/main/config.json |
A distill on a Llama 70B shape. Architecture is copied from Llama 3.3 70B and marked as an estimate: no separate config was fetched. | 70.6Bestimate | 131,072 | 43.0 | 52.2 | 79.0 | 144.7 | estimate no URL |