How to use the LLM GPU VRAM Calculator
- 1. Enter model sizeType the parameter count in billions, such as 7 or 70.
- 2. Set precision and useChoose weight precision and inference, LoRA or full fine-tuning.
- 3. Add context detailsEnter layers, hidden size, context and batch from the model config.
How the estimate is built
- Weights = parameters × bytes per parameter. FP16 is 2 bytes, INT8 1 byte, 4-bit 0.5 byte.
- KV cache = 2 × layers × hidden size × (KV heads ÷ heads) × context × batch × 2 bytes. It grows with every token of context.
- Full fine-tuning with Adam in mixed precision needs roughly 16 bytes per parameter for weights, gradients, optimizer states and an FP32 master copy, before activations.
- Overhead is an editable rule-of-thumb allowance for activations, the CUDA context and fragmentation.
Worked example and limits
A 7-billion-parameter model in FP16: 7 × 10⁹ × 2 bytes = 14 × 10⁹ bytes = 13.04 GiB for weights. With 32 layers, hidden size 4096, standard attention and a 4,096-token context, the KV cache adds 2 GiB. Adding 20% overhead gives about 18.05 GiB, so a 24 GB card fits it and a 16 GB card does not. In 4-bit, the weights drop to about 3.26 GiB.
Results are in GiB (2³⁰ bytes); GPU memory is usually advertised in GB but behaves close to GiB. Grouped-query attention, used in many recent models, shrinks the KV cache; pick the ratio from the model's config (num_key_value_heads ÷ num_attention_heads). Actual usage depends on the framework, kernels and batch settings, so treat this as planning guidance and leave headroom. Estimate prompt sizes with the LLM token counter.
Frequently asked questions
Where do I find layers and hidden size?
In the model's config.json on its model page: num_hidden_layers and hidden_size.
Why is full fine-tuning so much bigger?
Training stores gradients and optimizer states for every parameter, not just the weights.
Is the LLM GPU VRAM Calculator free?
Yes, the LLM GPU VRAM Calculator is free. The first time you see your first estimate, you are asked for your name and WhatsApp number once. After that, this and every other Uncodemy tool opens straight away without the form.