Local Inference Engineering
LLM VRAM, Quantization & Tokens/s Calculator
Verify memory requirements and inference throughput for Llama 3.1 8B (Meta) on GeForce RTX 3060 12GB.
ENGINEERING CONTROLS & INPUTS
Live Sync 1:14.85 bpw
8,192 tokens
2k8k16k32k64k128k
12 GB VRAM
32 GB RAM
TELEMETRY & MATH CONSOLE
Total VRAM Required
6.27GB / 12 GB
100% in VRAM (Max Performance)
GPU VRAM Occupancy52%
Detailed Mathematical Breakdown
Model Weights Footprint8.03B params at 4.85 bits per weight.
4.53GB
KV Cache FootprintContext: 8,192 tokens in FP16 format.
1GB
CUDA Overhead & RuntimeKernel driver overhead, scratchpad activations & context buffers.
0.73GB
GPU Layers vs CPU Layers100% of the network accelerated by VRAM.
32 / 32in GPU
Estimated Generation SpeedBounded by 360 GB/s memory bandwidth.
54 t/s
Applied Equation (Live Input):
VRAM_Total (6.27 GB) = Pesos (4.53 GB) + KV Cache (1.00 GB) + CUDA Buffer (0.73 GB)
Technical Methodology & Benchmark Guide
Quantization Matrix for Llama 3.1 8B (Meta) across all formats
| Format | Total Size | Fits VRAM | Est. Speed |
|---|---|---|---|
| FP16 / BF16 (Sin pérdida) | 17.11 GB | Offload CPU | 5.2 t/s |
| Q8_0 (Calidad Original) | 9.81 GB | ✓ 100% VRAM | 30.8 t/s |
| Q6_K (Excelente fidelidad) | 7.93 GB | ✓ 100% VRAM | 39.9 t/s |
| Q5_K_M (Equilibrado alto) | 7.08 GB | ✓ 100% VRAM | 46 t/s |
| Q4_K_M (Estándar Recomendado) | 6.27 GB | ✓ 100% VRAM | 54 t/s |
| IQ4_XS (Cuantización Inteligente) | 5.68 GB | ✓ 100% VRAM | 61.6 t/s |
| Q3_K_M (Ahorro agresivo) | 5.35 GB | ✓ 100% VRAM | 67 t/s |
| Q2_K (Ultra comprimido) | 4.81 GB | ✓ 100% VRAM | 78.2 t/s |
| IQ2_XXS (MÃnimo absoluto) | 3.55 GB | ✓ 100% VRAM | 127.1 t/s |
Frequently Asked Technical Questions
Can I run Llama 3.1 8B at full 128k context on an RTX 3060 12GB?
At 128k context in FP16, the KV cache alone demands 8.0 GB, which combined with 4.54 GB weights exceeds 12 GB. However, enabling Q4_0 KV cache quantization drops it to 2.0 GB, fitting entirely within VRAM.
What is the reasoning degradation of DeepSeek R1 70B at IQ2_XXS?
SOTA benchmarks confirm an IQ2-quantized 70B model retains substantially higher reasoning and coding competence than an unquantized 8B or 14B model in native FP16.