Local Inference Engineering
LLM VRAM, Quantization & Tokens/s Calculator
Verify memory requirements and inference throughput for DeepSeek-R1-Distill-Llama-70B on GeForce RTX 4090 24GB.
ENGINEERING CONTROLS & INPUTS
Live Sync 1:12.06 bpw
8,192 tokens
2k8k16k32k64k128k
24 GB VRAM
64 GB RAM
TELEMETRY & MATH CONSOLE
Total VRAM Required
18.78GB / 24 GB
100% in VRAM (Max Performance)
GPU VRAM Occupancy78%
Detailed Mathematical Breakdown
Model Weights Footprint70.6B params at 2.06 bits per weight.
16.93GB
KV Cache FootprintContext: 8,192 tokens in Q4_0 format.
0.63GB
CUDA Overhead & RuntimeKernel driver overhead, scratchpad activations & context buffers.
1.23GB
GPU Layers vs CPU Layers100% of the network accelerated by VRAM.
80 / 80in GPU
Estimated Generation SpeedBounded by 1008 GB/s memory bandwidth.
40.5 t/s
Applied Equation (Live Input):
VRAM_Total (18.78 GB) = Pesos (16.93 GB) + KV Cache (0.63 GB) + CUDA Buffer (1.23 GB)
Technical Methodology & Benchmark Guide
Quantization Matrix for DeepSeek-R1-Distill-Llama-70B across all formats
| Format | Total Size | Fits VRAM | Est. Speed |
|---|---|---|---|
| FP16 / BF16 (Sin pérdida) | 137.94 GB | Offload CPU | 0 t/s |
| Q8_0 (Calidad Original) | 73.83 GB | Offload CPU | 0 t/s |
| Q6_K (Excelente fidelidad) | 57.25 GB | Offload CPU | 0.9 t/s |
| Q5_K_M (Equilibrado alto) | 49.81 GB | Offload CPU | 1.2 t/s |
| Q4_K_M (Estándar Recomendado) | 42.63 GB | Offload CPU | 1.7 t/s |
| IQ4_XS (Cuantización Inteligente) | 37.5 GB | Offload CPU | 2.3 t/s |
| Q3_K_M (Ahorro agresivo) | 34.6 GB | Offload CPU | 2.9 t/s |
| Q2_K (Ultra comprimido) | 29.81 GB | Offload CPU | 5.3 t/s |
| IQ2_XXS (MÃnimo absoluto) | 18.78 GB | ✓ 100% VRAM | 40.5 t/s |
Frequently Asked Technical Questions
What is the reasoning degradation of DeepSeek R1 70B at IQ2_XXS?
SOTA benchmarks confirm an IQ2-quantized 70B model retains substantially higher reasoning and coding competence than an unquantized 8B or 14B model in native FP16.