Local AI Glossary

8-bit floating-point format for large-model weights and activations (E4M3 and E5M2 variants). Half the VRAM of FP16 at minimal quality loss on weights: recent models (e.g. Aleph Alpha's Kolibri) ship natively in FP8. Unlike NVFP4 or MXFP4, it is natively supported by base libraries (cuBLAS) without per-block micro-scaling.

The current generation of graphics memory: up to 32 Gbps per pin — 1 TB/s on a 256-bit bus (RTX 5090). For local inference, memory bandwidth sets generation speed: NVIDIA's Blackwell cards use it across the 50 series, and the 2026 GDDR7 price surge has pushed street prices up across the mid-range.

GGML Unified Format. The standard file format for distributing quantized LLM models, especially with llama.cpp. It supports storing the model, quantized weights, and tokens. It is the predominant format for local inference on consumer hardware.

Microscaling FP4: a 4-bit floating-point format where each block of 32 weights shares one scaling exponent. It is the native format of gpt-oss: weights ship pre-quantized, with no conversion step. Compared to Q4_K_M GGUF, MXFP4 is optimized for Blackwell tensor cores and NVIDIA hardware in particular.

NVIDIA's 4-bit quantization format (NVIDIA Floating Point 4): represents model weights with 4 bits plus a per-group scale factor, cutting memory by roughly a quarter compared to 16 bits. It is the native format of Blackwell tensor cores and lets 35B models run on a 32GB GPU.

Weight representation restricted to {−1, 0, +1} values, often with a shared scale per group. Brings the cost to ~1.58 bits per weight (plus the scale), enabling tens-of-billions-of-parameters models in a few GB, as in Bonsai-2 27B.