Memory bandwidth

The amount of data RAM can read or write per second (GB/s). It is the dominant factor in CPU decode speed: each token requires re-reading the model weights, so an 8B at Q4 (~5 GB) on 64 GB/s does roughly 11-12 tok/s. It matters less on GPU, because VRAM has far more bandwidth.