Local AI Glossary
Terms and definitions of local artificial intelligence, open source models, and computing.
8 terms in Hardware
Apple's family of chips (M1, M2, M3, M4 and later) that has equipped Macs since 2020. For local AI the key feature is the unified memory shared between CPU and GPU: a 32 GB Mac can run models that on a PC would require a dedicated GPU.
A single shared memory pool across CPU and GPUs where all cores read and write the same memory without copies. This is the model of NVIDIA's Grace-based systems (e.g. the DGX Station's 748GB) and of Apple's unified memory: the LLM lives in one place, with no copies between RAM and VRAM.
Graphics processing unit. For local AI the GPU is the component that accelerates inference: its video memory (VRAM) determines how large a model you can run. Apple's GPUs, built into Apple Silicon, run on unified memory instead of separate VRAM.
Memory bandwidth (GB/s). How fast the GPU reads the model's weights from VRAM. It is the factor that limits token generation speed: for the same model, a card with higher bandwidth produces more tokens per second. Measured in GB per second and listed in every card's official specs.
Non-Uniform Memory Access: the architecture of multi-socket systems where each CPU accesses its local memory faster than the other socket's memory. Inference servers should pin threads to a single NUMA node, otherwise performance drops by up to 30%.
The computer's main memory, used by the CPU for programs and — without a GPU — for loading model weights. In CPU inference, the amount of RAM determines which models you can load and its bandwidth determines the generation speed.
A family of CPU instructions that process multiple data elements in one operation (AVX, AVX-512, NEON, AMX). This is why modern CPUs can run quantized models at usable speeds: the same operation applies to 8 or 16 numbers at once.
Video RAM. The dedicated memory of the graphics card (GPU). It is the critical resource for local inference: a 7B model quantized at 4-bit requires about 5 GB of VRAM, a 13B model about 9 GB. When VRAM is insufficient, the system can use system RAM, but with reduced performance.