Local AI Glossary
Terms and definitions of local artificial intelligence, open source models, and computing.
3 terms in Optimization
Technique that reduces memory access during attention by splitting it into blocks and keeping intermediate results in fast memory. It lowers bandwidth demand and speeds up prefill and decode, especially on long contexts. In llama.cpp it is enabled with the -fa flag.
The unit cost you pay for inference on APIs and providers: a price per million input tokens and, separately, per million output tokens. Comparing it with your own hardware's per-token cost (electricity bill + amortization) tells you when owning beats calling an API.
A technique that reduces the numerical precision of a model to decrease its size and speed up inference. 4-bit quantization can shrink a model from 40 GB to about 10 GB while retaining most performance. GGUF and GPTQ are standard formats for quantized models.