Local AI Glossary

The maximum number of tokens a model can process simultaneously in a single interaction. A larger context window allows the model to read longer documents, more extensive conversations, and more complex code. Modern models support 8K to 128K+ context tokens.

Vector representation of text. A model transforms words, sentences, or documents into an array of numbers (vector) that captures the semantic meaning. Embeddings are used for semantic search, text similarity, and as a foundation for RAG systems.

The network interface of your own computer, reachable at address 127.0.0.1 (IPv4) or ::1 (IPv6). Services running locally — the Ollama server, LM Studio, web UIs — listen there: your applications connect to the local model without traversing the external network. A well-configured firewall that blocks external access to localhost is the first line of defense: no program can send your content outside without an outbound connection from the machine showing it.

Memory mapping: the mechanism by which the operating system maps a file into a program's memory, reading only the requested parts on demand. It's why models in GGUF format, with their aligned tensors, load in a few seconds: the whole file doesn't have to be copied into RAM.

Optical Character Recognition: the technique that converts images and scans into selectable text. For documents not "born digital", it's the step before the language model: without OCR, the model has no text to work on.

Remote Direct Memory Access: lets one machine read and write directly into another's memory without routing data through their CPUs. On Mac Studio it is used to cluster up to 4 units over Thunderbolt 5, speeding up distributed inference by up to 3x.

AMD's GPU compute software stack: the CUDA equivalent for Radeon hardware. Since ROCm 7, consumer RDNA 3 and 4 cards (RX 7900 XTX, RX 9070 XT) are officially supported on Linux with Ollama and llama.cpp, no workaround environment variables. The alternative path is Vulkan: less optimized but more cross-vendor, and it is the maintained path for Intel Arc cards as well.

A cloud service that runs code or models on GPUs only when there is demand, starting instances in seconds and dropping cost to zero when idle. You pay for actual compute seconds, not for an always-on server.

The technology that converts audio into text (speech recognition, ASR). Locally, with models like Whisper Large-v3-Turbo, transcription happens on your own hardware: the audio never leaves the machine, making it the reference option for sensitive documents. Quality is measured by WER (word error rate): turbo sits around 2.5% on clean English.

Speech synthesis: generating spoken audio from text. Local systems like Piper (VITS/ONNX-based) run in real time on CPU with no GPU: ideal for accessibility pipelines, voice commands and assistants. The trade-off is naturalness versus latency and resources.

The basic unit of text processed by an LLM. A token is typically a word or part of a word. In English, roughly 1 token ≈ 4 characters. Models have a maximum context measured in tokens: GPT-4 supports 128K tokens, while Llama 3 supports 128K. More tokens = more context but more resources needed.

The component that splits text into tokens, the units the model actually reads. A multilingual tokenizer encodes Italian in fewer tokens than an English-only one, reducing compute and context consumption.

A specialized database that stores document embeddings (numeric vectors) and answers similarity searches: given the embedding of a question, it returns the closest passages. It is the component that makes RAG possible.