Local AI Glossary

A model that, given a state and a typed question (yes/no, choice among options, score), returns a probability distribution over the options in a single pass, without generating tokens. It is the "System 1" of AI: fast and calibrated, used for routing, classification and quality control, in place of a generative LLM when you don't need to write.

Small, fast AI models designed to answer multiple-choice questions with a structured label and a probability per option, instead of generating free text. The name comes from Jev, the acronym identifying this class of models. They are built for low-latency triage, routing, classification and moderation: where a large LLM would generate a paragraph, a decision model returns the option and its confidence level.

The second phase of inference: the model generates the response token by token. Each token requires re-reading all model weights and the KV cache from memory, so the speed (tokens per second) is driven mostly by memory bandwidth.

A model in which every token passes through all parameters: there is no expert selection, the full capacity is active at each step. Heavier to decode than a same-total-size MoE, but without expert routing. Example: Qwen3.8-27B.

The largest-scale language models (OpenAI's GPT series, Anthropic's Claude, etc.) which today are almost always available only via paid APIs: the lowest per-token prices, but with provider dependency and rate limits.

Google DeepMind's open language model family, built on the same technology as the Gemini series. Sizes range from 1B to 27B parameters; Gemma 3 adds a 128K-token context, 140+ languages and multimedia input, while Gemma 4 (2026) moves to the Apache 2.0 license.

The process of using a trained model to generate responses or make predictions on new data. In local AI, inference runs on your computer instead of remote servers. This ensures privacy and consistent response times without relying on internet connections.

A version of a model adapted for conversation: it understands instructions and answers directly, recognizable by the 'instruct' or '-it' suffix. The base model, without this training, is a raw text completer used for fine-tuning.

Family of open-source LLM models by Meta. Llama 3 is the current generation, with 8B and 70B parameter versions. Llama is one of the most widely used models for local AI, with broad compatibility with Ollama, llama.cpp, and most inference frameworks.

Large Language Model. An AI model trained on massive amounts of text, capable of understanding and generating human language. LLMs like Llama, Mistral, and GPT power chatbots, virtual assistants, and automated writing tools.

Family of open-source LLM models developed by Mistral AI, a French company. Mistral 7B and Mixtral are among the most popular models for local inference due to their quality/performance ratio. They are available on Hugging Face and supported by Ollama, llama.cpp, and other tools.

The first phase of inference, where the model processes the input prompt (all tokens) in one pass. It determines time-to-first-token: near-instant on a GPU, it can take several seconds on a CPU for long prompts.

A multidimensional array of numbers: the matrix that makes up each layer of a neural model (weights, biases, embeddings). In a GGUF file every tensor has a name, dimensions, type, and offset; it is the alignment and type of each tensor that determine how the model is loaded and how much memory it takes.

Word Error Rate: the standard metric for comparing transcriptions. It compares the produced text against a reference transcript: fewer errors per word means a lower score and a better model.