Where local models learn: open weights explained without the fluff

The question comes up every time, and it makes sense: a model running on your PC — where does it get its abilities? No internet, no server it hooks into: yet it knows grammar, math, code. The answer in one sentence: it learned them once, during training, on public data — and since then everything lives inside the model's weights, which sit on a disk, not in a cloud.

The analogy is the encyclopedist. An encyclopedist who has read millions of pages doesn't need the library to answer: he knows Paris is the capital of France without thinking about it, because the information has become part of him. A pretrained model is exactly that: an "encyclopedist" who has read a significant fraction of the public text on the internet, and who since answers from memory. Your hardware downloads nothing in real time: it runs computations on what the model already learned.

The three phases: how a model is built

1. Pre-training — the model reads the world. Researchers assemble a huge corpus of public text (web, digitized books, Wikipedia, code on GitHub, forums) and do something that looks stupid: they train the network to predict the next word, billions of billions of times. To do it well, the model is forced to build an internal representation of the world: meanings, logical structures, the conventions of a programming language, rules of physics glimpsed across millions of pages. This is where the abilities are born.

2. Alignment — it learns to be useful. The raw pre-trained model is a text completer with the personality of an autocomplete. A second pass (instruction + feedback) turns it into an assistant that follows your questions instead of merely continuing the text.

3. Distribution — the weights come out (or not). This is where it's decided whether a model is "open weights". The lab publishes the weights file (tens or hundreds of GB, in formats like GGUF) and you download it once: from then on it's yours, it runs offline, nobody updates it or pulls it. OpenAI with ChatGPT does not publish the weights: the model never leaves its servers. Here is the real dividing line — not between "open source" and "commercial", but between weights you own and weights you rent.

Open weights is not open data

The part where confusion is most frequent: "open weight" does not mean the training data is public, nor that the model is open source in the legal sense (GPL license). It means the weights file is downloadable and modifiable: you can quantize it, fine-tune it, distribute it, delete the cloud. The data it was trained on, generally, remains a proprietary-and-public mix that the lab does not make fully available.

So no: local AI does not "pull global data" from Mistral or DeepSeek in real time. It takes what the weights memorized, and it executes it on your machine.

The models you could run, and where they come from

Model	Parameters	Declared pre-training	Published by
Llama 3.1 (Meta)	8B – 405B	~15 trillion tokens, 8+ languages	Meta
Mistral 7B (Mistral AI)	7B	~8 trillion tokens	Mistral AI
DeepSeek-V3 (DeepSeek)	671B (MoE, 37B active)	~14.8 trillion tokens	DeepSeek
Gemma 3 (Google)	1B – 27B	not disclosed	Google
gpt-oss (OpenAI)	20B – 120B (MoE)	not disclosed	OpenAI, Apache 2.0 license

Two readings that matter. First: open weights don't come from a single French lab — Meta, Mistral, Google, DeepSeek, and since 2025 OpenAI itself publish downloadable models. Second: the "global" abilities come from multilingual, multicultural corpora — that's why an 8B model speaks acceptable Italian and French despite being trained on mostly English data.

How much data does it actually take?

Pre-training of important models is measured in trillions of tokens: Llama 3 declared roughly 15 trillion for the 8B-405B family, DeepSeek-V3 roughly 14.8 trillion. A trillion tokens is about 700 billion words — the order of magnitude of "all public text on the internet, read a few times over". There is no way a single user does this: it requires clusters with thousands of GPUs for months. The good part is you don't need to: the dirty work is already done, and the file you download from Ollama or Hugging Face is the finished result.

What happens when you ask something

When you type a sentence, your hardware does something very concrete: multiplications and additions on the weights, token by token. Each generated token requires a full pass of your input data through the model's matrices. For an 8-bit-quantized 7B you need roughly 4.5 GB of RAM; for a 4-bit 70B about 40 GB. RAM is the bottleneck, not compute power — GGUF, the format that fits models on your hardware explains quantization in detail.

And the limits?

Honest ones, because they deserve a section.

• Knowledge frozen at training time. The model doesn't "know" what happened after its corpus cut-off. That's not a network limit: it's the price of owning the model.
• Quantization costs something. A 4-bit 7B is faster and lighter than the full-precision model, but on subtle tasks the difference shows.
• Complex cases still belong to the frontier. Multi-step reasoning, very long contexts, multimodal tasks: here the frontier models (ChatGPT, Claude, Gemini) are still ahead. The honest frontier vs local comparison measures where the difference really matters.

In summary

Local AI doesn't hook into the internet to "pull global abilities": it absorbed them once, during pre-training, on mostly public data, and it executes them entirely on your hardware. "Open weights" is not synonymous with "open data", the models come from many labs (Meta, Mistral, DeepSeek, Google, and since 2025 OpenAI too), and your only job is to download the file, pick the right quantization, and start the computations. The question that matters is not where the abilities come from, but what you do with them — data that never leaves your machine.

Sources

• Llama 3: Herd of Models (arXiv 2310.06825)
• Mistral 7B (arXiv 2401.04088)
• DeepSeek-V3 (arXiv 2407.21783)
• gpt-oss Technical Report (arXiv 2412.19437)
• OpenAI gpt-oss-120b — model card
• DeepSeek-V3 — Hugging Face

Article generated with AI