Ling 3.1 Flash: 560B open-weight with MIT license and 1M context — what changes if you have 24GB of VRAM

Think of a 560-horsepower car with fully open paperwork: you can do whatever you want with it, no subscription, no one to ask permission from. That's roughly what's happening with Ling 3.1 Flash, the 560-billion-parameter MoE model that inclusionAI just made available as open-weights under the MIT license with a one-million-token context. The real question, though, is a different one: if you have a 24 GB GPU, what can you actually do with it?

The honest answer: you can't run the whole model, but you can understand exactly what you'd need to try it and which realistic alternatives get you close to the same capability tier. Let's look at the numbers, because in this case the numbers decide everything.

What Ling 3.1 Flash is, and why it changes the picture

Ling 3.1 Flash is a hybrid reasoning model: it can reason at length (thinking) or answer instantly, and in both modes it supports tool calling and structured outputs. It's the successor to Ling 3.0 Flash (124B total / ~5B active), but it moves to a different scale: 560B total parameters, 25B active per token, and a native 1M-token context.

The key numbers, from the official model card:

Specification	Value
Architecture	Hybrid MoE (KDA linear attention + Gated MLA)
Total parameters	560B
Active parameters per token	25B
Native context	1,048,576 tokens (1M)
Experts	512 routed + 1 shared, 8 active per token
Native precision	FP8 (~572 GB checkpoint on disk)
License	MIT

The point that matters for you is the total/active ratio: 560B of weights, but only 25B of compute per token. That's the classic MoE pattern — memory forces you to load everything, but speed depends only on what activates.

What MIT means, concretely

MIT is the most permissive open-source license there is: using the model, commercially or otherwise, costs nothing, you can redistribute it, modify it, and integrate it into your products without asking anyone. It is not an "open model" with usage restrictions: it's open in the full sense. For anyone who manages sensitive data or wants infrastructure with no external dependency, that's the difference between a model you can adopt and a service you have to treat as a service.

The 1M-token context, translated

One million tokens is roughly 750,000 words in English — the equivalent of several novels, or an entire mid-size codebase. Most cloud models stop at 128K or 200K tokens; here the ceiling is five times higher. In practice that means you can load an entire codebase, a company knowledge base, or a very long conversation into a single prompt without fragmenting it into windows.

What 24 GB of VRAM can actually do

Here the numbers leave no room for interpretation. The native checkpoint is FP8: ~560B parameters × 1 byte ≈ 572 GB. The whole model does not fit in 24 GB — not with any standard quantization, because FP8 alone weighs more than 23× your VRAM.

So how many GPUs do you need? Let's do the math with reference quantizations:

Quantization	Weights on disk	2× 24 GB	4× 24 GB	8× 24 GB	2× 48 GB	4× 96 GB
FP8 (native)	~572 GB	No	No	No	No	No
Q8_0	~570 GB	No	No	No	No	No
Q4_K_M	~290 GB	No	No	No	No	Yes
Q3_K_M	~220 GB	No	No	Yes	Yes	Yes
Q2_K	~170 GB	No	No	Yes	Yes	Yes

The reading is simple: for Q4 you need at least 4 GPUs of 96 GB (or 8 of 24 GB with slow offloading); for Q3 just 8× 24 GB will do. The recipe officially recommended by inclusionAI is 8× H200 with tensor parallelism 8 + expert parallelism 8, or 4× GB300 with TP4/EP4.

In practice, with 24 GB of VRAM you have three honest options:

1. API or cloud: you use Ling 3.1 Flash as a service. On OpenRouter it costs $0.075 per million input tokens and $0.225 per million output tokens (cache read $0.015/M), with a 262K-token served context. It's the fastest way to experiment.
2. Multi-GPU at home: 2× 48 GB (A6000) or 4× 96 GB (H200/MI300X) for Q4. It's the "home" setup closest to the official recipe.
3. Smaller MoEs on 24 GB: the most realistic path. MoE models with 20–40B active parameters (for example Qwen3's 30B-A3B) run at Q4 on a single 24 GB GPU at 10–20 tokens/s. It's not Ling 3.1 Flash, but the MoE principle is the same and the cost is a fraction.

The key point: Ling 3.1 Flash is not "too big for 24 GB" in an absolute sense — it's the reference point against which to measure what your VRAM can actually support.

What the API costs, in euros

Let's do the math with today's OpenRouter prices (1 USD ≈ 0.93 EUR):

Scenario	Input tokens	Output tokens	Cost
1 request, 100K in / 10K out	100,000	10,000	~$10 (≈ €9)
1 request, 1M in / 30K out	1,000,000	30,000	~$305 (≈ €284)
10 requests/day of 100K	1,000,000	100,000	~$100/day (≈ €93)
100,000 requests/month (100K in each)	10,000,000,000	—	~$750,000/month

The last row is deliberately exaggerated: it's there to show that beyond a certain threshold the API becomes economically absurd. For occasional use (a few hundred thousand tokens per day), the API is still the most sensible choice and it won't cost you 40,000 € of GPUs.

And the limits?

Let's be honest, because honesty is worth as much as enthusiasm:

• 24 GB is not enough for the whole model, not in Q4 and not in Q3: you need multi-GPU or the API.
• The 1M context is native, but the KV cache grows with length: on consumer hardware, practical usage will stop well short of a million tokens, depending on remaining VRAM and quantization.
• MoE + offloading: loading weights to RAM/SSD works, but generation speed drops noticeably compared to a fully-VRAM setup.
• The official benchmarks (Terminal-Bench 4.0, SWE-bench Pro, FrontierSWE) are measured on H200/GB300: the consumer-hardware numbers are not declared and you can't expect the same performance.
• MIT license ≠ absence of responsibility: the license covers the weights, not the data you feed it. Responsibility for what you send to the model is still yours.

If your use case doesn't justify the hardware, the API is the pragmatic choice. If you're building a serious local infrastructure, Ling 3.1 Flash is the reference point to design against.

How to try it, if you have the right hardware

The officially recommended path by inclusionAI is SGLang, with a day-0 image that includes support and MTP speculative decoding. For an 8-GPU setup (TP8 + EP8):

docker pull lmsysorg/sglang:dev-ling-3.1-flash
docker run --gpus all --network host --ipc=host --rm \
  lmsysorg/sglang:dev-ling-3.1-flash \
  python3 -m sglang.launch_server \
  --model-path inclusionAI/Ling-3.1-flash \
  --tp 8 --ep 8 \
  --speculative-algorithm NEXTN \
  --host 0.0.0.0 --port 30000

And the verification:

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Ling-3.1-flash","messages":[{"role":"user","content":"Hello"}],"stream":false,"temperature":1.0,"top_k":20,"top_p":0.95,"chat_template_kwargs":{"enable_thinking":true}}'

If the port is already in use, change --port; if the server goes OOM, lower the context (--context-length) or use Q4 instead of FP8. vLLM works as an alternative, with --enable-expert-parallel and the tool parser ling3.

Sources

• inclusionAI/Ling-3.1-flash model card (Hugging Face) — full specs, architecture, SGLang/vLLM recipe.
• OpenRouter — inclusionAI/ling-3.1-flash — API pricing and served context.
• Pandaily — Ant Group open-sources Ling-3.1-flash weights under MIT — launch context.

Related articles

• Ollama vs LM Studio: which tool to choose for local AI — the practical comparison between the two most widely used tools.
• Mac Studio M5 Ultra: 512 GB of unified memory for the desktop — if you want to get close to big MoEs in a single chassis.
• Aleph Alpha's Kolibri: the 78B MoE that runs (almost) locally — the smaller counterpart of the same MoE principle.

In short

Ling 3.1 Flash is the largest open MoE model available under the MIT license: 560B total, 25B active, 1M context, native FP8. With 24 GB of VRAM you can't run the whole thing — you need 2× 48 GB or more, or the API. But it's exactly the reference point you need to understand where your hardware stops and what you can realistically do instead: smaller MoEs at 10–20 tokens/s, or the cloud for the peaks. MIT means you can adopt it without asking anyone. The numbers are on the table: choose according to your use case.

Article generated with AI