News

A near-frontier mobile agent is published with fully open training environments and recipes: MobileGym++ simulates 35 commercial apps with cross-app workflows, with controllable reset and verification. Relevant for anyone building on-device agents.

LLMspeed is an open-source tool (Hacker News showcase) that estimates Inference speed for 250+ model/hardware combinations: RTX 3060 to H100, Mac Studio M5 Ultra, CPU-only. Measured estimates come from community benchmarks (r/LocalLLaMA, localscore.ai); the rest are derived from similar setups or projected from specs. Useful for choosing hardware before spending.

Build b11717 adds Hadamard MUL_MAT and GET_ROW operations for Q1_0 and Q2_0 quantizations (flat and tiled) to the Hexagon backend (Snapdragon NPU), and reduces packet counts in the q8_1 quantizer. Qualcomm contribution: a further step toward mobile NPU Inference for the most aggressive quantizations.

Open-source tool (Rust) acting as a local gateway: it swaps outbound secrets with equal-length realistic mocks and restores them on the way back, so the LLM never touches a real value. One-line integration. Needs primary-source verification.

Moose 0.6 adds protected credential storage for cloud services, a Local-Only mode with an app-managed Ollama (model download and removal), per-chat or per-provider permissions, a GNOME Files browser to attach documents to conversations, and an expanded local model catalog. 0.6.1 adds Mistral and DeepSeek. Available on Flathub.

FLUX 3 Image Klein is a 5.6B open-weights model for text-to-image generation and editing: one model for prompting, single-reference edits and multi-reference composition, with placement boxes in the prompt. Ships in BF16, FP8 and NVFP4 (5.3GB).

Build b11716 adds Q1_0 and Q2_0 Quantization support to the Hexagon backend (Snapdragon NPU), extending mobile NPU Inference to the most aggressive quantizations. Pre-release, October 17.

The KyutaiSTT module brings Kyutai's streaming speech recognition to Apple Silicon via MLX. The reference checkpoint is the STT-1B (English and French) quantized to 8-bit, with a causal Mimi encoder producing 32 audio codes every 80ms. CC-BY-4.0 license.

sift is a zero-dependency Python tool that detects a file's real type by content (not just extension), extracts its structure, and asks a local LLM — Ollama, LM Studio or MLX-LM — to classify and summarize it. Nothing leaves the machine; it counts its own numbers and reports files that talk to the assistant.

A new Benchmark measuring how well video generation models keep a Coherent memory of the world they generate: previously observed content preserved and coherent evolution over time, alongside visual fidelity and duration.

NVIDIA has published MagpieTTS Multilingual on Hugging Face, a compact 357M-Parameter speech synthesis model covering 12 languages, including Italian and French. Encoder-decoder architecture, designed for low-Latency local execution.

An online index collects local models and shows which run on your machine based on available RAM and VRAM: useful before buying hardware for local AI.

Google DeepMind and GovTech Singapore use lightweight on-device Gemma models to intercept scams in private chats, without sending data to the cloud.

inclusionAI releases Ling 3.1 Flash: 560B total parameters, 25B active per Token, hybrid MoE architecture, up to a 1M-Token Context Window and an MIT license. At 4 bits it fits in 512GB of Unified memory — the edge case of today's Mac Studio M5 Ultra article.

NVIDIA publishes Kumo-Tabular on Hugging Face, a 500-million-Parameter foundation model for tabular data with OpenMDW 1.1 support and ready-to-use checkpoints: an example of local AI on structured business data.

Autotrust releases JEV-35B, a 35B-Parameter (3B active) MoE Decision model fine-tuned on Qwen3.5-35B-A3B for text classification and risk scoring. In NVFP4 Quantization it occupies about 25.6GB of VRAM: it runs on a 32GB GPU without going through the frontier.

ARDY is a 326M-Parameter autoregressive diffusion model for interactive real-time humanoid motion generation: online text prompting, flexible kinematic constraints (root paths, keyframes, sparse poses) and sequences up to 8 seconds long. The Core-RP-20FPS-Horizon40 checkpoint, trained on the Bones Rigplay 1 dataset, is on Hugging Face since October 7 and optimized for NVIDIA Ampere, Hopper and Blackwell GPUs.

ClawLoop (LongCat) trains sandbox LLM agents through a self-sustaining three-phase cycle — task synthesis, SFT/RL training, rollout diagnosis — where the diagnosis feeds the next task synthesis, removing the external model that normally synthesizes tasks and trajectories. With Qwen3.6-27B it improves on SkillsBench, Claw-Eval-Live and ClawGym-Bench; the refined tasks transfer with gains even to Qwen3.5-4B and 9B.

Now that AI assistants are part of daily use, Linux kernel patches with AI/LLM contributions had to declare it with the «Assisted-by:» tag. With this tooling now ubiquitous, kernel developers are looking at dropping the requirement — the declaration becomes a choice, not a mandate.

OpenBMB releases MiniCPM-V 4.7, 35B-A3B variant: the first open-source MoE able to Decode smoothly on a smartphone without a quality compromise — 13+ tokens/second with only 3GB of memory on an iPhone 17 Pro Max. The secret: Lookahead Expert Scheduling on llama.cpp-omni, combined with fine-grained layer-wise Quantization and speculative sampling. Day-0 support for Rockchip, Intel, AMD and Arm; Ollama, llama.cpp, vLLM and SGLang.

A new Hugging Face paper shows how terminal agents are moving beyond local operations toward real professional tasks - accumulating evidence, adjusting strategy, and turning intermediate results into final deliverables.

New paper on Hugging Face: aligning the latent representations of image generators with different architectures (diffusion and flow matching) is still an open problem. InstantFusion proposes a shared latent interface to make generators interoperable.

A GitHub project running multi-agent AutoML workflows entirely on Ollama - no API calls, no cloud. Verified against the primary source; early-stage (low HN signal).

Open-source project on GitHub: a local tutor that studies the books and documents you give it and teaches the content with quizzes and questions, using a local LLM. Designed for people who want to learn without sending their materials to a cloud.

A Transformer trained to predict the conditional marginals of missing variables, the key ingredient for computing Bayes risk and the value of one more observation before deciding under incomplete information.

Hugging Face hosts ej, a small 11.4 MB calibrated Decision model with open weights: given a state and typed questions, it returns one probability distribution per question in a single pass, without decoding tokens. Designed to run on CPU with a 128 MB resident-memory peak.

A new paper shows a single Transformer block, applied recurrently, matches a full-depth vision encoder's accuracy at comparable Inference FLOPs. The key is representing the FFN at each recurrent depth as a convex combination of a small shared expert bank, with no intermediate feature distillation.

A new paper shows how a multimodal model can internalize the spatial reasoning ability of a tool-using agent, cutting Inference cost: the model learns from verified traces of its own executions, with no reliance on external tools.

A method to monitor and intervene in real time on LLM agent trajectories, using streaming structure-aware optimal transport: it avoids the cost and Latency of a per-step safeguard agent while staying proactive rather than examining logs after the fact.

Memento 3, the continuation of the Memento series, lets frozen-LLM agents continually learn explicit world models in unfamiliar environments, using a reflective rulebook that updates as new evidence arrives.

A new Hugging Face paper proposes that LLM agents work directly on workspace files through dedicated tabs, with no indexing: the agent searches and reads documents from a terminal and produces deliverables from multiple files in one flow.

The Google AI Edge team released ML Drift, a cross-platform GPU compute engine for on-device Inference under the Apache 2.0 license. It covers OpenGL ES, OpenCL, Metal and WebGPU, supports 5D tensors for 3D networks and generative models, and is the new acceleration core of LiteRT.

On LocalLLaMA, one author describes building the architecture behind the Jev decision models a year ago and open-sourcing it completely: model, dataset and paper. Backstory that accompanies the arrival of the System One format in llama.cpp and the opening of the decision-model ecosystem to the community.

The llama.cpp server now supports decision models: send a state (text, JSON or a screenshot) plus typed questions, and the model returns a probability for each option in a single forward pass, without generating text. Three question types (choice, score, noul) and six models from 144M (Julia-1) to 27B (OpenJev, Clef), answering in 3-43 ms on an RTX PRO 6000, Apache 2.0 except OpenJev (CC BY-NC).

Aleph Alpha's new sovereign Open-weight model, trained from scratch in Germany: 78B parameters (3.46B active) with up to 1M tokens of context. Built for mission-critical environments - multi-step reasoning, structured extraction, RAG and reliable tool calling, with native German and English. Weights are on Hugging Face, with a full technical report.

A study of tool-using agents in a Retrieval environment with controlled source failures: when a tool keeps returning useless results, the agent judges them as useless but rarely turns that judgment into a stopping decision. The contrast between stopping after long and short runs of judged-useless results is zero for clock- or deadline-driven stopping — it measures the gap that separates today's models.

The executable harness around a GUI model — how observations are assembled, actions executed, and verification, recovery and termination controlled — determines most of an agent's behavior. GUI-HARVEST optimizes it automatically from execution evidence, tackling reconciling model intent with observed visual effects, diagnosing failures under variable execution, and controlling termination.

An open-source tool that schedules several local LLMs on the same GPU, keeping them from fighting over VRAM in turns. Useful for people running multiple models (an assistant, an embedder, a classifier) on one card who want predictable latencies.

The PaLoRA paper shows that the step-size restriction in LoRA-based continual learning should grow with the effective rank of accumulated knowledge, and proposes an adaptive pacing scheme gaining 4% accuracy on 50-task benchmarks.

Kandinsky Lab publishes the Kandinsky 6.0 Video paper: foundation models for synchronized video and audio generation, the day's #1 paper on Hugging Face with 8 models and 6 Spaces already linked.

Lightricks releases LTX-2.5, open weights: video generation with synchronized audio from text, image, or video, with native multishot that holds identity and style across scenes, a new diffusion video decoder replacing the VAE stage, and a Gemma 4 12B text encoder. The weights are built for running and Fine-tuning on your own infrastructure.

A native macOS app turns dictation into text with Whisper running on-device plus a local LLM (Qwen) rewrite pass on the M-series GPU: no byte leaves the computer, no cloud account. Audio goes through Whisper, then the local model cleans up punctuation and formatting before delivering text to the target app.

On GitHub, an experimental project offloads part of local LLM Inference to a paired iPhone, using the phone's GPU and memory as a secondary accelerator — an idea for anyone wanting more tokens per second without swapping graphics cards.

Moose is a new native GTK4/libadwaita app for GNOME, written in Rust under GPLv3, that lets you chat with local models via Ollama: it handles Ollama setup, model management and loading local documents and images. It's on Flathub, the code is on GitHub, and it was announced on This Week in GNOME.

A small open-source router (v0.4.0) that adds multimodal capabilities to a text-only LLM model: images and audio are handled at the routing layer, so the local model stays the one you already run, without swapping it out.

New research paper: when a user changes their mind during a multi-turn conversation, superseded parts of the intent keep influencing the final answer (intent drift). The paper introduces IntentFlux, an executable Benchmark that converts verifiable tasks into dialogues with controlled intent changes, and proposes methods to detect and repair the failure mode.

A Hugging Face community paper explores how the implicit priors of agent models can guide policy learning, reducing the cost of training LLM agents in structured domains.

llama.cpp b11337 widens the CPU/edge backend matrix: Snapdragon (Adreno and Hexagon), OpenVINO, SYCL FP16/FP32, CUDA 13, Android, and ROCm 10, plus server fixes. It is the eighth release in the October 2 batch.

A well-argued article explains why a local model that stalls, ignores the System prompt or keeps generating past its answer is almost always a chat-template or Token-metadata problem in the GGUF file, not a weights problem — and how to fix it with GGUF-surgeon without re-quantizing.

An open-source routing proxy (Apache 2.0) that cascades LLM requests: it starts with the local model on Ollama and climbs to the cloud only when needed, with semantic caching and per-hop reliability tracking. An interesting pattern for those who want local by default.

University of Central Florida researchers propose TACO, a ternary column-wise one-sparse optimizer for LLM Fine-tuning: it keeps first-order gradients while shrinking persistent optimizer state from 27.7 GB to 0.16 GB on OPT-13B.

The vLLM project tags proto-v0.4.0, the experimental development track that precedes the stable releases of the language-model Inference engine. One more step for anyone serving local models in production.

A new method, WUSH-KV, cuts the memory and bandwidth the KV cache takes up in long-context Inference by quantizing it at low precision with data-calibrated transforms. At 2 bits, the approach matches or beats the OSCAR transform across all tested models and tasks. It's a sign that memory optimization is no longer only about the weights, but also the context memory.

Launched on Hacker News (YC S25), Magnitude is a Rust Inference engine that compiles and tunes its kernels directly on your target hardware. On open models it claims up to 2× the speed of llama.cpp, on Apple Silicon, NVIDIA, AMD or CPU. Verified against the official repository.

With Ollama 0.35 you can use decision models based on TypeSafe's Jev API: send text as a state with a set of named questions and a local model answers them all in one request, with a probability per option and no free-form text generation. Built for ticket triage, model routing, and content moderation.

Ollama 0.35 adds decision-model support: Bespoke Labs' 9B nimble model runs entirely locally and answers typed questions (true/false, numeric scale, multiple choice) through the new systemone API.

A well-documented experiment: 67 course videos transcribed locally and offline with whisper-large-v3-turbo rebuilt on MLX for the Mac's GPU — 17 hours of video processed in 34 minutes, without a single byte leaving the machine.

The September 2026 ranking orders Open-weight models by VRAM tier: GPT-OSS 20B and ZAYA1-8B lead the 8-16 GB tier, Qwen3.6-27B (46.49) and Gemma 4 31B (44.84) dominate the 16-24 GB tier on a single RTX 4090 at Q4, and trillion-Parameter MoEs need 8x H100. A useful reference when choosing a model size for your hardware.

Qwen3Guard-Stream is a 0.6B moderation model with a Token-level classifier for real-time monitoring during incremental generation. Three severity tiers (safe, controversial, unsafe) and 119 languages: a small, local component to filter outputs without sending them to the cloud.

A creative experiment in Vilnius: a wall with microphones and speakers powered by a fully local LLM pipeline. Participants talk to the wall and get responses generated on site — no cloud, no external servers.

DeepSeek V4.1 Flash pushes the MoE architecture further: 552B total parameters but only 8-16B active per Token, a 1M-Token Context Window, and an MIT license. A model built for online serving, but useful as a reference for learning to read model spec sheets.

A series of 21 patches for the open-source AMDXDNA driver extends NPU3 (Ryzen AI 300) support: kernel-mode command submission, full suspend/resume handling and Telemetry, paving the way for NPU-based local AI on Linux.

Gemma 4, released in March 2026, marks a license change for Google's family: for the first time the weights are under Apache 2.0. Available in E2B, E4B, 26B A4B (MoE) and 31B sizes, with context windows up to 256K tokens and text, image, and audio input. The 12B Unified followed in June — an ideal size for PCs with consumer GPUs.

Prism ML releases Bonsai-2 27B in GGUF: a 27B reasoning model with Ternary weights at 1.72 bits per weight, 5.95 GB and a 262K-Token context, retaining 98.2% of the FP16 intelligence and reaching ~47 tok/s on an Apple M5 Max laptop.

The PISA paper proposes a pyramid Top-K selection strategy that brings block sparse attention to O(N log N) complexity, cutting the quadratic bottleneck of block selection; Triton kernels for training and Inference.

Open WebUI release v0.11.4, published on September 21, brings two direct RAG improvements: metadata attached to uploaded files now travels with each chunk and arrives with the retrieved sources, and hybrid search now runs natively in the database (pgvector) instead of loading the whole collection into memory, making large knowledge bases dramatically faster. The slim image also drops to around 175 MB.

A new open speech-to-text model: real-time streaming with low Latency, built for live dictation and captions. One more step toward audio that genuinely runs locally, without going through the cloud.

Ollama's v0.34.4 release applies structured outputs on thinking models in a single pass (faster and more reliable), fixes 'model not found' errors with large local libraries, and speeds up Qwen 3.8 prompt processing on Apple Silicon. A useful update if you run Ollama in production.

NemotronLabs VoiceChat 11B is an end-to-end full-duplex voice model: audio understanding and generation in a single architecture, without the cascaded ASR→LLM→TTS stack. It is the first open full-duplex model to support tool calling while keeping the natural flow: ~450ms responses, interruptions, and natural turns. The code is on GitHub.

Mozilla and Mistral announced a partnership integrating Mistral's AI — open, multilingual models with a privacy focus — into Firefox Smart Window, the browser's AI panel. A signal in the direction of today's article: bringing AI into everyday work tools, with data staying under the user's control as much as possible.

The weekly report on the llama.cpp project covers the current build b4991 and the week's activity: a bug where Gemma 4 in Thinking mode emitted progressively longer trailing garbage was resolved by trimming whitespace in the chat template; the Vulkan CI runners moved to the NVIDIA r615 driver, and the report analyzes the community's most active issues.

Qwen-Image 2.1 is a 7B Open-weight model (32-layer DiT) that unifies text-to-image generation up to 2048x2048, editing with up to 10 reference images, and native RGBA transparency. With ~14GB of VRAM in bfloat16 it runs on consumer hardware, and ComfyUI supports it natively: a concrete step for cloud-free image production.

The vLLM project released version 0.29.0, with 594 commits from 277 contributors. The main highlight is Model Runner V2, now the default for all models, with CUDA graph memory profiling for KV cache auto-sizing. New model support also lands: Qwen3.8-Flash-Next, Hy4-preview (a 770B MoE with native MTP), Granite SWA, and Kimi K3 checkpoints in NVFP4.

Alibaba's Qwen team released Qwen3.8-Omni-Flash, a native omnimodal model that jointly handles text, images, audio and video with a one-million-Token Context Window. It is built around agentic capabilities: it plans tasks, calls tools and produces finished audio-visual content. The Qwen family keeps growing right as this guide analyzes it for local use.

Open WebUI v0.11.1 adds tool-call approvals (a pause button between the model and its tools), a built-in ask_user function that lets the model pause and ask you a question mid-answer, a terminal file browser with in-place document previews, and Delta streaming that cuts reply traffic by roughly 99%. The full release includes more than ninety additions and over two hundred fixes.

Ollama v0.34.0 lets you use local Ollama models directly in ChatGPT Desktop on macOS, improves structured output performance on Apple Silicon, and adds OpenAI-compatible client tool search and response compaction.

Gemma4 can now process images and audio on the MLX engine. Also, cached prompt tokens are now reported correctly and default parameters defined in GGUF files are honored.

The Ollama app again follows the system appearance (dark mode restored), an already-running macOS instance is no longer duplicated, and the Claude Desktop proxy no longer interrupts in-flight requests when the model catalog updates.

New Qwen3.8 Flash Next support on the MLX engine, structured output for mlxrunner, and a fix for Metal GPU timeouts when loading models from slow storage.

The release adds Claude Desktop setup with Ollama as a third-party gateway provider, plus major caching improvements: Prefill restore points make retries far more efficient and fix a hang when agent clients cancel long prefills.

Meta released Llama 4, the new Open Source language model with 100 billion parameters. Performance surpasses GPT-4 on many benchmarks and the model is optimized for local execution with Quantization.

Apple updated the MLX framework with native support for 4-bit quantized GGUF models, allowing larger models to run on M-series chips.

The Ollama release adds support for new open-source models and measurable Inference performance improvements. Tip: after every update, restart the server and verify your models with `Ollama list`.