News

Open-source tool (Rust) acting as a local gateway: it swaps outbound secrets with equal-length realistic mocks and restores them on the way back, so the LLM never touches a real value. One-line integration. Needs primary-source verification.

LLMspeed is an open-source tool (Hacker News showcase) that estimates Inference speed for 250+ model/hardware combinations: RTX 3060 to H100, Mac Studio M5 Ultra, CPU-only. Measured estimates come from community benchmarks (r/LocalLLaMA, localscore.ai); the rest are derived from similar setups or projected from specs. Useful for choosing hardware before spending.

sift is a zero-dependency Python tool that detects a file's real type by content (not just extension), extracts its structure, and asks a local LLM — Ollama, LM Studio or MLX-LM — to classify and summarize it. Nothing leaves the machine; it counts its own numbers and reports files that talk to the assistant.

Build b11716 adds Q1_0 and Q2_0 Quantization support to the Hexagon backend (Snapdragon NPU), extending mobile NPU Inference to the most aggressive quantizations. Pre-release, October 17.

An online index collects local models and shows which run on your machine based on available RAM and VRAM: useful before buying hardware for local AI.

Now that AI assistants are part of daily use, Linux kernel patches with AI/LLM contributions had to declare it with the «Assisted-by:» tag. With this tooling now ubiquitous, kernel developers are looking at dropping the requirement — the declaration becomes a choice, not a mandate.

A GitHub project running multi-agent AutoML workflows entirely on Ollama - no API calls, no cloud. Verified against the primary source; early-stage (low HN signal).

An open-source tool that schedules several local LLMs on the same GPU, keeping them from fighting over VRAM in turns. Useful for people running multiple models (an assistant, an embedder, a classifier) on one card who want predictable latencies.

A well-argued article explains why a local model that stalls, ignores the System prompt or keeps generating past its answer is almost always a chat-template or Token-metadata problem in the GGUF file, not a weights problem — and how to fix it with GGUF-surgeon without re-quantizing.

An open-source routing proxy (Apache 2.0) that cascades LLM requests: it starts with the local model on Ollama and climbs to the cloud only when needed, with semantic caching and per-hop reliability tracking. An interesting pattern for those who want local by default.

Launched on Hacker News (YC S25), Magnitude is a Rust Inference engine that compiles and tunes its kernels directly on your target hardware. On open models it claims up to 2× the speed of llama.cpp, on Apple Silicon, NVIDIA, AMD or CPU. Verified against the official repository.

Ollama v0.34.0 lets you use local Ollama models directly in ChatGPT Desktop on macOS, improves structured output performance on Apple Silicon, and adds OpenAI-compatible client tool search and response compaction.

The Ollama app again follows the system appearance (dark mode restored), an already-running macOS instance is no longer duplicated, and the Claude Desktop proxy no longer interrupts in-flight requests when the model catalog updates.

The release adds Claude Desktop setup with Ollama as a third-party gateway provider, plus major caching improvements: Prefill restore points make retries far more efficient and fix a hang when agent clients cancel long prefills.

The Ollama release adds support for new open-source models and measurable Inference performance improvements. Tip: after every update, restart the server and verify your models with `Ollama list`.