Inference⚡
llama.cpp
Released Oct 15, 2022
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
LLMC++LocalEfficient
Resources
Tools, frameworks, and models for local AI
6 resources found for "LLM"
Tools, frameworks, and models for local AI
6 resources found for "LLM"
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
Open-source inference engine (Rust, YC S25) that compiles and tunes its kernels directly on the target hardware: Apple Silicon, NVIDIA, AMD or CPU. On open models it claims up to 2× the speed of llama.cpp. Also aimed at serving agents.
Open source tool to run LLMs locally with simplicity.
Cloud gateway for LLM inference: a single API endpoint for hundreds of models (including open-weight MoE), with per-token pricing and automatic routing across providers. For Ling 3.1 Flash: $0.075/M input, $0.225/M output.
Open-source serving engine for LLMs with RadixAttention, continuous batching and first-class speculative decoding (MTP, EAGLE, draft models, ngram). Apache-2.0 license.
High-throughput LLM serving with PagedAttention and hardware optimizations.