Inference⚡
llama.cpp
Released Oct 15, 2022
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
LLMC++LocalEfficient
Back to resources
Inference⚡
SGLang
Released on Jan 18, 2024
Open-source serving engine for LLMs with RadixAttention, continuous batching and first-class speculative decoding (MTP, EAGLE, draft models, ngram). Apache-2.0 license.
Features
- RadixAttention
- Continuous batching
- Speculative decoding (MTP/EAGLE/draft/ngram)
- Multi-GPU
- OpenAI API
- High-throughput serving
Version history
- v0.1.5Latest
Prima release pubblica di SGLang sul repository GitHub (sgl-project/sglang, creato a gennaio 2024).
Tags
Useful links
Discover SGLang
Visit the official site to download, learn more and start using SGLang.
Related resources — Inference
Inference🦙
Ollama
Released Jul 27, 2023v0.35.0
Open source tool to run LLMs locally with simplicity.
LLMLocalOpen SourceCLI
Inference🗣️
Piper TTS
Released Apr 1, 2022Piper
Sintesi vocale in locale a bassa latenza basata su ONNX: genera parlato naturale da testo in tempo quasi reale, anche su CPU. Multilingue (italiano incluso) e facile da integrare nei pipeline: download del modello, una riga di comando. MIT.
AudioTTSOpen SourceLocale