Inference🦙
Ollama
Released Jul 27, 2023v0.35.0
Open source tool to run LLMs locally with simplicity.
LLMLocalOpen SourceCLI
Back to resources
llama.cpp
Released on Oct 15, 2022
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
Features
- CPU/GPU offload
- GGUF format
- Low memory
- Fast inference
- AVX/AVX-512/AMX CPU support
Tags
Useful links
Discover llama.cpp
Visit the official site to download, learn more and start using llama.cpp.
Related resources — Inference
Inference🗣️
Piper TTS
Released Apr 1, 2022Piper
Sintesi vocale in locale a bassa latenza basata su ONNX: genera parlato naturale da testo in tempo quasi reale, anche su CPU. Multilingue (italiano incluso) e facile da integrare nei pipeline: download del modello, una riga di comando. MIT.
AudioTTSOpen SourceLocale
Inference🚀
vLLM
Released Jun 13, 2023v0.30.0
High-throughput LLM serving with PagedAttention and hardware optimizations.
LLMServingGPUHigh-throughput