Inference⚡
llama.cpp
Released Oct 15, 2022
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
LLMC++LocalEfficient
Back to resources
Inference
Yzma
Open-source inference engine in pure Go that embeds llama.cpp in-process via purego/ffi (no CGo). Reads GGUF models and runs on CUDA, Metal, Vulkan, OpenVINO and in-browser WebGPU. Covers roughly 96% of llama.cpp's functionality; optional server with an OpenAI-compatible API.
Features
- In-process llama.cpp
- GGUF
- OpenAI-compatible server
- WebGPU
Version history
- 1.30.0Latest
Version usata nei benchmark di riferimento pubblicati sul sito del progetto: embedding bge-small a 1,7 ms contro 4,5 ms di Ollama 0.40.5.
Tags
Useful links
Discover Yzma
Visit the official site to download, learn more and start using Yzma.
Related resources — Inference
Inference🦙
Ollama
Released Jul 27, 2023v0.35.0
Open source tool to run LLMs locally with simplicity.
LLMLocalOpen SourceCLI
Inference🗣️
Piper TTS
Released Apr 1, 2022Piper
Sintesi vocale in locale a bassa latenza basata su ONNX: genera parlato naturale da testo in tempo quasi reale, anche su CPU. Multilingue (italiano incluso) e facile da integrare nei pipeline: download del modello, una riga di comando. MIT.
AudioTTSOpen SourceLocale