Inference⚡
llama.cpp
Released Oct 15, 2022
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
LLMC++LocalEfficient
Back to resources
Inference⚡
ML Drift
Google's open-source (Apache 2.0) on-device GPU inference engine, successor to the TensorFlow Lite GPU delegate: tensor virtualization, stage-aware KV cache for prefill and decode, FP16/INT8/INT4. Backends: OpenCL (Android), Metal (Apple), WebGPU via Dawn (browser), OpenGL ES 3.1+. Integrates with LiteRT; in production on Chrome, YouTube, Photos and Meet.
Features
- Apache 2.0
- Tensor virtualization
- Stage-aware KV cache (prefill/decode)
- FP16 / INT8 / INT4
- OpenCL, Metal, WebGPU (Dawn), OpenGL ES 3.1+ backends
Tags
Useful links
Discover ML Drift
Visit the official site to download, learn more and start using ML Drift.
Related resources — Inference
Inference🦙
Ollama
Released Jul 27, 2023v0.35.0
Open source tool to run LLMs locally with simplicity.
LLMLocalOpen SourceCLI
Inference🗣️
Piper TTS
Released Apr 1, 2022Piper
Sintesi vocale in locale a bassa latenza basata su ONNX: genera parlato naturale da testo in tempo quasi reale, anche su CPU. Multilingue (italiano incluso) e facile da integrare nei pipeline: download del modello, una riga di comando. MIT.
AudioTTSOpen SourceLocale