Tools, frameworks, and models for local AI
8 resources found for "GPU"
Reimplementation of Whisper based on CTranslate2: up to 4 times faster than openai/whisper with the same accuracy and less memory. Supports 8-bit quantization on both CPU and GPU.
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
Serverless platform for running code and AI models on GPUs: you pay for real seconds of use, it scales to zero when idle, and you can run any open-weights model. One of the routes to inference on rented hardware without managing servers.
Inference API for OpenAI's frontier models (GPT series). You pay per token: a fixed price per million input and output tokens, no GPUs to manage, with rate limits per account tier.
Serverless platform for deploying and auto-scaling AI models with GPUs.
Open-source serving engine for LLMs with RadixAttention, continuous batching and first-class speculative decoding (MTP, EAGLE, draft models, ngram). Apache-2.0 license.
A marketplace for renting GPUs: GPU owners put their capacity up for sale and rent it out by the hour or by period, at prices that follow demand. A middle path between frontier cloud and your own hardware.
High-throughput LLM serving with PagedAttention and hardware optimizations.
