Resources

Tools, frameworks, and models for local AI

All Organizations Hardware Software Models Inference Frameworks

Sort by:Name (A-Z)Name (Z-A)Category Newest first Oldest first Price (low to high)Price (high to low)VRAM (highest first)Parameters (largest first)

2 resources found for "High-throughput"

Inference⚡

SGLang

Released Jan 18, 2024v0.1.5

Open-source serving engine for LLMs with RadixAttention, continuous batching and first-class speculative decoding (MTP, EAGLE, draft models, ngram). Apache-2.0 license.

LLMServingGPUHigh-throughput

Learn more

CompareSite

Inference🚀

vLLM

Released Jun 13, 2023v0.30.0

High-throughput LLM serving with PagedAttention and hardware optimizations.

LLMServingGPUHigh-throughput

Learn more

CompareSite