Agent infrastructure
sglang
sgl-project/sglang
A fast open-source inference framework for serving large language and multimodal models at scale.
Topics
Related projects
freetoken16.8kFreeToken is an edge-native Mixture-of-Experts inference engine that runs large open-weight models on consumer hardware, with a desktop app, CLI, and OpenAI/Anthropic-compatible APIs.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
LMCache12.3kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
Rapid-MLX4.1kA local LLM inference server and Mac app for Apple Silicon with OpenAI and Anthropic APIs, tuned for coding agents.
project-maya2.6kProject Maya is a local LLM inference engine that runs large GLM, DeepSeek, and MiniMax models on your own GPUs, with quantized weights, a web dashboard, and OpenAI/Anthropic-compatible APIs.
willow-inference-server511Willow Inference Server is a self-hosted, CUDA-optimized inference server exposing Whisper speech recognition, text-to-speech and LLM inference over REST, WebRTC and WebSocket APIs.