Agent infrastructure
vllm-omni
vllm-project/vllm-omni
Efficient inference-serving framework for omni-modal models, unifying text, image, audio, video, and action inputs under one serving architecture for developers deploying multimodal applications.
Topics
Related projects
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
whisper.cpp55.1kA dependency-free, high-performance C/C++ port of Whisper for on-device speech recognition, with quantization, broad hardware acceleration, and many language bindings.
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
freetoken16.8kFreeToken is an edge-native Mixture-of-Experts inference engine that runs large open-weight models on consumer hardware, with a desktop app, CLI, and OpenAI/Anthropic-compatible APIs.
LMCache12.3kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.