Agent infrastructure
nano-vllm
GeeeekExplorer/nano-vllm
A lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
Topics
Related projects
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
LMCache12.3kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.
vllm-omni7.5kEfficient inference-serving framework for omni-modal models, unifying text, image, audio, video, and action inputs under one serving architecture for developers deploying multimodal applications.
faster-whisper26.7kA fast Whisper transcription library built on CTranslate2, with quantization, batching, VAD filtering, and word timestamps.
deeplake9.3kAn AI database and multimodal data lake serving as serverless vector store for retrieval and training.