Agent infrastructure
lmdeploy
InternLM/lmdeploy
A toolkit for compressing, deploying, and serving LLMs with fast inference engines, quantization, and distributed serving.
Topics
Related projects
lemonade6.2kA local AI server that runs optimized LLMs on your own GPU and NPU and serves them to apps via standard OpenAI-compatible APIs.
LocalAI50.3kAn open-source engine to run LLMs, vision, and audio models locally with OpenAI-compatible APIs.
airllm42.3kA Python inference engine that streams huge language models layer by layer to run 70B to 671B models on small GPUs.
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
OpenLLM12.6kAn open-source platform for self-hosting LLMs and serving them as OpenAI-compatible APIs in the cloud.
shimmy6kA tiny pure-Rust local inference server for GGUF models with an OpenAI-compatible API and WebGPU acceleration.