Agent infrastructure
pooled
Nehanth/pooled
Peer-to-peer browser inference that pools devices to run large open models with an OpenAI-compatible API.
Topics
Related projects
web-llm19.3kA high-performance in-browser LLM inference engine with WebGPU acceleration and OpenAI-compatible APIs for building private web AI apps.
nano-vllm15.8kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
LMCache12.1kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
inference9.6kA unified model-serving library to deploy LLMs and multimodal models locally or in the cloud via OpenAI-compatible API.
rapid-mlx4kA local LLM inference server and Mac app for Apple Silicon with OpenAI and Anthropic APIs, tuned for coding agents.
qwen38-27b-rtx30902kA tuned vLLM serving stack that runs Qwen 3.8 27B fast on a single RTX 3090 with long context and an OpenAI-compatible API.