Agent infrastructure
freetoken
flashml-org/freetoken
FreeToken is an edge-native Mixture-of-Experts inference engine that runs large open-weight models on consumer hardware, with a desktop app, CLI, and OpenAI/Anthropic-compatible APIs.
Topics
Related projects
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
whisper.cpp55.1kA dependency-free, high-performance C/C++ port of Whisper for on-device speech recognition, with quantization, broad hardware acceleration, and many language bindings.
nano-vllm16.3kA lightweight vLLM reimplementation in about 1,200 lines of Python delivering fast, readable offline LLM inference.
server11.1kNVIDIA's open-source inference serving software for deploying AI models efficiently across cloud and edge.
vllm-omni7.5kEfficient inference-serving framework for omni-modal models, unifying text, image, audio, video, and action inputs under one serving architecture for developers deploying multimodal applications.
tokenspeed2.9kA blazing-fast open-source LLM inference engine tuned for agentic workloads, pairing TensorRT-class speed with vLLM-like ease for production deployments at scale.