Agent infrastructure
TensorRT-LLM
NVIDIA/TensorRT-LLM
NVIDIA's open-source library for optimizing LLM inference with custom kernels and a high-performance runtime.
Topics
Related projects
sglang38.7kA fast open-source inference framework for serving large language and multimodal models at scale.
LMCache12.3kA vendor-neutral KV cache management layer for LLM inference that reuses cached context to cut latency and boost throughput.
project-maya2.6kProject Maya is a local LLM inference engine that runs large GLM, DeepSeek, and MiniMax models on your own GPUs, with quantized weights, a web dashboard, and OpenAI/Anthropic-compatible APIs.
llm-algo-leetcode744A notebook-based tutorial lab for LLM algorithms and systems, from PyTorch to Triton and CUDA.
onnxruntime22.3kA cross-platform high-performance inference and training accelerator for ML models with hardware acceleration and graph optimizations.
freetoken16.8kFreeToken is an edge-native Mixture-of-Experts inference engine that runs large open-weight models on consumer hardware, with a desktop app, CLI, and OpenAI/Anthropic-compatible APIs.