mistral.rs

High-performance LLM inference engine written in Rust, supporting automatic loading of multiple models, quantization and multimodal inference through an OpenAI-compatible serving interface.