Libraries & frameworks
gigatoken
marcelroed/gigatoken
Delivers GB-per-second language model tokenization compatible with popular tokenizers, designed for high-throughput preprocessing and inference pipelines for researchers and engineers needing fast text encoding at scale.
Topics
Related projects
next.js142.4kProvides a popular React full-stack framework with current React features and Rust-based tooling, helping web developers build and deploy production applications.
node122kOpen-source, cross-platform JavaScript runtime for servers and tooling, providing a unified execution foundation used by backend developers and JavaScript build ecosystems worldwide.
rust119.1kThe main source for the Rust programming language, including the compiler, standard library, and documentation for reliable systems software.
typescript111.2kA typed superset of JavaScript for large-scale apps that compiles to clean JavaScript with rich tooling and type checking.