spider

A high-concurrency web crawling engine written in Rust for low-latency streaming data extraction, supporting smart rendering, respectful crawling controls, and distributed scaling for large datasets.