Libraries & frameworks
spider
spider-rs/spider
A concurrency-first Rust crawling engine that streams pages locally or at scale for search and RAG feeds.
Topics
Related projects
crawlee26.4kA Node.js web scraping and browser automation library for building reliable crawlers with proxies and storage.- crawlee-python9.7kA Python web-scraping and browser-automation library for building fast, reliable crawlers at scale.
firecrawl199.5kA developer API service that gives AI agents real-time web search, scraping, and crawling with LLM-ready structured output.
Scrapegraph-ai32.7kA Python web-scraping library that uses LLMs and graph pipelines to turn websites and documents into structured data.
Scrapling100.4kAn adaptive Python web-scraping framework with stealth fetching, spiders, proxy rotation and scaling for developers.
puppeteer95.8kA JavaScript library for automating headless Chrome and Firefox through the DevTools protocol.