Libraries & frameworks
scrapy-redis
rmax/scrapy-redis
Redis-based components for Scrapy that enable distributed crawling and post-processing by letting multiple spiders share a single Redis queue.
Topics
Related projects
redisson24.4kRedisson is a Java client and real-time data platform for Redis and Valkey. It adds distributed collections, locks, queues, caches, and other services on top of the core key-value store.
arq3karq is a Python job queue and RPC library built on asyncio and Redis. It lets async applications schedule background jobs and distributed tasks.
redislock1.8kredislock is a simplified distributed locking library for Go backed by Redis, offering lock acquisition, refresh, release and optional fencing tokens for safe concurrent access.
pottery1.2kPottery is a Pythonic Redis wrapper that exposes dicts, sets, lists, locks, caches, bloom filters and ID generators as familiar Python data structures.
CosId652A universal, high-performance distributed ID generator for Java with Snowflake, segment, periodic-segment, and UUIDv7 strategies over JDBC, Redis, MongoDB, or ZooKeeper stores.
wakaq597A minimal Redis-backed background task queue for Python, offering queue priorities, delayed and cron tasks, broadcast, soft/hard timeouts and retries as a lightweight Celery alternative.