3.7 KiB
3.7 KiB
Distributed Crawler
An enterprise-oriented distributed crawler scaffold built around aiohttp and asyncio.
Included capabilities
- Master + Worker topology
- Redis-backed priority queues
- Consistent-hash worker assignment
- Bloom-filter task dedupe
- Dead-letter queue for exhausted failures
- Checkpoint + state recovery
- Task/request/worker trace headers
- Exponential backoff retries
- Domain-aware rate limiting
- Domain-level circuit breaker
- Proxy pool with health verification
- Proxy scoring with weighted selection
- Shared session store with Redis + local backup
- Result aggregation with JSONL, MongoDB, or PostgreSQL sinks and URL-hash dedupe
- APScheduler-driven periodic task submission
Safety notes
This project intentionally keeps high-risk account automation, captcha bypass, and anti-protection evasion as extension points only. Use it only against systems you are authorized to access and collect from.
Directory layout
distributed_crawler/
config.example.yaml
requirements.txt
crawler/
auth.py
backoff.py
bloom.py
config.py
hash_ring.py
http.py
master.py
models.py
proxy.py
queue.py
rate_limit.py
results.py
scheduler.py
sessions.py
signatures.py
spider.py
state.py
worker.py
scripts/
run_master.py
run_worker.py
Quick start
- Copy
config.example.yamltoconfig.yaml. - Start Redis.
- Install dependencies from
requirements.txt. - Start the master:
python scripts/run_master.py --config config.yaml
- Start one or more workers:
python scripts/run_worker.py --config config.yaml --worker-id worker-1
python scripts/run_worker.py --config config.yaml --worker-id worker-2
- Submit a task:
curl -X POST http://127.0.0.1:8088/tasks ^
-H "Content-Type: application/json" ^
-d "{\"url\":\"https://httpbin.org/get\",\"spider\":\"generic_fetch\"}"
Master endpoints
POST /tasksPOST /heartbeatGET /workersGET /statsGET /healthGET /dashboard(feature and runtime panel)
Failure handling and tracing
- Bloom sizing uses
master.bloom_capacityandmaster.bloom_false_positive_rateto calculate bit count and hash count automatically. Setmaster.bloom_hashesonly when you need to override the computed optimum. - Final failed tasks are appended to
master.dead_letter_queue_namewith the original task, final result, retry count, worker ID, error, and HTTP status. - Worker requests include
X-Task-Id,X-Request-Id, andX-Worker-Idheaders. Result payloads also includerequest_idfor easier log/result correlation.
Stability controls
- Proxy endpoints track success count, failure count, latency, cooldown, and a computed score. Workers choose healthy or cooled-down proxies with weighted random selection.
circuit_breakerprotects each domain independently. After repeated transport errors or 5xx responses, the domain opens briefly, then moves to half-open for recovery probing.- Result storage writes a stable
url_hash. MongoDB and PostgreSQL enforce uniqueness with upsert behavior; JSONL keeps a sidecar.url_hashesindex.
Auto login and shared sessions
- Enable
auth.enabled: trueand configure per-domain credentials inconfig.yaml. - Worker marks sessions expired based on configured status codes/body keywords.
- When expired, worker triggers
AuthProvider.refresh_session()and writes refreshed cookie to Redis + local backup viaDualSessionStore. - All workers read the same session keys (
session.redis_prefix), so login state is shared.
Extension points
crawler.auth.AuthProvidercrawler.signatures.SignatureProvidercrawler.spider.BaseSpider