commit
9c5fef1a4d
412 files changed
+94582
No files matched your search
@@ -0,0 +1,116 @@
|
||||
# Distributed Crawler
|
||||
|
||||
An enterprise-oriented distributed crawler scaffold built around `aiohttp` and `asyncio`.
|
||||
|
||||
## Included capabilities
|
||||
|
||||
- Master + Worker topology
|
||||
- Redis-backed priority queues
|
||||
- Consistent-hash worker assignment
|
||||
- Bloom-filter task dedupe
|
||||
- Dead-letter queue for exhausted failures
|
||||
- Checkpoint + state recovery
|
||||
- Task/request/worker trace headers
|
||||
- Exponential backoff retries
|
||||
- Domain-aware rate limiting
|
||||
- Domain-level circuit breaker
|
||||
- Proxy pool with health verification
|
||||
- Proxy scoring with weighted selection
|
||||
- Shared session store with Redis + local backup
|
||||
- Result aggregation with JSONL, MongoDB, or PostgreSQL sinks and URL-hash dedupe
|
||||
- APScheduler-driven periodic task submission
|
||||
|
||||
## Safety notes
|
||||
|
||||
This project intentionally keeps high-risk account automation, captcha bypass, and anti-protection evasion as extension points only.
|
||||
Use it only against systems you are authorized to access and collect from.
|
||||
|
||||
## Directory layout
|
||||
|
||||
```text
|
||||
distributed_crawler/
|
||||
config.example.yaml
|
||||
requirements.txt
|
||||
crawler/
|
||||
auth.py
|
||||
backoff.py
|
||||
bloom.py
|
||||
config.py
|
||||
hash_ring.py
|
||||
http.py
|
||||
master.py
|
||||
models.py
|
||||
proxy.py
|
||||
queue.py
|
||||
rate_limit.py
|
||||
results.py
|
||||
scheduler.py
|
||||
sessions.py
|
||||
signatures.py
|
||||
spider.py
|
||||
state.py
|
||||
worker.py
|
||||
scripts/
|
||||
run_master.py
|
||||
run_worker.py
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
1. Copy `config.example.yaml` to `config.yaml`.
|
||||
2. Start Redis.
|
||||
3. Install dependencies from `requirements.txt`.
|
||||
4. Start the master:
|
||||
|
||||
```bash
|
||||
python scripts/run_master.py --config config.yaml
|
||||
```
|
||||
|
||||
5. Start one or more workers:
|
||||
|
||||
```bash
|
||||
python scripts/run_worker.py --config config.yaml --worker-id worker-1
|
||||
python scripts/run_worker.py --config config.yaml --worker-id worker-2
|
||||
```
|
||||
|
||||
6. Submit a task:
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:8088/tasks ^
|
||||
-H "Content-Type: application/json" ^
|
||||
-d "{\"url\":\"https://httpbin.org/get\",\"spider\":\"generic_fetch\"}"
|
||||
```
|
||||
|
||||
## Master endpoints
|
||||
|
||||
- `POST /tasks`
|
||||
- `POST /heartbeat`
|
||||
- `GET /workers`
|
||||
- `GET /stats`
|
||||
- `GET /health`
|
||||
- `GET /dashboard` (feature and runtime panel)
|
||||
|
||||
## Failure handling and tracing
|
||||
|
||||
- Bloom sizing uses `master.bloom_capacity` and `master.bloom_false_positive_rate` to calculate bit count and hash count automatically. Set `master.bloom_hashes` only when you need to override the computed optimum.
|
||||
- Final failed tasks are appended to `master.dead_letter_queue_name` with the original task, final result, retry count, worker ID, error, and HTTP status.
|
||||
- Worker requests include `X-Task-Id`, `X-Request-Id`, and `X-Worker-Id` headers. Result payloads also include `request_id` for easier log/result correlation.
|
||||
|
||||
## Stability controls
|
||||
|
||||
- Proxy endpoints track success count, failure count, latency, cooldown, and a computed score. Workers choose healthy or cooled-down proxies with weighted random selection.
|
||||
- `circuit_breaker` protects each domain independently. After repeated transport errors or 5xx responses, the domain opens briefly, then moves to half-open for recovery probing.
|
||||
- Result storage writes a stable `url_hash`. MongoDB and PostgreSQL enforce uniqueness with upsert behavior; JSONL keeps a sidecar `.url_hashes` index.
|
||||
|
||||
## Auto login and shared sessions
|
||||
|
||||
- Enable `auth.enabled: true` and configure per-domain credentials in `config.yaml`.
|
||||
- Worker marks sessions expired based on configured status codes/body keywords.
|
||||
- When expired, worker triggers `AuthProvider.refresh_session()` and writes refreshed cookie to Redis + local backup via `DualSessionStore`.
|
||||
- All workers read the same session keys (`session.redis_prefix`), so login state is shared.
|
||||
|
||||
## Extension points
|
||||
|
||||
- `crawler.auth.AuthProvider`
|
||||
- `crawler.signatures.SignatureProvider`
|
||||
- `crawler.spider.BaseSpider`
|
||||
@@ -0,0 +1,93 @@
|
||||
app_name: distributed-crawler
|
||||
redis_dsn: redis://127.0.0.1:6379/0
|
||||
checkpoint_dir: ./runtime/checkpoints
|
||||
|
||||
master:
|
||||
host: 127.0.0.1
|
||||
port: 8088
|
||||
worker_ttl_seconds: 30
|
||||
stale_task_seconds: 120
|
||||
bloom_capacity: 1000000
|
||||
bloom_false_positive_rate: 0.01
|
||||
bloom_hashes:
|
||||
queue_name: crawler
|
||||
result_queue_name: crawler:results
|
||||
dead_letter_queue_name: crawler:dead_letter
|
||||
default_spider: generic_fetch
|
||||
|
||||
worker:
|
||||
worker_id: worker-1
|
||||
concurrency: 100
|
||||
heartbeat_interval_seconds: 5
|
||||
request_timeout_seconds: 20
|
||||
connect_timeout_seconds: 10
|
||||
retries: 4
|
||||
base_backoff_seconds: 1
|
||||
max_backoff_seconds: 16
|
||||
download_dir: ./runtime/downloads
|
||||
|
||||
storage:
|
||||
backend: jsonl
|
||||
jsonl_path: ./runtime/results/results.jsonl
|
||||
mongodb_dsn: mongodb://127.0.0.1:27017
|
||||
mongodb_database: crawler
|
||||
mongodb_collection: results
|
||||
postgres_dsn: postgresql://postgres:postgres@127.0.0.1:5432/crawler
|
||||
postgres_table: crawl_results
|
||||
|
||||
proxy:
|
||||
enabled: false
|
||||
verify_url: https://httpbin.org/ip
|
||||
verify_timeout_seconds: 8
|
||||
refresh_interval_seconds: 300
|
||||
failure_threshold: 3
|
||||
cooldown_seconds: 60
|
||||
static_proxies: []
|
||||
provider_api_url: ""
|
||||
provider_auth_header: ""
|
||||
|
||||
circuit_breaker:
|
||||
enabled: true
|
||||
failure_threshold: 8
|
||||
recovery_timeout_seconds: 30
|
||||
half_open_success_threshold: 2
|
||||
|
||||
session:
|
||||
enabled: true
|
||||
redis_prefix: crawler:sessions
|
||||
local_backup_path: ./runtime/sessions/sessions.json
|
||||
|
||||
auth:
|
||||
enabled: false
|
||||
domains:
|
||||
example.com:
|
||||
login_url: https://example.com/api/login
|
||||
username: your_username
|
||||
password: your_password
|
||||
username_field: username
|
||||
password_field: password
|
||||
submit_type: json # json or form
|
||||
extra_payload: {}
|
||||
success_status_codes: [200]
|
||||
expire_status_codes: [401, 403]
|
||||
expire_body_keywords: ["login required", "please login", "unauthorized"]
|
||||
|
||||
rate_limit:
|
||||
default_qps: 3.0
|
||||
default_burst: 6
|
||||
per_domain:
|
||||
httpbin.org:
|
||||
qps: 2.0
|
||||
burst: 4
|
||||
|
||||
schedules:
|
||||
- name: sample-httpbin
|
||||
trigger: interval
|
||||
seconds: 300
|
||||
priority: medium
|
||||
task:
|
||||
spider: generic_fetch
|
||||
url: https://httpbin.org/get
|
||||
method: GET
|
||||
metadata:
|
||||
source: schedule
|
||||
@@ -0,0 +1,5 @@
|
||||
from .config import AppSettings, load_settings
|
||||
from .master import MasterNode
|
||||
from .worker import WorkerNode
|
||||
|
||||
__all__ = ["AppSettings", "MasterNode", "WorkerNode", "load_settings"]
|
||||
@@ -0,0 +1,75 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from .config import AuthSettings
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AuthProvider(ABC):
|
||||
@abstractmethod
|
||||
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
Provide an authorized and compliant session refresh implementation for the target system.
|
||||
"""
|
||||
|
||||
|
||||
class ManualAuthProvider(AuthProvider):
|
||||
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
|
||||
raise NotImplementedError(
|
||||
f"Session refresh for {domain} must be implemented with authorized credentials and target-specific logic."
|
||||
)
|
||||
|
||||
|
||||
class ConfigurableAuthProvider(AuthProvider):
|
||||
"""
|
||||
Generic username/password login provider.
|
||||
This intentionally keeps site-specific anti-bot behavior out of scope.
|
||||
"""
|
||||
|
||||
def __init__(self, settings: AuthSettings, timeout_seconds: int = 20) -> None:
|
||||
self.settings = settings
|
||||
self.timeout_seconds = timeout_seconds
|
||||
|
||||
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
|
||||
if not self.settings.enabled:
|
||||
return {**existing_session, "expired": False}
|
||||
cfg = self.settings.domains.get(domain)
|
||||
if cfg is None or not cfg.login_url or not cfg.username or not cfg.password:
|
||||
LOGGER.warning("No auth config for domain=%s, keeping existing session", domain)
|
||||
return {**existing_session, "expired": True}
|
||||
|
||||
payload: dict[str, Any] = dict(cfg.extra_payload)
|
||||
payload[cfg.username_field] = cfg.username
|
||||
payload[cfg.password_field] = cfg.password
|
||||
|
||||
timeout = aiohttp.ClientTimeout(total=self.timeout_seconds)
|
||||
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||
if cfg.submit_type.lower() == "json":
|
||||
async with session.post(cfg.login_url, json=payload) as response:
|
||||
body_text = await response.text()
|
||||
if response.status not in set(cfg.success_status_codes):
|
||||
raise RuntimeError(f"Login failed for {domain}: {response.status} {body_text[:180]}")
|
||||
else:
|
||||
async with session.post(cfg.login_url, data=payload) as response:
|
||||
body_text = await response.text()
|
||||
if response.status not in set(cfg.success_status_codes):
|
||||
raise RuntimeError(f"Login failed for {domain}: {response.status} {body_text[:180]}")
|
||||
|
||||
cookie_pairs = [f"{k}={v.value}" for k, v in session.cookie_jar.filter_cookies(cfg.login_url).items()]
|
||||
cookie_header = "; ".join(cookie_pairs)
|
||||
if not cookie_header:
|
||||
# Fallback: keep old cookies if target does not set cookies in login response.
|
||||
cookie_header = str(existing_session.get("cookie", ""))
|
||||
|
||||
return {
|
||||
**existing_session,
|
||||
"expired": False,
|
||||
"cookie": cookie_header,
|
||||
"updated_by": "ConfigurableAuthProvider",
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
|
||||
|
||||
def exponential_backoff(attempt: int, base_seconds: float, max_seconds: float) -> float:
|
||||
delay = min(max_seconds, base_seconds * (2 ** max(attempt, 0)))
|
||||
jitter = random.uniform(0, delay * 0.2 if delay > 0 else 0)
|
||||
return delay + jitter
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from hashlib import md5, sha1
|
||||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
|
||||
class RedisBloomFilter:
|
||||
def __init__(self, redis: Redis, key: str, bits: int = 8_000_000, hashes: int = 6) -> None:
|
||||
self.redis = redis
|
||||
self.key = key
|
||||
self.bits = max(int(bits), 1)
|
||||
self.hashes = max(int(hashes), 1)
|
||||
|
||||
@classmethod
|
||||
def from_capacity(
|
||||
cls,
|
||||
redis: Redis,
|
||||
key: str,
|
||||
*,
|
||||
capacity: int,
|
||||
false_positive_rate: float = 0.01,
|
||||
hashes: int | None = None,
|
||||
) -> "RedisBloomFilter":
|
||||
capacity = max(int(capacity), 1)
|
||||
false_positive_rate = min(max(float(false_positive_rate), 1e-9), 0.999999)
|
||||
bits = math.ceil(-(capacity * math.log(false_positive_rate)) / (math.log(2) ** 2))
|
||||
optimal_hashes = max(1, round((bits / capacity) * math.log(2)))
|
||||
return cls(redis, key, bits=bits, hashes=hashes or optimal_hashes)
|
||||
|
||||
def _positions(self, value: str) -> list[int]:
|
||||
digest_a = int(md5(value.encode("utf-8")).hexdigest(), 16)
|
||||
digest_b = int(sha1(value.encode("utf-8")).hexdigest(), 16)
|
||||
return [((digest_a + index * digest_b) % self.bits) for index in range(self.hashes)]
|
||||
|
||||
async def contains_or_add(self, value: str) -> bool:
|
||||
positions = self._positions(value)
|
||||
pipeline = self.redis.pipeline()
|
||||
for position in positions:
|
||||
pipeline.getbit(self.key, position)
|
||||
existing = await pipeline.execute()
|
||||
seen = all(bit == 1 for bit in existing)
|
||||
|
||||
pipeline = self.redis.pipeline()
|
||||
for position in positions:
|
||||
pipeline.setbit(self.key, position, 1)
|
||||
await pipeline.execute()
|
||||
return seen
|
||||
@@ -0,0 +1,67 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from time import monotonic
|
||||
|
||||
from .config import CircuitBreakerSettings
|
||||
|
||||
|
||||
class CircuitState(str, Enum):
|
||||
CLOSED = "closed"
|
||||
OPEN = "open"
|
||||
HALF_OPEN = "half_open"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Circuit:
|
||||
state: CircuitState = CircuitState.CLOSED
|
||||
failures: int = 0
|
||||
half_open_successes: int = 0
|
||||
opened_at: float = 0.0
|
||||
|
||||
|
||||
class CircuitOpenError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class DomainCircuitBreaker:
|
||||
def __init__(self, settings: CircuitBreakerSettings) -> None:
|
||||
self.settings = settings
|
||||
self._circuits: dict[str, Circuit] = {}
|
||||
|
||||
def before_request(self, domain: str) -> None:
|
||||
if not self.settings.enabled:
|
||||
return
|
||||
circuit = self._circuits.setdefault(domain, Circuit())
|
||||
if circuit.state != CircuitState.OPEN:
|
||||
return
|
||||
elapsed = monotonic() - circuit.opened_at
|
||||
if elapsed >= self.settings.recovery_timeout_seconds:
|
||||
circuit.state = CircuitState.HALF_OPEN
|
||||
circuit.half_open_successes = 0
|
||||
return
|
||||
raise CircuitOpenError(f"Circuit open for {domain}")
|
||||
|
||||
def record_success(self, domain: str) -> None:
|
||||
if not self.settings.enabled:
|
||||
return
|
||||
circuit = self._circuits.setdefault(domain, Circuit())
|
||||
if circuit.state == CircuitState.HALF_OPEN:
|
||||
circuit.half_open_successes += 1
|
||||
if circuit.half_open_successes < self.settings.half_open_success_threshold:
|
||||
return
|
||||
circuit.state = CircuitState.CLOSED
|
||||
circuit.failures = 0
|
||||
circuit.half_open_successes = 0
|
||||
circuit.opened_at = 0.0
|
||||
|
||||
def record_failure(self, domain: str) -> None:
|
||||
if not self.settings.enabled:
|
||||
return
|
||||
circuit = self._circuits.setdefault(domain, Circuit())
|
||||
circuit.failures += 1
|
||||
circuit.half_open_successes = 0
|
||||
if circuit.failures >= self.settings.failure_threshold or circuit.state == CircuitState.HALF_OPEN:
|
||||
circuit.state = CircuitState.OPEN
|
||||
circuit.opened_at = monotonic()
|
||||
@@ -0,0 +1,206 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
@dataclass
|
||||
class MasterSettings:
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 8088
|
||||
worker_ttl_seconds: int = 30
|
||||
stale_task_seconds: int = 120
|
||||
bloom_capacity: int = 1_000_000
|
||||
bloom_false_positive_rate: float = 0.01
|
||||
bloom_hashes: int | None = None
|
||||
queue_name: str = "crawler"
|
||||
result_queue_name: str = "crawler:results"
|
||||
dead_letter_queue_name: str = "crawler:dead_letter"
|
||||
default_spider: str = "generic_fetch"
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerSettings:
|
||||
worker_id: str = "worker-1"
|
||||
concurrency: int = 100
|
||||
heartbeat_interval_seconds: int = 5
|
||||
request_timeout_seconds: int = 20
|
||||
connect_timeout_seconds: int = 10
|
||||
retries: int = 4
|
||||
base_backoff_seconds: float = 1.0
|
||||
max_backoff_seconds: float = 16.0
|
||||
download_dir: str = "./runtime/downloads"
|
||||
|
||||
|
||||
@dataclass
|
||||
class StorageSettings:
|
||||
backend: str = "jsonl"
|
||||
jsonl_path: str = "./runtime/results/results.jsonl"
|
||||
mongodb_dsn: str = "mongodb://127.0.0.1:27017"
|
||||
mongodb_database: str = "crawler"
|
||||
mongodb_collection: str = "results"
|
||||
postgres_dsn: str = "postgresql://postgres:postgres@127.0.0.1:5432/crawler"
|
||||
postgres_table: str = "crawl_results"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProxySettings:
|
||||
enabled: bool = False
|
||||
verify_url: str = "https://httpbin.org/ip"
|
||||
verify_timeout_seconds: int = 8
|
||||
refresh_interval_seconds: int = 300
|
||||
failure_threshold: int = 3
|
||||
cooldown_seconds: int = 60
|
||||
static_proxies: list[str] = field(default_factory=list)
|
||||
provider_api_url: str = ""
|
||||
provider_auth_header: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class CircuitBreakerSettings:
|
||||
enabled: bool = True
|
||||
failure_threshold: int = 8
|
||||
recovery_timeout_seconds: int = 30
|
||||
half_open_success_threshold: int = 2
|
||||
|
||||
|
||||
@dataclass
|
||||
class SessionSettings:
|
||||
enabled: bool = True
|
||||
redis_prefix: str = "crawler:sessions"
|
||||
local_backup_path: str = "./runtime/sessions/sessions.json"
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthDomainSettings:
|
||||
login_url: str = ""
|
||||
username: str = ""
|
||||
password: str = ""
|
||||
username_field: str = "username"
|
||||
password_field: str = "password"
|
||||
submit_type: str = "form"
|
||||
extra_payload: dict[str, Any] = field(default_factory=dict)
|
||||
success_status_codes: list[int] = field(default_factory=lambda: [200])
|
||||
expire_status_codes: list[int] = field(default_factory=lambda: [401, 403])
|
||||
expire_body_keywords: list[str] = field(default_factory=lambda: ["login required", "please login", "unauthorized"])
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthSettings:
|
||||
enabled: bool = False
|
||||
domains: dict[str, AuthDomainSettings] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DomainRateLimitSettings:
|
||||
qps: float = 3.0
|
||||
burst: int = 6
|
||||
|
||||
|
||||
@dataclass
|
||||
class RateLimitSettings:
|
||||
default_qps: float = 3.0
|
||||
default_burst: int = 6
|
||||
per_domain: dict[str, DomainRateLimitSettings] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScheduleTaskSettings:
|
||||
name: str
|
||||
trigger: str
|
||||
task: dict[str, Any]
|
||||
priority: str = "medium"
|
||||
seconds: int | None = None
|
||||
minutes: int | None = None
|
||||
cron: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class AppSettings:
|
||||
app_name: str = "distributed-crawler"
|
||||
redis_dsn: str = "redis://127.0.0.1:6379/0"
|
||||
checkpoint_dir: str = "./runtime/checkpoints"
|
||||
master: MasterSettings = field(default_factory=MasterSettings)
|
||||
worker: WorkerSettings = field(default_factory=WorkerSettings)
|
||||
storage: StorageSettings = field(default_factory=StorageSettings)
|
||||
proxy: ProxySettings = field(default_factory=ProxySettings)
|
||||
circuit_breaker: CircuitBreakerSettings = field(default_factory=CircuitBreakerSettings)
|
||||
session: SessionSettings = field(default_factory=SessionSettings)
|
||||
auth: AuthSettings = field(default_factory=AuthSettings)
|
||||
rate_limit: RateLimitSettings = field(default_factory=RateLimitSettings)
|
||||
schedules: list[ScheduleTaskSettings] = field(default_factory=list)
|
||||
source_path: Path | None = None
|
||||
|
||||
@property
|
||||
def base_dir(self) -> Path:
|
||||
if self.source_path is None:
|
||||
return Path.cwd()
|
||||
return self.source_path.parent
|
||||
|
||||
def resolve_path(self, value: str) -> Path:
|
||||
candidate = Path(value)
|
||||
if candidate.is_absolute():
|
||||
return candidate
|
||||
return (self.base_dir / candidate).resolve()
|
||||
|
||||
|
||||
def _load_domain_limits(raw_limits: dict[str, Any]) -> dict[str, DomainRateLimitSettings]:
|
||||
result: dict[str, DomainRateLimitSettings] = {}
|
||||
for domain, item in raw_limits.items():
|
||||
result[domain] = DomainRateLimitSettings(
|
||||
qps=float(item.get("qps", 3.0)),
|
||||
burst=int(item.get("burst", 6)),
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _load_auth_domains(raw_domains: dict[str, Any]) -> dict[str, AuthDomainSettings]:
|
||||
result: dict[str, AuthDomainSettings] = {}
|
||||
for domain, item in raw_domains.items():
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
result[domain] = AuthDomainSettings(
|
||||
login_url=str(item.get("login_url", "")),
|
||||
username=str(item.get("username", "")),
|
||||
password=str(item.get("password", "")),
|
||||
username_field=str(item.get("username_field", "username")),
|
||||
password_field=str(item.get("password_field", "password")),
|
||||
submit_type=str(item.get("submit_type", "form")),
|
||||
extra_payload=dict(item.get("extra_payload", {})),
|
||||
success_status_codes=[int(v) for v in item.get("success_status_codes", [200])],
|
||||
expire_status_codes=[int(v) for v in item.get("expire_status_codes", [401, 403])],
|
||||
expire_body_keywords=[str(v).lower() for v in item.get("expire_body_keywords", ["login required", "please login", "unauthorized"])],
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def load_settings(path: str | Path) -> AppSettings:
|
||||
config_path = Path(path).resolve()
|
||||
data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||
|
||||
settings = AppSettings(
|
||||
app_name=str(data.get("app_name", "distributed-crawler")),
|
||||
redis_dsn=str(data.get("redis_dsn", "redis://127.0.0.1:6379/0")),
|
||||
checkpoint_dir=str(data.get("checkpoint_dir", "./runtime/checkpoints")),
|
||||
master=MasterSettings(**(data.get("master") or {})),
|
||||
worker=WorkerSettings(**(data.get("worker") or {})),
|
||||
storage=StorageSettings(**(data.get("storage") or {})),
|
||||
proxy=ProxySettings(**(data.get("proxy") or {})),
|
||||
circuit_breaker=CircuitBreakerSettings(**(data.get("circuit_breaker") or {})),
|
||||
session=SessionSettings(**(data.get("session") or {})),
|
||||
auth=AuthSettings(
|
||||
enabled=bool((data.get("auth") or {}).get("enabled", False)),
|
||||
domains=_load_auth_domains((data.get("auth") or {}).get("domains", {})),
|
||||
),
|
||||
rate_limit=RateLimitSettings(
|
||||
default_qps=float((data.get("rate_limit") or {}).get("default_qps", 3.0)),
|
||||
default_burst=int((data.get("rate_limit") or {}).get("default_burst", 6)),
|
||||
per_domain=_load_domain_limits((data.get("rate_limit") or {}).get("per_domain", {})),
|
||||
),
|
||||
schedules=[ScheduleTaskSettings(**item) for item in data.get("schedules", [])],
|
||||
source_path=config_path,
|
||||
)
|
||||
return settings
|
||||
@@ -0,0 +1,33 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from bisect import bisect
|
||||
from hashlib import md5
|
||||
|
||||
|
||||
class ConsistentHashRing:
|
||||
def __init__(self, replicas: int = 100) -> None:
|
||||
self.replicas = replicas
|
||||
self._keys: list[int] = []
|
||||
self._nodes: dict[int, str] = {}
|
||||
|
||||
def _hash(self, value: str) -> int:
|
||||
return int(md5(value.encode("utf-8")).hexdigest(), 16)
|
||||
|
||||
def rebuild(self, nodes: list[str]) -> None:
|
||||
self._keys.clear()
|
||||
self._nodes.clear()
|
||||
for node in sorted(set(nodes)):
|
||||
for replica in range(self.replicas):
|
||||
key = self._hash(f"{node}:{replica}")
|
||||
self._keys.append(key)
|
||||
self._nodes[key] = node
|
||||
self._keys.sort()
|
||||
|
||||
def get_node(self, key: str) -> str | None:
|
||||
if not self._keys:
|
||||
return None
|
||||
hashed = self._hash(key)
|
||||
index = bisect(self._keys, hashed)
|
||||
if index == len(self._keys):
|
||||
index = 0
|
||||
return self._nodes[self._keys[index]]
|
||||
@@ -0,0 +1,180 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from time import monotonic
|
||||
from typing import Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from .auth import AuthProvider
|
||||
from .circuit_breaker import DomainCircuitBreaker
|
||||
from .config import AppSettings
|
||||
from .models import CrawlTask
|
||||
from .proxy import ProxyPool
|
||||
from .rate_limit import DomainRateLimiter
|
||||
from .sessions import DualSessionStore
|
||||
from .signatures import SignatureProvider
|
||||
|
||||
|
||||
USER_AGENTS = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/126.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4) AppleWebKit/605.1.15 Version/17.4 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Edg/126.0.0.0 Safari/537.36",
|
||||
]
|
||||
|
||||
|
||||
class AsyncHttpClient:
|
||||
def __init__(
|
||||
self,
|
||||
settings: AppSettings,
|
||||
rate_limiter: DomainRateLimiter,
|
||||
proxy_pool: ProxyPool,
|
||||
session_store: DualSessionStore,
|
||||
auth_provider: AuthProvider,
|
||||
signature_provider: SignatureProvider,
|
||||
) -> None:
|
||||
self.settings = settings
|
||||
self.rate_limiter = rate_limiter
|
||||
self.proxy_pool = proxy_pool
|
||||
self.session_store = session_store
|
||||
self.auth_provider = auth_provider
|
||||
self.signature_provider = signature_provider
|
||||
self.session: aiohttp.ClientSession | None = None
|
||||
self.circuit_breaker = DomainCircuitBreaker(settings.circuit_breaker)
|
||||
|
||||
async def start(self) -> None:
|
||||
timeout = aiohttp.ClientTimeout(
|
||||
total=self.settings.worker.request_timeout_seconds,
|
||||
connect=self.settings.worker.connect_timeout_seconds,
|
||||
)
|
||||
connector = aiohttp.TCPConnector(limit=self.settings.worker.concurrency, ssl=False)
|
||||
self.session = aiohttp.ClientSession(timeout=timeout, connector=connector)
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.session is not None:
|
||||
await self.session.close()
|
||||
|
||||
async def fetch(self, task: CrawlTask) -> dict[str, Any]:
|
||||
if self.session is None:
|
||||
raise RuntimeError("HTTP client is not started")
|
||||
|
||||
await self.rate_limiter.acquire(task.domain)
|
||||
self.circuit_breaker.before_request(task.domain)
|
||||
proxy = await self.proxy_pool.acquire()
|
||||
session_payload = await self.session_store.load(task.domain)
|
||||
if self.settings.session.enabled and session_payload.get("expired"):
|
||||
session_payload = await self.auth_provider.refresh_session(task.domain, session_payload)
|
||||
await self.session_store.save(task.domain, session_payload)
|
||||
|
||||
headers = dict(task.headers)
|
||||
headers.setdefault("User-Agent", random.choice(USER_AGENTS))
|
||||
headers.setdefault("Accept", "*/*")
|
||||
if task.metadata.get("referer"):
|
||||
headers.setdefault("Referer", str(task.metadata["referer"]))
|
||||
if task.metadata.get("origin"):
|
||||
headers.setdefault("Origin", str(task.metadata["origin"]))
|
||||
headers.setdefault("X-Task-Id", task.task_id)
|
||||
if task.metadata.get("request_id"):
|
||||
headers.setdefault("X-Request-Id", str(task.metadata["request_id"]))
|
||||
if task.metadata.get("worker_id"):
|
||||
headers.setdefault("X-Worker-Id", str(task.metadata["worker_id"]))
|
||||
cookie_header = session_payload.get("cookie")
|
||||
if cookie_header:
|
||||
headers.setdefault("Cookie", str(cookie_header))
|
||||
|
||||
signature_headers = await self.signature_provider.sign(
|
||||
{
|
||||
"url": task.url,
|
||||
"method": task.method,
|
||||
"body": task.body,
|
||||
"metadata": task.metadata,
|
||||
}
|
||||
)
|
||||
headers.update(signature_headers)
|
||||
|
||||
started_at = monotonic()
|
||||
try:
|
||||
async with self.session.request(
|
||||
task.method,
|
||||
task.url,
|
||||
headers=headers,
|
||||
data=task.body.encode("utf-8") if isinstance(task.body, str) else None,
|
||||
proxy=proxy.url if proxy else None,
|
||||
) as response:
|
||||
content_type = response.headers.get("Content-Type", "")
|
||||
body_bytes = await response.read()
|
||||
if "application/json" in content_type:
|
||||
body: Any = json.loads(body_bytes.decode("utf-8", errors="ignore"))
|
||||
else:
|
||||
body = body_bytes.decode("utf-8", errors="ignore")
|
||||
response_data = {
|
||||
"status": response.status,
|
||||
"headers": dict(response.headers),
|
||||
"body": body,
|
||||
"body_bytes": body_bytes,
|
||||
"proxy": proxy.url if proxy else None,
|
||||
}
|
||||
if self.settings.session.enabled:
|
||||
if self._is_session_expired(task.domain, response.status, body):
|
||||
session_payload["expired"] = True
|
||||
await self.session_store.save(task.domain, session_payload)
|
||||
else:
|
||||
if "Set-Cookie" in response.headers:
|
||||
cookie_header = self._extract_cookie_header(response)
|
||||
if cookie_header:
|
||||
session_payload["cookie"] = cookie_header
|
||||
session_payload["expired"] = False
|
||||
await self.session_store.save(task.domain, session_payload)
|
||||
if response.status >= 500:
|
||||
self.circuit_breaker.record_failure(task.domain)
|
||||
else:
|
||||
self.circuit_breaker.record_success(task.domain)
|
||||
await self.proxy_pool.mark_success(proxy, (monotonic() - started_at) * 1000)
|
||||
return response_data
|
||||
except Exception:
|
||||
self.circuit_breaker.record_failure(task.domain)
|
||||
await self.proxy_pool.mark_failed(proxy)
|
||||
raise
|
||||
|
||||
async def save_binary(self, task: CrawlTask, file_name: str, body_bytes: bytes) -> str:
|
||||
save_dir = self.settings.resolve_path(self.settings.worker.download_dir)
|
||||
save_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = save_dir / file_name
|
||||
target.write_bytes(body_bytes)
|
||||
return str(target)
|
||||
|
||||
def _is_session_expired(self, domain: str, status_code: int, body: Any) -> bool:
|
||||
if not self.settings.auth.enabled:
|
||||
return False
|
||||
domain_cfg = self.settings.auth.domains.get(domain)
|
||||
if domain_cfg is None:
|
||||
return False
|
||||
if status_code in set(domain_cfg.expire_status_codes):
|
||||
return True
|
||||
if isinstance(body, str):
|
||||
lower_body = body.lower()
|
||||
return any(keyword in lower_body for keyword in domain_cfg.expire_body_keywords)
|
||||
if isinstance(body, dict):
|
||||
packed = json.dumps(body, ensure_ascii=False).lower()
|
||||
return any(keyword in packed for keyword in domain_cfg.expire_body_keywords)
|
||||
return False
|
||||
|
||||
def _extract_cookie_header(self, response: aiohttp.ClientResponse) -> str:
|
||||
values = response.headers.getall("Set-Cookie", [])
|
||||
cookie_parts: list[str] = []
|
||||
for value in values:
|
||||
head = value.split(";", 1)[0].strip()
|
||||
if "=" in head:
|
||||
cookie_parts.append(head)
|
||||
# keep order and remove duplicates
|
||||
seen: set[str] = set()
|
||||
deduped: list[str] = []
|
||||
for item in cookie_parts:
|
||||
key = item.split("=", 1)[0].strip()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
deduped.append(item)
|
||||
return "; ".join(deduped)
|
||||
@@ -0,0 +1,346 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from time import time
|
||||
from typing import Any
|
||||
from html import escape
|
||||
|
||||
from aiohttp import web
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from .bloom import RedisBloomFilter
|
||||
from .config import AppSettings
|
||||
from .hash_ring import ConsistentHashRing
|
||||
from .models import CrawlResult, CrawlTask, DeadLetterEntry, TaskPriority, TaskStatus, WorkerHeartbeat
|
||||
from .queue import RedisPriorityQueue
|
||||
from .results import ResultRepository, build_result_repository
|
||||
from .scheduler import ScheduleService
|
||||
from .state import CheckpointManager
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MasterNode:
|
||||
def __init__(self, settings: AppSettings) -> None:
|
||||
self.settings = settings
|
||||
self.redis = Redis.from_url(settings.redis_dsn, decode_responses=True)
|
||||
self.queue = RedisPriorityQueue(
|
||||
self.redis,
|
||||
settings.master.queue_name,
|
||||
settings.master.result_queue_name,
|
||||
settings.master.dead_letter_queue_name,
|
||||
)
|
||||
self.checkpoints = CheckpointManager(self.redis, settings.resolve_path(settings.checkpoint_dir), settings.master.queue_name)
|
||||
self.bloom = RedisBloomFilter.from_capacity(
|
||||
self.redis,
|
||||
key=f"{settings.master.queue_name}:bloom",
|
||||
capacity=settings.master.bloom_capacity,
|
||||
false_positive_rate=settings.master.bloom_false_positive_rate,
|
||||
hashes=settings.master.bloom_hashes,
|
||||
)
|
||||
self.hash_ring = ConsistentHashRing()
|
||||
self.result_repo: ResultRepository = build_result_repository(
|
||||
settings.storage,
|
||||
settings.resolve_path(settings.storage.jsonl_path),
|
||||
)
|
||||
self.scheduler = ScheduleService(settings, self.submit_task)
|
||||
self.workers: dict[str, WorkerHeartbeat] = {}
|
||||
self.app = web.Application()
|
||||
self.app.add_routes(
|
||||
[
|
||||
web.post("/tasks", self.handle_submit_task),
|
||||
web.post("/heartbeat", self.handle_heartbeat),
|
||||
web.get("/workers", self.handle_workers),
|
||||
web.get("/stats", self.handle_stats),
|
||||
web.get("/health", self.handle_health),
|
||||
web.get("/dashboard", self.handle_dashboard),
|
||||
]
|
||||
)
|
||||
self._background_tasks: list[asyncio.Task] = []
|
||||
|
||||
async def start(self) -> None:
|
||||
await self.result_repo.setup()
|
||||
await self._recover_local_checkpoints()
|
||||
self.scheduler.start()
|
||||
self._background_tasks = [
|
||||
asyncio.create_task(self.consume_results_loop(), name="consume-results"),
|
||||
asyncio.create_task(self.reap_stale_workers_loop(), name="reap-workers"),
|
||||
]
|
||||
|
||||
async def close(self) -> None:
|
||||
for task in self._background_tasks:
|
||||
task.cancel()
|
||||
await asyncio.gather(*self._background_tasks, return_exceptions=True)
|
||||
await self.scheduler.shutdown()
|
||||
await self.result_repo.close()
|
||||
await self.redis.aclose()
|
||||
|
||||
async def submit_task(self, task: CrawlTask) -> bool:
|
||||
duplicate = await self.bloom.contains_or_add(task.fingerprint)
|
||||
if duplicate:
|
||||
LOGGER.info("Skip duplicate task %s", task.url)
|
||||
return False
|
||||
worker_id = self._assign_worker(task)
|
||||
task.assigned_worker = worker_id
|
||||
task.status = TaskStatus.PENDING
|
||||
await self.checkpoints.save(task)
|
||||
await self.queue.enqueue_task(worker_id, task)
|
||||
LOGGER.info("Submitted task %s to %s", task.task_id, worker_id)
|
||||
return True
|
||||
|
||||
async def handle_submit_task(self, request: web.Request) -> web.Response:
|
||||
payload = await request.json()
|
||||
task = CrawlTask.create(
|
||||
url=payload["url"],
|
||||
spider=payload.get("spider", self.settings.master.default_spider),
|
||||
method=payload.get("method", "GET"),
|
||||
headers=payload.get("headers"),
|
||||
metadata=payload.get("metadata"),
|
||||
body=payload.get("body"),
|
||||
priority=payload.get("priority", TaskPriority.MEDIUM.value),
|
||||
max_retries=int(payload.get("max_retries", self.settings.worker.retries)),
|
||||
)
|
||||
accepted = await self.submit_task(task)
|
||||
return web.json_response({"accepted": accepted, "task_id": task.task_id, "worker_id": task.assigned_worker})
|
||||
|
||||
async def handle_heartbeat(self, request: web.Request) -> web.Response:
|
||||
payload = await request.json()
|
||||
heartbeat = WorkerHeartbeat(
|
||||
worker_id=str(payload["worker_id"]),
|
||||
concurrency=int(payload["concurrency"]),
|
||||
active_tasks=int(payload["active_tasks"]),
|
||||
host=payload.get("host"),
|
||||
tags=list(payload.get("tags", [])),
|
||||
)
|
||||
self.workers[heartbeat.worker_id] = heartbeat
|
||||
self.hash_ring.rebuild(list(self.workers.keys()))
|
||||
return web.json_response({"ok": True, "known_workers": len(self.workers)})
|
||||
|
||||
async def handle_workers(self, _: web.Request) -> web.Response:
|
||||
return web.json_response({"workers": [item.to_dict() for item in self.workers.values()]})
|
||||
|
||||
async def handle_stats(self, _: web.Request) -> web.Response:
|
||||
return web.json_response(
|
||||
{
|
||||
"workers": len(self.workers),
|
||||
"queues": {
|
||||
worker_id: {
|
||||
priority.value: await self.redis.llen(self.queue._task_key(worker_id, priority))
|
||||
for priority in TaskPriority
|
||||
}
|
||||
for worker_id in self.workers or {"default": None}
|
||||
},
|
||||
"dead_letters": await self.redis.llen(self.queue.dead_letter_queue_name),
|
||||
}
|
||||
)
|
||||
|
||||
async def handle_health(self, _: web.Request) -> web.Response:
|
||||
pong = await self.redis.ping()
|
||||
return web.json_response({"status": "ok", "redis": bool(pong), "workers": len(self.workers)})
|
||||
|
||||
async def handle_dashboard(self, _: web.Request) -> web.Response:
|
||||
queue_stats = await self._collect_queue_stats()
|
||||
session_count = await self._count_sessions()
|
||||
dead_letter_count = await self.redis.llen(self.queue.dead_letter_queue_name)
|
||||
html = self._render_dashboard_html(
|
||||
queue_stats=queue_stats,
|
||||
session_count=session_count,
|
||||
dead_letter_count=dead_letter_count,
|
||||
)
|
||||
return web.Response(text=html, content_type="text/html")
|
||||
|
||||
async def consume_results_loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
result = await self.queue.consume_result(timeout=5)
|
||||
if result is None:
|
||||
continue
|
||||
await self.process_result(result)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
LOGGER.exception("Result consumer failed")
|
||||
await asyncio.sleep(1)
|
||||
|
||||
async def process_result(self, result: CrawlResult) -> None:
|
||||
task = await self.checkpoints.load(result.task_id)
|
||||
if task is None:
|
||||
LOGGER.warning("Result for unknown task %s", result.task_id)
|
||||
return
|
||||
task.status = result.status
|
||||
await self.checkpoints.mark_status(task, result.status)
|
||||
await self.result_repo.store(result)
|
||||
if result.status == TaskStatus.FAILED:
|
||||
await self.queue.publish_dead_letter(DeadLetterEntry.from_task_result(task, result))
|
||||
LOGGER.warning(
|
||||
"Dead-lettered task task_id=%s worker_id=%s request_id=%s reason=%s",
|
||||
task.task_id,
|
||||
result.worker_id,
|
||||
result.data.get("request_id", task.metadata.get("request_id", task.task_id)),
|
||||
result.error,
|
||||
)
|
||||
|
||||
async def reap_stale_workers_loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(self.settings.master.worker_ttl_seconds)
|
||||
now = time()
|
||||
stale_workers = [
|
||||
worker_id
|
||||
for worker_id, heartbeat in self.workers.items()
|
||||
if now - self._heartbeat_epoch(heartbeat.last_seen) > self.settings.master.worker_ttl_seconds
|
||||
]
|
||||
for worker_id in stale_workers:
|
||||
LOGGER.warning("Worker %s is stale, recovering tasks", worker_id)
|
||||
recovered = await self.checkpoints.recover_stale(worker_id, self.settings.master.stale_task_seconds)
|
||||
for task in recovered:
|
||||
task.retries += 1
|
||||
if task.retries > task.max_retries:
|
||||
task.status = TaskStatus.FAILED
|
||||
await self.checkpoints.mark_status(task, TaskStatus.FAILED)
|
||||
result = CrawlResult(
|
||||
task_id=task.task_id,
|
||||
worker_id=worker_id,
|
||||
status=TaskStatus.FAILED,
|
||||
url=task.url,
|
||||
spider=task.spider,
|
||||
error="Task exceeded retry budget during worker recovery",
|
||||
)
|
||||
await self.result_repo.store(result)
|
||||
await self.queue.publish_dead_letter(DeadLetterEntry.from_task_result(task, result))
|
||||
continue
|
||||
task.assigned_worker = self._assign_worker(task, exclude=worker_id)
|
||||
await self.checkpoints.save(task)
|
||||
await self.queue.enqueue_task(task.assigned_worker, task)
|
||||
self.workers.pop(worker_id, None)
|
||||
self.hash_ring.rebuild(list(self.workers.keys()))
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
LOGGER.exception("Stale worker reaper failed")
|
||||
|
||||
async def _recover_local_checkpoints(self) -> None:
|
||||
tasks = await self.checkpoints.recover_local()
|
||||
for task in tasks:
|
||||
task.assigned_worker = self._assign_worker(task)
|
||||
task.status = TaskStatus.RETRY
|
||||
await self.checkpoints.save(task)
|
||||
await self.queue.enqueue_task(task.assigned_worker, task)
|
||||
|
||||
async def _collect_queue_stats(self) -> dict[str, dict[str, int]]:
|
||||
worker_keys = list(self.workers.keys()) or ["default"]
|
||||
result: dict[str, dict[str, int]] = {}
|
||||
for worker_id in worker_keys:
|
||||
result[worker_id] = {
|
||||
priority.value: await self.redis.llen(self.queue._task_key(worker_id, priority))
|
||||
for priority in TaskPriority
|
||||
}
|
||||
return result
|
||||
|
||||
async def _count_sessions(self) -> int:
|
||||
pattern = f"{self.settings.session.redis_prefix}:*"
|
||||
count = 0
|
||||
async for _ in self.redis.scan_iter(match=pattern):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
def _render_dashboard_html(self, queue_stats: dict[str, dict[str, int]], session_count: int, dead_letter_count: int) -> str:
|
||||
worker_rows = []
|
||||
for worker_id, hb in self.workers.items():
|
||||
load = hb.active_tasks / max(hb.concurrency, 1)
|
||||
worker_rows.append(
|
||||
f"<tr><td>{escape(worker_id)}</td><td>{hb.active_tasks}</td><td>{hb.concurrency}</td><td>{load:.2f}</td><td>{escape(hb.last_seen)}</td></tr>"
|
||||
)
|
||||
if not worker_rows:
|
||||
worker_rows.append("<tr><td colspan='5'>暂无在线 Worker</td></tr>")
|
||||
|
||||
queue_blocks = []
|
||||
for worker_id, levels in queue_stats.items():
|
||||
queue_blocks.append(
|
||||
"<div class='card'><h3>Queue - "
|
||||
+ escape(worker_id)
|
||||
+ "</h3><p>high: "
|
||||
+ str(levels.get("high", 0))
|
||||
+ " | medium: "
|
||||
+ str(levels.get("medium", 0))
|
||||
+ " | low: "
|
||||
+ str(levels.get("low", 0))
|
||||
+ "</p></div>"
|
||||
)
|
||||
queue_html = "\n".join(queue_blocks)
|
||||
|
||||
enabled_auth_domains = ", ".join(sorted(self.settings.auth.domains.keys())) if self.settings.auth.domains else "-"
|
||||
enabled_proxy = "enabled" if self.settings.proxy.enabled else "disabled"
|
||||
enabled_session = "enabled" if self.settings.session.enabled else "disabled"
|
||||
enabled_auth = "enabled" if self.settings.auth.enabled else "disabled"
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<title>Distributed Crawler Dashboard</title>
|
||||
<style>
|
||||
body {{ font-family: Segoe UI, Arial, sans-serif; background:#f6f8fb; margin:0; padding:24px; color:#1f2d3d; }}
|
||||
.grid {{ display:grid; grid-template-columns: repeat(auto-fill, minmax(280px,1fr)); gap:12px; margin-bottom:16px; }}
|
||||
.card {{ background:#fff; border:1px solid #dce6f4; border-radius:10px; padding:12px 14px; }}
|
||||
h1 {{ margin:0 0 14px 0; font-size:24px; }}
|
||||
h3 {{ margin:0 0 8px 0; font-size:16px; }}
|
||||
table {{ width:100%; border-collapse: collapse; background:#fff; border:1px solid #dce6f4; }}
|
||||
th, td {{ text-align:left; padding:10px; border-bottom:1px solid #eef2f8; font-size:13px; }}
|
||||
th {{ background:#f1f5fc; }}
|
||||
.muted {{ color:#607086; font-size:12px; }}
|
||||
code {{ background:#eef3fb; padding:2px 6px; border-radius:4px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Distributed Crawler Dashboard</h1>
|
||||
<div class="grid">
|
||||
<div class="card"><h3>Workers</h3><p>{len(self.workers)}</p></div>
|
||||
<div class="card"><h3>Session Keys</h3><p>{session_count}</p></div>
|
||||
<div class="card"><h3>Dead Letters</h3><p>{dead_letter_count}</p></div>
|
||||
<div class="card"><h3>Proxy</h3><p>{enabled_proxy}</p></div>
|
||||
<div class="card"><h3>Auto Login</h3><p>{enabled_auth}</p></div>
|
||||
<div class="card"><h3>Session Sharing</h3><p>{enabled_session}</p></div>
|
||||
<div class="card"><h3>Auth Domains</h3><p>{escape(enabled_auth_domains)}</p></div>
|
||||
</div>
|
||||
<div class="grid">
|
||||
{queue_html}
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Worker Runtime</h3>
|
||||
<table>
|
||||
<thead><tr><th>worker_id</th><th>active_tasks</th><th>concurrency</th><th>load</th><th>last_seen</th></tr></thead>
|
||||
<tbody>
|
||||
{''.join(worker_rows)}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p class="muted">API endpoints: <code>/tasks</code> <code>/workers</code> <code>/stats</code> <code>/health</code></p>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
def _assign_worker(self, task: CrawlTask, exclude: str | None = None) -> str:
|
||||
candidates = [worker_id for worker_id in self.workers if worker_id != exclude]
|
||||
if not candidates:
|
||||
return "default"
|
||||
self.hash_ring.rebuild(candidates)
|
||||
preferred = self.hash_ring.get_node(task.url) or candidates[0]
|
||||
preferred_heartbeat = self.workers.get(preferred)
|
||||
if preferred_heartbeat is None:
|
||||
return preferred
|
||||
|
||||
preferred_load = preferred_heartbeat.active_tasks / max(preferred_heartbeat.concurrency, 1)
|
||||
if preferred_load < 0.9:
|
||||
return preferred
|
||||
|
||||
least_loaded = min(
|
||||
candidates,
|
||||
key=lambda worker_id: self.workers[worker_id].active_tasks / max(self.workers[worker_id].concurrency, 1),
|
||||
)
|
||||
return least_loaded
|
||||
|
||||
def _heartbeat_epoch(self, timestamp: str) -> float:
|
||||
from datetime import datetime
|
||||
|
||||
return datetime.fromisoformat(timestamp).timestamp()
|
||||
@@ -0,0 +1,189 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from enum import Enum
|
||||
from hashlib import sha1
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
def make_url_hash(url: str) -> str:
|
||||
return sha1(url.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class TaskPriority(str, Enum):
|
||||
HIGH = "high"
|
||||
MEDIUM = "medium"
|
||||
LOW = "low"
|
||||
|
||||
|
||||
class TaskStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
SUCCESS = "success"
|
||||
FAILED = "failed"
|
||||
RETRY = "retry"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CrawlTask:
|
||||
task_id: str
|
||||
url: str
|
||||
spider: str
|
||||
method: str = "GET"
|
||||
headers: dict[str, str] = field(default_factory=dict)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
body: str | None = None
|
||||
priority: TaskPriority = TaskPriority.MEDIUM
|
||||
status: TaskStatus = TaskStatus.PENDING
|
||||
retries: int = 0
|
||||
max_retries: int = 4
|
||||
assigned_worker: str | None = None
|
||||
created_at: str = field(default_factory=utc_now)
|
||||
updated_at: str = field(default_factory=utc_now)
|
||||
schedule_name: str | None = None
|
||||
|
||||
@property
|
||||
def domain(self) -> str:
|
||||
return urlparse(self.url).netloc
|
||||
|
||||
@property
|
||||
def fingerprint(self) -> str:
|
||||
payload = f"{self.method}:{self.url}:{self.spider}:{self.body or ''}"
|
||||
return sha1(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
payload = asdict(self)
|
||||
payload["priority"] = self.priority.value
|
||||
payload["status"] = self.status.value
|
||||
return payload
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, payload: dict[str, Any]) -> "CrawlTask":
|
||||
return cls(
|
||||
task_id=str(payload["task_id"]),
|
||||
url=str(payload["url"]),
|
||||
spider=str(payload["spider"]),
|
||||
method=str(payload.get("method", "GET")),
|
||||
headers=dict(payload.get("headers", {})),
|
||||
metadata=dict(payload.get("metadata", {})),
|
||||
body=payload.get("body"),
|
||||
priority=TaskPriority(payload.get("priority", TaskPriority.MEDIUM.value)),
|
||||
status=TaskStatus(payload.get("status", TaskStatus.PENDING.value)),
|
||||
retries=int(payload.get("retries", 0)),
|
||||
max_retries=int(payload.get("max_retries", 4)),
|
||||
assigned_worker=payload.get("assigned_worker"),
|
||||
created_at=str(payload.get("created_at", utc_now())),
|
||||
updated_at=str(payload.get("updated_at", utc_now())),
|
||||
schedule_name=payload.get("schedule_name"),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
*,
|
||||
url: str,
|
||||
spider: str,
|
||||
method: str = "GET",
|
||||
headers: dict[str, str] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
body: str | None = None,
|
||||
priority: str = "medium",
|
||||
max_retries: int = 4,
|
||||
schedule_name: str | None = None,
|
||||
) -> "CrawlTask":
|
||||
return cls(
|
||||
task_id=uuid4().hex,
|
||||
url=url,
|
||||
spider=spider,
|
||||
method=method.upper(),
|
||||
headers=headers or {},
|
||||
metadata=metadata or {},
|
||||
body=body,
|
||||
priority=TaskPriority(priority),
|
||||
max_retries=max_retries,
|
||||
schedule_name=schedule_name,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CrawlResult:
|
||||
task_id: str
|
||||
worker_id: str
|
||||
status: TaskStatus
|
||||
url: str
|
||||
spider: str
|
||||
http_status: int | None = None
|
||||
data: dict[str, Any] = field(default_factory=dict)
|
||||
error: str | None = None
|
||||
finished_at: str = field(default_factory=utc_now)
|
||||
|
||||
@property
|
||||
def url_hash(self) -> str:
|
||||
return make_url_hash(self.url)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
payload = asdict(self)
|
||||
payload["status"] = self.status.value
|
||||
payload["url_hash"] = self.url_hash
|
||||
return payload
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, payload: dict[str, Any]) -> "CrawlResult":
|
||||
return cls(
|
||||
task_id=str(payload["task_id"]),
|
||||
worker_id=str(payload["worker_id"]),
|
||||
status=TaskStatus(payload["status"]),
|
||||
url=str(payload["url"]),
|
||||
spider=str(payload["spider"]),
|
||||
http_status=payload.get("http_status"),
|
||||
data=dict(payload.get("data", {})),
|
||||
error=payload.get("error"),
|
||||
finished_at=str(payload.get("finished_at", utc_now())),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DeadLetterEntry:
|
||||
task: dict[str, Any]
|
||||
result: dict[str, Any] | None = None
|
||||
reason: str = ""
|
||||
failed_at: str = field(default_factory=utc_now)
|
||||
last_error: str | None = None
|
||||
response_status: int | None = None
|
||||
retry_count: int = 0
|
||||
worker_id: str | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_task_result(cls, task: CrawlTask, result: CrawlResult, reason: str | None = None) -> "DeadLetterEntry":
|
||||
return cls(
|
||||
task=task.to_dict(),
|
||||
result=result.to_dict(),
|
||||
reason=reason or result.error or "Task failed",
|
||||
last_error=result.error,
|
||||
response_status=result.http_status,
|
||||
retry_count=task.retries,
|
||||
worker_id=result.worker_id,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerHeartbeat:
|
||||
worker_id: str
|
||||
concurrency: int
|
||||
active_tasks: int
|
||||
last_seen: str = field(default_factory=utc_now)
|
||||
host: str | None = None
|
||||
tags: list[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
@@ -0,0 +1,162 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from time import monotonic
|
||||
|
||||
import aiohttp
|
||||
|
||||
from .config import ProxySettings
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProxyEndpoint:
|
||||
url: str
|
||||
failures: int = 0
|
||||
successes: int = 0
|
||||
latency_ms: float | None = None
|
||||
last_verified_at: float = 0.0
|
||||
last_failed_at: float = 0.0
|
||||
healthy: bool = True
|
||||
metadata: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def success_rate(self) -> float:
|
||||
total = self.successes + self.failures
|
||||
if total == 0:
|
||||
return 1.0
|
||||
return self.successes / total
|
||||
|
||||
@property
|
||||
def score(self) -> float:
|
||||
latency_penalty = 1.0
|
||||
if self.latency_ms is not None:
|
||||
latency_penalty = max(0.1, min(1.0, 1000.0 / max(self.latency_ms, 1.0)))
|
||||
failure_penalty = 1.0 / (1.0 + self.failures)
|
||||
return max(0.01, self.success_rate * latency_penalty * failure_penalty)
|
||||
|
||||
|
||||
class ProxyProvider(ABC):
|
||||
@abstractmethod
|
||||
async def fetch(self) -> list[ProxyEndpoint]: ...
|
||||
|
||||
|
||||
class StaticProxyProvider(ProxyProvider):
|
||||
def __init__(self, proxies: list[str]) -> None:
|
||||
self.proxies = proxies
|
||||
|
||||
async def fetch(self) -> list[ProxyEndpoint]:
|
||||
return [ProxyEndpoint(url=value) for value in self.proxies]
|
||||
|
||||
|
||||
class ApiProxyProvider(ProxyProvider):
|
||||
def __init__(self, api_url: str, auth_header: str = "") -> None:
|
||||
self.api_url = api_url
|
||||
self.auth_header = auth_header
|
||||
|
||||
async def fetch(self) -> list[ProxyEndpoint]:
|
||||
if not self.api_url:
|
||||
return []
|
||||
headers = {"Authorization": self.auth_header} if self.auth_header else {}
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(self.api_url, headers=headers) as response:
|
||||
response.raise_for_status()
|
||||
text = await response.text()
|
||||
proxies: list[ProxyEndpoint] = []
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
proxies.append(ProxyEndpoint(url=line))
|
||||
return proxies
|
||||
|
||||
|
||||
class ProxyPool:
|
||||
def __init__(self, settings: ProxySettings) -> None:
|
||||
self.settings = settings
|
||||
self._proxies: list[ProxyEndpoint] = []
|
||||
self._lock = asyncio.Lock()
|
||||
self._next_index = 0
|
||||
|
||||
async def refresh(self) -> None:
|
||||
providers: list[ProxyProvider] = [StaticProxyProvider(self.settings.static_proxies)]
|
||||
if self.settings.provider_api_url:
|
||||
providers.append(ApiProxyProvider(self.settings.provider_api_url, self.settings.provider_auth_header))
|
||||
async with self._lock:
|
||||
existing = {endpoint.url: endpoint for endpoint in self._proxies}
|
||||
merged: dict[str, ProxyEndpoint] = {}
|
||||
for provider in providers:
|
||||
for endpoint in await provider.fetch():
|
||||
merged.setdefault(endpoint.url, existing.get(endpoint.url, endpoint))
|
||||
async with self._lock:
|
||||
self._proxies = list(merged.values())
|
||||
|
||||
async def verify(self) -> None:
|
||||
if not self.settings.enabled:
|
||||
return
|
||||
async with self._lock:
|
||||
proxies = list(self._proxies)
|
||||
if not proxies:
|
||||
return
|
||||
async with aiohttp.ClientSession() as session:
|
||||
tasks = [self._verify_proxy(session, proxy) for proxy in proxies]
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
async def acquire(self) -> ProxyEndpoint | None:
|
||||
if not self.settings.enabled:
|
||||
return None
|
||||
async with self._lock:
|
||||
now = monotonic()
|
||||
candidates = [
|
||||
item
|
||||
for item in self._proxies
|
||||
if item.healthy or now - item.last_failed_at >= self.settings.cooldown_seconds
|
||||
]
|
||||
if not candidates:
|
||||
return None
|
||||
proxy = random.choices(candidates, weights=[item.score for item in candidates], k=1)[0]
|
||||
self._next_index += 1
|
||||
return proxy
|
||||
|
||||
async def mark_success(self, proxy: ProxyEndpoint | None, latency_ms: float | None = None) -> None:
|
||||
if proxy is None:
|
||||
return
|
||||
async with self._lock:
|
||||
proxy.successes += 1
|
||||
proxy.failures = 0
|
||||
proxy.healthy = True
|
||||
if latency_ms is not None:
|
||||
proxy.latency_ms = latency_ms
|
||||
proxy.last_verified_at = monotonic()
|
||||
|
||||
async def mark_failed(self, proxy: ProxyEndpoint | None) -> None:
|
||||
if proxy is None:
|
||||
return
|
||||
async with self._lock:
|
||||
proxy.failures += 1
|
||||
proxy.last_failed_at = monotonic()
|
||||
if proxy.failures >= self.settings.failure_threshold:
|
||||
proxy.healthy = False
|
||||
|
||||
async def healthy_count(self) -> int:
|
||||
async with self._lock:
|
||||
return len([item for item in self._proxies if item.healthy])
|
||||
|
||||
async def _verify_proxy(self, session: aiohttp.ClientSession, proxy: ProxyEndpoint) -> None:
|
||||
try:
|
||||
timeout = aiohttp.ClientTimeout(total=self.settings.verify_timeout_seconds)
|
||||
started_at = monotonic()
|
||||
async with session.get(self.settings.verify_url, proxy=proxy.url, timeout=timeout) as response:
|
||||
response.raise_for_status()
|
||||
await response.read()
|
||||
proxy.healthy = True
|
||||
proxy.failures = 0
|
||||
proxy.successes += 1
|
||||
proxy.latency_ms = (monotonic() - started_at) * 1000
|
||||
proxy.last_verified_at = monotonic()
|
||||
except Exception:
|
||||
proxy.failures += 1
|
||||
proxy.last_failed_at = monotonic()
|
||||
if proxy.failures >= self.settings.failure_threshold:
|
||||
proxy.healthy = False
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from .models import CrawlResult, CrawlTask, DeadLetterEntry, TaskPriority
|
||||
|
||||
|
||||
class RedisPriorityQueue:
|
||||
def __init__(self, redis: Redis, namespace: str, result_queue_name: str, dead_letter_queue_name: str | None = None) -> None:
|
||||
self.redis = redis
|
||||
self.namespace = namespace
|
||||
self.result_queue_name = result_queue_name
|
||||
self.dead_letter_queue_name = dead_letter_queue_name or f"{namespace}:dead_letter"
|
||||
|
||||
def _task_key(self, worker_id: str, priority: TaskPriority) -> str:
|
||||
return f"{self.namespace}:tasks:{worker_id}:{priority.value}"
|
||||
|
||||
async def enqueue_task(self, worker_id: str, task: CrawlTask) -> None:
|
||||
await self.redis.rpush(self._task_key(worker_id, task.priority), json.dumps(task.to_dict()))
|
||||
|
||||
async def dequeue_task(self, worker_id: str, timeout: int = 5) -> CrawlTask | None:
|
||||
keys = [self._task_key(worker_id, priority) for priority in (TaskPriority.HIGH, TaskPriority.MEDIUM, TaskPriority.LOW)]
|
||||
result = await self.redis.blpop(keys, timeout=timeout)
|
||||
if not result:
|
||||
return None
|
||||
_, payload = result
|
||||
return CrawlTask.from_dict(json.loads(payload))
|
||||
|
||||
async def publish_result(self, result: CrawlResult) -> None:
|
||||
await self.redis.rpush(self.result_queue_name, json.dumps(result.to_dict()))
|
||||
|
||||
async def consume_result(self, timeout: int = 5) -> CrawlResult | None:
|
||||
result = await self.redis.blpop(self.result_queue_name, timeout=timeout)
|
||||
if not result:
|
||||
return None
|
||||
_, payload = result
|
||||
return CrawlResult.from_dict(json.loads(payload))
|
||||
|
||||
async def publish_dead_letter(self, entry: DeadLetterEntry) -> None:
|
||||
await self.redis.rpush(self.dead_letter_queue_name, json.dumps(entry.to_dict(), ensure_ascii=False))
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from time import monotonic
|
||||
|
||||
from .config import RateLimitSettings
|
||||
|
||||
|
||||
@dataclass
|
||||
class TokenBucket:
|
||||
qps: float
|
||||
burst: int
|
||||
tokens: float
|
||||
updated_at: float
|
||||
|
||||
|
||||
class DomainRateLimiter:
|
||||
def __init__(self, settings: RateLimitSettings) -> None:
|
||||
self.settings = settings
|
||||
self._buckets: dict[str, TokenBucket] = {}
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
async def acquire(self, domain: str) -> None:
|
||||
lock = self._locks.setdefault(domain, asyncio.Lock())
|
||||
async with lock:
|
||||
bucket = self._buckets.setdefault(domain, self._new_bucket(domain))
|
||||
while True:
|
||||
now = monotonic()
|
||||
elapsed = max(0.0, now - bucket.updated_at)
|
||||
bucket.tokens = min(bucket.burst, bucket.tokens + elapsed * bucket.qps)
|
||||
bucket.updated_at = now
|
||||
if bucket.tokens >= 1:
|
||||
bucket.tokens -= 1
|
||||
return
|
||||
await asyncio.sleep(max(0.05, 1 / bucket.qps))
|
||||
|
||||
def _new_bucket(self, domain: str) -> TokenBucket:
|
||||
domain_settings = self.settings.per_domain.get(domain)
|
||||
qps = domain_settings.qps if domain_settings else self.settings.default_qps
|
||||
burst = domain_settings.burst if domain_settings else self.settings.default_burst
|
||||
return TokenBucket(qps=qps, burst=burst, tokens=float(burst), updated_at=monotonic())
|
||||
@@ -0,0 +1,164 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
|
||||
import asyncpg
|
||||
from motor.motor_asyncio import AsyncIOMotorClient
|
||||
|
||||
from .config import StorageSettings
|
||||
from .models import CrawlResult, make_url_hash
|
||||
|
||||
|
||||
class ResultRepository(ABC):
|
||||
@abstractmethod
|
||||
async def setup(self) -> None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def store(self, result: CrawlResult) -> None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def close(self) -> None: ...
|
||||
|
||||
|
||||
class JsonlResultRepository(ResultRepository):
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = path
|
||||
self.index_path = path.with_suffix(path.suffix + ".url_hashes")
|
||||
self._seen_url_hashes: set[str] = set()
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def setup(self) -> None:
|
||||
if self.index_path.exists():
|
||||
self._seen_url_hashes = {
|
||||
line.strip()
|
||||
for line in self.index_path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
}
|
||||
return
|
||||
if self.path.exists():
|
||||
with self.path.open("r", encoding="utf-8") as file_handle:
|
||||
for line in file_handle:
|
||||
try:
|
||||
payload = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
url_hash = payload.get("url_hash") or (make_url_hash(str(payload["url"])) if payload.get("url") else None)
|
||||
if url_hash:
|
||||
self._seen_url_hashes.add(str(url_hash))
|
||||
if self._seen_url_hashes:
|
||||
self.index_path.write_text("\n".join(sorted(self._seen_url_hashes)) + "\n", encoding="utf-8")
|
||||
|
||||
async def store(self, result: CrawlResult) -> None:
|
||||
if result.url_hash in self._seen_url_hashes:
|
||||
return
|
||||
with self.path.open("a", encoding="utf-8") as file_handle:
|
||||
file_handle.write(json.dumps(result.to_dict(), ensure_ascii=False) + "\n")
|
||||
with self.index_path.open("a", encoding="utf-8") as file_handle:
|
||||
file_handle.write(result.url_hash + "\n")
|
||||
self._seen_url_hashes.add(result.url_hash)
|
||||
|
||||
async def close(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
class MongoResultRepository(ResultRepository):
|
||||
def __init__(self, settings: StorageSettings) -> None:
|
||||
self.client = AsyncIOMotorClient(settings.mongodb_dsn)
|
||||
self.collection = self.client[settings.mongodb_database][settings.mongodb_collection]
|
||||
|
||||
async def setup(self) -> None:
|
||||
await self.collection.create_index("task_id", unique=False)
|
||||
await self.collection.create_index(
|
||||
"url_hash",
|
||||
unique=True,
|
||||
partialFilterExpression={"url_hash": {"$exists": True}},
|
||||
)
|
||||
|
||||
async def store(self, result: CrawlResult) -> None:
|
||||
await self.collection.update_one(
|
||||
{"url_hash": result.url_hash},
|
||||
{"$set": result.to_dict()},
|
||||
upsert=True,
|
||||
)
|
||||
|
||||
async def close(self) -> None:
|
||||
self.client.close()
|
||||
|
||||
|
||||
class PostgresResultRepository(ResultRepository):
|
||||
def __init__(self, settings: StorageSettings) -> None:
|
||||
self.dsn = settings.postgres_dsn
|
||||
self.table = settings.postgres_table
|
||||
self.pool: asyncpg.Pool | None = None
|
||||
|
||||
async def setup(self) -> None:
|
||||
self.pool = await asyncpg.create_pool(self.dsn)
|
||||
async with self.pool.acquire() as connection:
|
||||
await connection.execute(
|
||||
f"""
|
||||
CREATE TABLE IF NOT EXISTS {self.table} (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
task_id TEXT NOT NULL,
|
||||
worker_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
url TEXT NOT NULL,
|
||||
url_hash TEXT NOT NULL,
|
||||
spider TEXT NOT NULL,
|
||||
http_status INTEGER,
|
||||
payload JSONB NOT NULL,
|
||||
error TEXT,
|
||||
finished_at TIMESTAMPTZ NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
await connection.execute(f"ALTER TABLE {self.table} ADD COLUMN IF NOT EXISTS url_hash TEXT")
|
||||
await connection.execute(
|
||||
f"CREATE UNIQUE INDEX IF NOT EXISTS {self.table}_url_hash_uidx ON {self.table} (url_hash)"
|
||||
)
|
||||
|
||||
async def store(self, result: CrawlResult) -> None:
|
||||
if self.pool is None:
|
||||
raise RuntimeError("Postgres repository is not initialized")
|
||||
async with self.pool.acquire() as connection:
|
||||
await connection.execute(
|
||||
f"""
|
||||
INSERT INTO {self.table}
|
||||
(task_id, worker_id, status, url, url_hash, spider, http_status, payload, error, finished_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10::timestamptz)
|
||||
ON CONFLICT (url_hash) DO UPDATE SET
|
||||
task_id = EXCLUDED.task_id,
|
||||
worker_id = EXCLUDED.worker_id,
|
||||
status = EXCLUDED.status,
|
||||
url = EXCLUDED.url,
|
||||
spider = EXCLUDED.spider,
|
||||
http_status = EXCLUDED.http_status,
|
||||
payload = EXCLUDED.payload,
|
||||
error = EXCLUDED.error,
|
||||
finished_at = EXCLUDED.finished_at
|
||||
""",
|
||||
result.task_id,
|
||||
result.worker_id,
|
||||
result.status.value,
|
||||
result.url,
|
||||
result.url_hash,
|
||||
result.spider,
|
||||
result.http_status,
|
||||
json.dumps(result.data, ensure_ascii=False),
|
||||
result.error,
|
||||
result.finished_at,
|
||||
)
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.pool is not None:
|
||||
await self.pool.close()
|
||||
|
||||
|
||||
def build_result_repository(settings: StorageSettings, jsonl_path: Path) -> ResultRepository:
|
||||
backend = settings.backend.lower()
|
||||
if backend == "mongodb":
|
||||
return MongoResultRepository(settings)
|
||||
if backend == "postgresql":
|
||||
return PostgresResultRepository(settings)
|
||||
return JsonlResultRepository(jsonl_path)
|
||||
@@ -0,0 +1,43 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.cron import CronTrigger
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
|
||||
from .config import AppSettings
|
||||
from .models import CrawlTask
|
||||
|
||||
|
||||
class ScheduleService:
|
||||
def __init__(self, settings: AppSettings, submit_callback) -> None:
|
||||
self.settings = settings
|
||||
self.submit_callback = submit_callback
|
||||
self.scheduler = AsyncIOScheduler()
|
||||
|
||||
def start(self) -> None:
|
||||
for item in self.settings.schedules:
|
||||
if item.trigger == "cron" and item.cron:
|
||||
trigger = CronTrigger.from_crontab(item.cron)
|
||||
else:
|
||||
seconds = item.seconds or 0
|
||||
minutes = item.minutes or 0
|
||||
trigger = IntervalTrigger(seconds=seconds, minutes=minutes)
|
||||
self.scheduler.add_job(self._submit_job, trigger=trigger, args=[item], id=item.name, replace_existing=True)
|
||||
self.scheduler.start()
|
||||
|
||||
async def shutdown(self) -> None:
|
||||
self.scheduler.shutdown(wait=False)
|
||||
|
||||
async def _submit_job(self, item) -> None:
|
||||
payload = dict(item.task)
|
||||
task = CrawlTask.create(
|
||||
url=payload["url"],
|
||||
spider=payload.get("spider", self.settings.master.default_spider),
|
||||
method=payload.get("method", "GET"),
|
||||
headers=payload.get("headers"),
|
||||
metadata=payload.get("metadata"),
|
||||
body=payload.get("body"),
|
||||
priority=item.priority,
|
||||
schedule_name=item.name,
|
||||
)
|
||||
await self.submit_callback(task)
|
||||
@@ -0,0 +1,35 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
|
||||
class DualSessionStore:
|
||||
def __init__(self, redis: Redis, redis_prefix: str, backup_path: Path) -> None:
|
||||
self.redis = redis
|
||||
self.redis_prefix = redis_prefix
|
||||
self.backup_path = backup_path
|
||||
self.backup_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _key(self, domain: str) -> str:
|
||||
return f"{self.redis_prefix}:{domain}"
|
||||
|
||||
async def load(self, domain: str) -> dict[str, Any]:
|
||||
raw = await self.redis.get(self._key(domain))
|
||||
if raw:
|
||||
return json.loads(raw)
|
||||
if self.backup_path.exists():
|
||||
data = json.loads(self.backup_path.read_text(encoding="utf-8"))
|
||||
return data.get(domain, {})
|
||||
return {}
|
||||
|
||||
async def save(self, domain: str, payload: dict[str, Any]) -> None:
|
||||
await self.redis.set(self._key(domain), json.dumps(payload))
|
||||
snapshot: dict[str, Any] = {}
|
||||
if self.backup_path.exists():
|
||||
snapshot = json.loads(self.backup_path.read_text(encoding="utf-8"))
|
||||
snapshot[domain] = payload
|
||||
self.backup_path.write_text(json.dumps(snapshot, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
|
||||
class SignatureProvider(ABC):
|
||||
@abstractmethod
|
||||
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]: ...
|
||||
|
||||
|
||||
class NoopSignatureProvider(SignatureProvider):
|
||||
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
|
||||
class RpcSignatureProvider(SignatureProvider):
|
||||
def __init__(self, endpoint: str) -> None:
|
||||
self.endpoint = endpoint
|
||||
|
||||
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]:
|
||||
if not self.endpoint:
|
||||
return {}
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(self.endpoint, json=request_payload) as response:
|
||||
response.raise_for_status()
|
||||
data = await response.json()
|
||||
return dict(data)
|
||||
@@ -0,0 +1,89 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .http import AsyncHttpClient
|
||||
from .models import CrawlResult, CrawlTask, TaskStatus
|
||||
|
||||
|
||||
class BaseSpider(ABC):
|
||||
name: str
|
||||
|
||||
@abstractmethod
|
||||
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult: ...
|
||||
|
||||
|
||||
class GenericFetchSpider(BaseSpider):
|
||||
name = "generic_fetch"
|
||||
|
||||
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
|
||||
response = await client.fetch(task)
|
||||
data: dict[str, Any] = {
|
||||
"headers": response["headers"],
|
||||
"body": response["body"],
|
||||
"proxy": response["proxy"],
|
||||
"request_id": task.metadata.get("request_id"),
|
||||
}
|
||||
status = TaskStatus.SUCCESS if 200 <= response["status"] < 400 else TaskStatus.FAILED
|
||||
return CrawlResult(
|
||||
task_id=task.task_id,
|
||||
worker_id=worker_id,
|
||||
status=status,
|
||||
url=task.url,
|
||||
spider=self.name,
|
||||
http_status=response["status"],
|
||||
data=data,
|
||||
error=None if status == TaskStatus.SUCCESS else f"Unexpected status {response['status']}",
|
||||
)
|
||||
|
||||
|
||||
class AssetDownloadSpider(BaseSpider):
|
||||
name = "asset_download"
|
||||
|
||||
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
|
||||
response = await client.fetch(task)
|
||||
parsed = urlparse(task.url)
|
||||
file_name = task.metadata.get("file_name") or Path(parsed.path or "resource.bin").name or "resource.bin"
|
||||
saved_to = await client.save_binary(task, file_name, response["body_bytes"])
|
||||
return CrawlResult(
|
||||
task_id=task.task_id,
|
||||
worker_id=worker_id,
|
||||
status=TaskStatus.SUCCESS if response["status"] < 400 else TaskStatus.FAILED,
|
||||
url=task.url,
|
||||
spider=self.name,
|
||||
http_status=response["status"],
|
||||
data={"saved_to": saved_to, "proxy": response["proxy"], "request_id": task.metadata.get("request_id")},
|
||||
error=None if response["status"] < 400 else f"Unexpected status {response['status']}",
|
||||
)
|
||||
|
||||
|
||||
class JsonApiSpider(BaseSpider):
|
||||
name = "json_api"
|
||||
|
||||
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
|
||||
response = await client.fetch(task)
|
||||
body = response["body"]
|
||||
if isinstance(body, str):
|
||||
try:
|
||||
body = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
body = {"raw": body}
|
||||
return CrawlResult(
|
||||
task_id=task.task_id,
|
||||
worker_id=worker_id,
|
||||
status=TaskStatus.SUCCESS if response["status"] < 400 else TaskStatus.FAILED,
|
||||
url=task.url,
|
||||
spider=self.name,
|
||||
http_status=response["status"],
|
||||
data={"payload": body, "proxy": response["proxy"], "request_id": task.metadata.get("request_id")},
|
||||
error=None if response["status"] < 400 else f"Unexpected status {response['status']}",
|
||||
)
|
||||
|
||||
|
||||
def build_spider_registry() -> dict[str, BaseSpider]:
|
||||
spiders: list[BaseSpider] = [GenericFetchSpider(), AssetDownloadSpider(), JsonApiSpider()]
|
||||
return {spider.name: spider for spider in spiders}
|
||||
@@ -0,0 +1,85 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from .models import CrawlTask, TaskStatus, utc_now
|
||||
|
||||
|
||||
class CheckpointManager:
|
||||
def __init__(self, redis: Redis, checkpoint_dir: Path, namespace: str) -> None:
|
||||
self.redis = redis
|
||||
self.checkpoint_dir = checkpoint_dir
|
||||
self.namespace = namespace
|
||||
self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _task_key(self, task_id: str) -> str:
|
||||
return f"{self.namespace}:task:{task_id}"
|
||||
|
||||
def _running_key(self, worker_id: str) -> str:
|
||||
return f"{self.namespace}:running:{worker_id}"
|
||||
|
||||
def _file_path(self, task_id: str) -> Path:
|
||||
return self.checkpoint_dir / f"{task_id}.json"
|
||||
|
||||
async def save(self, task: CrawlTask) -> None:
|
||||
task.updated_at = utc_now()
|
||||
await self.redis.set(self._task_key(task.task_id), json.dumps(task.to_dict()))
|
||||
await self._write_file(task)
|
||||
|
||||
async def mark_status(self, task: CrawlTask, status: TaskStatus) -> None:
|
||||
task.status = status
|
||||
await self.save(task)
|
||||
if task.assigned_worker:
|
||||
running_key = self._running_key(task.assigned_worker)
|
||||
if status == TaskStatus.RUNNING:
|
||||
await self.redis.zadd(running_key, {task.task_id: self._score_now()})
|
||||
elif status in {TaskStatus.SUCCESS, TaskStatus.FAILED}:
|
||||
await self.redis.zrem(running_key, task.task_id)
|
||||
self._cleanup_file(task.task_id)
|
||||
|
||||
async def load(self, task_id: str) -> CrawlTask | None:
|
||||
raw = await self.redis.get(self._task_key(task_id))
|
||||
if raw:
|
||||
return CrawlTask.from_dict(json.loads(raw))
|
||||
file_path = self._file_path(task_id)
|
||||
if file_path.exists():
|
||||
return CrawlTask.from_dict(json.loads(file_path.read_text(encoding="utf-8")))
|
||||
return None
|
||||
|
||||
async def recover_stale(self, worker_id: str, older_than_seconds: int) -> list[CrawlTask]:
|
||||
threshold = self._score_now() - older_than_seconds
|
||||
task_ids = await self.redis.zrangebyscore(self._running_key(worker_id), min=0, max=threshold)
|
||||
tasks: list[CrawlTask] = []
|
||||
for task_id in task_ids:
|
||||
task = await self.load(task_id)
|
||||
if task is None:
|
||||
continue
|
||||
if task.status == TaskStatus.RUNNING:
|
||||
task.status = TaskStatus.RETRY
|
||||
tasks.append(task)
|
||||
return tasks
|
||||
|
||||
async def recover_local(self) -> list[CrawlTask]:
|
||||
tasks: list[CrawlTask] = []
|
||||
for file_path in self.checkpoint_dir.glob("*.json"):
|
||||
payload = json.loads(file_path.read_text(encoding="utf-8"))
|
||||
task = CrawlTask.from_dict(payload)
|
||||
if task.status in {TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.RETRY}:
|
||||
tasks.append(task)
|
||||
return tasks
|
||||
|
||||
async def _write_file(self, task: CrawlTask) -> None:
|
||||
self._file_path(task.task_id).write_text(json.dumps(task.to_dict(), ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
def _cleanup_file(self, task_id: str) -> None:
|
||||
file_path = self._file_path(task_id)
|
||||
if file_path.exists():
|
||||
file_path.unlink()
|
||||
|
||||
def _score_now(self) -> float:
|
||||
from time import time
|
||||
|
||||
return time()
|
||||
@@ -0,0 +1,186 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import socket
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
import aiohttp
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from .auth import ConfigurableAuthProvider, ManualAuthProvider
|
||||
from .backoff import exponential_backoff
|
||||
from .config import AppSettings
|
||||
from .http import AsyncHttpClient
|
||||
from .models import CrawlResult, CrawlTask, TaskStatus, WorkerHeartbeat
|
||||
from .proxy import ProxyPool
|
||||
from .queue import RedisPriorityQueue
|
||||
from .rate_limit import DomainRateLimiter
|
||||
from .sessions import DualSessionStore
|
||||
from .signatures import NoopSignatureProvider
|
||||
from .spider import BaseSpider, build_spider_registry
|
||||
from .state import CheckpointManager
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WorkerNode:
|
||||
def __init__(self, settings: AppSettings) -> None:
|
||||
self.settings = settings
|
||||
self.worker_id = settings.worker.worker_id
|
||||
self.redis = Redis.from_url(settings.redis_dsn, decode_responses=True)
|
||||
self.queue = RedisPriorityQueue(
|
||||
self.redis,
|
||||
settings.master.queue_name,
|
||||
settings.master.result_queue_name,
|
||||
settings.master.dead_letter_queue_name,
|
||||
)
|
||||
self.checkpoints = CheckpointManager(self.redis, settings.resolve_path(settings.checkpoint_dir), settings.master.queue_name)
|
||||
self.rate_limiter = DomainRateLimiter(settings.rate_limit)
|
||||
self.proxy_pool = ProxyPool(settings.proxy)
|
||||
self.session_store = DualSessionStore(
|
||||
self.redis,
|
||||
settings.session.redis_prefix,
|
||||
settings.resolve_path(settings.session.local_backup_path),
|
||||
)
|
||||
auth_provider = ConfigurableAuthProvider(settings.auth) if settings.auth.enabled else ManualAuthProvider()
|
||||
self.client = AsyncHttpClient(
|
||||
settings,
|
||||
self.rate_limiter,
|
||||
self.proxy_pool,
|
||||
self.session_store,
|
||||
auth_provider,
|
||||
NoopSignatureProvider(),
|
||||
)
|
||||
self.spiders: dict[str, BaseSpider] = build_spider_registry()
|
||||
self.active_tasks: set[str] = set()
|
||||
self.semaphore = asyncio.Semaphore(settings.worker.concurrency)
|
||||
self.master_base_url = f"http://{settings.master.host}:{settings.master.port}"
|
||||
self._loops: list[asyncio.Task] = []
|
||||
self._http_session: aiohttp.ClientSession | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
self._http_session = aiohttp.ClientSession()
|
||||
await self.client.start()
|
||||
if self.settings.proxy.enabled:
|
||||
await self.proxy_pool.refresh()
|
||||
await self.proxy_pool.verify()
|
||||
recovered = await self.checkpoints.recover_local()
|
||||
for task in recovered:
|
||||
if task.assigned_worker in {None, self.worker_id, "default"}:
|
||||
await self.queue.enqueue_task(self.worker_id, task)
|
||||
self._loops = [
|
||||
asyncio.create_task(self.consume_loop(), name=f"{self.worker_id}-consume"),
|
||||
asyncio.create_task(self.heartbeat_loop(), name=f"{self.worker_id}-heartbeat"),
|
||||
asyncio.create_task(self.proxy_refresh_loop(), name=f"{self.worker_id}-proxy-refresh"),
|
||||
]
|
||||
|
||||
async def close(self) -> None:
|
||||
for task in self._loops:
|
||||
task.cancel()
|
||||
await asyncio.gather(*self._loops, return_exceptions=True)
|
||||
await self.client.close()
|
||||
if self._http_session is not None:
|
||||
await self._http_session.close()
|
||||
await self.redis.aclose()
|
||||
|
||||
async def consume_loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
task = await self.queue.dequeue_task(self.worker_id, timeout=5)
|
||||
if task is None:
|
||||
task = await self.queue.dequeue_task("default", timeout=1)
|
||||
if task is None:
|
||||
continue
|
||||
await self.semaphore.acquire()
|
||||
asyncio.create_task(self.execute_task(task))
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
LOGGER.exception("Worker %s consume loop failed", self.worker_id)
|
||||
await asyncio.sleep(1)
|
||||
|
||||
async def execute_task(self, task: CrawlTask) -> None:
|
||||
self.active_tasks.add(task.task_id)
|
||||
try:
|
||||
task.assigned_worker = self.worker_id
|
||||
task.metadata.setdefault("task_id", task.task_id)
|
||||
task.metadata["worker_id"] = self.worker_id
|
||||
task.metadata["request_id"] = f"{task.task_id}-{task.retries}-{uuid4().hex[:8]}"
|
||||
await self.checkpoints.mark_status(task, TaskStatus.RUNNING)
|
||||
spider = self.spiders.get(task.spider)
|
||||
if spider is None:
|
||||
raise RuntimeError(f"Unknown spider: {task.spider}")
|
||||
result = await spider.execute(task, self.client, self.worker_id)
|
||||
if result.status == TaskStatus.FAILED and task.retries < task.max_retries:
|
||||
await self._retry_task(task, result.error or "worker failure")
|
||||
else:
|
||||
await self.queue.publish_result(result)
|
||||
except Exception as exc:
|
||||
LOGGER.exception("Task %s failed", task.task_id)
|
||||
if task.retries < task.max_retries:
|
||||
await self._retry_task(task, str(exc))
|
||||
else:
|
||||
result = CrawlResult(
|
||||
task_id=task.task_id,
|
||||
worker_id=self.worker_id,
|
||||
status=TaskStatus.FAILED,
|
||||
url=task.url,
|
||||
spider=task.spider,
|
||||
error=str(exc),
|
||||
)
|
||||
await self.queue.publish_result(result)
|
||||
finally:
|
||||
self.active_tasks.discard(task.task_id)
|
||||
self.semaphore.release()
|
||||
|
||||
async def heartbeat_loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
await self.send_heartbeat()
|
||||
await asyncio.sleep(self.settings.worker.heartbeat_interval_seconds)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
LOGGER.exception("Worker %s heartbeat failed", self.worker_id)
|
||||
await asyncio.sleep(2)
|
||||
|
||||
async def send_heartbeat(self) -> None:
|
||||
if self._http_session is None:
|
||||
return
|
||||
heartbeat = WorkerHeartbeat(
|
||||
worker_id=self.worker_id,
|
||||
concurrency=self.settings.worker.concurrency,
|
||||
active_tasks=len(self.active_tasks),
|
||||
host=socket.gethostname(),
|
||||
)
|
||||
async with self._http_session.post(f"{self.master_base_url}/heartbeat", json=heartbeat.to_dict()) as response:
|
||||
response.raise_for_status()
|
||||
|
||||
async def proxy_refresh_loop(self) -> None:
|
||||
if not self.settings.proxy.enabled:
|
||||
return
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(self.settings.proxy.refresh_interval_seconds)
|
||||
await self.proxy_pool.refresh()
|
||||
await self.proxy_pool.verify()
|
||||
LOGGER.info("Worker %s healthy proxies: %s", self.worker_id, await self.proxy_pool.healthy_count())
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
LOGGER.exception("Proxy refresh loop failed")
|
||||
|
||||
async def _retry_task(self, task: CrawlTask, reason: str) -> None:
|
||||
task.retries += 1
|
||||
task.status = TaskStatus.RETRY
|
||||
await self.checkpoints.save(task)
|
||||
delay = exponential_backoff(
|
||||
task.retries - 1,
|
||||
self.settings.worker.base_backoff_seconds,
|
||||
self.settings.worker.max_backoff_seconds,
|
||||
)
|
||||
LOGGER.warning("Retry task %s in %.2fs: %s", task.task_id, delay, reason)
|
||||
await asyncio.sleep(delay)
|
||||
await self.queue.enqueue_task(self.worker_id, task)
|
||||
@@ -0,0 +1,6 @@
|
||||
aiohttp>=3.10.10
|
||||
redis>=5.0.8
|
||||
APScheduler>=3.10.4
|
||||
PyYAML>=6.0.2
|
||||
motor>=3.6.0
|
||||
asyncpg>=0.30.0
|
||||
@@ -0,0 +1,38 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from crawler.config import load_settings
|
||||
from crawler.master import MasterNode
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
settings = load_settings(args.config)
|
||||
node = MasterNode(settings)
|
||||
await node.start()
|
||||
|
||||
runner = web.AppRunner(node.app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, settings.master.host, settings.master.port)
|
||||
await site.start()
|
||||
logging.getLogger(__name__).info("Master started at http://%s:%s", settings.master.host, settings.master.port)
|
||||
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
await node.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from crawler.config import load_settings
|
||||
from crawler.worker import WorkerNode
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", required=True)
|
||||
parser.add_argument("--worker-id", required=False)
|
||||
args = parser.parse_args()
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
settings = load_settings(args.config)
|
||||
if args.worker_id:
|
||||
settings.worker.worker_id = args.worker_id
|
||||
|
||||
node = WorkerNode(settings)
|
||||
await node.start()
|
||||
logging.getLogger(__name__).info("Worker started: %s", settings.worker.worker_id)
|
||||
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
finally:
|
||||
await node.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Reference in new issue
Block a user