first commit

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
kunkunxiaomaoandClaude committed 2026-11-16 20:45:22 +08:00
commit 9c5fef1a4d
412 files changed
+94582

No files matched your search

+116
View File
@@ -0,0 +1,116 @@
# Distributed Crawler
An enterprise-oriented distributed crawler scaffold built around `aiohttp` and `asyncio`.
## Included capabilities
- Master + Worker topology
- Redis-backed priority queues
- Consistent-hash worker assignment
- Bloom-filter task dedupe
- Dead-letter queue for exhausted failures
- Checkpoint + state recovery
- Task/request/worker trace headers
- Exponential backoff retries
- Domain-aware rate limiting
- Domain-level circuit breaker
- Proxy pool with health verification
- Proxy scoring with weighted selection
- Shared session store with Redis + local backup
- Result aggregation with JSONL, MongoDB, or PostgreSQL sinks and URL-hash dedupe
- APScheduler-driven periodic task submission
## Safety notes
This project intentionally keeps high-risk account automation, captcha bypass, and anti-protection evasion as extension points only.
Use it only against systems you are authorized to access and collect from.
## Directory layout
```text
distributed_crawler/
config.example.yaml
requirements.txt
crawler/
auth.py
backoff.py
bloom.py
config.py
hash_ring.py
http.py
master.py
models.py
proxy.py
queue.py
rate_limit.py
results.py
scheduler.py
sessions.py
signatures.py
spider.py
state.py
worker.py
scripts/
run_master.py
run_worker.py
```
## Quick start
1. Copy `config.example.yaml` to `config.yaml`.
2. Start Redis.
3. Install dependencies from `requirements.txt`.
4. Start the master:
```bash
python scripts/run_master.py --config config.yaml
```
5. Start one or more workers:
```bash
python scripts/run_worker.py --config config.yaml --worker-id worker-1
python scripts/run_worker.py --config config.yaml --worker-id worker-2
```
6. Submit a task:
```bash
curl -X POST http://127.0.0.1:8088/tasks ^
-H "Content-Type: application/json" ^
-d "{\"url\":\"https://httpbin.org/get\",\"spider\":\"generic_fetch\"}"
```
## Master endpoints
- `POST /tasks`
- `POST /heartbeat`
- `GET /workers`
- `GET /stats`
- `GET /health`
- `GET /dashboard` (feature and runtime panel)
## Failure handling and tracing
- Bloom sizing uses `master.bloom_capacity` and `master.bloom_false_positive_rate` to calculate bit count and hash count automatically. Set `master.bloom_hashes` only when you need to override the computed optimum.
- Final failed tasks are appended to `master.dead_letter_queue_name` with the original task, final result, retry count, worker ID, error, and HTTP status.
- Worker requests include `X-Task-Id`, `X-Request-Id`, and `X-Worker-Id` headers. Result payloads also include `request_id` for easier log/result correlation.
## Stability controls
- Proxy endpoints track success count, failure count, latency, cooldown, and a computed score. Workers choose healthy or cooled-down proxies with weighted random selection.
- `circuit_breaker` protects each domain independently. After repeated transport errors or 5xx responses, the domain opens briefly, then moves to half-open for recovery probing.
- Result storage writes a stable `url_hash`. MongoDB and PostgreSQL enforce uniqueness with upsert behavior; JSONL keeps a sidecar `.url_hashes` index.
## Auto login and shared sessions
- Enable `auth.enabled: true` and configure per-domain credentials in `config.yaml`.
- Worker marks sessions expired based on configured status codes/body keywords.
- When expired, worker triggers `AuthProvider.refresh_session()` and writes refreshed cookie to Redis + local backup via `DualSessionStore`.
- All workers read the same session keys (`session.redis_prefix`), so login state is shared.
## Extension points
- `crawler.auth.AuthProvider`
- `crawler.signatures.SignatureProvider`
- `crawler.spider.BaseSpider`
+93
View File
@@ -0,0 +1,93 @@
app_name: distributed-crawler
redis_dsn: redis://127.0.0.1:6379/0
checkpoint_dir: ./runtime/checkpoints
master:
host: 127.0.0.1
port: 8088
worker_ttl_seconds: 30
stale_task_seconds: 120
bloom_capacity: 1000000
bloom_false_positive_rate: 0.01
bloom_hashes:
queue_name: crawler
result_queue_name: crawler:results
dead_letter_queue_name: crawler:dead_letter
default_spider: generic_fetch
worker:
worker_id: worker-1
concurrency: 100
heartbeat_interval_seconds: 5
request_timeout_seconds: 20
connect_timeout_seconds: 10
retries: 4
base_backoff_seconds: 1
max_backoff_seconds: 16
download_dir: ./runtime/downloads
storage:
backend: jsonl
jsonl_path: ./runtime/results/results.jsonl
mongodb_dsn: mongodb://127.0.0.1:27017
mongodb_database: crawler
mongodb_collection: results
postgres_dsn: postgresql://postgres:postgres@127.0.0.1:5432/crawler
postgres_table: crawl_results
proxy:
enabled: false
verify_url: https://httpbin.org/ip
verify_timeout_seconds: 8
refresh_interval_seconds: 300
failure_threshold: 3
cooldown_seconds: 60
static_proxies: []
provider_api_url: ""
provider_auth_header: ""
circuit_breaker:
enabled: true
failure_threshold: 8
recovery_timeout_seconds: 30
half_open_success_threshold: 2
session:
enabled: true
redis_prefix: crawler:sessions
local_backup_path: ./runtime/sessions/sessions.json
auth:
enabled: false
domains:
example.com:
login_url: https://example.com/api/login
username: your_username
password: your_password
username_field: username
password_field: password
submit_type: json # json or form
extra_payload: {}
success_status_codes: [200]
expire_status_codes: [401, 403]
expire_body_keywords: ["login required", "please login", "unauthorized"]
rate_limit:
default_qps: 3.0
default_burst: 6
per_domain:
httpbin.org:
qps: 2.0
burst: 4
schedules:
- name: sample-httpbin
trigger: interval
seconds: 300
priority: medium
task:
spider: generic_fetch
url: https://httpbin.org/get
method: GET
metadata:
source: schedule
+5
View File
@@ -0,0 +1,5 @@
from .config import AppSettings, load_settings
from .master import MasterNode
from .worker import WorkerNode
__all__ = ["AppSettings", "MasterNode", "WorkerNode", "load_settings"]
+75
View File
@@ -0,0 +1,75 @@
from __future__ import annotations
import logging
from abc import ABC, abstractmethod
from typing import Any
import aiohttp
from .config import AuthSettings
LOGGER = logging.getLogger(__name__)
class AuthProvider(ABC):
@abstractmethod
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
"""
Provide an authorized and compliant session refresh implementation for the target system.
"""
class ManualAuthProvider(AuthProvider):
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
raise NotImplementedError(
f"Session refresh for {domain} must be implemented with authorized credentials and target-specific logic."
)
class ConfigurableAuthProvider(AuthProvider):
"""
Generic username/password login provider.
This intentionally keeps site-specific anti-bot behavior out of scope.
"""
def __init__(self, settings: AuthSettings, timeout_seconds: int = 20) -> None:
self.settings = settings
self.timeout_seconds = timeout_seconds
async def refresh_session(self, domain: str, existing_session: dict[str, Any]) -> dict[str, Any]:
if not self.settings.enabled:
return {**existing_session, "expired": False}
cfg = self.settings.domains.get(domain)
if cfg is None or not cfg.login_url or not cfg.username or not cfg.password:
LOGGER.warning("No auth config for domain=%s, keeping existing session", domain)
return {**existing_session, "expired": True}
payload: dict[str, Any] = dict(cfg.extra_payload)
payload[cfg.username_field] = cfg.username
payload[cfg.password_field] = cfg.password
timeout = aiohttp.ClientTimeout(total=self.timeout_seconds)
async with aiohttp.ClientSession(timeout=timeout) as session:
if cfg.submit_type.lower() == "json":
async with session.post(cfg.login_url, json=payload) as response:
body_text = await response.text()
if response.status not in set(cfg.success_status_codes):
raise RuntimeError(f"Login failed for {domain}: {response.status} {body_text[:180]}")
else:
async with session.post(cfg.login_url, data=payload) as response:
body_text = await response.text()
if response.status not in set(cfg.success_status_codes):
raise RuntimeError(f"Login failed for {domain}: {response.status} {body_text[:180]}")
cookie_pairs = [f"{k}={v.value}" for k, v in session.cookie_jar.filter_cookies(cfg.login_url).items()]
cookie_header = "; ".join(cookie_pairs)
if not cookie_header:
# Fallback: keep old cookies if target does not set cookies in login response.
cookie_header = str(existing_session.get("cookie", ""))
return {
**existing_session,
"expired": False,
"cookie": cookie_header,
"updated_by": "ConfigurableAuthProvider",
}
+9
View File
@@ -0,0 +1,9 @@
from __future__ import annotations
import random
def exponential_backoff(attempt: int, base_seconds: float, max_seconds: float) -> float:
delay = min(max_seconds, base_seconds * (2 ** max(attempt, 0)))
jitter = random.uniform(0, delay * 0.2 if delay > 0 else 0)
return delay + jitter
+49
View File
@@ -0,0 +1,49 @@
from __future__ import annotations
import math
from hashlib import md5, sha1
from redis.asyncio import Redis
class RedisBloomFilter:
def __init__(self, redis: Redis, key: str, bits: int = 8_000_000, hashes: int = 6) -> None:
self.redis = redis
self.key = key
self.bits = max(int(bits), 1)
self.hashes = max(int(hashes), 1)
@classmethod
def from_capacity(
cls,
redis: Redis,
key: str,
*,
capacity: int,
false_positive_rate: float = 0.01,
hashes: int | None = None,
) -> "RedisBloomFilter":
capacity = max(int(capacity), 1)
false_positive_rate = min(max(float(false_positive_rate), 1e-9), 0.999999)
bits = math.ceil(-(capacity * math.log(false_positive_rate)) / (math.log(2) ** 2))
optimal_hashes = max(1, round((bits / capacity) * math.log(2)))
return cls(redis, key, bits=bits, hashes=hashes or optimal_hashes)
def _positions(self, value: str) -> list[int]:
digest_a = int(md5(value.encode("utf-8")).hexdigest(), 16)
digest_b = int(sha1(value.encode("utf-8")).hexdigest(), 16)
return [((digest_a + index * digest_b) % self.bits) for index in range(self.hashes)]
async def contains_or_add(self, value: str) -> bool:
positions = self._positions(value)
pipeline = self.redis.pipeline()
for position in positions:
pipeline.getbit(self.key, position)
existing = await pipeline.execute()
seen = all(bit == 1 for bit in existing)
pipeline = self.redis.pipeline()
for position in positions:
pipeline.setbit(self.key, position, 1)
await pipeline.execute()
return seen
@@ -0,0 +1,67 @@
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from time import monotonic
from .config import CircuitBreakerSettings
class CircuitState(str, Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
@dataclass
class Circuit:
state: CircuitState = CircuitState.CLOSED
failures: int = 0
half_open_successes: int = 0
opened_at: float = 0.0
class CircuitOpenError(RuntimeError):
pass
class DomainCircuitBreaker:
def __init__(self, settings: CircuitBreakerSettings) -> None:
self.settings = settings
self._circuits: dict[str, Circuit] = {}
def before_request(self, domain: str) -> None:
if not self.settings.enabled:
return
circuit = self._circuits.setdefault(domain, Circuit())
if circuit.state != CircuitState.OPEN:
return
elapsed = monotonic() - circuit.opened_at
if elapsed >= self.settings.recovery_timeout_seconds:
circuit.state = CircuitState.HALF_OPEN
circuit.half_open_successes = 0
return
raise CircuitOpenError(f"Circuit open for {domain}")
def record_success(self, domain: str) -> None:
if not self.settings.enabled:
return
circuit = self._circuits.setdefault(domain, Circuit())
if circuit.state == CircuitState.HALF_OPEN:
circuit.half_open_successes += 1
if circuit.half_open_successes < self.settings.half_open_success_threshold:
return
circuit.state = CircuitState.CLOSED
circuit.failures = 0
circuit.half_open_successes = 0
circuit.opened_at = 0.0
def record_failure(self, domain: str) -> None:
if not self.settings.enabled:
return
circuit = self._circuits.setdefault(domain, Circuit())
circuit.failures += 1
circuit.half_open_successes = 0
if circuit.failures >= self.settings.failure_threshold or circuit.state == CircuitState.HALF_OPEN:
circuit.state = CircuitState.OPEN
circuit.opened_at = monotonic()
+206
View File
@@ -0,0 +1,206 @@
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
import yaml
@dataclass
class MasterSettings:
host: str = "127.0.0.1"
port: int = 8088
worker_ttl_seconds: int = 30
stale_task_seconds: int = 120
bloom_capacity: int = 1_000_000
bloom_false_positive_rate: float = 0.01
bloom_hashes: int | None = None
queue_name: str = "crawler"
result_queue_name: str = "crawler:results"
dead_letter_queue_name: str = "crawler:dead_letter"
default_spider: str = "generic_fetch"
@dataclass
class WorkerSettings:
worker_id: str = "worker-1"
concurrency: int = 100
heartbeat_interval_seconds: int = 5
request_timeout_seconds: int = 20
connect_timeout_seconds: int = 10
retries: int = 4
base_backoff_seconds: float = 1.0
max_backoff_seconds: float = 16.0
download_dir: str = "./runtime/downloads"
@dataclass
class StorageSettings:
backend: str = "jsonl"
jsonl_path: str = "./runtime/results/results.jsonl"
mongodb_dsn: str = "mongodb://127.0.0.1:27017"
mongodb_database: str = "crawler"
mongodb_collection: str = "results"
postgres_dsn: str = "postgresql://postgres:postgres@127.0.0.1:5432/crawler"
postgres_table: str = "crawl_results"
@dataclass
class ProxySettings:
enabled: bool = False
verify_url: str = "https://httpbin.org/ip"
verify_timeout_seconds: int = 8
refresh_interval_seconds: int = 300
failure_threshold: int = 3
cooldown_seconds: int = 60
static_proxies: list[str] = field(default_factory=list)
provider_api_url: str = ""
provider_auth_header: str = ""
@dataclass
class CircuitBreakerSettings:
enabled: bool = True
failure_threshold: int = 8
recovery_timeout_seconds: int = 30
half_open_success_threshold: int = 2
@dataclass
class SessionSettings:
enabled: bool = True
redis_prefix: str = "crawler:sessions"
local_backup_path: str = "./runtime/sessions/sessions.json"
@dataclass
class AuthDomainSettings:
login_url: str = ""
username: str = ""
password: str = ""
username_field: str = "username"
password_field: str = "password"
submit_type: str = "form"
extra_payload: dict[str, Any] = field(default_factory=dict)
success_status_codes: list[int] = field(default_factory=lambda: [200])
expire_status_codes: list[int] = field(default_factory=lambda: [401, 403])
expire_body_keywords: list[str] = field(default_factory=lambda: ["login required", "please login", "unauthorized"])
@dataclass
class AuthSettings:
enabled: bool = False
domains: dict[str, AuthDomainSettings] = field(default_factory=dict)
@dataclass
class DomainRateLimitSettings:
qps: float = 3.0
burst: int = 6
@dataclass
class RateLimitSettings:
default_qps: float = 3.0
default_burst: int = 6
per_domain: dict[str, DomainRateLimitSettings] = field(default_factory=dict)
@dataclass
class ScheduleTaskSettings:
name: str
trigger: str
task: dict[str, Any]
priority: str = "medium"
seconds: int | None = None
minutes: int | None = None
cron: str | None = None
@dataclass
class AppSettings:
app_name: str = "distributed-crawler"
redis_dsn: str = "redis://127.0.0.1:6379/0"
checkpoint_dir: str = "./runtime/checkpoints"
master: MasterSettings = field(default_factory=MasterSettings)
worker: WorkerSettings = field(default_factory=WorkerSettings)
storage: StorageSettings = field(default_factory=StorageSettings)
proxy: ProxySettings = field(default_factory=ProxySettings)
circuit_breaker: CircuitBreakerSettings = field(default_factory=CircuitBreakerSettings)
session: SessionSettings = field(default_factory=SessionSettings)
auth: AuthSettings = field(default_factory=AuthSettings)
rate_limit: RateLimitSettings = field(default_factory=RateLimitSettings)
schedules: list[ScheduleTaskSettings] = field(default_factory=list)
source_path: Path | None = None
@property
def base_dir(self) -> Path:
if self.source_path is None:
return Path.cwd()
return self.source_path.parent
def resolve_path(self, value: str) -> Path:
candidate = Path(value)
if candidate.is_absolute():
return candidate
return (self.base_dir / candidate).resolve()
def _load_domain_limits(raw_limits: dict[str, Any]) -> dict[str, DomainRateLimitSettings]:
result: dict[str, DomainRateLimitSettings] = {}
for domain, item in raw_limits.items():
result[domain] = DomainRateLimitSettings(
qps=float(item.get("qps", 3.0)),
burst=int(item.get("burst", 6)),
)
return result
def _load_auth_domains(raw_domains: dict[str, Any]) -> dict[str, AuthDomainSettings]:
result: dict[str, AuthDomainSettings] = {}
for domain, item in raw_domains.items():
if not isinstance(item, dict):
continue
result[domain] = AuthDomainSettings(
login_url=str(item.get("login_url", "")),
username=str(item.get("username", "")),
password=str(item.get("password", "")),
username_field=str(item.get("username_field", "username")),
password_field=str(item.get("password_field", "password")),
submit_type=str(item.get("submit_type", "form")),
extra_payload=dict(item.get("extra_payload", {})),
success_status_codes=[int(v) for v in item.get("success_status_codes", [200])],
expire_status_codes=[int(v) for v in item.get("expire_status_codes", [401, 403])],
expire_body_keywords=[str(v).lower() for v in item.get("expire_body_keywords", ["login required", "please login", "unauthorized"])],
)
return result
def load_settings(path: str | Path) -> AppSettings:
config_path = Path(path).resolve()
data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
settings = AppSettings(
app_name=str(data.get("app_name", "distributed-crawler")),
redis_dsn=str(data.get("redis_dsn", "redis://127.0.0.1:6379/0")),
checkpoint_dir=str(data.get("checkpoint_dir", "./runtime/checkpoints")),
master=MasterSettings(**(data.get("master") or {})),
worker=WorkerSettings(**(data.get("worker") or {})),
storage=StorageSettings(**(data.get("storage") or {})),
proxy=ProxySettings(**(data.get("proxy") or {})),
circuit_breaker=CircuitBreakerSettings(**(data.get("circuit_breaker") or {})),
session=SessionSettings(**(data.get("session") or {})),
auth=AuthSettings(
enabled=bool((data.get("auth") or {}).get("enabled", False)),
domains=_load_auth_domains((data.get("auth") or {}).get("domains", {})),
),
rate_limit=RateLimitSettings(
default_qps=float((data.get("rate_limit") or {}).get("default_qps", 3.0)),
default_burst=int((data.get("rate_limit") or {}).get("default_burst", 6)),
per_domain=_load_domain_limits((data.get("rate_limit") or {}).get("per_domain", {})),
),
schedules=[ScheduleTaskSettings(**item) for item in data.get("schedules", [])],
source_path=config_path,
)
return settings
+33
View File
@@ -0,0 +1,33 @@
from __future__ import annotations
from bisect import bisect
from hashlib import md5
class ConsistentHashRing:
def __init__(self, replicas: int = 100) -> None:
self.replicas = replicas
self._keys: list[int] = []
self._nodes: dict[int, str] = {}
def _hash(self, value: str) -> int:
return int(md5(value.encode("utf-8")).hexdigest(), 16)
def rebuild(self, nodes: list[str]) -> None:
self._keys.clear()
self._nodes.clear()
for node in sorted(set(nodes)):
for replica in range(self.replicas):
key = self._hash(f"{node}:{replica}")
self._keys.append(key)
self._nodes[key] = node
self._keys.sort()
def get_node(self, key: str) -> str | None:
if not self._keys:
return None
hashed = self._hash(key)
index = bisect(self._keys, hashed)
if index == len(self._keys):
index = 0
return self._nodes[self._keys[index]]
+180
View File
@@ -0,0 +1,180 @@
from __future__ import annotations
import json
import random
from pathlib import Path
from time import monotonic
from typing import Any
import aiohttp
from .auth import AuthProvider
from .circuit_breaker import DomainCircuitBreaker
from .config import AppSettings
from .models import CrawlTask
from .proxy import ProxyPool
from .rate_limit import DomainRateLimiter
from .sessions import DualSessionStore
from .signatures import SignatureProvider
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/126.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4) AppleWebKit/605.1.15 Version/17.4 Safari/605.1.15",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Edg/126.0.0.0 Safari/537.36",
]
class AsyncHttpClient:
def __init__(
self,
settings: AppSettings,
rate_limiter: DomainRateLimiter,
proxy_pool: ProxyPool,
session_store: DualSessionStore,
auth_provider: AuthProvider,
signature_provider: SignatureProvider,
) -> None:
self.settings = settings
self.rate_limiter = rate_limiter
self.proxy_pool = proxy_pool
self.session_store = session_store
self.auth_provider = auth_provider
self.signature_provider = signature_provider
self.session: aiohttp.ClientSession | None = None
self.circuit_breaker = DomainCircuitBreaker(settings.circuit_breaker)
async def start(self) -> None:
timeout = aiohttp.ClientTimeout(
total=self.settings.worker.request_timeout_seconds,
connect=self.settings.worker.connect_timeout_seconds,
)
connector = aiohttp.TCPConnector(limit=self.settings.worker.concurrency, ssl=False)
self.session = aiohttp.ClientSession(timeout=timeout, connector=connector)
async def close(self) -> None:
if self.session is not None:
await self.session.close()
async def fetch(self, task: CrawlTask) -> dict[str, Any]:
if self.session is None:
raise RuntimeError("HTTP client is not started")
await self.rate_limiter.acquire(task.domain)
self.circuit_breaker.before_request(task.domain)
proxy = await self.proxy_pool.acquire()
session_payload = await self.session_store.load(task.domain)
if self.settings.session.enabled and session_payload.get("expired"):
session_payload = await self.auth_provider.refresh_session(task.domain, session_payload)
await self.session_store.save(task.domain, session_payload)
headers = dict(task.headers)
headers.setdefault("User-Agent", random.choice(USER_AGENTS))
headers.setdefault("Accept", "*/*")
if task.metadata.get("referer"):
headers.setdefault("Referer", str(task.metadata["referer"]))
if task.metadata.get("origin"):
headers.setdefault("Origin", str(task.metadata["origin"]))
headers.setdefault("X-Task-Id", task.task_id)
if task.metadata.get("request_id"):
headers.setdefault("X-Request-Id", str(task.metadata["request_id"]))
if task.metadata.get("worker_id"):
headers.setdefault("X-Worker-Id", str(task.metadata["worker_id"]))
cookie_header = session_payload.get("cookie")
if cookie_header:
headers.setdefault("Cookie", str(cookie_header))
signature_headers = await self.signature_provider.sign(
{
"url": task.url,
"method": task.method,
"body": task.body,
"metadata": task.metadata,
}
)
headers.update(signature_headers)
started_at = monotonic()
try:
async with self.session.request(
task.method,
task.url,
headers=headers,
data=task.body.encode("utf-8") if isinstance(task.body, str) else None,
proxy=proxy.url if proxy else None,
) as response:
content_type = response.headers.get("Content-Type", "")
body_bytes = await response.read()
if "application/json" in content_type:
body: Any = json.loads(body_bytes.decode("utf-8", errors="ignore"))
else:
body = body_bytes.decode("utf-8", errors="ignore")
response_data = {
"status": response.status,
"headers": dict(response.headers),
"body": body,
"body_bytes": body_bytes,
"proxy": proxy.url if proxy else None,
}
if self.settings.session.enabled:
if self._is_session_expired(task.domain, response.status, body):
session_payload["expired"] = True
await self.session_store.save(task.domain, session_payload)
else:
if "Set-Cookie" in response.headers:
cookie_header = self._extract_cookie_header(response)
if cookie_header:
session_payload["cookie"] = cookie_header
session_payload["expired"] = False
await self.session_store.save(task.domain, session_payload)
if response.status >= 500:
self.circuit_breaker.record_failure(task.domain)
else:
self.circuit_breaker.record_success(task.domain)
await self.proxy_pool.mark_success(proxy, (monotonic() - started_at) * 1000)
return response_data
except Exception:
self.circuit_breaker.record_failure(task.domain)
await self.proxy_pool.mark_failed(proxy)
raise
async def save_binary(self, task: CrawlTask, file_name: str, body_bytes: bytes) -> str:
save_dir = self.settings.resolve_path(self.settings.worker.download_dir)
save_dir.mkdir(parents=True, exist_ok=True)
target = save_dir / file_name
target.write_bytes(body_bytes)
return str(target)
def _is_session_expired(self, domain: str, status_code: int, body: Any) -> bool:
if not self.settings.auth.enabled:
return False
domain_cfg = self.settings.auth.domains.get(domain)
if domain_cfg is None:
return False
if status_code in set(domain_cfg.expire_status_codes):
return True
if isinstance(body, str):
lower_body = body.lower()
return any(keyword in lower_body for keyword in domain_cfg.expire_body_keywords)
if isinstance(body, dict):
packed = json.dumps(body, ensure_ascii=False).lower()
return any(keyword in packed for keyword in domain_cfg.expire_body_keywords)
return False
def _extract_cookie_header(self, response: aiohttp.ClientResponse) -> str:
values = response.headers.getall("Set-Cookie", [])
cookie_parts: list[str] = []
for value in values:
head = value.split(";", 1)[0].strip()
if "=" in head:
cookie_parts.append(head)
# keep order and remove duplicates
seen: set[str] = set()
deduped: list[str] = []
for item in cookie_parts:
key = item.split("=", 1)[0].strip()
if key in seen:
continue
seen.add(key)
deduped.append(item)
return "; ".join(deduped)
+346
View File
@@ -0,0 +1,346 @@
from __future__ import annotations
import asyncio
import logging
from time import time
from typing import Any
from html import escape
from aiohttp import web
from redis.asyncio import Redis
from .bloom import RedisBloomFilter
from .config import AppSettings
from .hash_ring import ConsistentHashRing
from .models import CrawlResult, CrawlTask, DeadLetterEntry, TaskPriority, TaskStatus, WorkerHeartbeat
from .queue import RedisPriorityQueue
from .results import ResultRepository, build_result_repository
from .scheduler import ScheduleService
from .state import CheckpointManager
LOGGER = logging.getLogger(__name__)
class MasterNode:
def __init__(self, settings: AppSettings) -> None:
self.settings = settings
self.redis = Redis.from_url(settings.redis_dsn, decode_responses=True)
self.queue = RedisPriorityQueue(
self.redis,
settings.master.queue_name,
settings.master.result_queue_name,
settings.master.dead_letter_queue_name,
)
self.checkpoints = CheckpointManager(self.redis, settings.resolve_path(settings.checkpoint_dir), settings.master.queue_name)
self.bloom = RedisBloomFilter.from_capacity(
self.redis,
key=f"{settings.master.queue_name}:bloom",
capacity=settings.master.bloom_capacity,
false_positive_rate=settings.master.bloom_false_positive_rate,
hashes=settings.master.bloom_hashes,
)
self.hash_ring = ConsistentHashRing()
self.result_repo: ResultRepository = build_result_repository(
settings.storage,
settings.resolve_path(settings.storage.jsonl_path),
)
self.scheduler = ScheduleService(settings, self.submit_task)
self.workers: dict[str, WorkerHeartbeat] = {}
self.app = web.Application()
self.app.add_routes(
[
web.post("/tasks", self.handle_submit_task),
web.post("/heartbeat", self.handle_heartbeat),
web.get("/workers", self.handle_workers),
web.get("/stats", self.handle_stats),
web.get("/health", self.handle_health),
web.get("/dashboard", self.handle_dashboard),
]
)
self._background_tasks: list[asyncio.Task] = []
async def start(self) -> None:
await self.result_repo.setup()
await self._recover_local_checkpoints()
self.scheduler.start()
self._background_tasks = [
asyncio.create_task(self.consume_results_loop(), name="consume-results"),
asyncio.create_task(self.reap_stale_workers_loop(), name="reap-workers"),
]
async def close(self) -> None:
for task in self._background_tasks:
task.cancel()
await asyncio.gather(*self._background_tasks, return_exceptions=True)
await self.scheduler.shutdown()
await self.result_repo.close()
await self.redis.aclose()
async def submit_task(self, task: CrawlTask) -> bool:
duplicate = await self.bloom.contains_or_add(task.fingerprint)
if duplicate:
LOGGER.info("Skip duplicate task %s", task.url)
return False
worker_id = self._assign_worker(task)
task.assigned_worker = worker_id
task.status = TaskStatus.PENDING
await self.checkpoints.save(task)
await self.queue.enqueue_task(worker_id, task)
LOGGER.info("Submitted task %s to %s", task.task_id, worker_id)
return True
async def handle_submit_task(self, request: web.Request) -> web.Response:
payload = await request.json()
task = CrawlTask.create(
url=payload["url"],
spider=payload.get("spider", self.settings.master.default_spider),
method=payload.get("method", "GET"),
headers=payload.get("headers"),
metadata=payload.get("metadata"),
body=payload.get("body"),
priority=payload.get("priority", TaskPriority.MEDIUM.value),
max_retries=int(payload.get("max_retries", self.settings.worker.retries)),
)
accepted = await self.submit_task(task)
return web.json_response({"accepted": accepted, "task_id": task.task_id, "worker_id": task.assigned_worker})
async def handle_heartbeat(self, request: web.Request) -> web.Response:
payload = await request.json()
heartbeat = WorkerHeartbeat(
worker_id=str(payload["worker_id"]),
concurrency=int(payload["concurrency"]),
active_tasks=int(payload["active_tasks"]),
host=payload.get("host"),
tags=list(payload.get("tags", [])),
)
self.workers[heartbeat.worker_id] = heartbeat
self.hash_ring.rebuild(list(self.workers.keys()))
return web.json_response({"ok": True, "known_workers": len(self.workers)})
async def handle_workers(self, _: web.Request) -> web.Response:
return web.json_response({"workers": [item.to_dict() for item in self.workers.values()]})
async def handle_stats(self, _: web.Request) -> web.Response:
return web.json_response(
{
"workers": len(self.workers),
"queues": {
worker_id: {
priority.value: await self.redis.llen(self.queue._task_key(worker_id, priority))
for priority in TaskPriority
}
for worker_id in self.workers or {"default": None}
},
"dead_letters": await self.redis.llen(self.queue.dead_letter_queue_name),
}
)
async def handle_health(self, _: web.Request) -> web.Response:
pong = await self.redis.ping()
return web.json_response({"status": "ok", "redis": bool(pong), "workers": len(self.workers)})
async def handle_dashboard(self, _: web.Request) -> web.Response:
queue_stats = await self._collect_queue_stats()
session_count = await self._count_sessions()
dead_letter_count = await self.redis.llen(self.queue.dead_letter_queue_name)
html = self._render_dashboard_html(
queue_stats=queue_stats,
session_count=session_count,
dead_letter_count=dead_letter_count,
)
return web.Response(text=html, content_type="text/html")
async def consume_results_loop(self) -> None:
while True:
try:
result = await self.queue.consume_result(timeout=5)
if result is None:
continue
await self.process_result(result)
except asyncio.CancelledError:
raise
except Exception:
LOGGER.exception("Result consumer failed")
await asyncio.sleep(1)
async def process_result(self, result: CrawlResult) -> None:
task = await self.checkpoints.load(result.task_id)
if task is None:
LOGGER.warning("Result for unknown task %s", result.task_id)
return
task.status = result.status
await self.checkpoints.mark_status(task, result.status)
await self.result_repo.store(result)
if result.status == TaskStatus.FAILED:
await self.queue.publish_dead_letter(DeadLetterEntry.from_task_result(task, result))
LOGGER.warning(
"Dead-lettered task task_id=%s worker_id=%s request_id=%s reason=%s",
task.task_id,
result.worker_id,
result.data.get("request_id", task.metadata.get("request_id", task.task_id)),
result.error,
)
async def reap_stale_workers_loop(self) -> None:
while True:
try:
await asyncio.sleep(self.settings.master.worker_ttl_seconds)
now = time()
stale_workers = [
worker_id
for worker_id, heartbeat in self.workers.items()
if now - self._heartbeat_epoch(heartbeat.last_seen) > self.settings.master.worker_ttl_seconds
]
for worker_id in stale_workers:
LOGGER.warning("Worker %s is stale, recovering tasks", worker_id)
recovered = await self.checkpoints.recover_stale(worker_id, self.settings.master.stale_task_seconds)
for task in recovered:
task.retries += 1
if task.retries > task.max_retries:
task.status = TaskStatus.FAILED
await self.checkpoints.mark_status(task, TaskStatus.FAILED)
result = CrawlResult(
task_id=task.task_id,
worker_id=worker_id,
status=TaskStatus.FAILED,
url=task.url,
spider=task.spider,
error="Task exceeded retry budget during worker recovery",
)
await self.result_repo.store(result)
await self.queue.publish_dead_letter(DeadLetterEntry.from_task_result(task, result))
continue
task.assigned_worker = self._assign_worker(task, exclude=worker_id)
await self.checkpoints.save(task)
await self.queue.enqueue_task(task.assigned_worker, task)
self.workers.pop(worker_id, None)
self.hash_ring.rebuild(list(self.workers.keys()))
except asyncio.CancelledError:
raise
except Exception:
LOGGER.exception("Stale worker reaper failed")
async def _recover_local_checkpoints(self) -> None:
tasks = await self.checkpoints.recover_local()
for task in tasks:
task.assigned_worker = self._assign_worker(task)
task.status = TaskStatus.RETRY
await self.checkpoints.save(task)
await self.queue.enqueue_task(task.assigned_worker, task)
async def _collect_queue_stats(self) -> dict[str, dict[str, int]]:
worker_keys = list(self.workers.keys()) or ["default"]
result: dict[str, dict[str, int]] = {}
for worker_id in worker_keys:
result[worker_id] = {
priority.value: await self.redis.llen(self.queue._task_key(worker_id, priority))
for priority in TaskPriority
}
return result
async def _count_sessions(self) -> int:
pattern = f"{self.settings.session.redis_prefix}:*"
count = 0
async for _ in self.redis.scan_iter(match=pattern):
count += 1
return count
def _render_dashboard_html(self, queue_stats: dict[str, dict[str, int]], session_count: int, dead_letter_count: int) -> str:
worker_rows = []
for worker_id, hb in self.workers.items():
load = hb.active_tasks / max(hb.concurrency, 1)
worker_rows.append(
f"<tr><td>{escape(worker_id)}</td><td>{hb.active_tasks}</td><td>{hb.concurrency}</td><td>{load:.2f}</td><td>{escape(hb.last_seen)}</td></tr>"
)
if not worker_rows:
worker_rows.append("<tr><td colspan='5'>暂无在线 Worker</td></tr>")
queue_blocks = []
for worker_id, levels in queue_stats.items():
queue_blocks.append(
"<div class='card'><h3>Queue - "
+ escape(worker_id)
+ "</h3><p>high: "
+ str(levels.get("high", 0))
+ " | medium: "
+ str(levels.get("medium", 0))
+ " | low: "
+ str(levels.get("low", 0))
+ "</p></div>"
)
queue_html = "\n".join(queue_blocks)
enabled_auth_domains = ", ".join(sorted(self.settings.auth.domains.keys())) if self.settings.auth.domains else "-"
enabled_proxy = "enabled" if self.settings.proxy.enabled else "disabled"
enabled_session = "enabled" if self.settings.session.enabled else "disabled"
enabled_auth = "enabled" if self.settings.auth.enabled else "disabled"
return f"""<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<title>Distributed Crawler Dashboard</title>
<style>
body {{ font-family: Segoe UI, Arial, sans-serif; background:#f6f8fb; margin:0; padding:24px; color:#1f2d3d; }}
.grid {{ display:grid; grid-template-columns: repeat(auto-fill, minmax(280px,1fr)); gap:12px; margin-bottom:16px; }}
.card {{ background:#fff; border:1px solid #dce6f4; border-radius:10px; padding:12px 14px; }}
h1 {{ margin:0 0 14px 0; font-size:24px; }}
h3 {{ margin:0 0 8px 0; font-size:16px; }}
table {{ width:100%; border-collapse: collapse; background:#fff; border:1px solid #dce6f4; }}
th, td {{ text-align:left; padding:10px; border-bottom:1px solid #eef2f8; font-size:13px; }}
th {{ background:#f1f5fc; }}
.muted {{ color:#607086; font-size:12px; }}
code {{ background:#eef3fb; padding:2px 6px; border-radius:4px; }}
</style>
</head>
<body>
<h1>Distributed Crawler Dashboard</h1>
<div class="grid">
<div class="card"><h3>Workers</h3><p>{len(self.workers)}</p></div>
<div class="card"><h3>Session Keys</h3><p>{session_count}</p></div>
<div class="card"><h3>Dead Letters</h3><p>{dead_letter_count}</p></div>
<div class="card"><h3>Proxy</h3><p>{enabled_proxy}</p></div>
<div class="card"><h3>Auto Login</h3><p>{enabled_auth}</p></div>
<div class="card"><h3>Session Sharing</h3><p>{enabled_session}</p></div>
<div class="card"><h3>Auth Domains</h3><p>{escape(enabled_auth_domains)}</p></div>
</div>
<div class="grid">
{queue_html}
</div>
<div class="card">
<h3>Worker Runtime</h3>
<table>
<thead><tr><th>worker_id</th><th>active_tasks</th><th>concurrency</th><th>load</th><th>last_seen</th></tr></thead>
<tbody>
{''.join(worker_rows)}
</tbody>
</table>
</div>
<p class="muted">API endpoints: <code>/tasks</code> <code>/workers</code> <code>/stats</code> <code>/health</code></p>
</body>
</html>"""
def _assign_worker(self, task: CrawlTask, exclude: str | None = None) -> str:
candidates = [worker_id for worker_id in self.workers if worker_id != exclude]
if not candidates:
return "default"
self.hash_ring.rebuild(candidates)
preferred = self.hash_ring.get_node(task.url) or candidates[0]
preferred_heartbeat = self.workers.get(preferred)
if preferred_heartbeat is None:
return preferred
preferred_load = preferred_heartbeat.active_tasks / max(preferred_heartbeat.concurrency, 1)
if preferred_load < 0.9:
return preferred
least_loaded = min(
candidates,
key=lambda worker_id: self.workers[worker_id].active_tasks / max(self.workers[worker_id].concurrency, 1),
)
return least_loaded
def _heartbeat_epoch(self, timestamp: str) -> float:
from datetime import datetime
return datetime.fromisoformat(timestamp).timestamp()
+189
View File
@@ -0,0 +1,189 @@
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from datetime import UTC, datetime
from enum import Enum
from hashlib import sha1
from typing import Any
from urllib.parse import urlparse
from uuid import uuid4
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def make_url_hash(url: str) -> str:
return sha1(url.encode("utf-8")).hexdigest()
class TaskPriority(str, Enum):
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
class TaskStatus(str, Enum):
PENDING = "pending"
RUNNING = "running"
SUCCESS = "success"
FAILED = "failed"
RETRY = "retry"
@dataclass
class CrawlTask:
task_id: str
url: str
spider: str
method: str = "GET"
headers: dict[str, str] = field(default_factory=dict)
metadata: dict[str, Any] = field(default_factory=dict)
body: str | None = None
priority: TaskPriority = TaskPriority.MEDIUM
status: TaskStatus = TaskStatus.PENDING
retries: int = 0
max_retries: int = 4
assigned_worker: str | None = None
created_at: str = field(default_factory=utc_now)
updated_at: str = field(default_factory=utc_now)
schedule_name: str | None = None
@property
def domain(self) -> str:
return urlparse(self.url).netloc
@property
def fingerprint(self) -> str:
payload = f"{self.method}:{self.url}:{self.spider}:{self.body or ''}"
return sha1(payload.encode("utf-8")).hexdigest()
def to_dict(self) -> dict[str, Any]:
payload = asdict(self)
payload["priority"] = self.priority.value
payload["status"] = self.status.value
return payload
@classmethod
def from_dict(cls, payload: dict[str, Any]) -> "CrawlTask":
return cls(
task_id=str(payload["task_id"]),
url=str(payload["url"]),
spider=str(payload["spider"]),
method=str(payload.get("method", "GET")),
headers=dict(payload.get("headers", {})),
metadata=dict(payload.get("metadata", {})),
body=payload.get("body"),
priority=TaskPriority(payload.get("priority", TaskPriority.MEDIUM.value)),
status=TaskStatus(payload.get("status", TaskStatus.PENDING.value)),
retries=int(payload.get("retries", 0)),
max_retries=int(payload.get("max_retries", 4)),
assigned_worker=payload.get("assigned_worker"),
created_at=str(payload.get("created_at", utc_now())),
updated_at=str(payload.get("updated_at", utc_now())),
schedule_name=payload.get("schedule_name"),
)
@classmethod
def create(
cls,
*,
url: str,
spider: str,
method: str = "GET",
headers: dict[str, str] | None = None,
metadata: dict[str, Any] | None = None,
body: str | None = None,
priority: str = "medium",
max_retries: int = 4,
schedule_name: str | None = None,
) -> "CrawlTask":
return cls(
task_id=uuid4().hex,
url=url,
spider=spider,
method=method.upper(),
headers=headers or {},
metadata=metadata or {},
body=body,
priority=TaskPriority(priority),
max_retries=max_retries,
schedule_name=schedule_name,
)
@dataclass
class CrawlResult:
task_id: str
worker_id: str
status: TaskStatus
url: str
spider: str
http_status: int | None = None
data: dict[str, Any] = field(default_factory=dict)
error: str | None = None
finished_at: str = field(default_factory=utc_now)
@property
def url_hash(self) -> str:
return make_url_hash(self.url)
def to_dict(self) -> dict[str, Any]:
payload = asdict(self)
payload["status"] = self.status.value
payload["url_hash"] = self.url_hash
return payload
@classmethod
def from_dict(cls, payload: dict[str, Any]) -> "CrawlResult":
return cls(
task_id=str(payload["task_id"]),
worker_id=str(payload["worker_id"]),
status=TaskStatus(payload["status"]),
url=str(payload["url"]),
spider=str(payload["spider"]),
http_status=payload.get("http_status"),
data=dict(payload.get("data", {})),
error=payload.get("error"),
finished_at=str(payload.get("finished_at", utc_now())),
)
@dataclass
class DeadLetterEntry:
task: dict[str, Any]
result: dict[str, Any] | None = None
reason: str = ""
failed_at: str = field(default_factory=utc_now)
last_error: str | None = None
response_status: int | None = None
retry_count: int = 0
worker_id: str | None = None
def to_dict(self) -> dict[str, Any]:
return asdict(self)
@classmethod
def from_task_result(cls, task: CrawlTask, result: CrawlResult, reason: str | None = None) -> "DeadLetterEntry":
return cls(
task=task.to_dict(),
result=result.to_dict(),
reason=reason or result.error or "Task failed",
last_error=result.error,
response_status=result.http_status,
retry_count=task.retries,
worker_id=result.worker_id,
)
@dataclass
class WorkerHeartbeat:
worker_id: str
concurrency: int
active_tasks: int
last_seen: str = field(default_factory=utc_now)
host: str | None = None
tags: list[str] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
return asdict(self)
+162
View File
@@ -0,0 +1,162 @@
from __future__ import annotations
import asyncio
import random
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from time import monotonic
import aiohttp
from .config import ProxySettings
@dataclass
class ProxyEndpoint:
url: str
failures: int = 0
successes: int = 0
latency_ms: float | None = None
last_verified_at: float = 0.0
last_failed_at: float = 0.0
healthy: bool = True
metadata: dict[str, str] = field(default_factory=dict)
@property
def success_rate(self) -> float:
total = self.successes + self.failures
if total == 0:
return 1.0
return self.successes / total
@property
def score(self) -> float:
latency_penalty = 1.0
if self.latency_ms is not None:
latency_penalty = max(0.1, min(1.0, 1000.0 / max(self.latency_ms, 1.0)))
failure_penalty = 1.0 / (1.0 + self.failures)
return max(0.01, self.success_rate * latency_penalty * failure_penalty)
class ProxyProvider(ABC):
@abstractmethod
async def fetch(self) -> list[ProxyEndpoint]: ...
class StaticProxyProvider(ProxyProvider):
def __init__(self, proxies: list[str]) -> None:
self.proxies = proxies
async def fetch(self) -> list[ProxyEndpoint]:
return [ProxyEndpoint(url=value) for value in self.proxies]
class ApiProxyProvider(ProxyProvider):
def __init__(self, api_url: str, auth_header: str = "") -> None:
self.api_url = api_url
self.auth_header = auth_header
async def fetch(self) -> list[ProxyEndpoint]:
if not self.api_url:
return []
headers = {"Authorization": self.auth_header} if self.auth_header else {}
async with aiohttp.ClientSession() as session:
async with session.get(self.api_url, headers=headers) as response:
response.raise_for_status()
text = await response.text()
proxies: list[ProxyEndpoint] = []
for line in text.splitlines():
line = line.strip()
if line:
proxies.append(ProxyEndpoint(url=line))
return proxies
class ProxyPool:
def __init__(self, settings: ProxySettings) -> None:
self.settings = settings
self._proxies: list[ProxyEndpoint] = []
self._lock = asyncio.Lock()
self._next_index = 0
async def refresh(self) -> None:
providers: list[ProxyProvider] = [StaticProxyProvider(self.settings.static_proxies)]
if self.settings.provider_api_url:
providers.append(ApiProxyProvider(self.settings.provider_api_url, self.settings.provider_auth_header))
async with self._lock:
existing = {endpoint.url: endpoint for endpoint in self._proxies}
merged: dict[str, ProxyEndpoint] = {}
for provider in providers:
for endpoint in await provider.fetch():
merged.setdefault(endpoint.url, existing.get(endpoint.url, endpoint))
async with self._lock:
self._proxies = list(merged.values())
async def verify(self) -> None:
if not self.settings.enabled:
return
async with self._lock:
proxies = list(self._proxies)
if not proxies:
return
async with aiohttp.ClientSession() as session:
tasks = [self._verify_proxy(session, proxy) for proxy in proxies]
await asyncio.gather(*tasks, return_exceptions=True)
async def acquire(self) -> ProxyEndpoint | None:
if not self.settings.enabled:
return None
async with self._lock:
now = monotonic()
candidates = [
item
for item in self._proxies
if item.healthy or now - item.last_failed_at >= self.settings.cooldown_seconds
]
if not candidates:
return None
proxy = random.choices(candidates, weights=[item.score for item in candidates], k=1)[0]
self._next_index += 1
return proxy
async def mark_success(self, proxy: ProxyEndpoint | None, latency_ms: float | None = None) -> None:
if proxy is None:
return
async with self._lock:
proxy.successes += 1
proxy.failures = 0
proxy.healthy = True
if latency_ms is not None:
proxy.latency_ms = latency_ms
proxy.last_verified_at = monotonic()
async def mark_failed(self, proxy: ProxyEndpoint | None) -> None:
if proxy is None:
return
async with self._lock:
proxy.failures += 1
proxy.last_failed_at = monotonic()
if proxy.failures >= self.settings.failure_threshold:
proxy.healthy = False
async def healthy_count(self) -> int:
async with self._lock:
return len([item for item in self._proxies if item.healthy])
async def _verify_proxy(self, session: aiohttp.ClientSession, proxy: ProxyEndpoint) -> None:
try:
timeout = aiohttp.ClientTimeout(total=self.settings.verify_timeout_seconds)
started_at = monotonic()
async with session.get(self.settings.verify_url, proxy=proxy.url, timeout=timeout) as response:
response.raise_for_status()
await response.read()
proxy.healthy = True
proxy.failures = 0
proxy.successes += 1
proxy.latency_ms = (monotonic() - started_at) * 1000
proxy.last_verified_at = monotonic()
except Exception:
proxy.failures += 1
proxy.last_failed_at = monotonic()
if proxy.failures >= self.settings.failure_threshold:
proxy.healthy = False
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
import json
from redis.asyncio import Redis
from .models import CrawlResult, CrawlTask, DeadLetterEntry, TaskPriority
class RedisPriorityQueue:
def __init__(self, redis: Redis, namespace: str, result_queue_name: str, dead_letter_queue_name: str | None = None) -> None:
self.redis = redis
self.namespace = namespace
self.result_queue_name = result_queue_name
self.dead_letter_queue_name = dead_letter_queue_name or f"{namespace}:dead_letter"
def _task_key(self, worker_id: str, priority: TaskPriority) -> str:
return f"{self.namespace}:tasks:{worker_id}:{priority.value}"
async def enqueue_task(self, worker_id: str, task: CrawlTask) -> None:
await self.redis.rpush(self._task_key(worker_id, task.priority), json.dumps(task.to_dict()))
async def dequeue_task(self, worker_id: str, timeout: int = 5) -> CrawlTask | None:
keys = [self._task_key(worker_id, priority) for priority in (TaskPriority.HIGH, TaskPriority.MEDIUM, TaskPriority.LOW)]
result = await self.redis.blpop(keys, timeout=timeout)
if not result:
return None
_, payload = result
return CrawlTask.from_dict(json.loads(payload))
async def publish_result(self, result: CrawlResult) -> None:
await self.redis.rpush(self.result_queue_name, json.dumps(result.to_dict()))
async def consume_result(self, timeout: int = 5) -> CrawlResult | None:
result = await self.redis.blpop(self.result_queue_name, timeout=timeout)
if not result:
return None
_, payload = result
return CrawlResult.from_dict(json.loads(payload))
async def publish_dead_letter(self, entry: DeadLetterEntry) -> None:
await self.redis.rpush(self.dead_letter_queue_name, json.dumps(entry.to_dict(), ensure_ascii=False))
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from time import monotonic
from .config import RateLimitSettings
@dataclass
class TokenBucket:
qps: float
burst: int
tokens: float
updated_at: float
class DomainRateLimiter:
def __init__(self, settings: RateLimitSettings) -> None:
self.settings = settings
self._buckets: dict[str, TokenBucket] = {}
self._locks: dict[str, asyncio.Lock] = {}
async def acquire(self, domain: str) -> None:
lock = self._locks.setdefault(domain, asyncio.Lock())
async with lock:
bucket = self._buckets.setdefault(domain, self._new_bucket(domain))
while True:
now = monotonic()
elapsed = max(0.0, now - bucket.updated_at)
bucket.tokens = min(bucket.burst, bucket.tokens + elapsed * bucket.qps)
bucket.updated_at = now
if bucket.tokens >= 1:
bucket.tokens -= 1
return
await asyncio.sleep(max(0.05, 1 / bucket.qps))
def _new_bucket(self, domain: str) -> TokenBucket:
domain_settings = self.settings.per_domain.get(domain)
qps = domain_settings.qps if domain_settings else self.settings.default_qps
burst = domain_settings.burst if domain_settings else self.settings.default_burst
return TokenBucket(qps=qps, burst=burst, tokens=float(burst), updated_at=monotonic())
+164
View File
@@ -0,0 +1,164 @@
from __future__ import annotations
import json
from abc import ABC, abstractmethod
from pathlib import Path
import asyncpg
from motor.motor_asyncio import AsyncIOMotorClient
from .config import StorageSettings
from .models import CrawlResult, make_url_hash
class ResultRepository(ABC):
@abstractmethod
async def setup(self) -> None: ...
@abstractmethod
async def store(self, result: CrawlResult) -> None: ...
@abstractmethod
async def close(self) -> None: ...
class JsonlResultRepository(ResultRepository):
def __init__(self, path: Path) -> None:
self.path = path
self.index_path = path.with_suffix(path.suffix + ".url_hashes")
self._seen_url_hashes: set[str] = set()
self.path.parent.mkdir(parents=True, exist_ok=True)
async def setup(self) -> None:
if self.index_path.exists():
self._seen_url_hashes = {
line.strip()
for line in self.index_path.read_text(encoding="utf-8").splitlines()
if line.strip()
}
return
if self.path.exists():
with self.path.open("r", encoding="utf-8") as file_handle:
for line in file_handle:
try:
payload = json.loads(line)
except json.JSONDecodeError:
continue
url_hash = payload.get("url_hash") or (make_url_hash(str(payload["url"])) if payload.get("url") else None)
if url_hash:
self._seen_url_hashes.add(str(url_hash))
if self._seen_url_hashes:
self.index_path.write_text("\n".join(sorted(self._seen_url_hashes)) + "\n", encoding="utf-8")
async def store(self, result: CrawlResult) -> None:
if result.url_hash in self._seen_url_hashes:
return
with self.path.open("a", encoding="utf-8") as file_handle:
file_handle.write(json.dumps(result.to_dict(), ensure_ascii=False) + "\n")
with self.index_path.open("a", encoding="utf-8") as file_handle:
file_handle.write(result.url_hash + "\n")
self._seen_url_hashes.add(result.url_hash)
async def close(self) -> None:
return None
class MongoResultRepository(ResultRepository):
def __init__(self, settings: StorageSettings) -> None:
self.client = AsyncIOMotorClient(settings.mongodb_dsn)
self.collection = self.client[settings.mongodb_database][settings.mongodb_collection]
async def setup(self) -> None:
await self.collection.create_index("task_id", unique=False)
await self.collection.create_index(
"url_hash",
unique=True,
partialFilterExpression={"url_hash": {"$exists": True}},
)
async def store(self, result: CrawlResult) -> None:
await self.collection.update_one(
{"url_hash": result.url_hash},
{"$set": result.to_dict()},
upsert=True,
)
async def close(self) -> None:
self.client.close()
class PostgresResultRepository(ResultRepository):
def __init__(self, settings: StorageSettings) -> None:
self.dsn = settings.postgres_dsn
self.table = settings.postgres_table
self.pool: asyncpg.Pool | None = None
async def setup(self) -> None:
self.pool = await asyncpg.create_pool(self.dsn)
async with self.pool.acquire() as connection:
await connection.execute(
f"""
CREATE TABLE IF NOT EXISTS {self.table} (
id BIGSERIAL PRIMARY KEY,
task_id TEXT NOT NULL,
worker_id TEXT NOT NULL,
status TEXT NOT NULL,
url TEXT NOT NULL,
url_hash TEXT NOT NULL,
spider TEXT NOT NULL,
http_status INTEGER,
payload JSONB NOT NULL,
error TEXT,
finished_at TIMESTAMPTZ NOT NULL
)
"""
)
await connection.execute(f"ALTER TABLE {self.table} ADD COLUMN IF NOT EXISTS url_hash TEXT")
await connection.execute(
f"CREATE UNIQUE INDEX IF NOT EXISTS {self.table}_url_hash_uidx ON {self.table} (url_hash)"
)
async def store(self, result: CrawlResult) -> None:
if self.pool is None:
raise RuntimeError("Postgres repository is not initialized")
async with self.pool.acquire() as connection:
await connection.execute(
f"""
INSERT INTO {self.table}
(task_id, worker_id, status, url, url_hash, spider, http_status, payload, error, finished_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10::timestamptz)
ON CONFLICT (url_hash) DO UPDATE SET
task_id = EXCLUDED.task_id,
worker_id = EXCLUDED.worker_id,
status = EXCLUDED.status,
url = EXCLUDED.url,
spider = EXCLUDED.spider,
http_status = EXCLUDED.http_status,
payload = EXCLUDED.payload,
error = EXCLUDED.error,
finished_at = EXCLUDED.finished_at
""",
result.task_id,
result.worker_id,
result.status.value,
result.url,
result.url_hash,
result.spider,
result.http_status,
json.dumps(result.data, ensure_ascii=False),
result.error,
result.finished_at,
)
async def close(self) -> None:
if self.pool is not None:
await self.pool.close()
def build_result_repository(settings: StorageSettings, jsonl_path: Path) -> ResultRepository:
backend = settings.backend.lower()
if backend == "mongodb":
return MongoResultRepository(settings)
if backend == "postgresql":
return PostgresResultRepository(settings)
return JsonlResultRepository(jsonl_path)
+43
View File
@@ -0,0 +1,43 @@
from __future__ import annotations
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger
from apscheduler.triggers.interval import IntervalTrigger
from .config import AppSettings
from .models import CrawlTask
class ScheduleService:
def __init__(self, settings: AppSettings, submit_callback) -> None:
self.settings = settings
self.submit_callback = submit_callback
self.scheduler = AsyncIOScheduler()
def start(self) -> None:
for item in self.settings.schedules:
if item.trigger == "cron" and item.cron:
trigger = CronTrigger.from_crontab(item.cron)
else:
seconds = item.seconds or 0
minutes = item.minutes or 0
trigger = IntervalTrigger(seconds=seconds, minutes=minutes)
self.scheduler.add_job(self._submit_job, trigger=trigger, args=[item], id=item.name, replace_existing=True)
self.scheduler.start()
async def shutdown(self) -> None:
self.scheduler.shutdown(wait=False)
async def _submit_job(self, item) -> None:
payload = dict(item.task)
task = CrawlTask.create(
url=payload["url"],
spider=payload.get("spider", self.settings.master.default_spider),
method=payload.get("method", "GET"),
headers=payload.get("headers"),
metadata=payload.get("metadata"),
body=payload.get("body"),
priority=item.priority,
schedule_name=item.name,
)
await self.submit_callback(task)
+35
View File
@@ -0,0 +1,35 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from redis.asyncio import Redis
class DualSessionStore:
def __init__(self, redis: Redis, redis_prefix: str, backup_path: Path) -> None:
self.redis = redis
self.redis_prefix = redis_prefix
self.backup_path = backup_path
self.backup_path.parent.mkdir(parents=True, exist_ok=True)
def _key(self, domain: str) -> str:
return f"{self.redis_prefix}:{domain}"
async def load(self, domain: str) -> dict[str, Any]:
raw = await self.redis.get(self._key(domain))
if raw:
return json.loads(raw)
if self.backup_path.exists():
data = json.loads(self.backup_path.read_text(encoding="utf-8"))
return data.get(domain, {})
return {}
async def save(self, domain: str, payload: dict[str, Any]) -> None:
await self.redis.set(self._key(domain), json.dumps(payload))
snapshot: dict[str, Any] = {}
if self.backup_path.exists():
snapshot = json.loads(self.backup_path.read_text(encoding="utf-8"))
snapshot[domain] = payload
self.backup_path.write_text(json.dumps(snapshot, ensure_ascii=False, indent=2), encoding="utf-8")
+30
View File
@@ -0,0 +1,30 @@
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Any
import aiohttp
class SignatureProvider(ABC):
@abstractmethod
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]: ...
class NoopSignatureProvider(SignatureProvider):
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]:
return {}
class RpcSignatureProvider(SignatureProvider):
def __init__(self, endpoint: str) -> None:
self.endpoint = endpoint
async def sign(self, request_payload: dict[str, Any]) -> dict[str, Any]:
if not self.endpoint:
return {}
async with aiohttp.ClientSession() as session:
async with session.post(self.endpoint, json=request_payload) as response:
response.raise_for_status()
data = await response.json()
return dict(data)
+89
View File
@@ -0,0 +1,89 @@
from __future__ import annotations
import json
from abc import ABC, abstractmethod
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
from .http import AsyncHttpClient
from .models import CrawlResult, CrawlTask, TaskStatus
class BaseSpider(ABC):
name: str
@abstractmethod
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult: ...
class GenericFetchSpider(BaseSpider):
name = "generic_fetch"
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
response = await client.fetch(task)
data: dict[str, Any] = {
"headers": response["headers"],
"body": response["body"],
"proxy": response["proxy"],
"request_id": task.metadata.get("request_id"),
}
status = TaskStatus.SUCCESS if 200 <= response["status"] < 400 else TaskStatus.FAILED
return CrawlResult(
task_id=task.task_id,
worker_id=worker_id,
status=status,
url=task.url,
spider=self.name,
http_status=response["status"],
data=data,
error=None if status == TaskStatus.SUCCESS else f"Unexpected status {response['status']}",
)
class AssetDownloadSpider(BaseSpider):
name = "asset_download"
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
response = await client.fetch(task)
parsed = urlparse(task.url)
file_name = task.metadata.get("file_name") or Path(parsed.path or "resource.bin").name or "resource.bin"
saved_to = await client.save_binary(task, file_name, response["body_bytes"])
return CrawlResult(
task_id=task.task_id,
worker_id=worker_id,
status=TaskStatus.SUCCESS if response["status"] < 400 else TaskStatus.FAILED,
url=task.url,
spider=self.name,
http_status=response["status"],
data={"saved_to": saved_to, "proxy": response["proxy"], "request_id": task.metadata.get("request_id")},
error=None if response["status"] < 400 else f"Unexpected status {response['status']}",
)
class JsonApiSpider(BaseSpider):
name = "json_api"
async def execute(self, task: CrawlTask, client: AsyncHttpClient, worker_id: str) -> CrawlResult:
response = await client.fetch(task)
body = response["body"]
if isinstance(body, str):
try:
body = json.loads(body)
except json.JSONDecodeError:
body = {"raw": body}
return CrawlResult(
task_id=task.task_id,
worker_id=worker_id,
status=TaskStatus.SUCCESS if response["status"] < 400 else TaskStatus.FAILED,
url=task.url,
spider=self.name,
http_status=response["status"],
data={"payload": body, "proxy": response["proxy"], "request_id": task.metadata.get("request_id")},
error=None if response["status"] < 400 else f"Unexpected status {response['status']}",
)
def build_spider_registry() -> dict[str, BaseSpider]:
spiders: list[BaseSpider] = [GenericFetchSpider(), AssetDownloadSpider(), JsonApiSpider()]
return {spider.name: spider for spider in spiders}
+85
View File
@@ -0,0 +1,85 @@
from __future__ import annotations
import json
from pathlib import Path
from redis.asyncio import Redis
from .models import CrawlTask, TaskStatus, utc_now
class CheckpointManager:
def __init__(self, redis: Redis, checkpoint_dir: Path, namespace: str) -> None:
self.redis = redis
self.checkpoint_dir = checkpoint_dir
self.namespace = namespace
self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
def _task_key(self, task_id: str) -> str:
return f"{self.namespace}:task:{task_id}"
def _running_key(self, worker_id: str) -> str:
return f"{self.namespace}:running:{worker_id}"
def _file_path(self, task_id: str) -> Path:
return self.checkpoint_dir / f"{task_id}.json"
async def save(self, task: CrawlTask) -> None:
task.updated_at = utc_now()
await self.redis.set(self._task_key(task.task_id), json.dumps(task.to_dict()))
await self._write_file(task)
async def mark_status(self, task: CrawlTask, status: TaskStatus) -> None:
task.status = status
await self.save(task)
if task.assigned_worker:
running_key = self._running_key(task.assigned_worker)
if status == TaskStatus.RUNNING:
await self.redis.zadd(running_key, {task.task_id: self._score_now()})
elif status in {TaskStatus.SUCCESS, TaskStatus.FAILED}:
await self.redis.zrem(running_key, task.task_id)
self._cleanup_file(task.task_id)
async def load(self, task_id: str) -> CrawlTask | None:
raw = await self.redis.get(self._task_key(task_id))
if raw:
return CrawlTask.from_dict(json.loads(raw))
file_path = self._file_path(task_id)
if file_path.exists():
return CrawlTask.from_dict(json.loads(file_path.read_text(encoding="utf-8")))
return None
async def recover_stale(self, worker_id: str, older_than_seconds: int) -> list[CrawlTask]:
threshold = self._score_now() - older_than_seconds
task_ids = await self.redis.zrangebyscore(self._running_key(worker_id), min=0, max=threshold)
tasks: list[CrawlTask] = []
for task_id in task_ids:
task = await self.load(task_id)
if task is None:
continue
if task.status == TaskStatus.RUNNING:
task.status = TaskStatus.RETRY
tasks.append(task)
return tasks
async def recover_local(self) -> list[CrawlTask]:
tasks: list[CrawlTask] = []
for file_path in self.checkpoint_dir.glob("*.json"):
payload = json.loads(file_path.read_text(encoding="utf-8"))
task = CrawlTask.from_dict(payload)
if task.status in {TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.RETRY}:
tasks.append(task)
return tasks
async def _write_file(self, task: CrawlTask) -> None:
self._file_path(task.task_id).write_text(json.dumps(task.to_dict(), ensure_ascii=False, indent=2), encoding="utf-8")
def _cleanup_file(self, task_id: str) -> None:
file_path = self._file_path(task_id)
if file_path.exists():
file_path.unlink()
def _score_now(self) -> float:
from time import time
return time()
+186
View File
@@ -0,0 +1,186 @@
from __future__ import annotations
import asyncio
import logging
import socket
from pathlib import Path
from uuid import uuid4
import aiohttp
from redis.asyncio import Redis
from .auth import ConfigurableAuthProvider, ManualAuthProvider
from .backoff import exponential_backoff
from .config import AppSettings
from .http import AsyncHttpClient
from .models import CrawlResult, CrawlTask, TaskStatus, WorkerHeartbeat
from .proxy import ProxyPool
from .queue import RedisPriorityQueue
from .rate_limit import DomainRateLimiter
from .sessions import DualSessionStore
from .signatures import NoopSignatureProvider
from .spider import BaseSpider, build_spider_registry
from .state import CheckpointManager
LOGGER = logging.getLogger(__name__)
class WorkerNode:
def __init__(self, settings: AppSettings) -> None:
self.settings = settings
self.worker_id = settings.worker.worker_id
self.redis = Redis.from_url(settings.redis_dsn, decode_responses=True)
self.queue = RedisPriorityQueue(
self.redis,
settings.master.queue_name,
settings.master.result_queue_name,
settings.master.dead_letter_queue_name,
)
self.checkpoints = CheckpointManager(self.redis, settings.resolve_path(settings.checkpoint_dir), settings.master.queue_name)
self.rate_limiter = DomainRateLimiter(settings.rate_limit)
self.proxy_pool = ProxyPool(settings.proxy)
self.session_store = DualSessionStore(
self.redis,
settings.session.redis_prefix,
settings.resolve_path(settings.session.local_backup_path),
)
auth_provider = ConfigurableAuthProvider(settings.auth) if settings.auth.enabled else ManualAuthProvider()
self.client = AsyncHttpClient(
settings,
self.rate_limiter,
self.proxy_pool,
self.session_store,
auth_provider,
NoopSignatureProvider(),
)
self.spiders: dict[str, BaseSpider] = build_spider_registry()
self.active_tasks: set[str] = set()
self.semaphore = asyncio.Semaphore(settings.worker.concurrency)
self.master_base_url = f"http://{settings.master.host}:{settings.master.port}"
self._loops: list[asyncio.Task] = []
self._http_session: aiohttp.ClientSession | None = None
async def start(self) -> None:
self._http_session = aiohttp.ClientSession()
await self.client.start()
if self.settings.proxy.enabled:
await self.proxy_pool.refresh()
await self.proxy_pool.verify()
recovered = await self.checkpoints.recover_local()
for task in recovered:
if task.assigned_worker in {None, self.worker_id, "default"}:
await self.queue.enqueue_task(self.worker_id, task)
self._loops = [
asyncio.create_task(self.consume_loop(), name=f"{self.worker_id}-consume"),
asyncio.create_task(self.heartbeat_loop(), name=f"{self.worker_id}-heartbeat"),
asyncio.create_task(self.proxy_refresh_loop(), name=f"{self.worker_id}-proxy-refresh"),
]
async def close(self) -> None:
for task in self._loops:
task.cancel()
await asyncio.gather(*self._loops, return_exceptions=True)
await self.client.close()
if self._http_session is not None:
await self._http_session.close()
await self.redis.aclose()
async def consume_loop(self) -> None:
while True:
try:
task = await self.queue.dequeue_task(self.worker_id, timeout=5)
if task is None:
task = await self.queue.dequeue_task("default", timeout=1)
if task is None:
continue
await self.semaphore.acquire()
asyncio.create_task(self.execute_task(task))
except asyncio.CancelledError:
raise
except Exception:
LOGGER.exception("Worker %s consume loop failed", self.worker_id)
await asyncio.sleep(1)
async def execute_task(self, task: CrawlTask) -> None:
self.active_tasks.add(task.task_id)
try:
task.assigned_worker = self.worker_id
task.metadata.setdefault("task_id", task.task_id)
task.metadata["worker_id"] = self.worker_id
task.metadata["request_id"] = f"{task.task_id}-{task.retries}-{uuid4().hex[:8]}"
await self.checkpoints.mark_status(task, TaskStatus.RUNNING)
spider = self.spiders.get(task.spider)
if spider is None:
raise RuntimeError(f"Unknown spider: {task.spider}")
result = await spider.execute(task, self.client, self.worker_id)
if result.status == TaskStatus.FAILED and task.retries < task.max_retries:
await self._retry_task(task, result.error or "worker failure")
else:
await self.queue.publish_result(result)
except Exception as exc:
LOGGER.exception("Task %s failed", task.task_id)
if task.retries < task.max_retries:
await self._retry_task(task, str(exc))
else:
result = CrawlResult(
task_id=task.task_id,
worker_id=self.worker_id,
status=TaskStatus.FAILED,
url=task.url,
spider=task.spider,
error=str(exc),
)
await self.queue.publish_result(result)
finally:
self.active_tasks.discard(task.task_id)
self.semaphore.release()
async def heartbeat_loop(self) -> None:
while True:
try:
await self.send_heartbeat()
await asyncio.sleep(self.settings.worker.heartbeat_interval_seconds)
except asyncio.CancelledError:
raise
except Exception:
LOGGER.exception("Worker %s heartbeat failed", self.worker_id)
await asyncio.sleep(2)
async def send_heartbeat(self) -> None:
if self._http_session is None:
return
heartbeat = WorkerHeartbeat(
worker_id=self.worker_id,
concurrency=self.settings.worker.concurrency,
active_tasks=len(self.active_tasks),
host=socket.gethostname(),
)
async with self._http_session.post(f"{self.master_base_url}/heartbeat", json=heartbeat.to_dict()) as response:
response.raise_for_status()
async def proxy_refresh_loop(self) -> None:
if not self.settings.proxy.enabled:
return
while True:
try:
await asyncio.sleep(self.settings.proxy.refresh_interval_seconds)
await self.proxy_pool.refresh()
await self.proxy_pool.verify()
LOGGER.info("Worker %s healthy proxies: %s", self.worker_id, await self.proxy_pool.healthy_count())
except asyncio.CancelledError:
raise
except Exception:
LOGGER.exception("Proxy refresh loop failed")
async def _retry_task(self, task: CrawlTask, reason: str) -> None:
task.retries += 1
task.status = TaskStatus.RETRY
await self.checkpoints.save(task)
delay = exponential_backoff(
task.retries - 1,
self.settings.worker.base_backoff_seconds,
self.settings.worker.max_backoff_seconds,
)
LOGGER.warning("Retry task %s in %.2fs: %s", task.task_id, delay, reason)
await asyncio.sleep(delay)
await self.queue.enqueue_task(self.worker_id, task)
+6
View File
@@ -0,0 +1,6 @@
aiohttp>=3.10.10
redis>=5.0.8
APScheduler>=3.10.4
PyYAML>=6.0.2
motor>=3.6.0
asyncpg>=0.30.0
+38
View File
@@ -0,0 +1,38 @@
from __future__ import annotations
import argparse
import asyncio
import logging
from aiohttp import web
from crawler.config import load_settings
from crawler.master import MasterNode
async def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", required=True)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
settings = load_settings(args.config)
node = MasterNode(settings)
await node.start()
runner = web.AppRunner(node.app)
await runner.setup()
site = web.TCPSite(runner, settings.master.host, settings.master.port)
await site.start()
logging.getLogger(__name__).info("Master started at http://%s:%s", settings.master.host, settings.master.port)
try:
while True:
await asyncio.sleep(3600)
finally:
await runner.cleanup()
await node.close()
if __name__ == "__main__":
asyncio.run(main())
+34
View File
@@ -0,0 +1,34 @@
from __future__ import annotations
import argparse
import asyncio
import logging
from crawler.config import load_settings
from crawler.worker import WorkerNode
async def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", required=True)
parser.add_argument("--worker-id", required=False)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
settings = load_settings(args.config)
if args.worker_id:
settings.worker.worker_id = args.worker_id
node = WorkerNode(settings)
await node.start()
logging.getLogger(__name__).info("Worker started: %s", settings.worker.worker_id)
try:
while True:
await asyncio.sleep(3600)
finally:
await node.close()
if __name__ == "__main__":
asyncio.run(main())