Files
SakuraHub/docs/三项核心改进-分页规则_搜索健康分_书架保存解耦.md
T
2026-11-16 20:45:22 +08:00

19 KiB
Raw Blame History

两项核心改进 — TOC 分页规则 · 搜索健康分

日期:2026-06-22 范围:仅移动端 Dart 侧(sakura_mobile/lib/src/core/) 涉及文件:

  • legado_models.dart — 数据模型
  • legado_rule_parser.dart — 规则解析器
  • legado_engine.dart — 搜索引擎入口 原则:不改 Python 插件,不改 UI 层,只动 core 层。

一、改进 1:TOC 章节目录分页规则完整支持

1.1 问题

fetchToc() 只拉第一页目录。很多书源第一页只给 50~100 章,剩下的靠 分页目录规则 指向下一页。不支持分页规则 → 用户看到的目录永远不全。

1.2 兼容的字段名

书源 JSON 中,下一页 URL 的字段名可能是以下任意一种,按优先级遍历取第一个非空值:

// legado_models.dart — BookSource / TocRule 中新增字段解析

static const List<String> _tocNextPageCandidates = [
  'tocNextPageRule',  // Legado 标准名
  'nextTocUrl',       // 常见别名
  'nextUrl',          // 通用别名
  'nextPage',         // 页码式
  'tocNextPage',      // 变体
];

String? get tocNextPageRule {
  for (final name in _tocNextPageCandidates) {
    final v = rawJson[name];
    if (v is String && v.trim().isNotEmpty) return v.trim();
  }
  return null;
}

1.3 分页规则语法全集

规则类型 语法示例 说明
JS 表达式 <js>nextUrl</js> 执行 JS 获取下一页 URL
JS 缩写 @js:nextUrl 同上,简写形式
模板拼接 https://host/list?page={{page}} {{page}} 从 1 自增
偏移模板 {{page+1}} / {{page-1}} 页号偏移
CSS 选择器 text.下一页@href 从当前页 HTML 提取下一页链接
class 选择器 class.next@href 同上
id 选择器 id.pagenext@href 同上
属性选择器 @href 取当前元素 href
XPath //a[contains(text(),'下一页')]/@href XPath 提取
字符串切片 !0:-1 对当前 URL 切片(去掉末尾字符)
offset 翻页 offset=50 按 offset 递增翻页
纯文本 URL https://... 直接跳到指定 URL
相对路径 /list/2.html 基于当前 URL 拼接

1.4 实现 — TocPaginationResolver

// 放在 legado_engine.dart 或新建 legado_toc_pagination.dart

class TocPaginationResolver {
  final LegadoRuleExecutor _ruleExecutor;
  final HttpClient _client;

  TocPaginationResolver(this._ruleExecutor, this._client);

  /// 拉取全部分页章节,合并去重。
  Future<List<ChapterInfo>> fetchAllChapters({
    required String firstPageUrl,
    required String? tocNextPageRule,
    required String tocRule,
    required Map<String, String>? headers,
    int maxPages = 20,
  }) async {
    final all = <ChapterInfo>[];
    final seenPageUrls = <String>{};
    final seenChapterUrls = <String>{};

    String? url = firstPageUrl;
    int page = 0;

    while (url != null && page < maxPages) {
      if (!seenPageUrls.add(url)) break; // 死循环保护

      final html = await _client.get(url, headers: headers);
      final batch = _ruleExecutor.selectList(html, url, rule: tocRule);

      int added = 0;
      for (final ch in batch) {
        if (seenChapterUrls.add(ch.url)) {
          all.add(ch);
          added++;
        }
      }

      if (added == 0) break; // 本页无新章节,终止翻页

      url = _resolveNextPage(
        currentUrl: url,
        currentHtml: html,
        rule: tocNextPageRule,
        pageIndex: page,
      );
      page++;
    }

    return all;
  }

  // ---------- 下一页 URL 解析 ----------

  String? _resolveNextPage({
    required String currentUrl,
    required String currentHtml,
    required String? rule,
    required int pageIndex,
  }) {
    if (rule == null || rule.trim().isEmpty) return null;
    final r = rule.trim();

    // 1) JS: <js>...</js> 或 @js:...
    if (r.startsWith('<js>') || r.startsWith('@js:')) {
      final code = r
          .replaceFirst(RegExp(r'^<js>'), '')
          .replaceFirst(RegExp(r'</js>$'), '')
          .replaceFirst(RegExp(r'^@js:'), '');
      return _ruleExecutor.evalJs(code, context: {
        'result': currentHtml,
        'baseUrl': currentUrl,
        'page': pageIndex,
      });
    }

    // 2) 切片: !start:end
    if (r.startsWith('!')) {
      return _applySlice(currentUrl, r.substring(1));
    }

    // 3) CSS 选择器 / XPath
    if (_isSelector(r)) {
      return _ruleExecutor.select(currentHtml, currentUrl, rule: r);
    }

    // 4) 模板替换 {{page}} / {{page+1}} / {{page-1}}
    if (r.contains('{{page')) {
      return _renderPageTemplate(r, pageIndex);
    }

    // 5) offset / start 参数
    if (RegExp(r'[?&](offset|start)=').hasMatch(r)) {
      return _renderOffsetTemplate(r, pageIndex);
    }

    // 6) 绝对 URL
    if (r.startsWith('http://') || r.startsWith('https://')) {
      return _renderPageTemplate(r, pageIndex);
    }

    // 7) 相对路径
    return Uri.parse(currentUrl).resolve(r).toString();
  }

  // ---------- 规则判定 ----------

  bool _isSelector(String s) {
    return s.startsWith('text.') ||
        s.startsWith('class.') ||
        s.startsWith('id.') ||
        s.startsWith('@') ||
        s.startsWith('//');  // XPath
  }

  // ---------- 模板渲染 ----------

  String _renderPageTemplate(String tpl, int page) {
    return tpl
        .replaceAll('{{page+1}}', '${page + 1}')
        .replaceAll('{{page-1}}', '${page - 1}')
        .replaceAll('{{page}}', '$page');
  }

  String _renderOffsetTemplate(String tpl, int page) {
    return tpl
        .replaceAllMapped(
          RegExp(r'offset=(\d+)'),
          (m) => 'offset=${int.parse(m.group(1)!) * page}',
        )
        .replaceAllMapped(
          RegExp(r'start=(\d+)'),
          (m) => 'start=${int.parse(m.group(1)!) * page}',
        );
  }

  String? _applySlice(String url, String expr) {
    final parts = expr.split(':');
    if (parts.length != 2) return null;
    final a = int.tryParse(parts[0]) ?? 0;
    final b = int.tryParse(parts[1]);
    final start = a < 0 ? url.length + a : a;
    if (b != null) {
      final end = b < 0 ? url.length + b : b;
      return url.substring(start, end);
    }
    return url.substring(start);
  }
}

1.5 集成到 LegadoEngine.fetchToc

// legado_engine.dart — 修改 fetchToc

Future<List<ChapterInfo>> fetchToc(BookSource source, String tocUrl) async {
  final rule = source.tocRule;
  if (rule == null || rule.isEmpty) return [];

  final resolver = TocPaginationResolver(_ruleExecutor, _client);
  return resolver.fetchAllChapters(
    firstPageUrl: tocUrl,
    tocNextPageRule: source.tocNextPageRule, // ← 新增
    tocRule: rule,
    headers: _buildHeaders(source),
    maxPages: 20,
  );
}

1.6 死循环防护(三层)

  1. maxPages = 20 — 单书最多翻 20 页目录
  2. seenPageUrls — 同一 URL 不重复请求
  3. added == 0 终止 — 当前页没有新增章节则停止

二、改进 2:搜索源健康分 + 短超时 + 分阶段返回

2.1 问题

当前 search() 对所有启用源并发请求,每个源 timeout 20~30 秒。10 个源一批,坏源多时整批拖到 20+ 秒才出结果。

2.2 源健康分模型

// legado_models.dart — 新增 SourceHealth

class SourceHealth {
  final String sourceId;

  int successCount = 0;
  int failCount = 0;
  int timeoutCount = 0;
  int totalLatencyMs = 0;
  DateTime? lastSuccessAt;
  DateTime? lastFailAt;
  int _consecutiveFails = 0;

  SourceHealth({required this.sourceId});

  // ---- 判定 ----

  /// 连续失败 ≥ 5 次 → 临时禁用
  bool get isUsable {
    if (_consecutiveFails >= 5) return false;
    if (failCount >= 5 && successCount == 0) return false;
    return true;
  }

  int get consecutiveFails => _consecutiveFails;

  /// 健康分 0~100
  double get score {
    final total = successCount + failCount + timeoutCount;
    if (total == 0) return 50.0;          // 新源默认 50

    double s = 0.0;

    // 成功率 60%
    s += 60.0 * (successCount / total);

    // 平均延迟 25%
    if (successCount > 0 && totalLatencyMs > 0) {
      final avg = totalLatencyMs / successCount;
      if (avg < 500) {
        s += 25.0;
      } else if (avg < 2000) {
        s += 25.0 * (1.0 - (avg - 500) / 1500);
      }
    }

    // 新鲜度 15%(24h 内有成功记录)
    if (lastSuccessAt != null) {
      final h = DateTime.now().difference(lastSuccessAt!).inHours;
      if (h < 1) {
        s += 15.0;
      } else if (h < 24) {
        s += 15.0 * (1.0 - h / 24);
      }
    }

    return s.clamp(0.0, 100.0);
  }

  // ---- 记录 ----

  void recordSuccess(int latencyMs) {
    successCount++;
    totalLatencyMs += latencyMs;
    lastSuccessAt = DateTime.now();
    _consecutiveFails = 0;
  }

  void recordFailure() {
    failCount++;
    lastFailAt = DateTime.now();
    _consecutiveFails++;
  }

  void recordTimeout() {
    timeoutCount++;
    lastFailAt = DateTime.now();
    _consecutiveFails++;
  }

  void resetConsecutiveFails() => _consecutiveFails = 0;

  // ---- 序列化(持久化到 SharedPreferences)----

  Map<String, dynamic> toJson() => {
    'sourceId': sourceId,
    'successCount': successCount,
    'failCount': failCount,
    'timeoutCount': timeoutCount,
    'totalLatencyMs': totalLatencyMs,
    'lastSuccessAt': lastSuccessAt?.toIso8601String(),
    'lastFailAt': lastFailAt?.toIso8601String(),
    'consecutiveFails': _consecutiveFails,
  };

  factory SourceHealth.fromJson(Map<String, dynamic> json) {
    final h = SourceHealth(sourceId: json['sourceId']);
    h.successCount = json['successCount'] ?? 0;
    h.failCount = json['failCount'] ?? 0;
    h.timeoutCount = json['timeoutCount'] ?? 0;
    h.totalLatencyMs = json['totalLatencyMs'] ?? 0;
    h.lastSuccessAt = json['lastSuccessAt'] != null
        ? DateTime.tryParse(json['lastSuccessAt'])
        : null;
    h.lastFailAt = json['lastFailAt'] != null
        ? DateTime.tryParse(json['lastFailAt'])
        : null;
    h._consecutiveFails = json['consecutiveFails'] ?? 0;
    return h;
  }
}

2.3 健康分持久化

// legado_engine.dart — SourceHealthStore

class SourceHealthStore {
  static const _key = 'source_health_v1';
  final Map<String, SourceHealth> _data = {};

  Future<void> load() async {
    final prefs = await SharedPreferences.getInstance();
    final raw = prefs.getStringList(_key);
    if (raw == null) return;
    for (final s in raw) {
      try {
        final h = SourceHealth.fromJson(jsonDecode(s));
        _data[h.sourceId] = h;
      } catch (_) {}
    }
  }

  Future<void> save() async {
    final prefs = await SharedPreferences.getInstance();
    final raw = _data.values.map((h) => jsonEncode(h.toJson())).toList();
    await prefs.setStringList(_key, raw);
  }

  bool isUsable(String id) => _data[id]?.isUsable ?? true;
  double score(String id) => _data[id]?.score ?? 50.0;

  void _ensure(String id) {
    _data.putIfAbsent(id, () => SourceHealth(sourceId: id));
  }

  void recordSuccess(String id, int latencyMs) {
    _ensure(id);
    _data[id]!.recordSuccess(latencyMs);
  }

  void recordFailure(String id) {
    _ensure(id);
    _data[id]!.recordFailure();
  }

  void recordTimeout(String id) {
    _ensure(id);
    _data[id]!.recordTimeout();
  }

  /// 用户手动重置所有源健康状态
  void resetAll() {
    for (final h in _data.values) {
      h.resetConsecutiveFails();
    }
  }
}

2.4 分阶段搜索调度器

// legado_engine.dart — SearchScheduler

class SearchScheduler {
  static const fastTimeout = Duration(seconds: 5);
  static const slowTimeout = Duration(seconds: 8);
  static const maxConcurrent = 6;

  final SourceHealthStore _health;
  final LegadoEngine _engine;

  SearchScheduler(this._health, this._engine);

  /// 返回 Stream,UI 层逐阶段渲染结果
  Stream<SearchPhase> search(String keyword, List<BookSource> allSources) async* {
    // 过滤 + 按健康分排序
    final usable = allSources
        .where((s) => _health.isUsable(s.id))
        .toList()
      ..sort((a, b) => _health.score(b.id).compareTo(_health.score(a.id)));

    if (usable.isEmpty) {
      yield SearchPhase(name: 'error', message: '没有可用的搜索源');
      return;
    }

    // 分三档
    final fast = usable.where((s) => _health.score(s.id) >= 70).toList();
    final normal = usable.where((s) {
      final sc = _health.score(s.id);
      return sc >= 40 && sc < 70;
    }).toList();
    final slow = usable.where((s) => _health.score(s.id) < 40).toList();

    final results = <SearchItem>[];
    final doneIds = <String>{};
    final seen = <String>{};

    // ---- 阶段 1:快速源,超时 5s ----
    yield SearchPhase(name: 'searching', message: '正在搜索 ${usable.length} 个源...');
    final r1 = await _batch(fast, keyword, timeout: fastTimeout, onDone: doneIds.add);
    _merge(results, r1, seen);
    yield SearchPhase(
      name: 'partial',
      items: List.of(results),
      doneCount: doneIds.length,
      totalCount: usable.length,
    );

    // ---- 阶段 2:普通源,超时 8s ----
    final r2 = await _batch(normal, keyword, timeout: slowTimeout, onDone: doneIds.add);
    _merge(results, r2, seen);
    yield SearchPhase(
      name: 'partial',
      items: List.of(results),
      doneCount: doneIds.length,
      totalCount: usable.length,
    );

    // ---- 阶段 3:慢源后台继续,不阻塞 ----
    if (slow.isNotEmpty) {
      // 放入后台队列,不 await
      _batch(slow, keyword, timeout: slowTimeout, onDone: doneIds.add)
          .then((r3) {
        _merge(results, r3, seen);
        // TODO: EventBus 通知 UI 追加
      });
      yield SearchPhase(
        name: 'complete_with_pending',
        items: List.of(results),
        doneCount: doneIds.length,
        totalCount: usable.length,
        message: '${slow.length} 个慢源将在后台继续搜索',
      );
    } else {
      yield SearchPhase(
        name: 'complete',
        items: List.of(results),
        doneCount: doneIds.length,
        totalCount: usable.length,
      );
    }
  }

  // ---- 内部分批并发 ----

  Future<List<SearchItem>> _batch(
    List<BookSource> sources,
    String keyword, {
    required Duration timeout,
    required void Function(String) onDone,
  }) async {
    final out = <SearchItem>[];
    for (var i = 0; i < sources.length; i += maxConcurrent) {
      final chunk = sources.skip(i).take(maxConcurrent).toList();
      final futures = chunk.map((s) => _searchOne(s, keyword, timeout));
      final results = await Future.wait(futures);
      for (var j = 0; j < results.length; j++) {
        if (results[j] != null) out.addAll(results[j]!);
        onDone(chunk[j].id);
      }
    }
    return out;
  }

  Future<List<SearchItem>?> _searchOne(
    BookSource source, String keyword, Duration timeout,
  ) async {
    final sw = Stopwatch()..start();
    try {
      final items = await _engine.search(source, keyword).timeout(timeout);
      sw.stop();
      _health.recordSuccess(source.id, sw.elapsedMilliseconds);
      return items;
    } on TimeoutException {
      _health.recordTimeout(source.id);
      return null;
    } catch (_) {
      _health.recordFailure(source.id);
      return null;
    }
  }

  void _merge(List<SearchItem> all, List<SearchItem> incoming, Set<String> seen) {
    for (final item in incoming) {
      if (seen.add(item.id)) all.add(item);
    }
  }
}

// ---- 阶段结果 ----

class SearchPhase {
  final String name;   // searching | partial | complete | complete_with_pending | error
  final List<SearchItem> items;
  final int doneCount;
  final int totalCount;
  final String? message;

  SearchPhase({
    required this.name,
    this.items = const [],
    this.doneCount = 0,
    this.totalCount = 0,
    this.message,
  });
}

2.5 UI 增量渲染时序

t=0s    → 显示 "正在搜索 N 个源..."
t=0~5s  → 第 1 批结果到达 → 首屏渲染(最快的 5~10 条)
t=5~8s  → 第 2 批结果到达 → 列表追加
t=8s    → 停止等待 → 提示"慢源已跳过,后台补充中"
t=8~15s → 第 3 批结果到达 → 静默追加到列表末尾

2.6 健康分自动衰减

每次 save() 时做一次衰减:

// SourceHealthStore.save() 追加衰减逻辑
Future<void> save() async {
  // 超过 7 天无任何记录 → 衰减 successCount 和 failCount 各减半
  for (final h in _data.values) {
    final latest = h.lastSuccessAt ?? h.lastFailAt;
    if (latest != null &&
        DateTime.now().difference(latest).inDays >= 7) {
      h.successCount = (h.successCount * 0.5).round();
      h.failCount = (h.failCount * 0.5).round();
      h.timeoutCount = (h.timeoutCount * 0.5).round();
    }
  }
  // ... 写入 SharedPreferences
}

三、两改进的协作关系

用户搜索一本书
    │
    ├─ SearchScheduler.search(keyword)
    │     ├─ 按健康分排序源
    │     ├─ 快速源先搜(5s 超时)
    │     ├─ 普通源补搜(8s 超时)
    │     └─ 返回结果列表
    │
    ├─ 用户点击某条结果 → fetchBookInfo → fetchToc
    │     └─ TocPaginationResolver 拉全章节(支持分页规则)
    │
    └─ 用户阅读某章 → fetchChapterContent(按需加载)

改进 1 保证"搜到书后章节不断尾",改进 2 保证"搜索本身不卡"。


四、实施顺序

第 1 步:改进 2 的 SourceHealth + SourceHealthStore
         → 只加数据模型 + 持久化,不影响现有搜索流程
         → 可以先跑几天积累健康数据

第 2 步:改进 2 的 SearchScheduler + 短超时
         → 替换现有 search() 的并发逻辑
         → UI 层适配 Stream<SearchPhase>

第 3 步:改进 1 的 TocPaginationResolver
         → 依赖 RuleParser 链式语法的基础支持
         → 见《Legado 规则引擎重构方案》第 1~3 步

第 4 步:联调 + 回归

五、风险控制

风险 应对
TOC 分页死循环 maxPages=20 + seenUrls 去重 + added==0 终止
健康分误杀快源 只降权不禁用;连续失败 ≥5 才标记不可用 + 手动可重置
短超时漏结果 慢源后台继续,event bus 异步追加,不丢数据
新源全部 50 分 第一轮搜索后就有真实分,冷启动无影响
SharedPreferences 过大 只存 String list,每个源约 200 字节,100 个源 ≈ 20KB