Files
SakuraHub/docs/三项核心改进-分页规则_搜索健康分_书架保存解耦.md
2026-11-16 20:45:22 +08:00

657 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 两项核心改进 — TOC 分页规则 · 搜索健康分
> 日期:2026-06-22
> 范围:仅移动端 Dart 侧(`sakura_mobile/lib/src/core/`)
> 涉及文件:
>
> - `legado_models.dart` — 数据模型
> - `legado_rule_parser.dart` — 规则解析器
> - `legado_engine.dart` — 搜索引擎入口
> 原则:不改 Python 插件,不改 UI 层,只动 core 层。
---
## 一、改进 1:TOC 章节目录分页规则完整支持
### 1.1 问题
`fetchToc()` 只拉第一页目录。很多书源第一页只给 50~100 章,剩下的靠 **分页目录规则** 指向下一页。不支持分页规则 → 用户看到的目录永远不全。
### 1.2 兼容的字段名
书源 JSON 中,下一页 URL 的字段名可能是以下任意一种,按优先级遍历取第一个非空值:
```dart
// legado_models.dart — BookSource / TocRule 中新增字段解析
static const List<String> _tocNextPageCandidates = [
'tocNextPageRule', // Legado 标准名
'nextTocUrl', // 常见别名
'nextUrl', // 通用别名
'nextPage', // 页码式
'tocNextPage', // 变体
];
String? get tocNextPageRule {
for (final name in _tocNextPageCandidates) {
final v = rawJson[name];
if (v is String && v.trim().isNotEmpty) return v.trim();
}
return null;
}
```
### 1.3 分页规则语法全集
| 规则类型 | 语法示例 | 说明 |
| --------- | ----------------------------------- | ------------------ |
| JS 表达式 | `<js>nextUrl</js>` | 执行 JS 获取下一页 URL |
| JS 缩写 | `@js:nextUrl` | 同上,简写形式 |
| 模板拼接 | `https://host/list?page={{page}}` | `{{page}}` 从 1 自增 |
| 偏移模板 | `{{page+1}}` / `{{page-1}}` | 页号偏移 |
| CSS 选择器 | `text.下一页@href` | 从当前页 HTML 提取下一页链接 |
| class 选择器 | `class.next@href` | 同上 |
| id 选择器 | `id.pagenext@href` | 同上 |
| 属性选择器 | `@href` | 取当前元素 href |
| XPath | `//a[contains(text(),'下一页')]/@href` | XPath 提取 |
| 字符串切片 | `!0:-1` | 对当前 URL 切片(去掉末尾字符) |
| offset 翻页 | `offset=50` | 按 offset 递增翻页 |
| 纯文本 URL | `https://...` | 直接跳到指定 URL |
| 相对路径 | `/list/2.html` | 基于当前 URL 拼接 |
### 1.4 实现 — `TocPaginationResolver`
```dart
// 放在 legado_engine.dart 或新建 legado_toc_pagination.dart
class TocPaginationResolver {
final LegadoRuleExecutor _ruleExecutor;
final HttpClient _client;
TocPaginationResolver(this._ruleExecutor, this._client);
/// 拉取全部分页章节,合并去重。
Future<List<ChapterInfo>> fetchAllChapters({
required String firstPageUrl,
required String? tocNextPageRule,
required String tocRule,
required Map<String, String>? headers,
int maxPages = 20,
}) async {
final all = <ChapterInfo>[];
final seenPageUrls = <String>{};
final seenChapterUrls = <String>{};
String? url = firstPageUrl;
int page = 0;
while (url != null && page < maxPages) {
if (!seenPageUrls.add(url)) break; // 死循环保护
final html = await _client.get(url, headers: headers);
final batch = _ruleExecutor.selectList(html, url, rule: tocRule);
int added = 0;
for (final ch in batch) {
if (seenChapterUrls.add(ch.url)) {
all.add(ch);
added++;
}
}
if (added == 0) break; // 本页无新章节,终止翻页
url = _resolveNextPage(
currentUrl: url,
currentHtml: html,
rule: tocNextPageRule,
pageIndex: page,
);
page++;
}
return all;
}
// ---------- 下一页 URL 解析 ----------
String? _resolveNextPage({
required String currentUrl,
required String currentHtml,
required String? rule,
required int pageIndex,
}) {
if (rule == null || rule.trim().isEmpty) return null;
final r = rule.trim();
// 1) JS: <js>...</js> 或 @js:...
if (r.startsWith('<js>') || r.startsWith('@js:')) {
final code = r
.replaceFirst(RegExp(r'^<js>'), '')
.replaceFirst(RegExp(r'</js>$'), '')
.replaceFirst(RegExp(r'^@js:'), '');
return _ruleExecutor.evalJs(code, context: {
'result': currentHtml,
'baseUrl': currentUrl,
'page': pageIndex,
});
}
// 2) 切片: !start:end
if (r.startsWith('!')) {
return _applySlice(currentUrl, r.substring(1));
}
// 3) CSS 选择器 / XPath
if (_isSelector(r)) {
return _ruleExecutor.select(currentHtml, currentUrl, rule: r);
}
// 4) 模板替换 {{page}} / {{page+1}} / {{page-1}}
if (r.contains('{{page')) {
return _renderPageTemplate(r, pageIndex);
}
// 5) offset / start 参数
if (RegExp(r'[?&](offset|start)=').hasMatch(r)) {
return _renderOffsetTemplate(r, pageIndex);
}
// 6) 绝对 URL
if (r.startsWith('http://') || r.startsWith('https://')) {
return _renderPageTemplate(r, pageIndex);
}
// 7) 相对路径
return Uri.parse(currentUrl).resolve(r).toString();
}
// ---------- 规则判定 ----------
bool _isSelector(String s) {
return s.startsWith('text.') ||
s.startsWith('class.') ||
s.startsWith('id.') ||
s.startsWith('@') ||
s.startsWith('//'); // XPath
}
// ---------- 模板渲染 ----------
String _renderPageTemplate(String tpl, int page) {
return tpl
.replaceAll('{{page+1}}', '${page + 1}')
.replaceAll('{{page-1}}', '${page - 1}')
.replaceAll('{{page}}', '$page');
}
String _renderOffsetTemplate(String tpl, int page) {
return tpl
.replaceAllMapped(
RegExp(r'offset=(\d+)'),
(m) => 'offset=${int.parse(m.group(1)!) * page}',
)
.replaceAllMapped(
RegExp(r'start=(\d+)'),
(m) => 'start=${int.parse(m.group(1)!) * page}',
);
}
String? _applySlice(String url, String expr) {
final parts = expr.split(':');
if (parts.length != 2) return null;
final a = int.tryParse(parts[0]) ?? 0;
final b = int.tryParse(parts[1]);
final start = a < 0 ? url.length + a : a;
if (b != null) {
final end = b < 0 ? url.length + b : b;
return url.substring(start, end);
}
return url.substring(start);
}
}
```
### 1.5 集成到 `LegadoEngine.fetchToc`
```dart
// legado_engine.dart — 修改 fetchToc
Future<List<ChapterInfo>> fetchToc(BookSource source, String tocUrl) async {
final rule = source.tocRule;
if (rule == null || rule.isEmpty) return [];
final resolver = TocPaginationResolver(_ruleExecutor, _client);
return resolver.fetchAllChapters(
firstPageUrl: tocUrl,
tocNextPageRule: source.tocNextPageRule, // ← 新增
tocRule: rule,
headers: _buildHeaders(source),
maxPages: 20,
);
}
```
### 1.6 死循环防护(三层)
1. **`maxPages = 20`** — 单书最多翻 20 页目录
2. **`seenPageUrls`** — 同一 URL 不重复请求
3. **`added == 0` 终止** — 当前页没有新增章节则停止
---
## 二、改进 2:搜索源健康分 + 短超时 + 分阶段返回
### 2.1 问题
当前 `search()` 对所有启用源并发请求,每个源 timeout 20~30 秒。10 个源一批,坏源多时整批拖到 20+ 秒才出结果。
### 2.2 源健康分模型
```dart
// legado_models.dart — 新增 SourceHealth
class SourceHealth {
final String sourceId;
int successCount = 0;
int failCount = 0;
int timeoutCount = 0;
int totalLatencyMs = 0;
DateTime? lastSuccessAt;
DateTime? lastFailAt;
int _consecutiveFails = 0;
SourceHealth({required this.sourceId});
// ---- 判定 ----
/// 连续失败 ≥ 5 次 → 临时禁用
bool get isUsable {
if (_consecutiveFails >= 5) return false;
if (failCount >= 5 && successCount == 0) return false;
return true;
}
int get consecutiveFails => _consecutiveFails;
/// 健康分 0~100
double get score {
final total = successCount + failCount + timeoutCount;
if (total == 0) return 50.0; // 新源默认 50
double s = 0.0;
// 成功率 60%
s += 60.0 * (successCount / total);
// 平均延迟 25%
if (successCount > 0 && totalLatencyMs > 0) {
final avg = totalLatencyMs / successCount;
if (avg < 500) {
s += 25.0;
} else if (avg < 2000) {
s += 25.0 * (1.0 - (avg - 500) / 1500);
}
}
// 新鲜度 15%(24h 内有成功记录)
if (lastSuccessAt != null) {
final h = DateTime.now().difference(lastSuccessAt!).inHours;
if (h < 1) {
s += 15.0;
} else if (h < 24) {
s += 15.0 * (1.0 - h / 24);
}
}
return s.clamp(0.0, 100.0);
}
// ---- 记录 ----
void recordSuccess(int latencyMs) {
successCount++;
totalLatencyMs += latencyMs;
lastSuccessAt = DateTime.now();
_consecutiveFails = 0;
}
void recordFailure() {
failCount++;
lastFailAt = DateTime.now();
_consecutiveFails++;
}
void recordTimeout() {
timeoutCount++;
lastFailAt = DateTime.now();
_consecutiveFails++;
}
void resetConsecutiveFails() => _consecutiveFails = 0;
// ---- 序列化(持久化到 SharedPreferences)----
Map<String, dynamic> toJson() => {
'sourceId': sourceId,
'successCount': successCount,
'failCount': failCount,
'timeoutCount': timeoutCount,
'totalLatencyMs': totalLatencyMs,
'lastSuccessAt': lastSuccessAt?.toIso8601String(),
'lastFailAt': lastFailAt?.toIso8601String(),
'consecutiveFails': _consecutiveFails,
};
factory SourceHealth.fromJson(Map<String, dynamic> json) {
final h = SourceHealth(sourceId: json['sourceId']);
h.successCount = json['successCount'] ?? 0;
h.failCount = json['failCount'] ?? 0;
h.timeoutCount = json['timeoutCount'] ?? 0;
h.totalLatencyMs = json['totalLatencyMs'] ?? 0;
h.lastSuccessAt = json['lastSuccessAt'] != null
? DateTime.tryParse(json['lastSuccessAt'])
: null;
h.lastFailAt = json['lastFailAt'] != null
? DateTime.tryParse(json['lastFailAt'])
: null;
h._consecutiveFails = json['consecutiveFails'] ?? 0;
return h;
}
}
```
### 2.3 健康分持久化
```dart
// legado_engine.dart — SourceHealthStore
class SourceHealthStore {
static const _key = 'source_health_v1';
final Map<String, SourceHealth> _data = {};
Future<void> load() async {
final prefs = await SharedPreferences.getInstance();
final raw = prefs.getStringList(_key);
if (raw == null) return;
for (final s in raw) {
try {
final h = SourceHealth.fromJson(jsonDecode(s));
_data[h.sourceId] = h;
} catch (_) {}
}
}
Future<void> save() async {
final prefs = await SharedPreferences.getInstance();
final raw = _data.values.map((h) => jsonEncode(h.toJson())).toList();
await prefs.setStringList(_key, raw);
}
bool isUsable(String id) => _data[id]?.isUsable ?? true;
double score(String id) => _data[id]?.score ?? 50.0;
void _ensure(String id) {
_data.putIfAbsent(id, () => SourceHealth(sourceId: id));
}
void recordSuccess(String id, int latencyMs) {
_ensure(id);
_data[id]!.recordSuccess(latencyMs);
}
void recordFailure(String id) {
_ensure(id);
_data[id]!.recordFailure();
}
void recordTimeout(String id) {
_ensure(id);
_data[id]!.recordTimeout();
}
/// 用户手动重置所有源健康状态
void resetAll() {
for (final h in _data.values) {
h.resetConsecutiveFails();
}
}
}
```
### 2.4 分阶段搜索调度器
```dart
// legado_engine.dart — SearchScheduler
class SearchScheduler {
static const fastTimeout = Duration(seconds: 5);
static const slowTimeout = Duration(seconds: 8);
static const maxConcurrent = 6;
final SourceHealthStore _health;
final LegadoEngine _engine;
SearchScheduler(this._health, this._engine);
/// 返回 Stream,UI 层逐阶段渲染结果
Stream<SearchPhase> search(String keyword, List<BookSource> allSources) async* {
// 过滤 + 按健康分排序
final usable = allSources
.where((s) => _health.isUsable(s.id))
.toList()
..sort((a, b) => _health.score(b.id).compareTo(_health.score(a.id)));
if (usable.isEmpty) {
yield SearchPhase(name: 'error', message: '没有可用的搜索源');
return;
}
// 分三档
final fast = usable.where((s) => _health.score(s.id) >= 70).toList();
final normal = usable.where((s) {
final sc = _health.score(s.id);
return sc >= 40 && sc < 70;
}).toList();
final slow = usable.where((s) => _health.score(s.id) < 40).toList();
final results = <SearchItem>[];
final doneIds = <String>{};
final seen = <String>{};
// ---- 阶段 1:快速源,超时 5s ----
yield SearchPhase(name: 'searching', message: '正在搜索 ${usable.length} 个源...');
final r1 = await _batch(fast, keyword, timeout: fastTimeout, onDone: doneIds.add);
_merge(results, r1, seen);
yield SearchPhase(
name: 'partial',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
// ---- 阶段 2:普通源,超时 8s ----
final r2 = await _batch(normal, keyword, timeout: slowTimeout, onDone: doneIds.add);
_merge(results, r2, seen);
yield SearchPhase(
name: 'partial',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
// ---- 阶段 3:慢源后台继续,不阻塞 ----
if (slow.isNotEmpty) {
// 放入后台队列,不 await
_batch(slow, keyword, timeout: slowTimeout, onDone: doneIds.add)
.then((r3) {
_merge(results, r3, seen);
// TODO: EventBus 通知 UI 追加
});
yield SearchPhase(
name: 'complete_with_pending',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
message: '${slow.length} 个慢源将在后台继续搜索',
);
} else {
yield SearchPhase(
name: 'complete',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
}
}
// ---- 内部分批并发 ----
Future<List<SearchItem>> _batch(
List<BookSource> sources,
String keyword, {
required Duration timeout,
required void Function(String) onDone,
}) async {
final out = <SearchItem>[];
for (var i = 0; i < sources.length; i += maxConcurrent) {
final chunk = sources.skip(i).take(maxConcurrent).toList();
final futures = chunk.map((s) => _searchOne(s, keyword, timeout));
final results = await Future.wait(futures);
for (var j = 0; j < results.length; j++) {
if (results[j] != null) out.addAll(results[j]!);
onDone(chunk[j].id);
}
}
return out;
}
Future<List<SearchItem>?> _searchOne(
BookSource source, String keyword, Duration timeout,
) async {
final sw = Stopwatch()..start();
try {
final items = await _engine.search(source, keyword).timeout(timeout);
sw.stop();
_health.recordSuccess(source.id, sw.elapsedMilliseconds);
return items;
} on TimeoutException {
_health.recordTimeout(source.id);
return null;
} catch (_) {
_health.recordFailure(source.id);
return null;
}
}
void _merge(List<SearchItem> all, List<SearchItem> incoming, Set<String> seen) {
for (final item in incoming) {
if (seen.add(item.id)) all.add(item);
}
}
}
// ---- 阶段结果 ----
class SearchPhase {
final String name; // searching | partial | complete | complete_with_pending | error
final List<SearchItem> items;
final int doneCount;
final int totalCount;
final String? message;
SearchPhase({
required this.name,
this.items = const [],
this.doneCount = 0,
this.totalCount = 0,
this.message,
});
}
```
### 2.5 UI 增量渲染时序
```
t=0s → 显示 "正在搜索 N 个源..."
t=0~5s → 第 1 批结果到达 → 首屏渲染(最快的 5~10 条)
t=5~8s → 第 2 批结果到达 → 列表追加
t=8s → 停止等待 → 提示"慢源已跳过,后台补充中"
t=8~15s → 第 3 批结果到达 → 静默追加到列表末尾
```
### 2.6 健康分自动衰减
每次 `save()` 时做一次衰减:
```dart
// SourceHealthStore.save() 追加衰减逻辑
Future<void> save() async {
// 超过 7 天无任何记录 → 衰减 successCount 和 failCount 各减半
for (final h in _data.values) {
final latest = h.lastSuccessAt ?? h.lastFailAt;
if (latest != null &&
DateTime.now().difference(latest).inDays >= 7) {
h.successCount = (h.successCount * 0.5).round();
h.failCount = (h.failCount * 0.5).round();
h.timeoutCount = (h.timeoutCount * 0.5).round();
}
}
// ... 写入 SharedPreferences
}
```
---
## 三、两改进的协作关系
```
用户搜索一本书
│
├─ SearchScheduler.search(keyword)
│ ├─ 按健康分排序源
│ ├─ 快速源先搜(5s 超时)
│ ├─ 普通源补搜(8s 超时)
│ └─ 返回结果列表
│
├─ 用户点击某条结果 → fetchBookInfo → fetchToc
│ └─ TocPaginationResolver 拉全章节(支持分页规则)
│
└─ 用户阅读某章 → fetchChapterContent(按需加载)
```
改进 1 保证"搜到书后章节不断尾",改进 2 保证"搜索本身不卡"。
---
## 四、实施顺序
```
第 1 步:改进 2 的 SourceHealth + SourceHealthStore
→ 只加数据模型 + 持久化,不影响现有搜索流程
→ 可以先跑几天积累健康数据
第 2 步:改进 2 的 SearchScheduler + 短超时
→ 替换现有 search() 的并发逻辑
→ UI 层适配 Stream<SearchPhase>
第 3 步:改进 1 的 TocPaginationResolver
→ 依赖 RuleParser 链式语法的基础支持
→ 见《Legado 规则引擎重构方案》第 1~3 步
第 4 步:联调 + 回归
```
---
## 五、风险控制
| 风险 | 应对 |
| -------------------- | --------------------------------------------- |
| TOC 分页死循环 | `maxPages=20` + `seenUrls` 去重 + `added==0` 终止 |
| 健康分误杀快源 | 只降权不禁用;连续失败 ≥5 才标记不可用 + 手动可重置 |
| 短超时漏结果 | 慢源后台继续,event bus 异步追加,不丢数据 |
| 新源全部 50 分 | 第一轮搜索后就有真实分,冷启动无影响 |
| SharedPreferences 过大 | 只存 String list,每个源约 200 字节,100 个源 ≈ 20KB |