Files
SakuraHub/docs/三项核心改进-分页规则_搜索健康分_书架保存解耦.md
T

656 lines
19 KiB
Markdown
Raw Normal View History

2026-11-16 20:45:22 +08:00
# 两项核心改进 — TOC 分页规则 · 搜索健康分
> 日期:2026-06-22
> 范围:仅移动端 Dart 侧(`sakura_mobile/lib/src/core/`)
> 涉及文件:
>
> - `legado_models.dart` — 数据模型
> - `legado_rule_parser.dart` — 规则解析器
> - `legado_engine.dart` — 搜索引擎入口
> 原则:不改 Python 插件,不改 UI 层,只动 core 层。
---
## 一、改进 1:TOC 章节目录分页规则完整支持
### 1.1 问题
`fetchToc()` 只拉第一页目录。很多书源第一页只给 50~100 章,剩下的靠 **分页目录规则** 指向下一页。不支持分页规则 → 用户看到的目录永远不全。
### 1.2 兼容的字段名
书源 JSON 中,下一页 URL 的字段名可能是以下任意一种,按优先级遍历取第一个非空值:
```dart
// legado_models.dart — BookSource / TocRule 中新增字段解析
static const List<String> _tocNextPageCandidates = [
'tocNextPageRule', // Legado 标准名
'nextTocUrl', // 常见别名
'nextUrl', // 通用别名
'nextPage', // 页码式
'tocNextPage', // 变体
];
String? get tocNextPageRule {
for (final name in _tocNextPageCandidates) {
final v = rawJson[name];
if (v is String && v.trim().isNotEmpty) return v.trim();
}
return null;
}
```
### 1.3 分页规则语法全集
| 规则类型 | 语法示例 | 说明 |
| --------- | ----------------------------------- | ------------------ |
| JS 表达式 | `<js>nextUrl</js>` | 执行 JS 获取下一页 URL |
| JS 缩写 | `@js:nextUrl` | 同上,简写形式 |
| 模板拼接 | `https://host/list?page={{page}}` | `{{page}}` 从 1 自增 |
| 偏移模板 | `{{page+1}}` / `{{page-1}}` | 页号偏移 |
| CSS 选择器 | `text.下一页@href` | 从当前页 HTML 提取下一页链接 |
| class 选择器 | `class.next@href` | 同上 |
| id 选择器 | `id.pagenext@href` | 同上 |
| 属性选择器 | `@href` | 取当前元素 href |
| XPath | `//a[contains(text(),'下一页')]/@href` | XPath 提取 |
| 字符串切片 | `!0:-1` | 对当前 URL 切片(去掉末尾字符) |
| offset 翻页 | `offset=50` | 按 offset 递增翻页 |
| 纯文本 URL | `https://...` | 直接跳到指定 URL |
| 相对路径 | `/list/2.html` | 基于当前 URL 拼接 |
### 1.4 实现 — `TocPaginationResolver`
```dart
// 放在 legado_engine.dart 或新建 legado_toc_pagination.dart
class TocPaginationResolver {
final LegadoRuleExecutor _ruleExecutor;
final HttpClient _client;
TocPaginationResolver(this._ruleExecutor, this._client);
/// 拉取全部分页章节,合并去重。
Future<List<ChapterInfo>> fetchAllChapters({
required String firstPageUrl,
required String? tocNextPageRule,
required String tocRule,
required Map<String, String>? headers,
int maxPages = 20,
}) async {
final all = <ChapterInfo>[];
final seenPageUrls = <String>{};
final seenChapterUrls = <String>{};
String? url = firstPageUrl;
int page = 0;
while (url != null && page < maxPages) {
if (!seenPageUrls.add(url)) break; // 死循环保护
final html = await _client.get(url, headers: headers);
final batch = _ruleExecutor.selectList(html, url, rule: tocRule);
int added = 0;
for (final ch in batch) {
if (seenChapterUrls.add(ch.url)) {
all.add(ch);
added++;
}
}
if (added == 0) break; // 本页无新章节,终止翻页
url = _resolveNextPage(
currentUrl: url,
currentHtml: html,
rule: tocNextPageRule,
pageIndex: page,
);
page++;
}
return all;
}
// ---------- 下一页 URL 解析 ----------
String? _resolveNextPage({
required String currentUrl,
required String currentHtml,
required String? rule,
required int pageIndex,
}) {
if (rule == null || rule.trim().isEmpty) return null;
final r = rule.trim();
// 1) JS: <js>...</js> 或 @js:...
if (r.startsWith('<js>') || r.startsWith('@js:')) {
final code = r
.replaceFirst(RegExp(r'^<js>'), '')
.replaceFirst(RegExp(r'</js>$'), '')
.replaceFirst(RegExp(r'^@js:'), '');
return _ruleExecutor.evalJs(code, context: {
'result': currentHtml,
'baseUrl': currentUrl,
'page': pageIndex,
});
}
// 2) 切片: !start:end
if (r.startsWith('!')) {
return _applySlice(currentUrl, r.substring(1));
}
// 3) CSS 选择器 / XPath
if (_isSelector(r)) {
return _ruleExecutor.select(currentHtml, currentUrl, rule: r);
}
// 4) 模板替换 {{page}} / {{page+1}} / {{page-1}}
if (r.contains('{{page')) {
return _renderPageTemplate(r, pageIndex);
}
// 5) offset / start 参数
if (RegExp(r'[?&](offset|start)=').hasMatch(r)) {
return _renderOffsetTemplate(r, pageIndex);
}
// 6) 绝对 URL
if (r.startsWith('http://') || r.startsWith('https://')) {
return _renderPageTemplate(r, pageIndex);
}
// 7) 相对路径
return Uri.parse(currentUrl).resolve(r).toString();
}
// ---------- 规则判定 ----------
bool _isSelector(String s) {
return s.startsWith('text.') ||
s.startsWith('class.') ||
s.startsWith('id.') ||
s.startsWith('@') ||
s.startsWith('//'); // XPath
}
// ---------- 模板渲染 ----------
String _renderPageTemplate(String tpl, int page) {
return tpl
.replaceAll('{{page+1}}', '${page + 1}')
.replaceAll('{{page-1}}', '${page - 1}')
.replaceAll('{{page}}', '$page');
}
String _renderOffsetTemplate(String tpl, int page) {
return tpl
.replaceAllMapped(
RegExp(r'offset=(\d+)'),
(m) => 'offset=${int.parse(m.group(1)!) * page}',
)
.replaceAllMapped(
RegExp(r'start=(\d+)'),
(m) => 'start=${int.parse(m.group(1)!) * page}',
);
}
String? _applySlice(String url, String expr) {
final parts = expr.split(':');
if (parts.length != 2) return null;
final a = int.tryParse(parts[0]) ?? 0;
final b = int.tryParse(parts[1]);
final start = a < 0 ? url.length + a : a;
if (b != null) {
final end = b < 0 ? url.length + b : b;
return url.substring(start, end);
}
return url.substring(start);
}
}
```
### 1.5 集成到 `LegadoEngine.fetchToc`
```dart
// legado_engine.dart — 修改 fetchToc
Future<List<ChapterInfo>> fetchToc(BookSource source, String tocUrl) async {
final rule = source.tocRule;
if (rule == null || rule.isEmpty) return [];
final resolver = TocPaginationResolver(_ruleExecutor, _client);
return resolver.fetchAllChapters(
firstPageUrl: tocUrl,
tocNextPageRule: source.tocNextPageRule, // ← 新增
tocRule: rule,
headers: _buildHeaders(source),
maxPages: 20,
);
}
```
### 1.6 死循环防护(三层)
1. **`maxPages = 20`** — 单书最多翻 20 页目录
2. **`seenPageUrls`** — 同一 URL 不重复请求
3. **`added == 0` 终止** — 当前页没有新增章节则停止
---
## 二、改进 2:搜索源健康分 + 短超时 + 分阶段返回
### 2.1 问题
当前 `search()` 对所有启用源并发请求,每个源 timeout 20~30 秒。10 个源一批,坏源多时整批拖到 20+ 秒才出结果。
### 2.2 源健康分模型
```dart
// legado_models.dart — 新增 SourceHealth
class SourceHealth {
final String sourceId;
int successCount = 0;
int failCount = 0;
int timeoutCount = 0;
int totalLatencyMs = 0;
DateTime? lastSuccessAt;
DateTime? lastFailAt;
int _consecutiveFails = 0;
SourceHealth({required this.sourceId});
// ---- 判定 ----
/// 连续失败 ≥ 5 次 → 临时禁用
bool get isUsable {
if (_consecutiveFails >= 5) return false;
if (failCount >= 5 && successCount == 0) return false;
return true;
}
int get consecutiveFails => _consecutiveFails;
/// 健康分 0~100
double get score {
final total = successCount + failCount + timeoutCount;
if (total == 0) return 50.0; // 新源默认 50
double s = 0.0;
// 成功率 60%
s += 60.0 * (successCount / total);
// 平均延迟 25%
if (successCount > 0 && totalLatencyMs > 0) {
final avg = totalLatencyMs / successCount;
if (avg < 500) {
s += 25.0;
} else if (avg < 2000) {
s += 25.0 * (1.0 - (avg - 500) / 1500);
}
}
// 新鲜度 15%(24h 内有成功记录)
if (lastSuccessAt != null) {
final h = DateTime.now().difference(lastSuccessAt!).inHours;
if (h < 1) {
s += 15.0;
} else if (h < 24) {
s += 15.0 * (1.0 - h / 24);
}
}
return s.clamp(0.0, 100.0);
}
// ---- 记录 ----
void recordSuccess(int latencyMs) {
successCount++;
totalLatencyMs += latencyMs;
lastSuccessAt = DateTime.now();
_consecutiveFails = 0;
}
void recordFailure() {
failCount++;
lastFailAt = DateTime.now();
_consecutiveFails++;
}
void recordTimeout() {
timeoutCount++;
lastFailAt = DateTime.now();
_consecutiveFails++;
}
void resetConsecutiveFails() => _consecutiveFails = 0;
// ---- 序列化(持久化到 SharedPreferences)----
Map<String, dynamic> toJson() => {
'sourceId': sourceId,
'successCount': successCount,
'failCount': failCount,
'timeoutCount': timeoutCount,
'totalLatencyMs': totalLatencyMs,
'lastSuccessAt': lastSuccessAt?.toIso8601String(),
'lastFailAt': lastFailAt?.toIso8601String(),
'consecutiveFails': _consecutiveFails,
};
factory SourceHealth.fromJson(Map<String, dynamic> json) {
final h = SourceHealth(sourceId: json['sourceId']);
h.successCount = json['successCount'] ?? 0;
h.failCount = json['failCount'] ?? 0;
h.timeoutCount = json['timeoutCount'] ?? 0;
h.totalLatencyMs = json['totalLatencyMs'] ?? 0;
h.lastSuccessAt = json['lastSuccessAt'] != null
? DateTime.tryParse(json['lastSuccessAt'])
: null;
h.lastFailAt = json['lastFailAt'] != null
? DateTime.tryParse(json['lastFailAt'])
: null;
h._consecutiveFails = json['consecutiveFails'] ?? 0;
return h;
}
}
```
### 2.3 健康分持久化
```dart
// legado_engine.dart — SourceHealthStore
class SourceHealthStore {
static const _key = 'source_health_v1';
final Map<String, SourceHealth> _data = {};
Future<void> load() async {
final prefs = await SharedPreferences.getInstance();
final raw = prefs.getStringList(_key);
if (raw == null) return;
for (final s in raw) {
try {
final h = SourceHealth.fromJson(jsonDecode(s));
_data[h.sourceId] = h;
} catch (_) {}
}
}
Future<void> save() async {
final prefs = await SharedPreferences.getInstance();
final raw = _data.values.map((h) => jsonEncode(h.toJson())).toList();
await prefs.setStringList(_key, raw);
}
bool isUsable(String id) => _data[id]?.isUsable ?? true;
double score(String id) => _data[id]?.score ?? 50.0;
void _ensure(String id) {
_data.putIfAbsent(id, () => SourceHealth(sourceId: id));
}
void recordSuccess(String id, int latencyMs) {
_ensure(id);
_data[id]!.recordSuccess(latencyMs);
}
void recordFailure(String id) {
_ensure(id);
_data[id]!.recordFailure();
}
void recordTimeout(String id) {
_ensure(id);
_data[id]!.recordTimeout();
}
/// 用户手动重置所有源健康状态
void resetAll() {
for (final h in _data.values) {
h.resetConsecutiveFails();
}
}
}
```
### 2.4 分阶段搜索调度器
```dart
// legado_engine.dart — SearchScheduler
class SearchScheduler {
static const fastTimeout = Duration(seconds: 5);
static const slowTimeout = Duration(seconds: 8);
static const maxConcurrent = 6;
final SourceHealthStore _health;
final LegadoEngine _engine;
SearchScheduler(this._health, this._engine);
/// 返回 Stream,UI 层逐阶段渲染结果
Stream<SearchPhase> search(String keyword, List<BookSource> allSources) async* {
// 过滤 + 按健康分排序
final usable = allSources
.where((s) => _health.isUsable(s.id))
.toList()
..sort((a, b) => _health.score(b.id).compareTo(_health.score(a.id)));
if (usable.isEmpty) {
yield SearchPhase(name: 'error', message: '没有可用的搜索源');
return;
}
// 分三档
final fast = usable.where((s) => _health.score(s.id) >= 70).toList();
final normal = usable.where((s) {
final sc = _health.score(s.id);
return sc >= 40 && sc < 70;
}).toList();
final slow = usable.where((s) => _health.score(s.id) < 40).toList();
final results = <SearchItem>[];
final doneIds = <String>{};
final seen = <String>{};
// ---- 阶段 1:快速源,超时 5s ----
yield SearchPhase(name: 'searching', message: '正在搜索 ${usable.length} 个源...');
final r1 = await _batch(fast, keyword, timeout: fastTimeout, onDone: doneIds.add);
_merge(results, r1, seen);
yield SearchPhase(
name: 'partial',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
// ---- 阶段 2:普通源,超时 8s ----
final r2 = await _batch(normal, keyword, timeout: slowTimeout, onDone: doneIds.add);
_merge(results, r2, seen);
yield SearchPhase(
name: 'partial',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
// ---- 阶段 3:慢源后台继续,不阻塞 ----
if (slow.isNotEmpty) {
// 放入后台队列,不 await
_batch(slow, keyword, timeout: slowTimeout, onDone: doneIds.add)
.then((r3) {
_merge(results, r3, seen);
// TODO: EventBus 通知 UI 追加
});
yield SearchPhase(
name: 'complete_with_pending',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
message: '${slow.length} 个慢源将在后台继续搜索',
);
} else {
yield SearchPhase(
name: 'complete',
items: List.of(results),
doneCount: doneIds.length,
totalCount: usable.length,
);
}
}
// ---- 内部分批并发 ----
Future<List<SearchItem>> _batch(
List<BookSource> sources,
String keyword, {
required Duration timeout,
required void Function(String) onDone,
}) async {
final out = <SearchItem>[];
for (var i = 0; i < sources.length; i += maxConcurrent) {
final chunk = sources.skip(i).take(maxConcurrent).toList();
final futures = chunk.map((s) => _searchOne(s, keyword, timeout));
final results = await Future.wait(futures);
for (var j = 0; j < results.length; j++) {
if (results[j] != null) out.addAll(results[j]!);
onDone(chunk[j].id);
}
}
return out;
}
Future<List<SearchItem>?> _searchOne(
BookSource source, String keyword, Duration timeout,
) async {
final sw = Stopwatch()..start();
try {
final items = await _engine.search(source, keyword).timeout(timeout);
sw.stop();
_health.recordSuccess(source.id, sw.elapsedMilliseconds);
return items;
} on TimeoutException {
_health.recordTimeout(source.id);
return null;
} catch (_) {
_health.recordFailure(source.id);
return null;
}
}
void _merge(List<SearchItem> all, List<SearchItem> incoming, Set<String> seen) {
for (final item in incoming) {
if (seen.add(item.id)) all.add(item);
}
}
}
// ---- 阶段结果 ----
class SearchPhase {
final String name; // searching | partial | complete | complete_with_pending | error
final List<SearchItem> items;
final int doneCount;
final int totalCount;
final String? message;
SearchPhase({
required this.name,
this.items = const [],
this.doneCount = 0,
this.totalCount = 0,
this.message,
});
}
```
### 2.5 UI 增量渲染时序
```
t=0s → 显示 "正在搜索 N 个源..."
t=0~5s → 第 1 批结果到达 → 首屏渲染(最快的 5~10 条)
t=5~8s → 第 2 批结果到达 → 列表追加
t=8s → 停止等待 → 提示"慢源已跳过,后台补充中"
t=8~15s → 第 3 批结果到达 → 静默追加到列表末尾
```
### 2.6 健康分自动衰减
每次 `save()` 时做一次衰减:
```dart
// SourceHealthStore.save() 追加衰减逻辑
Future<void> save() async {
// 超过 7 天无任何记录 → 衰减 successCount 和 failCount 各减半
for (final h in _data.values) {
final latest = h.lastSuccessAt ?? h.lastFailAt;
if (latest != null &&
DateTime.now().difference(latest).inDays >= 7) {
h.successCount = (h.successCount * 0.5).round();
h.failCount = (h.failCount * 0.5).round();
h.timeoutCount = (h.timeoutCount * 0.5).round();
}
}
// ... 写入 SharedPreferences
}
```
---
## 三、两改进的协作关系
```
用户搜索一本书
│
├─ SearchScheduler.search(keyword)
│ ├─ 按健康分排序源
│ ├─ 快速源先搜(5s 超时)
│ ├─ 普通源补搜(8s 超时)
│ └─ 返回结果列表
│
├─ 用户点击某条结果 → fetchBookInfo → fetchToc
│ └─ TocPaginationResolver 拉全章节(支持分页规则)
│
└─ 用户阅读某章 → fetchChapterContent(按需加载)
```
改进 1 保证"搜到书后章节不断尾",改进 2 保证"搜索本身不卡"。
---
## 四、实施顺序
```
第 1 步:改进 2 的 SourceHealth + SourceHealthStore
→ 只加数据模型 + 持久化,不影响现有搜索流程
→ 可以先跑几天积累健康数据
第 2 步:改进 2 的 SearchScheduler + 短超时
→ 替换现有 search() 的并发逻辑
→ UI 层适配 Stream<SearchPhase>
第 3 步:改进 1 的 TocPaginationResolver
→ 依赖 RuleParser 链式语法的基础支持
→ 见《Legado 规则引擎重构方案》第 1~3 步
第 4 步:联调 + 回归
```
---
## 五、风险控制
| 风险 | 应对 |
| -------------------- | --------------------------------------------- |
| TOC 分页死循环 | `maxPages=20` + `seenUrls` 去重 + `added==0` 终止 |
| 健康分误杀快源 | 只降权不禁用;连续失败 ≥5 才标记不可用 + 手动可重置 |
| 短超时漏结果 | 慢源后台继续,event bus 异步追加,不丢数据 |
| 新源全部 50 分 | 第一轮搜索后就有真实分,冷启动无影响 |
| SharedPreferences 过大 | 只存 String list,每个源约 200 字节,100 个源 ≈ 20KB |