simon/news
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: cninfo 抓取限制浏览器并发,修复整机冻结 (06:00 任务压穿内存)

Browse Source 
根因: _render_page 每次新建完整 headless Chromium(实测 857MB/实例),
crawl_watchlist 对 15 只股票全量并发 → 峰值需求 ≈12.5GB ≫ 7.9GB RAM,
2026-09-06/08/10 三次 06:0x 整机冻结(load 65 → 看门狗复位)。

- crawler/cninfo.py: 新增 MAX_RENDER_CONCURRENCY(默认 2,env 可配)
  + 模块级 _render_sem 信号量,渲染全程持槽;顺带清理 3 个死导入
- tests/test_cninfo.py: 新增 8 个(并发上限/串行/异常释放/env 解析)
- 实测: 峰值 1796MB / 22 进程 / load 2.75 / 耗时 441s(上限 900s)
- 全量 291 passed
This commit is contained in:
simon
2026-09-10 20:54:30 +08:00
parent 6dede790c6
commit 2eaea2ee81
4 changed files with 251 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scheduler/pipeline.py
+1 -1
View File
@@ -57,7 +57,7 @@ STEP_TIMEOUTS: dict[str, int] = {
"embedding": 300, # M5 向量化
"qdrant": 300, # M6 入库(数据量大时需较长时间)
"report": 30, # 日报生成+上传
"cninfo_crawl": 900, # cninfo watchlist URL 驱动(SPA 渲染,每只约 25s)
"cninfo_crawl": 900, # cninfo watchlist URL 驱动(SPA 渲染;限流 2 并发后实测约 440s)
}
# 步骤对应的 uv run 命令(参数中 {date} 会被替换为实际日期)