Commit Graph
3 Commits
Author SHA1 Message Date
simon 9aa44e610d feat: LLM/Embedding 重试次数纳入 system.yaml 统一配置
- system.yaml: llm.max_retries(死配置) 改为 llm.max_attempts;embedding 段新增 max_attempts
- llm/client.py: LLMConfig 新增 max_attempts(默认 3 兜底),load_llm_config 读取配置
- llm/extractor.py: translate_and_extract(_async) max_attempts=None 时取 config.max_attempts
- embedding/client.py: load_embedding_config 读取 embedding.max_attempts
- scheduler/reporter.py: _call_llm_simple max_retries=None 时取 max_attempts-1
- 新增 2 个配置读取测试;已同步 pi5 验证
2026-08-05 08:48:43 +08:00
simonandClaude 5c106120bf feat: 增强反爬策略 + forexlive 迁移 investinglive
- crawler.py: 启用 magic/stealth 全自动反检测, --disable-webrtc,
  delay_before_return_html=3s, remove_consent_popups, 过滤列表页URL
- rss_crawler.py: 新增 _make_rss_client() 支持 httpx 代理,
  Google News RSS 跳过 article_url_pattern 域名过滤
- sources.yaml: investing.com 改用 Google News RSS 代理绕过Cloudflare;
  forexlive -> investinglive (域名已301重定向)
- domestic_crawl_8g.sh: HTTP_PROXY/HTTPS_PROXY 在 .env 加载前设置

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 09:25:23 +08:00
simon fe8b417ab6 初始化 2026-07-18 16:13:52 +08:00