feat: 增强反爬策略 + forexlive 迁移 investinglive

- crawler.py: 启用 magic/stealth 全自动反检测, --disable-webrtc,
  delay_before_return_html=3s, remove_consent_popups, 过滤列表页URL
- rss_crawler.py: 新增 _make_rss_client() 支持 httpx 代理,
  Google News RSS 跳过 article_url_pattern 域名过滤
- sources.yaml: investing.com 改用 Google News RSS 代理绕过Cloudflare;
  forexlive -> investinglive (域名已301重定向)
- domestic_crawl_8g.sh: HTTP_PROXY/HTTPS_PROXY 在 .env 加载前设置

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-23 09:25:23 +08:00
co-authored by Claude
parent fe8b417ab6
commit 5c106120bf
4 changed files with 56 additions and 21 deletions
+12 -9
View File
@@ -75,7 +75,9 @@ sources:
article_url_pattern: "/news/"
js_render: true
max_articles_per_run: 25
rss_url: "https://www.investing.com/rss/news_1063.rss" # RSS 绕过反爬
rss_url: "https://news.google.com/rss/search?q=site:investing.com+news&hl=en-US&gl=US&ceid=US:en" # Google News RSS 代理(绕过 Cloudflare,仅摘要)
anti_bot_mode: "stealth" # Cloudflare 反爬(RSS Google News 代理绕过,Web headful 兜底)
# ⚠️ Google News RSS 仅获取标题+摘要,全文需 Web 回退 headful Playwright 抓取
- id: "seekingalpha"
name: "Seeking Alpha"
@@ -122,15 +124,16 @@ sources:
anti_bot_mode: "stealth"
rss_url: "https://www.economist.com/finance-and-economics/rss.xml" # Finance & economics RSS(含标题+摘要+链接)
- id: "forexlive"
name: "ForexLive"
- id: "investinglive"
name: "InvestingLive"
enabled: true
homepage: "https://www.forexlive.com/"
article_url_pattern: "/(news|technical-analysis|Education)/"
js_render: false
max_articles_per_run: 20
rss_url: "https://www.forexlive.com/feed" # Breaking News RSS(含正文,~91KB/次)
homepage: "https://investinglive.com/"
article_url_pattern: "/(news|central-banks|commodities|stocks|technical-analysis)/"
js_render: true
max_articles_per_run: 25
anti_bot_mode: "stealth"
rss_url: "https://investinglive.com/feed" # RSS 含完整正文(~120KB/次,25篇)
# 2026-07-23 更新: forexlive.com 已 301 重定向到 investinglive.com
- id: "zerohedge"
name: "ZeroHedge"
enabled: true