feat: 增强反爬策略 + forexlive 迁移 investinglive
- crawler.py: 启用 magic/stealth 全自动反检测, --disable-webrtc, delay_before_return_html=3s, remove_consent_popups, 过滤列表页URL - rss_crawler.py: 新增 _make_rss_client() 支持 httpx 代理, Google News RSS 跳过 article_url_pattern 域名过滤 - sources.yaml: investing.com 改用 Google News RSS 代理绕过Cloudflare; forexlive -> investinglive (域名已301重定向) - domestic_crawl_8g.sh: HTTP_PROXY/HTTPS_PROXY 在 .env 加载前设置 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
+12
-9
@@ -75,7 +75,9 @@ sources:
|
||||
article_url_pattern: "/news/"
|
||||
js_render: true
|
||||
max_articles_per_run: 25
|
||||
rss_url: "https://www.investing.com/rss/news_1063.rss" # RSS 绕过反爬
|
||||
rss_url: "https://news.google.com/rss/search?q=site:investing.com+news&hl=en-US&gl=US&ceid=US:en" # Google News RSS 代理(绕过 Cloudflare,仅摘要)
|
||||
anti_bot_mode: "stealth" # Cloudflare 反爬(RSS Google News 代理绕过,Web headful 兜底)
|
||||
# ⚠️ Google News RSS 仅获取标题+摘要,全文需 Web 回退 headful Playwright 抓取
|
||||
|
||||
- id: "seekingalpha"
|
||||
name: "Seeking Alpha"
|
||||
@@ -122,15 +124,16 @@ sources:
|
||||
anti_bot_mode: "stealth"
|
||||
rss_url: "https://www.economist.com/finance-and-economics/rss.xml" # Finance & economics RSS(含标题+摘要+链接)
|
||||
|
||||
- id: "forexlive"
|
||||
name: "ForexLive"
|
||||
- id: "investinglive"
|
||||
name: "InvestingLive"
|
||||
enabled: true
|
||||
homepage: "https://www.forexlive.com/"
|
||||
article_url_pattern: "/(news|technical-analysis|Education)/"
|
||||
js_render: false
|
||||
max_articles_per_run: 20
|
||||
rss_url: "https://www.forexlive.com/feed" # Breaking News RSS(含正文,~91KB/次)
|
||||
|
||||
homepage: "https://investinglive.com/"
|
||||
article_url_pattern: "/(news|central-banks|commodities|stocks|technical-analysis)/"
|
||||
js_render: true
|
||||
max_articles_per_run: 25
|
||||
anti_bot_mode: "stealth"
|
||||
rss_url: "https://investinglive.com/feed" # RSS 含完整正文(~120KB/次,25篇)
|
||||
# 2026-07-23 更新: forexlive.com 已 301 重定向到 investinglive.com
|
||||
- id: "zerohedge"
|
||||
name: "ZeroHedge"
|
||||
enabled: true
|
||||
|
||||
Reference in New Issue
Block a user