feat: 增强反爬策略 + forexlive 迁移 investinglive
- crawler.py: 启用 magic/stealth 全自动反检测, --disable-webrtc, delay_before_return_html=3s, remove_consent_popups, 过滤列表页URL - rss_crawler.py: 新增 _make_rss_client() 支持 httpx 代理, Google News RSS 跳过 article_url_pattern 域名过滤 - sources.yaml: investing.com 改用 Google News RSS 代理绕过Cloudflare; forexlive -> investinglive (域名已301重定向) - domestic_crawl_8g.sh: HTTP_PROXY/HTTPS_PROXY 在 .env 加载前设置 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,8 @@ LOG "浏览器模式: headful (Xvfb :99)"
|
||||
LOG "HTTP 代理: 127.0.0.1:3128 (Privoxy → SOCKS5)"
|
||||
|
||||
# ── 加载 .env ──
|
||||
export HTTP_PROXY=http://127.0.0.1:3128
|
||||
export HTTPS_PROXY=http://127.0.0.1:3128
|
||||
export $(grep -v '^#' .env | grep -v '^$' | xargs 2>/dev/null || true)
|
||||
|
||||
# ── 启动 Xvfb(如果尚未运行)──
|
||||
|
||||
Reference in New Issue
Block a user