simon
3b44f64f66
- docs: 删除 CLAUDE.md / continuation.md / english-news-plan.md 及旧版 intlnews_usage.*，
统一迁移到 docs/{README,architecture,quickstart,usage,pipeline,configuration,deployment,development,faq}.md
- README: 精简为仓库入口，指向 docs/
- configs/sources.yaml: 更新注释指向新文档
- .env.example: 修正 DashScope Embedding 端点说明
Pipeline 修复：
- dedup/llm/embedding/vectorstore/reporter: 过滤 M2 no_content / 空正文，避免污染下游与 Qdrant
- dedup/pipeline: 改为先写唯一文件再写指纹，避免崩溃导致文章永久丢失
- crawler/orchestrator: sources_crawled 改为“尝试数”，成功数 = crawled - failed
- crawler/storage: write_index_jsonl 从文章路径推断日期，修复跨天/测试路径问题
- scheduler/pipeline: STEP_TIMEOUTS 实际生效（SIGALRM）
- scheduler/reporter: emb_count 排除 index.json；日报跳过无原文事件
- vectorstore/pipeline: payload 增加 source_ids；--recreate --all 时空日期也重建 collection
- app/cli: extract/dedup/translate/embed/index/pipeline 支持 --date；embed/index 支持 --all；crawl 全源失败返回非零
- scripts: domestic_full/crawl_8g/crawl_2g/pipeline 安全加载 .env；M1 全失败不标记且最终退出码=1
239 lines
6.0 KiB
Markdown
239 lines
6.0 KiB
Markdown
# 配置说明
|
||
|
||
## 1. 配置文件总览
|
||
|
||
| 文件 | 用途 |
|
||
|------|------|
|
||
| `configs/system.yaml` | 系统功能配置：代理、抓取、提取、去重、LLM、Embedding、Qdrant、日报、调度、日志 |
|
||
| `configs/sources.yaml` | 新闻源列表与抓取规则 |
|
||
| `configs/profiles/8g_headful.yaml` | Pi 8G 有头浏览器 + HTTP 代理配置 |
|
||
| `configs/profiles/2g_headless.yaml` | 2G 轻量 headless stealth 配置 |
|
||
| `.env` | 敏感配置：API Key、数据库密码、远程 Qdrant 地址 |
|
||
| `.env.example` | 环境变量模板 |
|
||
| `prompts/translation_and_extraction.md` | M4 翻译+事件抽取 Prompt 模板 |
|
||
|
||
## 2. `.env` 环境变量
|
||
|
||
```env
|
||
# DeepSeek（M4 / M7 默认 LLM）
|
||
DEEPSEEK_API_KEY=sk-your-deepseek-key
|
||
DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||
|
||
# Qwen（可选，切换 LLM provider 时使用）
|
||
QWEN_API_KEY=sk-your-qwen-key
|
||
QWEN_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
||
|
||
# DashScope（M5 Embedding）
|
||
# 使用 OpenAI 兼容端点，默认复用 QWEN_BASE_URL；如需独立端点可设置 DASHSCOPE_EMBEDDING_BASE_URL
|
||
DASHSCOPE_API_KEY=sk-your-dashscope-key
|
||
# DASHSCOPE_EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
||
|
||
# Qdrant（可选；默认本地文件模式）
|
||
QDRANT_URL=http://localhost:6333
|
||
QDRANT_API_KEY=
|
||
|
||
# MySQL（M9 日报入库）
|
||
NEWS_DB_HOST=192.168.1.10
|
||
NEWS_DB_PORT=13306
|
||
NEWS_DB_USER=myquant
|
||
NEWS_DB_PASSWORD=
|
||
NEWS_DB_NAME=myquant
|
||
```
|
||
|
||
## 3. `configs/system.yaml`
|
||
|
||
### 3.1 servers
|
||
|
||
```yaml
|
||
servers:
|
||
overseas_host: "ecs-user@8.217.19.253"
|
||
overseas_path: "/opt/intlgrab"
|
||
domestic_host: "pi@192.168.1.160"
|
||
domestic_path: "/home/pi/intlnews"
|
||
```
|
||
|
||
海外服务器已停用，字段保留为兼容参考。
|
||
|
||
### 3.2 proxy
|
||
|
||
```yaml
|
||
proxy:
|
||
enabled: false
|
||
url: "http://127.0.0.1:3128"
|
||
bypass_domains: []
|
||
```
|
||
|
||
- 默认关闭。
|
||
- `8g_headful` Profile 会开启并通过 `http://127.0.0.1:3128`（Privoxy → SOCKS5）访问海外。
|
||
|
||
### 3.3 crawler
|
||
|
||
| 字段 | 默认 | 说明 |
|
||
|------|------|------|
|
||
| `max_memory_mb` | 1800 | 内存上限，超限触发 GC/保护逻辑 |
|
||
| `source_timeout_sec` | 7200 | 单源超时 |
|
||
| `article_delay_sec` | 3.0 | 文章间冷却 |
|
||
| `page_timeout_sec` | 45 | 单页加载超时 |
|
||
| `viewport_width/height` | 1024/768 | 浏览器视口 |
|
||
| `headful` | false | true=有头浏览器 |
|
||
| `xvfb_display` | ":99" | Xvfb 虚拟显示器 |
|
||
|
||
### 3.4 extractor / dedup
|
||
|
||
```yaml
|
||
extractor:
|
||
min_content_words: 50
|
||
trafilatura_fallback: true
|
||
|
||
dedup:
|
||
hamming_distance_threshold: 3
|
||
simhash_window_days: 30
|
||
min_content_length: 100
|
||
```
|
||
|
||
### 3.5 llm 与 llm_scenes
|
||
|
||
```yaml
|
||
llm:
|
||
provider: "deepseek"
|
||
deepseek_model: "deepseek-v4-flash"
|
||
qwen_model: "qwen-plus"
|
||
timeout_sec: 60
|
||
max_attempts: 3
|
||
max_tokens: 8192
|
||
temperature: 0.1
|
||
concurrency: 3
|
||
|
||
llm_scenes:
|
||
translation:
|
||
provider: "deepseek"
|
||
model: "deepseek-v4-flash"
|
||
temperature: 0.1
|
||
max_tokens: 8192
|
||
daily_report:
|
||
provider: "deepseek"
|
||
model: "deepseek-v4-flash"
|
||
temperature: 0.3
|
||
max_tokens: 1500
|
||
```
|
||
|
||
- `translation`：M4 使用。
|
||
- `daily_report`：M7 日报摘要使用。
|
||
- 场景未声明的字段会回退 `llm`。
|
||
|
||
### 3.6 embedding
|
||
|
||
```yaml
|
||
embedding:
|
||
provider: "dashscope"
|
||
dashscope_model: "text-embedding-v3"
|
||
dimension: 1024
|
||
batch_size: 10
|
||
max_attempts: 3
|
||
timeout_sec: 30
|
||
```
|
||
|
||
注意：Embedding 不支持按场景拆分配置，因为入库向量与查询向量必须同模型。
|
||
|
||
### 3.7 qdrant / report / schedule / logging
|
||
|
||
```yaml
|
||
qdrant:
|
||
collection: "en_finance_news"
|
||
|
||
report:
|
||
max_events: 20
|
||
summary_max_chars: 500
|
||
importance_threshold: 4
|
||
upload_host: "simon@doorcome.cn"
|
||
upload_path: "/var/www/html/echart/research"
|
||
|
||
schedule:
|
||
day_cutoff_hour: 6
|
||
times: ["06:00", "12:00", "18:00", "22:00"]
|
||
|
||
logging:
|
||
level: "INFO"
|
||
dir: "logs"
|
||
```
|
||
|
||
`report.upload_host/upload_path` 是 M9 之前 HTML 日报上传的旧配置，当前仅作保留。
|
||
|
||
## 4. `configs/sources.yaml`
|
||
|
||
### 4.1 顶层 settings
|
||
|
||
```yaml
|
||
settings:
|
||
concurrency: 5
|
||
request_delay_sec: 2
|
||
user_agent: "..."
|
||
```
|
||
|
||
### 4.2 单源字段
|
||
|
||
| 字段 | 说明 |
|
||
|------|------|
|
||
| `id` | 唯一标识，如 `reuters` |
|
||
| `name` | 展示名，如 `Reuters` |
|
||
| `enabled` | 是否启用 |
|
||
| `homepage` | 主页，用于回退/域名校验 |
|
||
| `article_url_pattern` | URL 正则，过滤非文章链接 |
|
||
| `js_render` | 是否需要 JS 渲染 |
|
||
| `max_articles_per_run` | 单次最多抓取篇数 |
|
||
| `rss_url` | RSS/Atom/Google News RSS 地址 |
|
||
| `anti_bot_mode` | `stealth` / `headful` 等 Web 回退模式 |
|
||
|
||
### 4.3 当前源列表
|
||
|
||
| ID | 名称 | 抓取方式 | 状态 |
|
||
|----|------|----------|------|
|
||
| `reuters` | Reuters | Google News RSS（可能只有摘要） | ⚠️ DataDome |
|
||
| `cnbc` | CNBC | CNBC RSS | ✅ |
|
||
| `marketwatch` | MarketWatch | RSS + stealth 回退 | ✅ |
|
||
| `ft` | Financial Times | FT RSS（摘要） | ⚠️ 付费墙 |
|
||
| `yahoo_finance` | Yahoo Finance | Yahoo RSS | ✅ |
|
||
| `investing` | Investing.com | Google News RSS + stealth | ✅（摘要） |
|
||
| `seekingalpha` | Seeking Alpha | RSS + stealth | ✅ |
|
||
| `barrons` | Barron's | Dow Jones RSS + stealth | ✅ |
|
||
| `wsj` | WSJ | Dow Jones RSS + stealth | ✅ |
|
||
| `economist` | The Economist | RSS + stealth | ✅ |
|
||
| `investinglive` | InvestingLive | RSS（全文） | ✅ |
|
||
| `zerohedge` | ZeroHedge | FeedBurner RSS（全文） | ✅ |
|
||
|
||
> 历史 `forexlive` 已迁移为 `investinglive`。
|
||
|
||
## 5. Profiles
|
||
|
||
### 5.1 `8g_headful.yaml`
|
||
|
||
适合国内 Pi（内存充裕）：
|
||
- 开启 HTTP 代理
|
||
- `headful: true`
|
||
- 更大视口/超时
|
||
- 通过 Xvfb `:99` 运行
|
||
|
||
### 5.2 `2g_headless.yaml`
|
||
|
||
适合低配/海外轻量：
|
||
- headless stealth
|
||
- 内存上限 1800MB
|
||
- 默认不开启代理
|
||
|
||
通过环境变量切换：
|
||
|
||
```bash
|
||
EN_NEWS_PROFILE=8g_headful uv run en-news crawl
|
||
# 或直接使用对应脚本
|
||
bash scripts/domestic_crawl_8g.sh
|
||
```
|
||
|
||
## 6. Prompt 模板
|
||
|
||
`prompts/translation_and_extraction.md` 包含：
|
||
|
||
- `## System Prompt`：LLM System 指令
|
||
- `## User Input`：带 `{title}`、`{source_name}`、`{publish_time}`、`{content}` 占位符
|
||
|
||
如需调整事件类型、输出格式或翻译风格，应修改此文件并同步测试。
|