662 lines
26 KiB
HTML
662 lines
26 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>国际财经 Deep Research 平台 — 使用手册 v1.0</title>
|
||
<style>
|
||
:root {
|
||
--bg: #f8f9fa; --card: #fff; --text: #212529; --muted: #6c757d;
|
||
--accent: #2563eb; --accent2: #0891b2; --border: #dee2e6;
|
||
--code-bg: #f1f5f9; --pre-bg: #1e293b; --pre-text: #e2e8f0;
|
||
--pos: #059669; --neg: #dc2626; --neu: #6b7280;
|
||
--radius: 8px;
|
||
}
|
||
* { box-sizing: border-box; margin: 0; padding: 0; }
|
||
body {
|
||
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", sans-serif;
|
||
background: var(--bg); color: var(--text); line-height: 1.75; padding-bottom: 4rem;
|
||
}
|
||
|
||
/* ── 导航栏 ── */
|
||
nav {
|
||
position: sticky; top: 0; z-index: 100;
|
||
background: linear-gradient(135deg, #0f172a, #1e3a5f);
|
||
color: #fff; padding: .8rem 0;
|
||
box-shadow: 0 2px 8px rgba(0,0,0,.15);
|
||
}
|
||
nav .nav-inner { max-width: 1100px; margin: 0 auto; padding: 0 1.5rem; display: flex; gap: 1.5rem; flex-wrap: wrap; align-items: center; }
|
||
nav a { color: #cbd5e1; text-decoration: none; font-size: .9em; transition: color .2s; }
|
||
nav a:hover { color: #fff; }
|
||
nav strong { font-size: 1.1em; margin-right: 1rem; }
|
||
|
||
/* ── 头部 ── */
|
||
header {
|
||
background: linear-gradient(135deg, #0f172a, #1e3a5f); color: #fff;
|
||
text-align: center; padding: 3rem 0 2.5rem;
|
||
}
|
||
header h1 { font-size: 2rem; margin-bottom: .5rem; }
|
||
header p { color: #94a3b8; font-size: .95em; }
|
||
|
||
/* ── 内容 ── */
|
||
.container { max-width: 1000px; margin: 0 auto; padding: 0 1.5rem; }
|
||
|
||
h2 {
|
||
font-size: 1.4rem; margin: 2.5rem 0 1rem; padding-bottom: .4rem;
|
||
border-bottom: 2px solid var(--accent); color: #1e293b;
|
||
}
|
||
h3 {
|
||
font-size: 1.1rem; margin: 1.5rem 0 .6rem; color: #334155;
|
||
}
|
||
h4 { font-size: 1rem; margin: 1rem 0 .4rem; color: #475569; }
|
||
|
||
p { margin: .6rem 0; }
|
||
a { color: var(--accent); text-decoration: none; }
|
||
a:hover { text-decoration: underline; }
|
||
|
||
/* ── 表格 ── */
|
||
table {
|
||
width: 100%; border-collapse: collapse; margin: .8rem 0 1.2rem; font-size: .93em;
|
||
}
|
||
th, td { border: 1px solid var(--border); padding: .5rem .8rem; text-align: left; }
|
||
th { background: #f1f5f9; font-weight: 600; white-space: nowrap; }
|
||
tr:nth-child(even) { background: #fafbfc; }
|
||
|
||
/* ── 代码 ── */
|
||
code {
|
||
background: var(--code-bg); padding: .15em .4em; border-radius: 4px;
|
||
font-family: "SF Mono", "Fira Code", "Cascadia Code", monospace;
|
||
font-size: .88em; color: #1e293b;
|
||
}
|
||
pre {
|
||
background: var(--pre-bg); color: var(--pre-text); padding: 1rem 1.2rem;
|
||
border-radius: var(--radius); overflow-x: auto; margin: .8rem 0;
|
||
font-family: "SF Mono", "Fira Code", "Cascadia Code", monospace;
|
||
font-size: .85em; line-height: 1.6;
|
||
}
|
||
pre code { background: none; padding: 0; color: inherit; font-size: inherit; }
|
||
|
||
/* ── 卡片 ── */
|
||
.card {
|
||
background: var(--card); border: 1px solid var(--border); border-radius: var(--radius);
|
||
padding: 1rem 1.2rem; margin: .8rem 0;
|
||
}
|
||
.card-grid {
|
||
display: grid; grid-template-columns: repeat(auto-fill, minmax(280px, 1fr));
|
||
gap: 1rem; margin: 1rem 0;
|
||
}
|
||
|
||
/* ── 标签 ── */
|
||
.badge {
|
||
display: inline-block; padding: .1em .5em; border-radius: 10px;
|
||
font-size: .78em; font-weight: 600;
|
||
}
|
||
.badge-pos { background: #d1fae5; color: #065f46; }
|
||
.badge-neg { background: #fee2e2; color: #991b1b; }
|
||
.badge-neu { background: #e5e7eb; color: #374151; }
|
||
.badge-blue { background: #dbeafe; color: #1e40af; }
|
||
|
||
/* ── 步骤流程 ── */
|
||
.flow {
|
||
display: flex; flex-wrap: wrap; gap: .5rem; align-items: center;
|
||
margin: .8rem 0; font-size: .9em;
|
||
}
|
||
.flow-step {
|
||
background: var(--accent); color: #fff; padding: .3rem .8rem;
|
||
border-radius: 20px; font-weight: 600; font-size: .85em;
|
||
}
|
||
.flow-arrow { color: var(--muted); font-size: 1.2em; }
|
||
|
||
/* ── 提示框 ── */
|
||
.tip {
|
||
background: #eff6ff; border-left: 4px solid var(--accent);
|
||
padding: .8rem 1rem; margin: .8rem 0; border-radius: 0 var(--radius) var(--radius) 0;
|
||
font-size: .92em;
|
||
}
|
||
.warn {
|
||
background: #fffbeb; border-left: 4px solid #f59e0b;
|
||
padding: .8rem 1rem; margin: .8rem 0; border-radius: 0 var(--radius) var(--radius) 0;
|
||
font-size: .92em;
|
||
}
|
||
|
||
/* ── TOC ── */
|
||
.toc { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); padding: 1.2rem 1.5rem; margin: 1.5rem 0; }
|
||
.toc ol { padding-left: 1.5rem; columns: 2; }
|
||
.toc li { margin: .2rem 0; font-size: .93em; }
|
||
|
||
/* ── 页脚 ── */
|
||
footer {
|
||
text-align: center; color: var(--muted); font-size: .82em;
|
||
margin-top: 4rem; padding-top: 1.5rem; border-top: 1px solid var(--border);
|
||
}
|
||
|
||
@media (max-width: 768px) {
|
||
header h1 { font-size: 1.4rem; }
|
||
.toc ol { columns: 1; }
|
||
.card-grid { grid-template-columns: 1fr; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
|
||
<nav>
|
||
<div class="nav-inner">
|
||
<strong>🌍 intlnews</strong>
|
||
<a href="#section-1">概述</a>
|
||
<a href="#section-2">部署</a>
|
||
<a href="#section-3">配置</a>
|
||
<a href="#section-4">CLI</a>
|
||
<a href="#section-5">M1 抓取</a>
|
||
<a href="#section-6">M2 提取</a>
|
||
<a href="#section-7">M3 去重</a>
|
||
<a href="#section-8">M4 翻译</a>
|
||
<a href="#section-9">M5 向量</a>
|
||
<a href="#section-10">M6 Qdrant</a>
|
||
<a href="#section-11">M7 管道</a>
|
||
<a href="#section-12">M8 MCP</a>
|
||
<a href="#section-15">FAQ</a>
|
||
</div>
|
||
</nav>
|
||
|
||
<header>
|
||
<div class="container">
|
||
<h1>🌍 国际财经 Deep Research 平台</h1>
|
||
<p>使用手册 v1.0 · 2026-06-21 · 私有化部署</p>
|
||
</div>
|
||
</header>
|
||
|
||
<main class="container">
|
||
|
||
<!-- ====== 目录 ====== -->
|
||
<div class="toc">
|
||
<strong>📑 目录</strong>
|
||
<ol>
|
||
<li><a href="#section-1">项目概述</a></li>
|
||
<li><a href="#section-2">部署拓扑</a></li>
|
||
<li><a href="#section-3">环境配置</a></li>
|
||
<li><a href="#section-4">CLI 命令参考</a></li>
|
||
<li><a href="#section-5">M1 — 新闻抓取</a></li>
|
||
<li><a href="#section-6">M2 — 正文提取</a></li>
|
||
<li><a href="#section-7">M3 — 三层去重</a></li>
|
||
<li><a href="#section-8">M4 — 翻译 + 事件抽取</a></li>
|
||
<li><a href="#section-9">M5 — 向量生成</a></li>
|
||
<li><a href="#section-10">M6 — Qdrant 入库与检索</a></li>
|
||
<li><a href="#section-11">M7 — 全链路管道 + 日报</a></li>
|
||
<li><a href="#section-12">M8 — MCP 服务</a></li>
|
||
<li><a href="#section-13">数据目录结构</a></li>
|
||
<li><a href="#section-14">定时任务</a></li>
|
||
<li><a href="#section-15">常见问题</a></li>
|
||
</ol>
|
||
</div>
|
||
|
||
<!-- ====== 1. 项目概述 ====== -->
|
||
<h2 id="section-1">1. 项目概述</h2>
|
||
|
||
<p>本项目构建面向<strong>国际英文财经新闻</strong>的私有化 Deep Research 平台。</p>
|
||
|
||
<div class="card-grid">
|
||
<div class="card">
|
||
<strong>✅ 核心能力</strong>
|
||
<ul style="padding-left:1.2rem;margin:.5rem 0 0;font-size:.93em;">
|
||
<li>英文财经新闻抓取（Crawl4AI，12 源）</li>
|
||
<li>正文提取（trafilatura）</li>
|
||
<li>全文英译中（DeepSeek LLM）</li>
|
||
<li>投资事件抽取 + 美股代码识别</li>
|
||
<li>双语向量知识库（Qdrant 1024维）</li>
|
||
<li>中文语义检索</li>
|
||
<li>MCP 服务（Agent 深度研究）</li>
|
||
<li>每日 AI 摘要日报（HTML）</li>
|
||
</ul>
|
||
</div>
|
||
<div class="card" style="border-color:#fca5a5;">
|
||
<strong>❌ 本项目不是</strong>
|
||
<ul style="padding-left:1.2rem;margin:.5rem 0 0;font-size:.93em;">
|
||
<li>自动交易系统</li>
|
||
<li>股票预测系统</li>
|
||
<li>投资顾问系统</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="tip">
|
||
<strong>💡 对标项目</strong>：A 股 Deep Research（<code>news/</code>），架构模式可复用。
|
||
</div>
|
||
|
||
<!-- ====== 2. 部署拓扑 ====== -->
|
||
<h2 id="section-2">2. 部署拓扑</h2>
|
||
|
||
<pre><code>┌──────────────────────────────────────────────────┐
|
||
│ Overseas Server (海外) │
|
||
│ M1 Crawl4AI 抓取 → data/raw/ │
|
||
│ 每天 4 次打包 → rsync 推送 │
|
||
│ SSH: <海外服务器> │
|
||
│ 路径: /opt/intlgrab │
|
||
└────────────────────┬─────────────────────────────┘
|
||
│ rsync
|
||
▼
|
||
┌──────────────────────────────────────────────────┐
|
||
│ Domestic Server (国内) │
|
||
│ M2 正文提取 → M3 去重 → M4 翻译+事件 │
|
||
│ → M5 向量生成 → M6 Qdrant 入库 │
|
||
│ → M7 调度 + 日报 → M8 MCP 服务 │
|
||
│ SSH: <国内服务器> │
|
||
│ 路径: /home/pi/intlnews │
|
||
└──────────────────────────────────────────────────┘</code></pre>
|
||
|
||
<!-- ====== 3. 环境配置 ====== -->
|
||
<h2 id="section-3">3. 环境配置</h2>
|
||
|
||
<h3>3.1 依赖安装</h3>
|
||
<pre><code>cd /home/pi/intlnews
|
||
uv sync</code></pre>
|
||
|
||
<h3>3.2 配置文件</h3>
|
||
|
||
<table>
|
||
<tr><th>文件</th><th>用途</th><th>注意</th></tr>
|
||
<tr><td><code>.env</code></td><td>API Key / URL</td><td>不入 Git，从 <code>.env.example</code> 复制</td></tr>
|
||
<tr><td><code>configs/system.yaml</code></td><td>功能参数</td><td>模型名、阈值、超时、调度</td></tr>
|
||
<tr><td><code>configs/sources.yaml</code></td><td>新闻源定义</td><td>12 个英文财经源</td></tr>
|
||
</table>
|
||
|
||
<h3>3.3 必需环境变量</h3>
|
||
|
||
<pre><code># DeepSeek（M4 翻译+事件抽取）
|
||
DEEPSEEK_API_KEY=sk-your-key
|
||
DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||
|
||
# Qwen（备选 LLM）
|
||
QWEN_API_KEY=sk-your-key
|
||
QWEN_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
||
|
||
# DashScope（M5 向量生成）
|
||
DASHSCOPE_API_KEY=sk-your-key
|
||
|
||
# Qdrant（留空使用本地文件模式）
|
||
QDRANT_URL=http://localhost:6333
|
||
QDRANT_API_KEY=</code></pre>
|
||
|
||
<h3>3.4 关键配置项</h3>
|
||
|
||
<pre><code># configs/system.yaml（关键项）
|
||
crawler.max_memory_mb: 1800 # 内存上限
|
||
dedup.hamming_distance_threshold: 3
|
||
dedup.simhash_window_days: 30
|
||
llm.provider: "deepseek"
|
||
llm.deepseek_model: "deepseek-v4-flash"
|
||
llm.concurrency: 3
|
||
embedding.dashscope_model: "text-embedding-v3"
|
||
embedding.dimension: 1024
|
||
qdrant.collection: "en_finance_news"
|
||
schedule.day_cutoff_hour: 6 # 新闻日切分点</code></pre>
|
||
|
||
<!-- ====== 4. CLI 命令参考 ====== -->
|
||
<h2 id="section-4">4. CLI 命令参考</h2>
|
||
|
||
<p>所有命令通过 <code>uv run en-news</code> 执行：</p>
|
||
|
||
<table>
|
||
<tr><th>命令</th><th>Milestone</th><th>功能</th></tr>
|
||
<tr><td><code>crawl</code></td><td>M1</td><td>抓取英文财经新闻</td></tr>
|
||
<tr><td><code>extract</code></td><td>M2</td><td>正文提取</td></tr>
|
||
<tr><td><code>dedup</code></td><td>M3</td><td>三层去重</td></tr>
|
||
<tr><td><code>translate</code></td><td>M4</td><td>翻译 + 事件抽取</td></tr>
|
||
<tr><td><code>embed</code></td><td>M5</td><td>向量生成</td></tr>
|
||
<tr><td><code>index</code></td><td>M6</td><td>Qdrant 入库</td></tr>
|
||
<tr><td><code>search <query></code></td><td>M6</td><td>语义检索</td></tr>
|
||
<tr><td><code>pipeline</code></td><td>M7</td><td>一键全链路 M2→M6</td></tr>
|
||
<tr><td><code>report</code></td><td>M7</td><td>日报生成</td></tr>
|
||
<tr><td><code>mcp-server</code></td><td>M8</td><td>启动 MCP 服务</td></tr>
|
||
</table>
|
||
|
||
<pre><code># 常用选项
|
||
uv run en-news crawl --source forexlive # 指定源
|
||
uv run en-news search "美联储" --top-k 5 # 指定返回数
|
||
uv run en-news index --recreate # 重建 Qdrant
|
||
uv run en-news pipeline --skip-report # 跳过日报</code></pre>
|
||
|
||
<!-- ====== 5. M1 ====== -->
|
||
<h2 id="section-5">5. M1 — 新闻抓取</h2>
|
||
|
||
<h3>5.1 手动抓取</h3>
|
||
<pre><code>uv run en-news crawl # 全部源
|
||
uv run en-news crawl -s forexlive # 指定源</code></pre>
|
||
|
||
<h3>5.2 12 个英文财经源</h3>
|
||
|
||
<table>
|
||
<tr><th>源 ID</th><th>名称</th><th>类型</th></tr>
|
||
<tr><td>reuters</td><td>Reuters</td><td>综合财经</td></tr>
|
||
<tr><td>cnbc</td><td>CNBC</td><td>市场新闻</td></tr>
|
||
<tr><td>marketwatch</td><td>MarketWatch</td><td>市场数据</td></tr>
|
||
<tr><td>ft</td><td>Financial Times</td><td>财经深度</td></tr>
|
||
<tr><td>yahoo_finance</td><td>Yahoo Finance</td><td>综合</td></tr>
|
||
<tr><td>investing</td><td>Investing.com</td><td>全球市场</td></tr>
|
||
<tr><td>seekingalpha</td><td>Seeking Alpha</td><td>投资分析</td></tr>
|
||
<tr><td>barrons</td><td>Barrons</td><td>市场评论</td></tr>
|
||
<tr><td>wsj</td><td>WSJ</td><td>综合财经</td></tr>
|
||
<tr><td>economist</td><td>The Economist</td><td>经济分析</td></tr>
|
||
<tr><td>forexlive</td><td>ForexLive</td><td>外汇新闻</td></tr>
|
||
<tr><td>zerohedge</td><td>ZeroHedge</td><td>另类财经</td></tr>
|
||
</table>
|
||
|
||
<h3>5.3 增量抓取机制</h3>
|
||
|
||
<p>抓取采用<strong>两层增量</strong>确保不重复下载和存储：</p>
|
||
|
||
<table>
|
||
<tr><th>层级</th><th>位置</th><th>机制</th></tr>
|
||
<tr><td>抓取层</td><td><code>_extract_article_urls</code></td><td>读取当日 index.jsonl 中已抓取的 url_hash，跳过已存在的 URL，不重复下载</td></tr>
|
||
<tr><td>存储层</td><td><code>write_index_jsonl</code></td><td>追加写入前再次检查 url_hash，已存在则跳过</td></tr>
|
||
</table>
|
||
|
||
<p>同一天内多次执行 <code>crawl</code>，只有新文章才会被下载和存储。</p>
|
||
|
||
<h3>5.4 产物</h3>
|
||
<pre><code>data/raw/{source_id}/{YYYYMMDD}/
|
||
├── {url_hash}.html # 原始 HTML
|
||
├── {url_hash}.md # Crawl4AI Markdown
|
||
└── index.jsonl # 文章索引</code></pre>
|
||
|
||
<h3>5.5 反爬策略</h3>
|
||
|
||
<p>部分新闻源有反爬保护（如 DataDome）。系统支持三种策略，按优先级自动选择：</p>
|
||
|
||
<table>
|
||
<tr><th>优先级</th><th>策略</th><th>配置</th><th>说明</th></tr>
|
||
<tr><td>1</td><td><strong>RSS 抓取</strong></td><td><code>rss_url</code></td><td>通过 Feed 获取文章，完全绕过反爬 ✅ MarketWatch</td></tr>
|
||
<tr><td>2</td><td><strong>Stealth</strong></td><td><code>anti_bot_mode: "stealth"</code></td><td>隐藏 webdriver 特征</td></tr>
|
||
<tr><td>3</td><td><strong>Headful</strong></td><td><code>anti_bot_mode: "headful"</code></td><td>非 headless 浏览器，最像真人</td></tr>
|
||
</table>
|
||
|
||
<p><strong>已验证</strong>：MarketWatch RSS 10/10 ✅ · WSJ RSS 20/20 ✅ · ForexLive 17/17 ✅ · ZeroHedge 15/15 ✅ · Barron's RSS 10/10 ✅ · Reuters ⚠️ · SeekingAlpha ⚠️</p>
|
||
|
||
<!-- ====== 6. M2 ====== -->
|
||
<h2 id="section-6">6. M2 — 正文提取</h2>
|
||
|
||
<pre><code>uv run en-news extract # 全部源
|
||
uv run en-news extract -s forexlive # 指定源</code></pre>
|
||
|
||
<p><strong>技术方案</strong>：优先 Crawl4AI Markdown → 回退 trafilatura（英文优化）。最小正文 50 词。</p>
|
||
|
||
<pre><code>data/processed/{source_id}/{YYYYMMDD}/
|
||
└── {url_hash}.json # ProcessedArticle</code></pre>
|
||
|
||
<!-- ====== 7. M3 ====== -->
|
||
<h2 id="section-7">7. M3 — 三层去重</h2>
|
||
|
||
<pre><code>uv run en-news dedup</code></pre>
|
||
|
||
<table>
|
||
<tr><th>层级</th><th>方法</th><th>说明</th></tr>
|
||
<tr><td>L1</td><td>URL Hash</td><td>完全相同 URL</td></tr>
|
||
<tr><td>L2</td><td>内容 Hash</td><td>SHA1[:16] 去标点空白后匹配</td></tr>
|
||
<tr><td>L3</td><td>SimHash</td><td>字符 3-gram，汉明距离 ≤ 3，30 天窗口</td></tr>
|
||
</table>
|
||
|
||
<pre><code>data/dedup/fingerprints.sqlite3 # SQLite 指纹库
|
||
data/deduped/{YYYYMMDD}/uniques/ # 唯一文章</code></pre>
|
||
|
||
<!-- ====== 8. M4 ====== -->
|
||
<h2 id="section-8">8. M4 — 翻译 + 事件抽取</h2>
|
||
|
||
<pre><code>uv run en-news translate</code></pre>
|
||
|
||
<div class="card">
|
||
<strong>⚙️ 技术规格</strong>
|
||
<ul style="padding-left:1.2rem;margin:.3rem 0 0;font-size:.93em;">
|
||
<li>Provider: DeepSeek v4-flash（默认）/ Qwen 备选</li>
|
||
<li>单次调用完成翻译+抽取，节省 token</li>
|
||
<li>并发 3 线程，3 次指数退避重试</li>
|
||
<li>正文截断 8000 字符</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<h3>14 种事件类型</h3>
|
||
<p>
|
||
<span class="badge badge-blue">财报披露</span>
|
||
<span class="badge badge-blue">并购收购</span>
|
||
<span class="badge badge-blue">产品发布</span>
|
||
<span class="badge badge-blue">监管政策</span>
|
||
<span class="badge badge-blue">宏观经济</span>
|
||
<span class="badge badge-blue">央行决议</span>
|
||
<span class="badge badge-blue">行业动态</span>
|
||
<span class="badge badge-blue">技术突破</span>
|
||
<span class="badge badge-blue">高管变动</span>
|
||
<span class="badge badge-blue">诉讼法律</span>
|
||
<span class="badge badge-blue">市场异动</span>
|
||
<span class="badge badge-blue">地缘政治</span>
|
||
<span class="badge badge-blue">大宗商品</span>
|
||
<span class="badge badge-blue">外汇波动</span>
|
||
</p>
|
||
|
||
<h3>情绪标签</h3>
|
||
<p>
|
||
<span class="badge badge-pos">🟢 positive 利好</span>
|
||
<span class="badge badge-neu">⚪ neutral 中性</span>
|
||
<span class="badge badge-neg">🔴 negative 利空</span>
|
||
</p>
|
||
|
||
<!-- ====== 9. M5 ====== -->
|
||
<h2 id="section-9">9. M5 — 向量生成</h2>
|
||
|
||
<pre><code>uv run en-news embed</code></pre>
|
||
|
||
<p><strong>技术方案</strong>：DashScope <code>text-embedding-v3</code>，1024 维。嵌入文本 = 中文标题 + 事件标签 + 中文正文（4000 字符截断）。</p>
|
||
|
||
<pre><code>data/embeddings/{YYYYMMDD}/
|
||
└── {url_hash}.json # EmbeddingResult (1024 floats)</code></pre>
|
||
|
||
<!-- ====== 10. M6 ====== -->
|
||
<h2 id="section-10">10. M6 — Qdrant 入库与检索</h2>
|
||
|
||
<h3>入库</h3>
|
||
<pre><code>uv run en-news index # 增量入库
|
||
uv run en-news index --recreate # 重建 collection</code></pre>
|
||
|
||
<h3>语义搜索</h3>
|
||
<pre><code>uv run en-news search "美联储利率决议"
|
||
uv run en-news search "伊朗霍尔木兹海峡" --top-k 5</code></pre>
|
||
|
||
<div class="card">
|
||
<strong>📦 Qdrant 规格</strong>
|
||
<ul style="padding-left:1.2rem;margin:.3rem 0 0;font-size:.93em;">
|
||
<li>模式: 本地文件（<code>data/qdrant_storage/</code>），无需 Docker</li>
|
||
<li>Collection: <code>en_finance_news</code></li>
|
||
<li>距离: Cosine · 维度: 1024</li>
|
||
<li>Payload: title / title_zh / url / events / content_zh_preview</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<!-- ====== 11. M7 ====== -->
|
||
<h2 id="section-11">11. M7 — 全链路管道 + 日报</h2>
|
||
|
||
<h3>手动全链路运行（完整流程）</h3>
|
||
|
||
<p>当需要手工执行完整的数据处理流程时，按顺序执行：</p>
|
||
|
||
<pre><code># 步骤 1（海外）: 抓取英文财经新闻
|
||
ssh <海外服务器> "cd /opt/intlgrab && uv run en-news crawl"
|
||
|
||
# 步骤 2（海外）: 打包 raw 数据
|
||
ssh <海外服务器> "cd /opt/intlgrab && bash scripts/overseas_pack.sh"
|
||
|
||
# 步骤 3（国内）: 拉取海外数据
|
||
cd /home/pi/intlnews && bash scripts/domestic_sync.sh
|
||
|
||
# 步骤 4: 一键全链路 M2→M6 + 日报
|
||
cd /home/pi/intlnews && uv run en-news pipeline</code></pre>
|
||
|
||
<p>也可以分步执行（适合调试）：</p>
|
||
|
||
<pre><code>uv run en-news extract # M2: 正文提取
|
||
uv run en-news dedup # M3: 三层去重
|
||
uv run en-news translate # M4: 翻译 + 事件抽取
|
||
uv run en-news embed # M5: 向量生成
|
||
uv run en-news index # M6: Qdrant 入库
|
||
uv run en-news report # 日报生成</code></pre>
|
||
|
||
<h3>一键管道（自动）</h3>
|
||
<pre><code>uv run en-news pipeline # 全链路 M2→M6 + 日报
|
||
uv run en-news pipeline --skip-report # 跳过日报
|
||
uv run en-news report # 仅日报</code></pre>
|
||
|
||
<h3>管道流程</h3>
|
||
<div class="flow">
|
||
<span class="flow-step">M2 extract</span>
|
||
<span class="flow-arrow">→</span>
|
||
<span class="flow-step">M3 dedup</span>
|
||
<span class="flow-arrow">→</span>
|
||
<span class="flow-step">M4 translate</span>
|
||
<span class="flow-arrow">→</span>
|
||
<span class="flow-step">M5 embed</span>
|
||
<span class="flow-arrow">→</span>
|
||
<span class="flow-step">M6 index</span>
|
||
<span class="flow-arrow">→</span>
|
||
<span class="flow-step">📰 日报</span>
|
||
</div>
|
||
|
||
<div class="warn">
|
||
<strong>⚠️ 降级策略</strong>：每步失败记录日志但不阻断后续步骤。
|
||
</div>
|
||
|
||
<h3>HTML 日报五板块</h3>
|
||
<ol style="padding-left:1.5rem;">
|
||
<li>🤖 <strong>AI 摘要</strong> — LLM 根据 important≥4 事件生成要点总结</li>
|
||
<li>🔥 <strong>重要事件</strong> — 事件表格（情绪/重要度/摘要/链接）</li>
|
||
<li>📊 <strong>数据总览</strong> — M1→M6 管道统计数字</li>
|
||
<li>📈 <strong>情绪分布</strong> — 利好/利空/中性比例条</li>
|
||
<li>📋 <strong>事件类型 TOP 10</strong></li>
|
||
</ol>
|
||
|
||
<p>本地: <code>data/reports/intl_news_daily_{YYYYMMDD}.html</code></p>
|
||
<p>线上: 自动上传到 <code>https://echart.doorcome.cn/research/{YYYYMMDD}/</code></p>
|
||
<p>上传配置在 <code>configs/system.yaml</code> → <code>report.upload_host</code> / <code>report.upload_path</code></p>
|
||
|
||
<!-- ====== 12. M8 ====== -->
|
||
<h2 id="section-12">12. M8 — MCP 服务</h2>
|
||
|
||
<h3>启动</h3>
|
||
<pre><code>uv run en-news mcp-server</code></pre>
|
||
|
||
<h3>5 个 MCP Tool</h3>
|
||
|
||
<table>
|
||
<tr><th>Tool</th><th>功能</th><th>示例调用</th></tr>
|
||
<tr><td><code>search_news</code></td><td>语义检索新闻</td><td><code>search_news("美联储利率决议")</code></td></tr>
|
||
<tr><td><code>search_by_stock</code></td><td>美股代码检索</td><td><code>search_by_stock("AAPL")</code></td></tr>
|
||
<tr><td><code>search_by_sentiment</code></td><td>情绪过滤检索</td><td><code>search_by_sentiment("加息", sentiment="negative")</code></td></tr>
|
||
<tr><td><code>get_today_events</code></td><td>当日重要事件</td><td><code>get_today_events(importance_min=4)</code></td></tr>
|
||
<tr><td><code>get_stats</code></td><td>系统统计</td><td><code>get_stats()</code></td></tr>
|
||
</table>
|
||
|
||
<h3>Claude Code MCP 配置</h3>
|
||
<pre><code>{
|
||
"mcpServers": {
|
||
"intl-news": {
|
||
"command": "uv",
|
||
"args": ["run", "en-news", "mcp-server"],
|
||
"cwd": "/home/pi/intlnews"
|
||
}
|
||
}
|
||
}</code></pre>
|
||
|
||
<!-- ====== 13. 数据目录 ====== -->
|
||
<h2 id="section-13">13. 数据目录结构</h2>
|
||
|
||
<pre><code>data/
|
||
├── raw/ # M1: 原始抓取
|
||
│ └── {source_id}/{YYYYMMDD}/
|
||
├── processed/ # M2: 正文提取
|
||
│ └── {source_id}/{YYYYMMDD}/
|
||
├── dedup/ # M3: 指纹库 + 去重结果
|
||
│ ├── fingerprints.sqlite3
|
||
│ └── {YYYYMMDD}/uniques/
|
||
├── events/ # M4: 翻译+事件
|
||
│ └── {YYYYMMDD}/
|
||
├── embeddings/ # M5: 向量
|
||
│ └── {YYYYMMDD}/
|
||
├── qdrant_storage/ # M6: Qdrant 本地存储
|
||
└── reports/ # M7: 日报
|
||
└── intl_news_daily_{YYYYMMDD}.html</code></pre>
|
||
|
||
<!-- ====== 14. 定时任务 ====== -->
|
||
<h2 id="section-14">14. 定时任务</h2>
|
||
|
||
<h3>时间线</h3>
|
||
|
||
<table>
|
||
<tr><th>时间</th><th>服务器</th><th>动作</th></tr>
|
||
<tr><td>05:55</td><td>海外</td><td>打包 data/raw/</td></tr>
|
||
<tr><td>06:30</td><td>国内</td><td>拉取 → M2→M6 管道 → 📰 日报</td></tr>
|
||
<tr><td>11:55</td><td>海外</td><td>打包</td></tr>
|
||
<tr><td>12:00</td><td>国内</td><td>拉取 → 管道</td></tr>
|
||
<tr><td>17:55</td><td>海外</td><td>打包</td></tr>
|
||
<tr><td>18:00</td><td>国内</td><td>拉取 → 管道</td></tr>
|
||
<tr><td>21:55</td><td>海外</td><td>打包</td></tr>
|
||
<tr><td>22:00</td><td>国内</td><td>拉取 → 管道 → 📰 日报</td></tr>
|
||
</table>
|
||
|
||
<h3>新闻日定义</h3>
|
||
<p>切分点 <code>day_cutoff_hour: 6</code>。当天 06:00 至次日 05:59 属于同一个新闻日。</p>
|
||
|
||
<!-- ====== 15. FAQ ====== -->
|
||
<h2 id="section-15">15. 常见问题</h2>
|
||
|
||
<h3>Q: 如何新增新闻源？</h3>
|
||
<p>编辑 <code>configs/sources.yaml</code>：</p>
|
||
<pre><code>- id: "new_source"
|
||
name: "New Source Name"
|
||
enabled: true
|
||
homepage: "https://example.com/finance/"
|
||
article_url_pattern: "/news/[^/]+/"
|
||
js_render: false
|
||
max_articles_per_run: 30</code></pre>
|
||
|
||
<h3>Q: 翻译质量不好怎么办？</h3>
|
||
<ol style="padding-left:1.5rem;">
|
||
<li>调整 <code>configs/system.yaml</code> 中 <code>llm.temperature</code>（降低更保守）</li>
|
||
<li>编辑 <code>prompts/translation_and_extraction.md</code> 优化 Prompt</li>
|
||
<li>切换 Provider：<code>llm.provider: "qwen"</code></li>
|
||
</ol>
|
||
|
||
<h3>Q: Qdrant 检索太慢？</h3>
|
||
<p>本地文件模式 17 条 < 0.01s，足够快。数据量 > 10 万条时切换到 Docker 模式。</p>
|
||
|
||
<h3>Q: 如何查看日志？</h3>
|
||
<pre><code>tail -f logs/sync.log
|
||
tail -f logs/crawler.log # 海外</code></pre>
|
||
|
||
<h3>Q: 数据如何备份？</h3>
|
||
<pre><code>tar czf intlnews_backup_$(date +%Y%m%d).tar.gz data/</code></pre>
|
||
|
||
<!-- ====== 附录 ====== -->
|
||
<h2>附录：技术栈</h2>
|
||
|
||
<table>
|
||
<tr><th>组件</th><th>技术</th></tr>
|
||
<tr><td>语言</td><td>Python 3.11</td></tr>
|
||
<tr><td>包管理</td><td>uv + pyproject.toml</td></tr>
|
||
<tr><td>抓取</td><td>Crawl4AI + Playwright</td></tr>
|
||
<tr><td>正文提取</td><td>trafilatura</td></tr>
|
||
<tr><td>去重</td><td>SimHash + SQLite</td></tr>
|
||
<tr><td>LLM</td><td>DeepSeek v4-flash（OpenAI SDK）</td></tr>
|
||
<tr><td>Embedding</td><td>DashScope text-embedding-v3</td></tr>
|
||
<tr><td>向量库</td><td>Qdrant（本地文件模式）</td></tr>
|
||
<tr><td>MCP</td><td>FastMCP</td></tr>
|
||
<tr><td>CLI</td><td>Typer</td></tr>
|
||
<tr><td>配置</td><td>YAML + .env</td></tr>
|
||
<tr><td>数据模型</td><td>Pydantic v2</td></tr>
|
||
</table>
|
||
|
||
</main>
|
||
|
||
<footer>
|
||
<div class="container">
|
||
<p>国际财经 Deep Research 私有投研平台 · 使用手册 v1.0 · 2026-06-21</p>
|
||
</div>
|
||
</footer>
|
||
|
||
</body>
|
||
</html>
|