fix: 日报摘要可靠性(去模型兜底+重试) 与取数逻辑优化
- llm/client: 移除内置默认模型兜底(deepseek-chat/qwen-plus), 模型必须显式配置否则报错 - reporter._llm_call: 指数退避重试(LLM_RETRY_TIMES 默认3 / LLM_RETRY_BACKOFF_SEC 默认2s) - pipeline report: report_date 改为当天(原昨天+回溯3天) - reporter._collect_news_events: 读当天+前一天目录, publish_time 30h 回溯(NEWS_LOOKBACK_HOURS=30), 统一时区 - reporter._collect_xwlb: 固定取 day_str 前一日(已播出联播), source_date 同步 - 公告/调研/互动保持近15日设置(CNINFO_DAYS_BACK), 不受 30h 影响 - 测试: 新增 30h回溯/带时区/重试/模型缺失/xwlb 前一日 用例
This commit is contained in:
@@ -77,3 +77,7 @@ NEWS_DB_USER=myquant
|
|||||||
NEWS_DB_PASSWORD= # 填真实值,禁止写入源码/文档
|
NEWS_DB_PASSWORD= # 填真实值,禁止写入源码/文档
|
||||||
NEWS_DB_NAME=myquant
|
NEWS_DB_NAME=myquant
|
||||||
REPORT_HISTORY_DIR=data/reports_history
|
REPORT_HISTORY_DIR=data/reports_history
|
||||||
|
|
||||||
|
# ---- LLM 日报摘要重试 ----
|
||||||
|
LLM_RETRY_TIMES=3 # AI 摘要调用失败重试次数(默认 3)
|
||||||
|
LLM_RETRY_BACKOFF_SEC=2.0 # 指数退避基数,秒(默认 2.0: 2s,4s,8s...)
|
||||||
|
|||||||
+52
-5
@@ -1,6 +1,6 @@
|
|||||||
# continuation.md
|
# continuation.md
|
||||||
|
|
||||||
> `checkpoint` @ 2026-08-03 21:30
|
> `checkpoint` @ 2026-08-05 08:30
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -10,14 +10,61 @@
|
|||||||
| --- | --- |
|
| --- | --- |
|
||||||
| 新闻源 | 14 个(13 Web + 1 API: xwlb 新闻联播) |
|
| 新闻源 | 14 个(13 Web + 1 API: xwlb 新闻联播) |
|
||||||
| Qdrant | 本地文件模式 `data/qdrant_storage/` |
|
| Qdrant | 本地文件模式 `data/qdrant_storage/` |
|
||||||
| 日报 | **M10 完成: 结构化入库 MySQL(myquant 库 news_report/news_event 表),不再生成 HTML** |
|
| 日报 | **M10 完成并已部署 pi5: 结构化入库 MySQL;日报按当天日期生成(新闻 30h 回溯 / xwlb 取前一日 / 公告调研近 15 日)** |
|
||||||
| 调度器 | APScheduler,systemd `a-share-research.service` |
|
| DB 连接 | pi 上 systemd 服务 `a-share-db-tunnel` 常驻(0.0.0.0:13306 → doorcome.cn:3306);**pi5 直连 192.168.1.10:13306** |
|
||||||
| LLM | `deepseek-v4-flash`(绝不允许擅自修改) |
|
| 调度器 | APScheduler,systemd `a-share-research.service`(pi5);每天 07:00 首次任务生成日报(12/18/22 点不生成) |
|
||||||
| 服务器 | `pi@192.168.1.160`,项目 `/home/pi/news/` |
|
| LLM | `deepseek-v4-flash`(绝不允许擅自修改;模型必须显式配置,无内置兜底) |
|
||||||
|
| 服务器 | `pi@192.168.1.160`(生产)/ `pi@192.168.1.10`(DB 隧道宿主) |
|
||||||
| 抓取方式 | js_render=false → httpx 直连;js_render=true → Playwright |
|
| 抓取方式 | js_render=false → httpx 直连;js_render=true → Playwright |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 本次完成 (2026-08-05) — 日报可靠性修复与取数逻辑优化
|
||||||
|
|
||||||
|
**目标:** 解决日报 AI 摘要偶发失败;修正日报日期与 xwlb/新闻取数语义。
|
||||||
|
|
||||||
|
**1. AI 摘要可靠性(llm/client.py + scheduler/reporter.py):**
|
||||||
|
- 去掉内置默认模型兜底(`deepseek-chat`/`qwen-plus`),模型必须显式配置(`DEEPSEEK_MODEL`/`QWEN_MODEL` → `LLM_MODEL`),缺失即报错,避免静默用错模型
|
||||||
|
- `_llm_call` 增加指数退避重试:`_LLM_RETRY_TIMES`(默认 3 次)、`_LLM_RETRY_BACKOFF_SEC`(默认 2.0s,可 .env 覆盖),全部失败才抛异常
|
||||||
|
- 确认 AI 摘要模型:`deepseek` + `deepseek-v4-flash`(生产实测)
|
||||||
|
|
||||||
|
**2. 日报取数逻辑(scheduler/pipeline.py + reporter.py):**
|
||||||
|
- pipeline report 步骤:`report_date = date.today()`(原为昨天+回溯 3 天)
|
||||||
|
- `_collect_news_events`:读当天+前一天事件目录,按 `publish_time` 过滤最近 30 小时(`_NEWS_LOOKBACK_HOURS=30`);时区统一(naive 假定本地时区);无时间戳事件保留
|
||||||
|
- `_collect_xwlb`:固定查 `day_str` 前一日(《新闻联播》19:00 播出,早间日报只能取昨晚已播出的一期);`source_date` 同步为实际来源日
|
||||||
|
- **公告/调研/互动保持原设置:近 15 日(`STOCK_REPORT_DAYS=15`),irm 互动仍跳过**——未受 30h 改动影响
|
||||||
|
|
||||||
|
**验证(pi5):**
|
||||||
|
- 单测 9 个(30h 回溯/带时区时间戳/重试/模型缺失报错/xwlb 前一日)全部通过;全量 212 passed + 3 crawler 预存在失败
|
||||||
|
- 生产端到端 report_id=191(2026-08-05):news 20 + cninfo 20 + xwlb 34(08-04 联播),AI 摘要 2076 字
|
||||||
|
- 生产服务已重启生效
|
||||||
|
|
||||||
|
**本次代码尚未 git 提交(见待办)。**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 本次完成 (2026-08-03 22:00) — pi5 部署与生产测试
|
||||||
|
|
||||||
|
**操作:** M10 代码全量同步 pi5 + 生产环境验收(所有测试在 pi5 执行,Mac 不再作为测试环境)。
|
||||||
|
|
||||||
|
**文件同步:** rsync 本地 → `pi5:/home/pi/news/`(排除 .venv/data/logs/.git/.env/configs/*.yaml);清除 pi5 根目录 6 月 17 日旧版散文件(已 tar 备份 /tmp/news_backup_20260803.tar.gz);pi5 `uv sync` 装 pymysql。
|
||||||
|
|
||||||
|
**DB 隧道修复:** pi 原 autossh 参数 `-L 13306:0.0.0.0:3306` 未生效(0.0.0.0 被当远端目标),改为 `-L 0.0.0.0:13306:127.0.0.1:3306` 并持久化为 systemd 服务 `a-share-db-tunnel`(enabled + active)。
|
||||||
|
|
||||||
|
**测试结果(pi5):**
|
||||||
|
- 全量 pytest:**208 passed, 3 failed**(3 个失败均为 crawler retry mock 预先存在问题,与 M10 无关)
|
||||||
|
- `report-import` 幂等:已存在文件正确 skipped
|
||||||
|
- 生产端到端:`a-share report --date 20260802/20260803` → report_id=181/182 入库成功(AI 摘要正常,57/58 事件)
|
||||||
|
- DB 总量:finance 52 + intl 128 = 180 行(历史 177 + 端到端测试 2 + 本机 1)
|
||||||
|
- 生产服务 `a-share-research` 已重启 active,22:00 定时任务起用新代码
|
||||||
|
|
||||||
|
**已知问题:**
|
||||||
|
- `tests/test_crawler.py` 3 个 retry 测试失败(预先存在,crawl4ai mock 行为)
|
||||||
|
- Mac 本机 anaconda/uv python 出站到 192.168.1.10 被拦截(EHOSTUNREACH;nc/bash/系统 python 正常)——仅影响本机,pi5 不受影响;Mac 本地用 `127.0.0.1` + ssh 隧道绕过
|
||||||
|
- 生产 pi5 的 `.env` 里 `NEWS_DB_HOST=192.168.1.10`(直连);Mac 本地 `.env` 为 `127.0.0.1`(隧道)——**两处 .env 不同,勿互相覆盖**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 本次完成 (2026-08-03) — M10 日报结构化入库
|
## 本次完成 (2026-08-03) — M10 日报结构化入库
|
||||||
|
|
||||||
**目标:** 日报前后端分离的数据层——日报内容结构化存入 MySQL(`news_` 前缀表),历史 178 份日报 HTML 解析入库;本项目不做 API/前端(用户另行实现)。
|
**目标:** 日报前后端分离的数据层——日报内容结构化存入 MySQL(`news_` 前缀表),历史 178 份日报 HTML 解析入库;本项目不做 API/前端(用户另行实现)。
|
||||||
|
|||||||
+10
-9
@@ -7,7 +7,8 @@
|
|||||||
DeepSeek: DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL / DEEPSEEK_MODEL
|
DeepSeek: DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL / DEEPSEEK_MODEL
|
||||||
Qwen: QWEN_API_KEY / QWEN_BASE_URL / QWEN_MODEL
|
Qwen: QWEN_API_KEY / QWEN_BASE_URL / QWEN_MODEL
|
||||||
(QWEN_API_KEY -> DASHSCOPE_API_KEY 兜底)
|
(QWEN_API_KEY -> DASHSCOPE_API_KEY 兜底)
|
||||||
LLM_MODEL (兜底) LLM_TEMPERATURE / LLM_TIMEOUT_SEC
|
模型必须显式配置(provider 对应的 *_MODEL 或 LLM_MODEL),不再提供内置默认模型。
|
||||||
|
LLM_TEMPERATURE / LLM_TIMEOUT_SEC
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -22,10 +23,6 @@ from openai import AsyncOpenAI, OpenAI
|
|||||||
_DEEPSEEK_DEFAULT_BASE = "https://api.deepseek.com"
|
_DEEPSEEK_DEFAULT_BASE = "https://api.deepseek.com"
|
||||||
_QWEN_DEFAULT_BASE = "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
_QWEN_DEFAULT_BASE = "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||||
|
|
||||||
# 默认模型
|
|
||||||
_DEEPSEEK_DEFAULT_MODEL = "deepseek-chat"
|
|
||||||
_QWEN_DEFAULT_MODEL = "qwen-plus"
|
|
||||||
|
|
||||||
# 抽取任务默认参数
|
# 抽取任务默认参数
|
||||||
DEFAULT_TIMEOUT_SEC = 60.0
|
DEFAULT_TIMEOUT_SEC = 60.0
|
||||||
DEFAULT_TEMPERATURE = 0.1
|
DEFAULT_TEMPERATURE = 0.1
|
||||||
@@ -69,13 +66,17 @@ def load_llm_config(
|
|||||||
if p == "deepseek":
|
if p == "deepseek":
|
||||||
api_key = _read_env("DEEPSEEK_API_KEY") or ""
|
api_key = _read_env("DEEPSEEK_API_KEY") or ""
|
||||||
base = _read_env("DEEPSEEK_BASE_URL", _DEEPSEEK_DEFAULT_BASE) or _DEEPSEEK_DEFAULT_BASE
|
base = _read_env("DEEPSEEK_BASE_URL", _DEEPSEEK_DEFAULT_BASE) or _DEEPSEEK_DEFAULT_BASE
|
||||||
# DEEPSEEK_MODEL → LLM_MODEL(兜底) → 默认
|
# DEEPSEEK_MODEL → LLM_MODEL;模型必须显式配置,不提供内置默认
|
||||||
m = model or _read_env("DEEPSEEK_MODEL") or _read_env("LLM_MODEL") or _DEEPSEEK_DEFAULT_MODEL
|
m = model or _read_env("DEEPSEEK_MODEL") or _read_env("LLM_MODEL")
|
||||||
|
if not m:
|
||||||
|
raise ValueError("未配置 LLM 模型: 请设置 DEEPSEEK_MODEL 或 LLM_MODEL")
|
||||||
elif p in ("qwen", "dashscope"):
|
elif p in ("qwen", "dashscope"):
|
||||||
api_key = _read_env("QWEN_API_KEY") or _read_env("DASHSCOPE_API_KEY") or ""
|
api_key = _read_env("QWEN_API_KEY") or _read_env("DASHSCOPE_API_KEY") or ""
|
||||||
base = _read_env("QWEN_BASE_URL", _QWEN_DEFAULT_BASE) or _QWEN_DEFAULT_BASE
|
base = _read_env("QWEN_BASE_URL", _QWEN_DEFAULT_BASE) or _QWEN_DEFAULT_BASE
|
||||||
# QWEN_MODEL → LLM_MODEL(兜底) → 默认
|
# QWEN_MODEL → LLM_MODEL;模型必须显式配置,不提供内置默认
|
||||||
m = model or _read_env("QWEN_MODEL") or _read_env("LLM_MODEL") or _QWEN_DEFAULT_MODEL
|
m = model or _read_env("QWEN_MODEL") or _read_env("LLM_MODEL")
|
||||||
|
if not m:
|
||||||
|
raise ValueError("未配置 LLM 模型: 请设置 QWEN_MODEL 或 LLM_MODEL")
|
||||||
p = "qwen" # 内部统一用 qwen
|
p = "qwen" # 内部统一用 qwen
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"未知 LLM provider: {p!r},仅支持 deepseek / qwen")
|
raise ValueError(f"未知 LLM provider: {p!r},仅支持 deepseek / qwen")
|
||||||
|
|||||||
+4
-28
@@ -74,39 +74,15 @@ def run_step(name: str, date_str: str) -> StepResult:
|
|||||||
返回: StepResult。
|
返回: StepResult。
|
||||||
"""
|
"""
|
||||||
# report 步骤:内部函数,不走子进程
|
# report 步骤:内部函数,不走子进程
|
||||||
# 日报默认统计"昨天"的数据(因为今天的数据由当天的定时任务处理)。
|
# 日报按当天日期生成: 新闻由 _collect_news_events 回溯过去 30 小时,
|
||||||
# 如果昨天没有事件数据,向前回溯最多 3 天,取最近有数据的日期。
|
# xwlb 由 _collect_xwlb 固定取前一日(已播出)联播。
|
||||||
if name == "report":
|
if name == "report":
|
||||||
started = datetime.now()
|
started = datetime.now()
|
||||||
try:
|
try:
|
||||||
from datetime import timedelta # noqa: E402
|
|
||||||
from pathlib import Path # noqa: E402
|
|
||||||
|
|
||||||
from .reporter import generate_report # noqa: E402
|
from .reporter import generate_report # noqa: E402
|
||||||
|
|
||||||
# 向前回溯找最近有事件数据的日期(最多回溯 3 天)
|
report_date = date.today().strftime("%Y%m%d")
|
||||||
report_date: str | None = None
|
logger.info("日报: report_date={} (新闻 30h 回溯, xwlb 前一日)", report_date)
|
||||||
for offset in range(1, 4):
|
|
||||||
candidate = (date.today() - timedelta(days=offset)).strftime("%Y%m%d")
|
|
||||||
ev_dir = Path(f"data/events/{candidate}")
|
|
||||||
if ev_dir.is_dir() and list(ev_dir.glob("*.json")):
|
|
||||||
report_date = candidate
|
|
||||||
break
|
|
||||||
|
|
||||||
if report_date is None:
|
|
||||||
# 没有任何事件数据,仍然尝试生成昨天日报(至少展示管道统计)
|
|
||||||
report_date = (date.today() - timedelta(days=1)).strftime("%Y%m%d")
|
|
||||||
logger.warning(
|
|
||||||
"日报: 近 3 日均无事件数据 ({} ~ {}), 日报将只含管道统计",
|
|
||||||
(date.today() - timedelta(days=3)).strftime("%Y%m%d"),
|
|
||||||
(date.today() - timedelta(days=1)).strftime("%Y%m%d"),
|
|
||||||
)
|
|
||||||
elif report_date != (date.today() - timedelta(days=1)).strftime("%Y%m%d"):
|
|
||||||
logger.warning(
|
|
||||||
"日报: 昨天 ({}) 无事件数据, 回退使用 {}",
|
|
||||||
(date.today() - timedelta(days=1)).strftime("%Y%m%d"),
|
|
||||||
report_date,
|
|
||||||
)
|
|
||||||
|
|
||||||
path = generate_report(report_date, upload=True)
|
path = generate_report(report_date, upload=True)
|
||||||
elapsed = (datetime.now() - started).total_seconds()
|
elapsed = (datetime.now() - started).total_seconds()
|
||||||
|
|||||||
+70
-26
@@ -13,6 +13,7 @@ import json
|
|||||||
import os as _os
|
import os as _os
|
||||||
import re as _re
|
import re as _re
|
||||||
import subprocess
|
import subprocess
|
||||||
|
import time
|
||||||
from collections import Counter
|
from collections import Counter
|
||||||
from datetime import date, datetime, timedelta
|
from datetime import date, datetime, timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -38,6 +39,13 @@ NEWS_DAYS_BACK = 1 # 新闻回溯天数
|
|||||||
|
|
||||||
_MAX_HIGH_EVENTS = 20
|
_MAX_HIGH_EVENTS = 20
|
||||||
|
|
||||||
|
# LLM 摘要调用重试参数(环境变量可覆盖)
|
||||||
|
_LLM_RETRY_TIMES = int(_os.environ.get("LLM_RETRY_TIMES", "3"))
|
||||||
|
_LLM_RETRY_BACKOFF_SEC = float(_os.environ.get("LLM_RETRY_BACKOFF_SEC", "2.0"))
|
||||||
|
|
||||||
|
# 日报新闻回溯窗口(小时):07:00 生成当日日报时覆盖昨日全天至今晨的新闻
|
||||||
|
_NEWS_LOOKBACK_HOURS = 30
|
||||||
|
|
||||||
|
|
||||||
def _load_source_names() -> dict[str, str]:
|
def _load_source_names() -> dict[str, str]:
|
||||||
import yaml
|
import yaml
|
||||||
@@ -102,16 +110,31 @@ def _load_events_from_dir(day_str: str) -> list[dict]:
|
|||||||
def _collect_news_events(day_str: str) -> dict[str, Any]:
|
def _collect_news_events(day_str: str) -> dict[str, Any]:
|
||||||
"""收集新闻事件(排除 cninfo)。
|
"""收集新闻事件(排除 cninfo)。
|
||||||
|
|
||||||
事件已按日期目录组织(data/events/{day_str}/),
|
读取 `day_str` 与前一天两个事件目录,按 publish_time 过滤最近
|
||||||
不再用 datetime.now() 做 24h 二次过滤,
|
`_NEWS_LOOKBACK_HOURS`(默认 30)小时内的新闻——07:00 生成当日日报时
|
||||||
避免日报早上 8 点跑时前一天新闻被全部过滤掉。
|
可覆盖昨日全天至今晨的新闻。无 publish_time 的事件保留(容错)。
|
||||||
"""
|
"""
|
||||||
all_ev = _load_events_from_dir(day_str)
|
day = datetime.strptime(day_str, "%Y%m%d").date()
|
||||||
|
prev_day = (day - timedelta(days=1)).strftime("%Y%m%d")
|
||||||
|
all_ev = _load_events_from_dir(day_str) + _load_events_from_dir(prev_day)
|
||||||
|
|
||||||
|
# publish_time 过滤: 最近 30 小时(时间缺失/格式异常的事件保留)
|
||||||
|
cutoff = (datetime.now() - timedelta(hours=_NEWS_LOOKBACK_HOURS)).astimezone()
|
||||||
news_ev: list[dict] = []
|
news_ev: list[dict] = []
|
||||||
for e in all_ev:
|
for e in all_ev:
|
||||||
if e["source_id"] == "cninfo":
|
if e["source_id"] == "cninfo":
|
||||||
continue
|
continue
|
||||||
|
pt = e.get("publish_time")
|
||||||
|
if pt:
|
||||||
|
try:
|
||||||
|
# naive 时间假定为本地时区, 与带时区(aware)的 cutoff 统一比较
|
||||||
|
t = datetime.fromisoformat(pt)
|
||||||
|
if t.tzinfo is None:
|
||||||
|
t = t.astimezone()
|
||||||
|
if t < cutoff:
|
||||||
|
continue
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass # 时间格式异常时保留
|
||||||
news_ev.append(e)
|
news_ev.append(e)
|
||||||
|
|
||||||
sentiments: Counter = Counter()
|
sentiments: Counter = Counter()
|
||||||
@@ -311,16 +334,21 @@ def _score_xwlb_importance(title: str, content: str = "") -> int:
|
|||||||
def _collect_xwlb(day_str: str) -> dict[str, Any]:
|
def _collect_xwlb(day_str: str) -> dict[str, Any]:
|
||||||
"""收集新闻联播要闻(从 doorcome API /api/xwlbFine/ 获取)。
|
"""收集新闻联播要闻(从 doorcome API /api/xwlbFine/ 获取)。
|
||||||
|
|
||||||
|
《新闻联播》每天 19:00 播出:日报在早上生成时当日联播尚未播出,
|
||||||
|
因此固定取 `day_str` 前一日(最近一期已播出)的联播数据。
|
||||||
|
|
||||||
API 返回 AI 精编后的独立新闻条目(含标题+正文),
|
API 返回 AI 精编后的独立新闻条目(含标题+正文),
|
||||||
跳过第 1 条"内容提要"(仅为节目开场白)。
|
跳过第 1 条"内容提要"(仅为节目开场白)。
|
||||||
|
|
||||||
返回: {"items": [event_dict, ...], "date": "MM月DD日", "source_date": "20260622"}
|
返回: {"items": [event_dict, ...], "date": "MM月DD日", "source_date": "前一日"}
|
||||||
"""
|
"""
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
result: dict[str, Any] = {"items": [], "date": "", "source_date": day_str}
|
# 取前一晚(已播出)的联播:day_str 前一天
|
||||||
|
prev_day = (datetime.strptime(day_str, "%Y%m%d") - timedelta(days=1)).strftime("%Y%m%d")
|
||||||
|
result: dict[str, Any] = {"items": [], "date": "", "source_date": prev_day}
|
||||||
|
|
||||||
api_url = f"https://api.doorcome.cn/api/xwlbFine/?start_date={day_str}&end_date={day_str}"
|
api_url = f"https://api.doorcome.cn/api/xwlbFine/?start_date={prev_day}&end_date={prev_day}"
|
||||||
try:
|
try:
|
||||||
req = urllib.request.Request(api_url)
|
req = urllib.request.Request(api_url)
|
||||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||||
@@ -338,7 +366,7 @@ def _collect_xwlb(day_str: str) -> dict[str, Any]:
|
|||||||
if dates:
|
if dates:
|
||||||
d = min(dates)
|
d = min(dates)
|
||||||
result["date"] = f"{d[5:7]}月{d[8:10]}日"
|
result["date"] = f"{d[5:7]}月{d[8:10]}日"
|
||||||
result["source_date"] = day_str
|
result["source_date"] = prev_day
|
||||||
|
|
||||||
# 转换为事件格式,跳过第 1 条(内容提要/开场白)
|
# 转换为事件格式,跳过第 1 条(内容提要/开场白)
|
||||||
events: list[dict] = []
|
events: list[dict] = []
|
||||||
@@ -595,27 +623,43 @@ def _build_prompt(lines: list[str], day_str: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _llm_call(client, model: str, prompt: str, max_tokens: int = 1500) -> str:
|
def _llm_call(client, model: str, prompt: str, max_tokens: int = 1500) -> str:
|
||||||
"""单次 LLM 调用,返回 strip 后的文本。
|
"""单次 LLM 调用(带重试),返回 strip 后的文本。
|
||||||
|
|
||||||
|
失败按指数退避重试 `_LLM_RETRY_TIMES` 次(默认 3),全部失败则抛出最后一次异常。
|
||||||
若 finish_reason 为 'length' 则说明达到 max_tokens 上限被截断。
|
若 finish_reason 为 'length' 则说明达到 max_tokens 上限被截断。
|
||||||
"""
|
"""
|
||||||
resp = client.chat.completions.create(
|
last_exc: Exception | None = None
|
||||||
model=model,
|
for attempt in range(_LLM_RETRY_TIMES):
|
||||||
messages=[
|
try:
|
||||||
{"role": "system", "content": "你是 A 股日报撰写助手,输出简洁、有洞察的新闻摘要。"},
|
resp = client.chat.completions.create(
|
||||||
{"role": "user", "content": prompt},
|
model=model,
|
||||||
],
|
messages=[
|
||||||
temperature=0.3,
|
{"role": "system", "content": "你是 A 股日报撰写助手,输出简洁、有洞察的新闻摘要。"},
|
||||||
max_tokens=max_tokens,
|
{"role": "user", "content": prompt},
|
||||||
)
|
],
|
||||||
content = (resp.choices[0].message.content or "").strip()
|
temperature=0.3,
|
||||||
finish = getattr(resp.choices[0], "finish_reason", None)
|
max_tokens=max_tokens,
|
||||||
if finish == "length":
|
)
|
||||||
logger.warning(
|
content = (resp.choices[0].message.content or "").strip()
|
||||||
"AI 摘要可能被截断: max_tokens={} finish_reason=length 实际输出 {} 字符",
|
finish = getattr(resp.choices[0], "finish_reason", None)
|
||||||
max_tokens, len(content),
|
if finish == "length":
|
||||||
)
|
logger.warning(
|
||||||
return content
|
"AI 摘要可能被截断: max_tokens={} finish_reason=length 实际输出 {} 字符",
|
||||||
|
max_tokens, len(content),
|
||||||
|
)
|
||||||
|
return content
|
||||||
|
except Exception as e:
|
||||||
|
last_exc = e
|
||||||
|
if attempt < _LLM_RETRY_TIMES - 1:
|
||||||
|
wait = _LLM_RETRY_BACKOFF_SEC * (2 ** attempt)
|
||||||
|
logger.warning(
|
||||||
|
"AI 摘要 LLM 调用失败(第 {}/{} 次): {}; {} 秒后重试",
|
||||||
|
attempt + 1, _LLM_RETRY_TIMES, e, round(wait, 2),
|
||||||
|
)
|
||||||
|
time.sleep(wait)
|
||||||
|
logger.error("AI 摘要 LLM 调用重试 {} 次仍失败: {}", _LLM_RETRY_TIMES, last_exc)
|
||||||
|
assert last_exc is not None
|
||||||
|
raise last_exc
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
+15
-2
@@ -322,21 +322,24 @@ async def test_extract_event_async_retries(fake_config: LLMConfig) -> None:
|
|||||||
def test_load_llm_config_deepseek_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_load_llm_config_deepseek_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
monkeypatch.setenv("LLM_PROVIDER", "deepseek")
|
monkeypatch.setenv("LLM_PROVIDER", "deepseek")
|
||||||
monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-test-deepseek")
|
monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-test-deepseek")
|
||||||
|
monkeypatch.setenv("DEEPSEEK_MODEL", "deepseek-v4-flash")
|
||||||
monkeypatch.delenv("LLM_MODEL", raising=False)
|
monkeypatch.delenv("LLM_MODEL", raising=False)
|
||||||
cfg = load_llm_config()
|
cfg = load_llm_config()
|
||||||
assert cfg.provider == "deepseek"
|
assert cfg.provider == "deepseek"
|
||||||
assert cfg.api_key == "sk-test-deepseek"
|
assert cfg.api_key == "sk-test-deepseek"
|
||||||
assert cfg.model.startswith("deepseek")
|
assert cfg.model == "deepseek-v4-flash"
|
||||||
assert "deepseek" in cfg.base_url
|
assert "deepseek" in cfg.base_url
|
||||||
|
|
||||||
|
|
||||||
def test_load_llm_config_qwen_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_load_llm_config_qwen_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
monkeypatch.setenv("LLM_PROVIDER", "qwen")
|
monkeypatch.setenv("LLM_PROVIDER", "qwen")
|
||||||
monkeypatch.setenv("DASHSCOPE_API_KEY", "sk-test-qwen")
|
monkeypatch.setenv("DASHSCOPE_API_KEY", "sk-test-qwen")
|
||||||
|
monkeypatch.setenv("QWEN_MODEL", "qwen-plus")
|
||||||
monkeypatch.delenv("LLM_MODEL", raising=False)
|
monkeypatch.delenv("LLM_MODEL", raising=False)
|
||||||
cfg = load_llm_config()
|
cfg = load_llm_config()
|
||||||
assert cfg.provider == "qwen"
|
assert cfg.provider == "qwen"
|
||||||
assert cfg.api_key == "sk-test-qwen"
|
assert cfg.api_key == "sk-test-qwen"
|
||||||
|
assert cfg.model == "qwen-plus"
|
||||||
assert "dashscope" in cfg.base_url or "aliyuncs" in cfg.base_url
|
assert "dashscope" in cfg.base_url or "aliyuncs" in cfg.base_url
|
||||||
|
|
||||||
|
|
||||||
@@ -347,5 +350,15 @@ def test_load_llm_config_unknown_provider_raises(monkeypatch: pytest.MonkeyPatch
|
|||||||
|
|
||||||
def test_load_llm_config_missing_key_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_load_llm_config_missing_key_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False)
|
monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False)
|
||||||
with pytest.raises(ValueError):
|
monkeypatch.setenv("DEEPSEEK_MODEL", "deepseek-v4-flash")
|
||||||
|
with pytest.raises(ValueError, match="API key"):
|
||||||
|
load_llm_config(provider="deepseek")
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_llm_config_missing_model_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""去掉内置默认模型后:未显式配置模型必须报错(不再回退 deepseek-chat)。"""
|
||||||
|
monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-test")
|
||||||
|
monkeypatch.delenv("DEEPSEEK_MODEL", raising=False)
|
||||||
|
monkeypatch.delenv("LLM_MODEL", raising=False)
|
||||||
|
with pytest.raises(ValueError, match="模型"):
|
||||||
load_llm_config(provider="deepseek")
|
load_llm_config(provider="deepseek")
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ from __future__ import annotations
|
|||||||
import json
|
import json
|
||||||
from datetime import date
|
from datetime import date
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
from scheduler.reporter import _build_report_data
|
from scheduler.reporter import _build_report_data
|
||||||
|
|
||||||
|
|
||||||
@@ -85,3 +87,139 @@ class TestBuildReportData:
|
|||||||
r = _build_report_data(news, cninfo, {}, None, "20260710")
|
r = _build_report_data(news, cninfo, {}, None, "20260710")
|
||||||
assert r.events == []
|
assert r.events == []
|
||||||
assert r.ai_summary is None
|
assert r.ai_summary is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestLlmCallRetry:
|
||||||
|
"""_llm_call 重试逻辑(纯逻辑,mock client)。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _fake_client(failures: int):
|
||||||
|
"""构造 mock client:前 failures 次抛 ConnectionError,之后成功。"""
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
n = {"count": 0}
|
||||||
|
|
||||||
|
class Completions:
|
||||||
|
def create(self, **kwargs):
|
||||||
|
n["count"] += 1
|
||||||
|
if n["count"] <= failures:
|
||||||
|
raise ConnectionError("transient")
|
||||||
|
return SimpleNamespace(
|
||||||
|
choices=[SimpleNamespace(
|
||||||
|
message=SimpleNamespace(content="今日要点摘要"),
|
||||||
|
finish_reason="stop",
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
|
||||||
|
return SimpleNamespace(chat=SimpleNamespace(completions=Completions())), n
|
||||||
|
|
||||||
|
def test_success_first_try(self) -> None:
|
||||||
|
from scheduler.reporter import _llm_call
|
||||||
|
client, n = self._fake_client(0)
|
||||||
|
out = _llm_call(client, "deepseek-v4-flash", "p")
|
||||||
|
assert out == "今日要点摘要"
|
||||||
|
assert n["count"] == 1
|
||||||
|
|
||||||
|
def test_retry_then_success(self, monkeypatch) -> None:
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
monkeypatch.setattr(rep, "_LLM_RETRY_TIMES", 3)
|
||||||
|
monkeypatch.setattr(rep, "_LLM_RETRY_BACKOFF_SEC", 0.01)
|
||||||
|
client, n = self._fake_client(2) # 前 2 次失败,第 3 次成功
|
||||||
|
out = rep._llm_call(client, "deepseek-v4-flash", "p")
|
||||||
|
assert out == "今日要点摘要"
|
||||||
|
assert n["count"] == 3
|
||||||
|
|
||||||
|
def test_exhausts_retries_raises(self, monkeypatch) -> None:
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
monkeypatch.setattr(rep, "_LLM_RETRY_TIMES", 2)
|
||||||
|
monkeypatch.setattr(rep, "_LLM_RETRY_BACKOFF_SEC", 0.01)
|
||||||
|
client, n = self._fake_client(99) # 一直失败
|
||||||
|
with pytest.raises(ConnectionError):
|
||||||
|
rep._llm_call(client, "deepseek-v4-flash", "p")
|
||||||
|
assert n["count"] == 2 # 重试 2 次后放弃
|
||||||
|
|
||||||
|
|
||||||
|
class TestCollectXwlb:
|
||||||
|
"""_collect_xwlb 取数逻辑:应查询日报前一日(已播出的联播),并跳过内容提要。"""
|
||||||
|
|
||||||
|
def test_queries_previous_day_and_skips_toc(self, monkeypatch) -> None:
|
||||||
|
import json as _json
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
captured: dict[str, str] = {}
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout=15): # noqa: ARG001
|
||||||
|
captured["url"] = req.full_url
|
||||||
|
|
||||||
|
class Resp:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return _json.dumps({"data": {"news": [
|
||||||
|
{"daily_sub_id": 1, "news_title": "内容提要", "news_days": "2026-08-04", "news_improve": "开场白"},
|
||||||
|
{"daily_sub_id": 2, "news_title": "联播要闻A", "news_days": "2026-08-04", "news_improve": "正文A"},
|
||||||
|
]}}).encode("utf-8")
|
||||||
|
|
||||||
|
return Resp()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||||
|
from scheduler.reporter import _collect_xwlb
|
||||||
|
|
||||||
|
result = _collect_xwlb("20260805")
|
||||||
|
# 查询的是前一日(20260804)而非当日
|
||||||
|
assert "start_date=20260804" in captured["url"]
|
||||||
|
assert "end_date=20260804" in captured["url"]
|
||||||
|
# 跳过第 1 条内容提要
|
||||||
|
assert len(result["items"]) == 1
|
||||||
|
assert result["items"][0]["title"] == "联播要闻A"
|
||||||
|
assert result["source_date"] == "20260804"
|
||||||
|
assert result["date"] == "08月04日"
|
||||||
|
|
||||||
|
|
||||||
|
class TestCollectNewsEventsLookback:
|
||||||
|
"""_collect_news_events 30 小时回溯逻辑。"""
|
||||||
|
|
||||||
|
def test_filters_30h_and_excludes_cninfo(self, monkeypatch) -> None:
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
|
||||||
|
now = datetime.now().astimezone()
|
||||||
|
|
||||||
|
def fake_load(day_str: str) -> list[dict]: # noqa: ARG001
|
||||||
|
def ev(title: str, hours_ago: float | None, source: str = "cls",
|
||||||
|
importance: int = 5, aware: bool = False) -> dict:
|
||||||
|
pt = None
|
||||||
|
if hours_ago is not None:
|
||||||
|
t = now - timedelta(hours=hours_ago)
|
||||||
|
pt = t.isoformat() if not aware else t.astimezone().isoformat()
|
||||||
|
return {
|
||||||
|
"title": title, "url": "u", "source_id": source,
|
||||||
|
"publish_time": pt,
|
||||||
|
"event": {"importance": importance, "sentiment": "neutral",
|
||||||
|
"event_type": "其他", "summary": "s"},
|
||||||
|
}
|
||||||
|
|
||||||
|
return [
|
||||||
|
ev("窗口内新闻", 10),
|
||||||
|
ev("窗口内新闻带时区", 12, aware=True),
|
||||||
|
ev("窗口外旧闻", 40),
|
||||||
|
ev("无时间戳", None),
|
||||||
|
ev("公告排除", 5, source="cninfo", importance=2),
|
||||||
|
]
|
||||||
|
|
||||||
|
monkeypatch.setattr(rep, "_load_events_from_dir", fake_load)
|
||||||
|
result = rep._collect_news_events("20260805")
|
||||||
|
|
||||||
|
# 两个日期目录各返回 5 条(共 10): 旧闻×2、公告×2 被滤, 保留 6 条
|
||||||
|
assert result["total"] == 6
|
||||||
|
titles = {e["title"] for e in result["high"]}
|
||||||
|
assert "窗口内新闻" in titles
|
||||||
|
assert "窗口内新闻带时区" in titles
|
||||||
|
assert "无时间戳" in titles
|
||||||
|
assert "窗口外旧闻" not in titles
|
||||||
|
assert "公告排除" not in titles
|
||||||
|
|||||||
Reference in New Issue
Block a user