feat: Token Plan 迁移与 .env 热加载,并修复日报 AI 摘要为空
Token Plan 迁移 / 配置热加载:
- configs/llm_models.yaml: 各场景切到 Token Plan(deepseek-v4.1-flash / qwen3.6-flash)
- 新增 configs/runtime_env.py: .env 按 (mtime_ns, size) 热加载并同步 os.environ,
统一 env_get 取值;llm / embedding / vectorstore / mcp / pipeline 改用 env_get
- configs/loader.py / scripts/run_scheduler.py 等配套调整
- 新增 tests/test_hot_reload.py
日报 AI 摘要为空修复(2026-09-25):
- 根因: 推理模型的 reasoning token 与正文共用 max_tokens, 预算 1500 被"思考"
占满 -> text_tokens=0 / finish_reason=length, 摘要静默为空且不重试
- daily_report 场景新增 max_tokens(默认 4000, YAML 保存即热生效);
LLMConfig 支持可选 max_tokens; 分块预算 800 -> 2000
- _llm_call 拆出 _call_once, 正文为空时自动加倍预算重试(上限 16000),
用尽才降级返回空串; 网络异常重试语义不变
- docs/user-guide.md 新增 FAQ; continuation.md 记录本次排查
- 已重跑 2026-09-25 日报(report_id=357)补回 466 字摘要
测试: 相关用例 56 passed(test_hot_reload 12 passed);
ruff 无新增问题; 3 个 crawler 既有失败与本改动无关
This commit is contained in:
@@ -41,6 +41,11 @@ reasonix.toml
|
|||||||
.env
|
.env
|
||||||
.env.local
|
.env.local
|
||||||
.env.*.local
|
.env.*.local
|
||||||
|
# 迁移/切换服务商时留下的 .env 备份(含密钥,禁止入库)
|
||||||
|
.env*
|
||||||
|
!.env.example
|
||||||
|
*.bak
|
||||||
|
*.bak*
|
||||||
*.key
|
*.key
|
||||||
*.pem
|
*.pem
|
||||||
|
|
||||||
|
|||||||
+19
-15
@@ -18,7 +18,8 @@
|
|||||||
# LLM_MODEL);若全部缺失则直接报错,绝不静默使用内置默认模型。
|
# LLM_MODEL);若全部缺失则直接报错,绝不静默使用内置默认模型。
|
||||||
# · api_key_env / base_url_env 为可选字段,填写存放 API Key / 服务地址的
|
# · api_key_env / base_url_env 为可选字段,填写存放 API Key / 服务地址的
|
||||||
# 环境变量名;API Key 一律放 .env,禁止写入本文件(安全规范)。
|
# 环境变量名;API Key 一律放 .env,禁止写入本文件(安全规范)。
|
||||||
# · 修改后无需重启常驻服务即可生效(每次调用重新读取;如需热更新缓存可重启)。
|
# · 修改后无需重启常驻服务即可生效:configs/loader.py 以 (mtime, size) 失效缓存,
|
||||||
|
# 保存后下一次调用即读到新值;.env 的改动由 configs/runtime_env.py 在约 2s 内热更新。
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
# ---- 全局默认参数(各场景可覆盖;低于 .env,高于代码内置默认)----
|
# ---- 全局默认参数(各场景可覆盖;低于 .env,高于代码内置默认)----
|
||||||
@@ -49,9 +50,9 @@ scenes:
|
|||||||
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus / qwen-max
|
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus / qwen-max
|
||||||
event_extraction:
|
event_extraction:
|
||||||
provider: qwen # 建议 deepseek | qwen;留空则回退 .env 的 LLM_PROVIDER
|
provider: qwen # 建议 deepseek | qwen;留空则回退 .env 的 LLM_PROVIDER
|
||||||
model: qwen3.7-flash # 留空则回退 .env(DEEPSEEK_MODEL → LLM_MODEL)
|
model: qwen3.6-flash # Token Plan 模型;注意 Token Plan 无 qwen3.7-flash
|
||||||
api_key_env: DASHSCOPE_API_KEY # 例如: DEEPSEEK_API_KEY / QWEN_API_KEY / DASHSCOPE_API_KEY
|
api_key_env: QWEN_API_KEY # Token Plan 计费账号(sk-sp-…);勿用 DASHSCOPE_API_KEY
|
||||||
base_url_env: QWEN_BASE_URL # 例如: DEEPSEEK_BASE_URL / QWEN_BASE_URL
|
base_url_env: QWEN_BASE_URL # .env 指向 token-plan.*.maas.aliyuncs.com
|
||||||
temperature: 0.1
|
temperature: 0.1
|
||||||
timeout_sec: 60
|
timeout_sec: 60
|
||||||
max_attempts: 3 # 单篇解析失败的最大重试次数
|
max_attempts: 3 # 单篇解析失败的最大重试次数
|
||||||
@@ -66,20 +67,23 @@ scenes:
|
|||||||
# 使用方式:无需手动触发,定时任务自动执行;失败自动降级(日报留空,不影响入库)。
|
# 使用方式:无需手动触发,定时任务自动执行;失败自动降级(日报留空,不影响入库)。
|
||||||
# 对模型的要求:
|
# 对模型的要求:
|
||||||
# · OpenAI 兼容 chat 接口(不需要 JSON 输出);
|
# · OpenAI 兼容 chat 接口(不需要 JSON 输出);
|
||||||
# · 输出长度 ≥ 1500 tokens(max_tokens=1500,输出超长会被截断并记 WARNING);
|
# · 输出长度 ≥ max_tokens 配置值(见下,输出超长会被截断并记 WARNING);
|
||||||
# · 中文摘要能力强、要点化输出稳定(每条一行,以 "- " 开头);
|
# · 中文摘要能力强、要点化输出稳定(每条一行,以 "- " 开头);
|
||||||
# · 上下文窗口 ≥ 8K tokens(素材按 3000 字符/块分块,多块先分段再合并);
|
# · 上下文窗口 ≥ 8K tokens(素材按 3000 字符/块分块,多块先分段再合并);
|
||||||
# · temperature 0.3 左右,兼顾稳定与表达;网络失败按指数退避重试 3 次。
|
# · temperature 0.3 左右,兼顾稳定与表达;网络失败按指数退避重试 3 次。
|
||||||
# · 输出长度需求:分段摘要约 800 tokens、合并摘要约 1500 tokens(代码内置,
|
# · max_tokens 说明:推理模型(deepseek-v4.1-flash 等)的 reasoning token 与
|
||||||
# 不在本文件配置),模型应能稳定输出 1500+ tokens 的中文要点。
|
# 正文共用该预算;预算过小时"思考"会占满配额导致正文为空
|
||||||
|
# (finish_reason=length、0 字符,日报因此没有 AI 摘要)。代码兜底见
|
||||||
|
# scheduler/reporter.py: 正文为空时自动加倍预算重试(最多 2 次,上限 16000)。
|
||||||
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus
|
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus
|
||||||
daily_report:
|
daily_report:
|
||||||
provider: # 建议 deepseek | qwen;留空则回退 .env 的 LLM_PROVIDER
|
provider: qwen # Token Plan 计费账号
|
||||||
model:
|
model: deepseek-v4.1-flash
|
||||||
api_key_env:
|
api_key_env: QWEN_API_KEY
|
||||||
base_url_env:
|
base_url_env: QWEN_BASE_URL
|
||||||
temperature: 0.3
|
temperature: 0.3
|
||||||
timeout_sec: 60
|
timeout_sec: 60
|
||||||
|
max_tokens: 4000 # 单块/合并摘要输出预算(需为 reasoning token 预留余量)
|
||||||
|
|
||||||
# ------------------------------------------------------------------------- #
|
# ------------------------------------------------------------------------- #
|
||||||
# 场景 3: 个股 AI 要点分析
|
# 场景 3: 个股 AI 要点分析
|
||||||
@@ -97,10 +101,10 @@ scenes:
|
|||||||
# · 输出长度需求:约 500 tokens(代码内置,不在本文件配置)。
|
# · 输出长度需求:约 500 tokens(代码内置,不在本文件配置)。
|
||||||
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus
|
# 建议模型: deepseek-v4-flash(生产实测) / deepseek-chat / qwen-plus
|
||||||
stock_report:
|
stock_report:
|
||||||
provider: # 建议 deepseek | qwen;留空则回退 .env 的 LLM_PROVIDER
|
provider: qwen # Token Plan(个股日报当前禁用,配置好以防将来启用时漏计费)
|
||||||
model:
|
model: qwen3.6-flash
|
||||||
api_key_env:
|
api_key_env: QWEN_API_KEY
|
||||||
base_url_env:
|
base_url_env: QWEN_BASE_URL
|
||||||
temperature: 0.3
|
temperature: 0.3
|
||||||
timeout_sec: 60
|
timeout_sec: 60
|
||||||
|
|
||||||
|
|||||||
+59
-16
@@ -8,33 +8,75 @@
|
|||||||
3. 环境变量 / .env(LLM_PROVIDER、DEEPSEEK_MODEL 等,向后兼容)
|
3. 环境变量 / .env(LLM_PROVIDER、DEEPSEEK_MODEL 等,向后兼容)
|
||||||
4. 代码内置默认值
|
4. 代码内置默认值
|
||||||
|
|
||||||
|
热加载: 缓存以 ``(mtime_ns, size)`` 为准 —— 改完 YAML 保存后,下一次读取即生效,
|
||||||
|
常驻进程(调度器 / MCP server)无需重启。
|
||||||
|
|
||||||
说明:API Key 一律放 .env,本文件只保存环境变量名(api_key_env),禁止写密钥。
|
说明:API Key 一律放 .env,本文件只保存环境变量名(api_key_env),禁止写密钥。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from functools import lru_cache
|
import os
|
||||||
|
import threading
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
DEFAULT_CONFIG_PATH = Path("configs/llm_models.yaml")
|
#: 指定替代的模型配置文件路径(测试 / 多环境部署用)
|
||||||
|
MODELS_CONFIG_OVERRIDE = "A_SHARE_MODELS_CONFIG"
|
||||||
|
|
||||||
|
#: 默认模型配置文件(绝对路径,不依赖当前工作目录)
|
||||||
|
DEFAULT_CONFIG_PATH = Path(__file__).resolve().parents[1] / "configs" / "llm_models.yaml"
|
||||||
|
|
||||||
|
_lock = threading.Lock()
|
||||||
|
_cache: dict[Path, tuple[tuple[int, int] | None, dict]] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def config_path() -> Path:
|
||||||
|
"""返回当前使用的 ``llm_models.yaml`` 路径。"""
|
||||||
|
override = os.environ.get(MODELS_CONFIG_OVERRIDE)
|
||||||
|
if override:
|
||||||
|
return Path(override).expanduser()
|
||||||
|
return DEFAULT_CONFIG_PATH
|
||||||
|
|
||||||
|
|
||||||
|
def _signature(path: Path) -> tuple[int, int] | None:
|
||||||
|
"""返回 ``(mtime_ns, size)``;文件不存在时返回 None。"""
|
||||||
|
try:
|
||||||
|
st = path.stat()
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
return (st.st_mtime_ns, st.st_size)
|
||||||
|
|
||||||
|
|
||||||
@lru_cache(maxsize=8)
|
|
||||||
def _load_yaml(path: Path) -> dict:
|
def _load_yaml(path: Path) -> dict:
|
||||||
"""读取 YAML 文件为 dict;文件缺失或解析失败返回空 dict(走兜底配置)。"""
|
"""读取 YAML 为 dict;文件缺失或解析失败返回空 dict(走兜底配置)。
|
||||||
|
|
||||||
|
按 ``(mtime_ns, size)`` 失效缓存:文件一旦变化,下次调用即重新解析。
|
||||||
|
"""
|
||||||
|
sig = _signature(path)
|
||||||
|
|
||||||
|
with _lock:
|
||||||
|
cached = _cache.get(path)
|
||||||
|
if cached is not None and cached[0] == sig:
|
||||||
|
return cached[1]
|
||||||
|
|
||||||
if not path.is_file():
|
if not path.is_file():
|
||||||
logger.debug("配置文件不存在,使用内置/环境变量兜底: {}", path)
|
logger.debug("配置文件不存在,使用内置/环境变量兜底: {}", path)
|
||||||
return {}
|
data: dict = {}
|
||||||
try:
|
else:
|
||||||
import yaml
|
try:
|
||||||
|
import yaml
|
||||||
|
|
||||||
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
||||||
except Exception as e: # noqa: BLE001 - YAML 语法错误等
|
data = raw if isinstance(raw, dict) else {}
|
||||||
logger.error("解析 {} 失败: {}", path, e)
|
except Exception as e: # noqa: BLE001 - YAML 语法错误等
|
||||||
return {}
|
logger.error("解析 {} 失败: {}", path, e)
|
||||||
return data if isinstance(data, dict) else {}
|
data = {}
|
||||||
|
|
||||||
|
with _lock:
|
||||||
|
_cache[path] = (sig, data)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
def load_scene_config(scene: str) -> dict:
|
def load_scene_config(scene: str) -> dict:
|
||||||
@@ -45,7 +87,7 @@ def load_scene_config(scene: str) -> dict:
|
|||||||
"""
|
"""
|
||||||
if not scene:
|
if not scene:
|
||||||
return {}
|
return {}
|
||||||
data = _load_yaml(DEFAULT_CONFIG_PATH)
|
data = _load_yaml(config_path())
|
||||||
scenes = data.get("scenes") or {}
|
scenes = data.get("scenes") or {}
|
||||||
cfg = scenes.get(scene)
|
cfg = scenes.get(scene)
|
||||||
if cfg is None:
|
if cfg is None:
|
||||||
@@ -59,11 +101,12 @@ def load_scene_config(scene: str) -> dict:
|
|||||||
|
|
||||||
def load_defaults() -> dict:
|
def load_defaults() -> dict:
|
||||||
"""读取 llm_models.yaml 顶层 defaults(全局默认参数)。"""
|
"""读取 llm_models.yaml 顶层 defaults(全局默认参数)。"""
|
||||||
data = _load_yaml(DEFAULT_CONFIG_PATH)
|
data = _load_yaml(config_path())
|
||||||
d = data.get("defaults") or {}
|
d = data.get("defaults") or {}
|
||||||
return d if isinstance(d, dict) else {}
|
return d if isinstance(d, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
def clear_cache() -> None:
|
def clear_cache() -> None:
|
||||||
"""清空 YAML 缓存(测试或热更新配置时使用)。"""
|
"""清空 YAML 缓存(测试或强制重载时使用;正常热加载无需调用)。"""
|
||||||
_load_yaml.cache_clear()
|
with _lock:
|
||||||
|
_cache.clear()
|
||||||
|
|||||||
@@ -0,0 +1,188 @@
|
|||||||
|
"""运行期配置热加载:改 ``.env`` 后无需重启进程即生效。
|
||||||
|
|
||||||
|
为什么需要它
|
||||||
|
------------
|
||||||
|
常驻进程(``scripts/run_scheduler.py``、``mcp_server``)启动时把 ``.env`` 读进
|
||||||
|
``os.environ``,之后再改 ``.env`` 不会生效——子进程虽然会 ``load_dotenv()``,但
|
||||||
|
它继承的是父进程那份旧环境,而 python-dotenv 默认不覆盖已存在的键,于是
|
||||||
|
"改了配置却没反应"。
|
||||||
|
|
||||||
|
做法
|
||||||
|
----
|
||||||
|
- :func:`ensure_env_loaded`:先比对 ``.env`` 的 ``(mtime_ns, size)``。文件没变时
|
||||||
|
只做一次 ``stat``;变了才重新解析并同步到 ``os.environ``。
|
||||||
|
- 删除语义:上一轮由 ``.env`` 带入、这一轮已从文件里删掉的键会被清除,
|
||||||
|
保证"文件即事实源",而不是只能加不能减。
|
||||||
|
- :func:`env_get`:先热加载再读取,供各模块统一取配置(空字符串视为未设置)。
|
||||||
|
- :func:`start_env_watcher`:守护线程周期性刷新,照顾那些仍直接读
|
||||||
|
``os.environ`` 的历史代码路径。
|
||||||
|
|
||||||
|
优先级(从高到低)
|
||||||
|
------------------
|
||||||
|
显式参数 / CLI > configs/llm_models.yaml 场景 > 进程环境(shell / systemd)
|
||||||
|
> .env 文件 > 内置默认值
|
||||||
|
|
||||||
|
其中「进程环境」与「.env」的关系是:
|
||||||
|
|
||||||
|
- 进程环境里**显式设置且与文件不同**的键优先,热加载不会覆盖它
|
||||||
|
(例如 ``LLM_PROVIDER=qwen python -m a_share_cli`` 这种一次性覆盖);
|
||||||
|
- 其余键由本模块托管,跟随 ``.env`` 文件变化即时更新;
|
||||||
|
- 从 ``.env`` 里删掉的托管键,会同步从进程环境移除。
|
||||||
|
|
||||||
|
测试或部署可用 ``A_SHARE_ENV_FILE`` 指定其它 ``.env`` 路径。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from dotenv import dotenv_values
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
#: 指定替代的 .env 路径(测试 / 多环境部署用)
|
||||||
|
ENV_FILE_OVERRIDE = "A_SHARE_ENV_FILE"
|
||||||
|
|
||||||
|
#: watcher 轮询间隔(秒)
|
||||||
|
WATCH_INTERVAL_SEC = 2.0
|
||||||
|
|
||||||
|
_lock = threading.Lock()
|
||||||
|
_signature: tuple[int, int] | None = None
|
||||||
|
#: 由本模块写入 os.environ 的键 -> 写入值;用于识别"外部显式覆盖"
|
||||||
|
_managed: dict[str, str] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def dotenv_path() -> Path:
|
||||||
|
"""返回当前使用的 ``.env`` 路径。"""
|
||||||
|
override = os.environ.get(ENV_FILE_OVERRIDE)
|
||||||
|
if override:
|
||||||
|
return Path(override).expanduser()
|
||||||
|
return Path(__file__).resolve().parents[1] / ".env"
|
||||||
|
|
||||||
|
|
||||||
|
def file_signature(path: Path) -> tuple[int, int] | None:
|
||||||
|
"""返回 ``(mtime_ns, size)``;文件不存在时返回 None。"""
|
||||||
|
try:
|
||||||
|
st = path.stat()
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
return (st.st_mtime_ns, st.st_size)
|
||||||
|
|
||||||
|
|
||||||
|
def _release_all() -> None:
|
||||||
|
"""撤下所有仍由本模块托管的键(.env 消失时)。"""
|
||||||
|
for key, managed_value in list(_managed.items()):
|
||||||
|
if os.environ.get(key) == managed_value:
|
||||||
|
os.environ.pop(key, None)
|
||||||
|
del _managed[key]
|
||||||
|
|
||||||
|
|
||||||
|
def _apply(values: dict[str, str]) -> None:
|
||||||
|
"""把文件值同步到 ``os.environ``,尊重外部显式覆盖。"""
|
||||||
|
# 1) 已从文件移除的托管键 → 同步删除
|
||||||
|
for key in list(_managed):
|
||||||
|
if key in values:
|
||||||
|
continue
|
||||||
|
if os.environ.get(key) == _managed[key]:
|
||||||
|
os.environ.pop(key, None)
|
||||||
|
del _managed[key]
|
||||||
|
|
||||||
|
# 2) 应用文件中的键
|
||||||
|
for key, value in values.items():
|
||||||
|
current = os.environ.get(key)
|
||||||
|
if current is None or current == value:
|
||||||
|
# 未设置,或与文件一致 → 交给文件托管(后续可热更新)
|
||||||
|
os.environ[key] = value
|
||||||
|
_managed[key] = value
|
||||||
|
elif _managed.get(key) == current:
|
||||||
|
# 当前值正是本模块上一轮写入的 → 跟随文件热更新
|
||||||
|
os.environ[key] = value
|
||||||
|
_managed[key] = value
|
||||||
|
else:
|
||||||
|
# 进程环境里显式设置且与文件不同 → 外部优先,不接管
|
||||||
|
_managed.pop(key, None)
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_env_loaded(force: bool = False) -> bool:
|
||||||
|
"""确保 ``os.environ`` 与 ``.env`` 文件一致。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
force: 忽略签名缓存,强制重新解析(首次加载 / 测试用)。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
本次是否真的重新加载了文件。
|
||||||
|
"""
|
||||||
|
global _signature
|
||||||
|
|
||||||
|
path = dotenv_path()
|
||||||
|
sig = file_signature(path)
|
||||||
|
|
||||||
|
with _lock:
|
||||||
|
if not force and sig == _signature:
|
||||||
|
return False
|
||||||
|
|
||||||
|
if sig is None:
|
||||||
|
removed = len(_managed)
|
||||||
|
_release_all()
|
||||||
|
_signature = None
|
||||||
|
if removed:
|
||||||
|
logger.warning("{} 不可读,已回退 {} 个环境变量", path, removed)
|
||||||
|
return True
|
||||||
|
|
||||||
|
values = {k: v for k, v in dotenv_values(path).items() if v is not None}
|
||||||
|
_apply(values)
|
||||||
|
_signature = sig
|
||||||
|
|
||||||
|
logger.debug("已加载/热更新 {}({} 项)", path, len(values))
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def env_get(key: str, default: str | None = None) -> str | None:
|
||||||
|
"""读取配置项(读取前自动热加载 ``.env``);空字符串视为未设置。"""
|
||||||
|
ensure_env_loaded()
|
||||||
|
value = os.environ.get(key)
|
||||||
|
if value is None or value.strip() == "":
|
||||||
|
return default
|
||||||
|
return value.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def env_raw(key: str, default: str | None = None) -> str | None:
|
||||||
|
"""读取配置项原始值(读取前自动热加载 ``.env``)。
|
||||||
|
|
||||||
|
与 :func:`env_get` 的区别:**不把空字符串当作未设置**。
|
||||||
|
用于"显式留空表示禁用"这类开关,例如 ``STOCK_REPORT_TIME=``。
|
||||||
|
"""
|
||||||
|
ensure_env_loaded()
|
||||||
|
value = os.environ.get(key)
|
||||||
|
if value is None:
|
||||||
|
return default
|
||||||
|
return value.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def start_env_watcher(interval: float = WATCH_INTERVAL_SEC) -> threading.Thread:
|
||||||
|
"""启动守护线程:周期性检查 ``.env``,变了就热更新 ``os.environ``。
|
||||||
|
|
||||||
|
只对**常驻进程**有意义;短命的一次性脚本按需读取即可。
|
||||||
|
"""
|
||||||
|
def _loop() -> None:
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
ensure_env_loaded()
|
||||||
|
except Exception: # noqa: BLE001 - 热加载失败不应拖垮主进程
|
||||||
|
logger.exception("热加载 .env 失败")
|
||||||
|
time.sleep(interval)
|
||||||
|
|
||||||
|
thread = threading.Thread(target=_loop, name="env-watcher", daemon=True)
|
||||||
|
thread.start()
|
||||||
|
logger.info("已启动 .env 热加载监听(每 {:.0f}s 检查一次)", interval)
|
||||||
|
return thread
|
||||||
|
|
||||||
|
|
||||||
|
def reset_cache() -> None:
|
||||||
|
"""仅供测试:撤下托管键并清空签名缓存。"""
|
||||||
|
global _signature
|
||||||
|
with _lock:
|
||||||
|
_release_all()
|
||||||
|
_signature = None
|
||||||
@@ -4,6 +4,44 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 本次完成 (2026-09-25) — 日报 AI 摘要为空修复(推理模型 reasoning 占满 max_tokens)
|
||||||
|
|
||||||
|
**现象**:用户反馈 2026-09-25 日报没有 AI 摘要。`news_report` 中 `id=357`(2026-09-25, finance) `ai_summary` 为 `NULL`;当天 07:13:55 日志:
|
||||||
|
|
||||||
|
```
|
||||||
|
WARNING | scheduler.reporter:_llm_call - AI 摘要可能被截断: max_tokens=1500 finish_reason=length 实际输出 0 字符
|
||||||
|
```
|
||||||
|
|
||||||
|
**根因(证据链闭合)**:
|
||||||
|
- 日报场景(daily_report)调用 `deepseek-v4.1-flash`(Token Plan),这是**推理模型**:`reasoning_content` 的 token 与正文**共用** `max_tokens` 预算
|
||||||
|
- 用当天真实素材原样复现:**`reasoning_tokens=1500` / `text_tokens=0` / `content=0 字符` / `finish_reason=length`** —— 预算被"思考"全部吃掉,正文为空
|
||||||
|
- 同素材把预算提到 4000:`finish_reason=stop`、reasoning 938 + text 337、摘要 523 字 ✓
|
||||||
|
- 代码缺陷:`scheduler/reporter.py:_llm_call` 只取 `message.content`,空内容**不抛异常** → `_generate_ai_summary` 返回 `""` → `ai_summary=None` 入库,pipeline 仍标 report ✅(**静默失败、无重试**)
|
||||||
|
- **非本次 Token Plan 迁移引入**:历史同为空的还有 9-12 / 9-13 / 9-15 / 9-18,当时用的是 `deepseek-v4-flash`(provider=deepseek),同样是推理类模型 → 长期间歇性缺陷
|
||||||
|
|
||||||
|
**修复(方案 B:配置化 + 代码兜底)**:
|
||||||
|
1. `configs/llm_models.yaml`:`daily_report` 新增 `max_tokens: 4000`(注释说明 reasoning 共用预算);YAML 保存即热生效
|
||||||
|
2. `llm/client.py`:`LLMConfig` 新增可选字段 `max_tokens`;`load_llm_config` 用新增的 `_pick_optional_int` 读取场景配置(未配置 = `None`,调用方回退内置默认)
|
||||||
|
3. `scheduler/reporter.py`:
|
||||||
|
- 新增常量 `DEFAULT_SUMMARY_MAX_TOKENS=4000` / `DEFAULT_SUMMARY_CHUNK_MAX_TOKENS=2000` / `MAX_SUMMARY_MAX_TOKENS=16000` / `_MAX_BUDGET_ESCALATIONS=2`
|
||||||
|
- 新增 `_summary_max_tokens()`(场景配置 > 内置默认)、`_chunk_max_tokens()`(不超过合并预算)
|
||||||
|
- `_llm_call` 拆出 `_call_once`;保持网络异常指数退避重试语义不变;新增**空正文 + finish_reason=length 时自动加倍预算重试**(上限 16000),用尽后返回空串降级(不抛异常)
|
||||||
|
- 分块预算 800 → 2000;合并预算 1500 → 配置值(4000)
|
||||||
|
|
||||||
|
**验证**:
|
||||||
|
- 线上复现 → 修复后回归:`uv run a-share report --date 20260925` 重跑,**无截断告警**,`report_id=357` 原地更新(幂等 upsert),`ai_summary` 466 字 ✓
|
||||||
|
- 测试:新增 `TestReasoningBudgetEscalation`(升级恢复 / 场景值优先 / 用尽降级返回空 / 上限)+ `test_llm.py` 的 `max_tokens` 场景配置、`_pick_optional_int`、真实 YAML 预算 ≥4000 回归保护 → `tests/test_report_builder.py tests/test_llm.py` **56 passed**
|
||||||
|
- `ruff` 改动文件无新增问题(5 条 N806/SIM115 为 reporter.py 既有);`mypy` 仅剩 `llm/client.py:86` 既有告警
|
||||||
|
|
||||||
|
**运维动作**:`report` 步骤由调度器**进程内**执行,已 `sudo systemctl restart a-share-research`(10:55,重启后任务同步正常),明早 07:00 起生效。
|
||||||
|
|
||||||
|
**遗留与后续**:
|
||||||
|
- `scheduler/stock_reporter.py:285` 个股日报 `max_tokens=500`,配置模型 `qwen3.6-flash` 亦属推理类,**同类隐患**(当前个股日报禁用);启用前建议一并按本方案处理
|
||||||
|
- 空摘要目前只降级为"无摘要",未做告警;可考虑连续 N 天为空时推送通知
|
||||||
|
- 本次改动尚未 git commit(工作区还混有 9-22 Token Plan 迁移的未提交改动,避免混提)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 本次完成 (2026-09-10) — cninfo 抓取压穿内存导致整机冻结的修复
|
## 本次完成 (2026-09-10) — cninfo 抓取压穿内存导致整机冻结的修复
|
||||||
|
|
||||||
**现象**:2026-09-06 / 09-08 / 09-10 连续三次早上 06:0x 整机冻结,看门狗(硬件 2min)硬复位。
|
**现象**:2026-09-06 / 09-08 / 09-10 连续三次早上 06:0x 整机冻结,看门狗(硬件 2min)硬复位。
|
||||||
|
|||||||
@@ -543,9 +543,15 @@ ReportData (Pydantic)
|
|||||||
| 文件 | 格式 | 用途 | 热更新 |
|
| 文件 | 格式 | 用途 | 热更新 |
|
||||||
|------|------|------|--------|
|
|------|------|------|--------|
|
||||||
| `configs/sources.yaml` | YAML | 14 个新闻源配置 | 每次抓取重读 |
|
| `configs/sources.yaml` | YAML | 14 个新闻源配置 | 每次抓取重读 |
|
||||||
| `configs/llm_models.yaml` | YAML | 4 个 LLM 场景配置 | 每次调用重读 |
|
| `configs/llm_models.yaml` | YAML | 4 个 LLM 场景配置 | 每次调用重读(mtime 缓存失效) |
|
||||||
| `configs/watchlist.yaml` | YAML | cninfo 公告关注列表 | 每次操作重读 |
|
| `configs/watchlist.yaml` | YAML | cninfo 公告关注列表 | 每次操作重读 |
|
||||||
| `.env` | dotenv | API Key + 调度/超时/DB 配置 | 重启服务生效 |
|
| `.env` | dotenv | API Key + 调度/超时/DB 配置 | ~2s 内自动生效(常驻进程无需重启) |
|
||||||
|
|
||||||
|
> 热加载实现见 `configs/runtime_env.py`:常驻进程(调度器 / MCP server)启动后
|
||||||
|
> 会以 2s 周期比对 `.env` 的 `(mtime, size)`,变化即写入 `os.environ`;
|
||||||
|
> `configs/loader.py` 对 YAML 做同样的 mtime 失效。
|
||||||
|
> 进程环境里**显式设置且与文件不同**的变量优先(如 `LLM_PROVIDER=qwen ...`),
|
||||||
|
> `.env` 中删除的键也会同步从环境中移除。
|
||||||
|
|
||||||
### 5.2 配置优先级(LLM 场景)
|
### 5.2 配置优先级(LLM 场景)
|
||||||
|
|
||||||
|
|||||||
+21
-3
@@ -507,17 +507,25 @@ tail -f logs/scheduler.log # 文件日志
|
|||||||
| 18:00 | crawler→xwlb→extractor→dedup→llm→embedding→qdrant |
|
| 18:00 | crawler→xwlb→extractor→dedup→llm→embedding→qdrant |
|
||||||
| 22:00 | crawler→xwlb→extractor→dedup→llm→embedding→qdrant |
|
| 22:00 | crawler→xwlb→extractor→dedup→llm→embedding→qdrant |
|
||||||
|
|
||||||
### 7.4 修改调度时间
|
### 7.4 修改调度时间(无需重启)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 编辑 .env 中的 SCHEDULE_TIMES,格式: HH:MM,HH:MM,...
|
# 编辑 .env 中的 SCHEDULE_TIMES,格式: HH:MM,HH:MM,...
|
||||||
nano /home/pi/news/.env
|
nano /home/pi/news/.env
|
||||||
# 例: SCHEDULE_TIMES=08:00,14:00,20:00
|
# 例: SCHEDULE_TIMES=08:00,14:00,20:00
|
||||||
|
|
||||||
# 重启生效
|
# 保存即生效:常驻调度器每 30s 比对一次,自动重新注册定时任务
|
||||||
sudo systemctl restart a-share-research
|
# 日志确认:grep "定时任务已同步" /home/pi/news/logs/scheduler.log
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **配置热加载**:`.env`(模型 / Key / 端点 / 超时 / DB / 调度时间)与
|
||||||
|
> `configs/llm_models.yaml`(各场景 provider / model / 温度)都是**改完保存即生效**,
|
||||||
|
> 不需要 `systemctl restart`。调度时间最长 30s 生效,其余配置约 2s 生效。
|
||||||
|
>
|
||||||
|
> 只有两种情况需要重启:
|
||||||
|
> 1. 部署/更新了 Python 代码本身;
|
||||||
|
> 2. 正在执行中的那一步(子进程)会继续用旧配置跑完,下一步才用新配置。
|
||||||
|
|
||||||
### 7.5 前台守护模式(调试)
|
### 7.5 前台守护模式(调试)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -703,6 +711,16 @@ uv run a-share stock-report
|
|||||||
|
|
||||||
检查 `.env` 中 `DASHSCOPE_API_KEY` 是否填写。可用 `--provider qwen` 切换到百炼测试。模型名缺失时直接报错,检查 `configs/llm_models.yaml` 中 `event_extraction` 场景的 `model` 字段。
|
检查 `.env` 中 `DASHSCOPE_API_KEY` 是否填写。可用 `--provider qwen` 切换到百炼测试。模型名缺失时直接报错,检查 `configs/llm_models.yaml` 中 `event_extraction` 场景的 `model` 字段。
|
||||||
|
|
||||||
|
**Q: 日报没有 AI 摘要(ai_summary 为空)?**
|
||||||
|
|
||||||
|
先查 `logs/scheduler.log` 是否有 `AI 摘要可能被截断: ... finish_reason=length 实际输出 0 字符`。根因通常是**推理模型的 reasoning token 与正文共用 `max_tokens`**:预算过小时"思考"占满配额,正文一个字都没有。处理办法:
|
||||||
|
|
||||||
|
1. 调大 `configs/llm_models.yaml` 中 `daily_report.max_tokens`(默认 4000,YAML 保存即热生效,无需重启);
|
||||||
|
2. 代码已内置兜底:正文为空时自动加倍预算重试(上限 16000),仍失败才降级为无摘要;
|
||||||
|
3. 补生成某天摘要:`uv run a-share report --date <YYYYMMDD>`(按 `(report_date, report_type, file_name)` 幂等 upsert,不会新增记录)。
|
||||||
|
|
||||||
|
注意:`report` 步骤由调度器**进程内**执行(`scheduler/pipeline.py`),改动 Python 代码后需 `sudo systemctl restart a-share-research` 才会生效;只改 YAML / `.env` 则无需重启。
|
||||||
|
|
||||||
**Q: Qdrant 搜索不到结果?**
|
**Q: Qdrant 搜索不到结果?**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -5,9 +5,8 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
|
||||||
|
|
||||||
from configs.loader import load_scene_config
|
from configs.loader import load_scene_config
|
||||||
|
from configs.runtime_env import env_get
|
||||||
|
|
||||||
from .base import AsyncEmbeddingProvider, EmbeddingProvider
|
from .base import AsyncEmbeddingProvider, EmbeddingProvider
|
||||||
from .models import EmbeddingError, EmbeddingProviderType
|
from .models import EmbeddingError, EmbeddingProviderType
|
||||||
@@ -18,10 +17,8 @@ from .remote import (
|
|||||||
|
|
||||||
|
|
||||||
def _read_env(key: str, default: str | None = None) -> str | None:
|
def _read_env(key: str, default: str | None = None) -> str | None:
|
||||||
val = os.environ.get(key)
|
"""读取环境变量(先热加载 .env,改文件后无需重启进程)。"""
|
||||||
if val is None or val.strip() == "":
|
return env_get(key, default)
|
||||||
return default
|
|
||||||
return val.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_provider_type(provider: str | None = None) -> EmbeddingProviderType:
|
def resolve_provider_type(provider: str | None = None) -> EmbeddingProviderType:
|
||||||
|
|||||||
+3
-5
@@ -10,7 +10,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import os
|
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
@@ -22,16 +21,15 @@ if TYPE_CHECKING:
|
|||||||
from sentence_transformers import SentenceTransformer
|
from sentence_transformers import SentenceTransformer
|
||||||
|
|
||||||
from configs.loader import load_scene_config
|
from configs.loader import load_scene_config
|
||||||
|
from configs.runtime_env import env_get
|
||||||
|
|
||||||
LOCAL_DEFAULT_MODEL = "BAAI/bge-m3"
|
LOCAL_DEFAULT_MODEL = "BAAI/bge-m3"
|
||||||
LOCAL_DEFAULT_DIM = 1024
|
LOCAL_DEFAULT_DIM = 1024
|
||||||
|
|
||||||
|
|
||||||
def _read_env(key: str, default: str | None = None) -> str | None:
|
def _read_env(key: str, default: str | None = None) -> str | None:
|
||||||
val = os.environ.get(key)
|
"""读取环境变量(先热加载 .env,改文件后无需重启进程)。"""
|
||||||
if val is None or val.strip() == "":
|
return env_get(key, default)
|
||||||
return default
|
|
||||||
return val.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _try_import_st() -> type[SentenceTransformer]:
|
def _try_import_st() -> type[SentenceTransformer]:
|
||||||
|
|||||||
+3
-5
@@ -18,12 +18,12 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import os
|
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from openai import AsyncOpenAI, OpenAI
|
from openai import AsyncOpenAI, OpenAI
|
||||||
|
|
||||||
from configs.loader import load_scene_config
|
from configs.loader import load_scene_config
|
||||||
|
from configs.runtime_env import env_get
|
||||||
|
|
||||||
from .base import AsyncEmbeddingProvider, EmbeddingProvider
|
from .base import AsyncEmbeddingProvider, EmbeddingProvider
|
||||||
from .models import EmbeddingError
|
from .models import EmbeddingError
|
||||||
@@ -43,10 +43,8 @@ SCENE_EMBEDDING = "embedding"
|
|||||||
|
|
||||||
|
|
||||||
def _read_env(key: str, default: str | None = None) -> str | None:
|
def _read_env(key: str, default: str | None = None) -> str | None:
|
||||||
val = os.environ.get(key)
|
"""读取环境变量(先热加载 .env,改文件后无需重启进程)。"""
|
||||||
if val is None or val.strip() == "":
|
return env_get(key, default)
|
||||||
return default
|
|
||||||
return val.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _scene() -> dict:
|
def _scene() -> dict:
|
||||||
|
|||||||
+14
-5
@@ -19,13 +19,13 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from openai import AsyncOpenAI, OpenAI
|
from openai import AsyncOpenAI, OpenAI
|
||||||
|
|
||||||
from configs.loader import load_defaults, load_scene_config
|
from configs.loader import load_defaults, load_scene_config
|
||||||
|
from configs.runtime_env import env_get
|
||||||
|
|
||||||
# 默认基址
|
# 默认基址
|
||||||
_DEEPSEEK_DEFAULT_BASE = "https://api.deepseek.com"
|
_DEEPSEEK_DEFAULT_BASE = "https://api.deepseek.com"
|
||||||
@@ -53,6 +53,9 @@ class LLMConfig:
|
|||||||
timeout_sec: float = DEFAULT_TIMEOUT_SEC
|
timeout_sec: float = DEFAULT_TIMEOUT_SEC
|
||||||
temperature: float = DEFAULT_TEMPERATURE
|
temperature: float = DEFAULT_TEMPERATURE
|
||||||
max_attempts: int = DEFAULT_MAX_ATTEMPTS # 单次任务失败重试次数
|
max_attempts: int = DEFAULT_MAX_ATTEMPTS # 单次任务失败重试次数
|
||||||
|
# 单次输出预算(可选,场景配置 scenes.<scene>.max_tokens);None = 调用方用内置默认。
|
||||||
|
# 注意:推理模型(deepseek-v4.1-flash 等)的 reasoning token 与正文共用该预算。
|
||||||
|
max_tokens: int | None = None
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
if not self.api_key:
|
if not self.api_key:
|
||||||
@@ -60,10 +63,8 @@ class LLMConfig:
|
|||||||
|
|
||||||
|
|
||||||
def _read_env(key: str, default: str | None = None) -> str | None:
|
def _read_env(key: str, default: str | None = None) -> str | None:
|
||||||
val = os.environ.get(key)
|
"""读取环境变量(先热加载 .env,改文件后无需重启进程)。"""
|
||||||
if val is None or val.strip() == "":
|
return env_get(key, default)
|
||||||
return default
|
|
||||||
return val.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _first_env(keys: list[str | None]) -> str | None:
|
def _first_env(keys: list[str | None]) -> str | None:
|
||||||
@@ -147,6 +148,7 @@ def load_llm_config(
|
|||||||
timeout = _pick_float(sc, dflt, "timeout_sec", "LLM_TIMEOUT_SEC", DEFAULT_TIMEOUT_SEC)
|
timeout = _pick_float(sc, dflt, "timeout_sec", "LLM_TIMEOUT_SEC", DEFAULT_TIMEOUT_SEC)
|
||||||
temperature = _pick_float(sc, dflt, "temperature", "LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
|
temperature = _pick_float(sc, dflt, "temperature", "LLM_TEMPERATURE", DEFAULT_TEMPERATURE)
|
||||||
max_attempts = _pick_int(sc, "max_attempts", DEFAULT_MAX_ATTEMPTS)
|
max_attempts = _pick_int(sc, "max_attempts", DEFAULT_MAX_ATTEMPTS)
|
||||||
|
max_tokens = _pick_optional_int(sc, "max_tokens")
|
||||||
|
|
||||||
return LLMConfig(
|
return LLMConfig(
|
||||||
provider=p,
|
provider=p,
|
||||||
@@ -156,6 +158,7 @@ def load_llm_config(
|
|||||||
timeout_sec=timeout,
|
timeout_sec=timeout,
|
||||||
temperature=temperature,
|
temperature=temperature,
|
||||||
max_attempts=max_attempts,
|
max_attempts=max_attempts,
|
||||||
|
max_tokens=max_tokens,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -182,6 +185,12 @@ def _pick_int(sc: dict, sc_key: str, default: int) -> int:
|
|||||||
return int(v) if v is not None else default
|
return int(v) if v is not None else default
|
||||||
|
|
||||||
|
|
||||||
|
def _pick_optional_int(sc: dict, sc_key: str) -> int | None:
|
||||||
|
"""可选整数场景配置;未配置或非法时返回 None(调用方回退各自内置默认)。"""
|
||||||
|
v = _num(sc.get(sc_key))
|
||||||
|
return int(v) if v is not None else None
|
||||||
|
|
||||||
|
|
||||||
def make_sync_client(config: LLMConfig) -> OpenAI:
|
def make_sync_client(config: LLMConfig) -> OpenAI:
|
||||||
"""构造同步 OpenAI 客户端(指向 DeepSeek/Qwen 兼容端点)。"""
|
"""构造同步 OpenAI 客户端(指向 DeepSeek/Qwen 兼容端点)。"""
|
||||||
logger.debug(
|
logger.debug(
|
||||||
|
|||||||
+22
-5
@@ -9,15 +9,22 @@ from __future__ import annotations
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from dotenv import load_dotenv
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from mcp.server.fastmcp import FastMCP
|
from mcp.server.fastmcp import FastMCP
|
||||||
|
|
||||||
|
from configs.loader import config_path
|
||||||
|
from configs.runtime_env import (
|
||||||
|
dotenv_path,
|
||||||
|
ensure_env_loaded,
|
||||||
|
file_signature,
|
||||||
|
start_env_watcher,
|
||||||
|
)
|
||||||
from embedding import make_sync_provider
|
from embedding import make_sync_provider
|
||||||
from vectorstore import SearchFilter, VectorStore, make_qdrant_client
|
from vectorstore import SearchFilter, VectorStore, make_qdrant_client
|
||||||
|
|
||||||
# 加载 .env(API key 等)
|
# 加载 .env 并开启热加载:MCP server 是常驻进程,改配置无需重启
|
||||||
load_dotenv()
|
ensure_env_loaded()
|
||||||
|
start_env_watcher()
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# 单例(模块加载时初始化,所有工具共用)
|
# 单例(模块加载时初始化,所有工具共用)
|
||||||
@@ -29,17 +36,27 @@ class _Backend:
|
|||||||
vector_store: VectorStore
|
vector_store: VectorStore
|
||||||
|
|
||||||
_backend: _Backend | None = None
|
_backend: _Backend | None = None
|
||||||
|
_backend_sig: tuple[Any, ...] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _config_signature() -> tuple[Any, ...]:
|
||||||
|
"""(.env, llm_models.yaml) 的 mtime 签名;变化即表示需要重建后端。"""
|
||||||
|
return (file_signature(dotenv_path()), file_signature(config_path()))
|
||||||
|
|
||||||
|
|
||||||
def _get_backend() -> _Backend:
|
def _get_backend() -> _Backend:
|
||||||
global _backend
|
global _backend, _backend_sig
|
||||||
if _backend is None:
|
sig = _config_signature()
|
||||||
|
if _backend is None or sig != _backend_sig:
|
||||||
|
if _backend is not None:
|
||||||
|
logger.info("检测到配置变化, 重建 MCP 后端")
|
||||||
emb = make_sync_provider() # 读取 EMBEDDING_PROVIDER 环境变量
|
emb = make_sync_provider() # 读取 EMBEDDING_PROVIDER 环境变量
|
||||||
logger.info("MCP embedder 就绪: dim={}", emb.dim)
|
logger.info("MCP embedder 就绪: dim={}", emb.dim)
|
||||||
client = make_qdrant_client()
|
client = make_qdrant_client()
|
||||||
store = VectorStore(client)
|
store = VectorStore(client)
|
||||||
logger.info("MCP vector_store 就绪: count={}", store.count())
|
logger.info("MCP vector_store 就绪: count={}", store.count())
|
||||||
_backend = _Backend(embedder=emb, vector_store=store)
|
_backend = _Backend(embedder=emb, vector_store=store)
|
||||||
|
_backend_sig = sig
|
||||||
return _backend
|
return _backend
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+14
-17
@@ -12,7 +12,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
@@ -21,6 +20,8 @@ from pathlib import Path
|
|||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from configs.runtime_env import env_get, env_raw
|
||||||
|
|
||||||
from .timeutil import today_str
|
from .timeutil import today_str
|
||||||
|
|
||||||
# 断点状态文件(按日期隔离,记录每步骤结果)
|
# 断点状态文件(按日期隔离,记录每步骤结果)
|
||||||
@@ -159,20 +160,16 @@ def _llm_scene_desc(scene: str) -> str | None:
|
|||||||
try:
|
try:
|
||||||
from configs.loader import load_scene_config
|
from configs.loader import load_scene_config
|
||||||
|
|
||||||
def _env(key: str) -> str | None:
|
|
||||||
v = os.environ.get(key)
|
|
||||||
return v.strip() if v else None
|
|
||||||
|
|
||||||
sc = load_scene_config(scene)
|
sc = load_scene_config(scene)
|
||||||
p = (sc.get("provider") or _env("LLM_PROVIDER") or "deepseek").lower()
|
p = (sc.get("provider") or env_get("LLM_PROVIDER") or "deepseek").lower()
|
||||||
if p in ("qwen", "dashscope"):
|
if p in ("qwen", "dashscope"):
|
||||||
p = "qwen"
|
p = "qwen"
|
||||||
model = sc.get("model")
|
model = sc.get("model")
|
||||||
if not model:
|
if not model:
|
||||||
if p == "qwen":
|
if p == "qwen":
|
||||||
model = _env("QWEN_MODEL") or _env("LLM_MODEL")
|
model = env_get("QWEN_MODEL") or env_get("LLM_MODEL")
|
||||||
else:
|
else:
|
||||||
model = _env("DEEPSEEK_MODEL") or _env("LLM_MODEL")
|
model = env_get("DEEPSEEK_MODEL") or env_get("LLM_MODEL")
|
||||||
if not model:
|
if not model:
|
||||||
return None
|
return None
|
||||||
return f"provider={p}, model={model}"
|
return f"provider={p}, model={model}"
|
||||||
@@ -192,9 +189,9 @@ def _embedding_desc() -> str | None:
|
|||||||
model = sc.get("model")
|
model = sc.get("model")
|
||||||
if not model:
|
if not model:
|
||||||
if pt == "dashscope":
|
if pt == "dashscope":
|
||||||
model = os.environ.get("DASHSCOPE_EMBEDDING_MODEL") or "text-embedding-v3"
|
model = env_get("DASHSCOPE_EMBEDDING_MODEL") or "text-embedding-v3"
|
||||||
else:
|
else:
|
||||||
model = os.environ.get("LOCAL_EMBEDDING_MODEL") or "BAAI/bge-m3"
|
model = env_get("LOCAL_EMBEDDING_MODEL") or "BAAI/bge-m3"
|
||||||
return f"provider={pt}, model={model}"
|
return f"provider={pt}, model={model}"
|
||||||
except Exception as e: # noqa: BLE001
|
except Exception as e: # noqa: BLE001
|
||||||
logger.debug("embedding 描述解析失败: {}", e)
|
logger.debug("embedding 描述解析失败: {}", e)
|
||||||
@@ -281,14 +278,14 @@ def run_step(name: str, date_str: str) -> StepResult:
|
|||||||
# 2. PIPELINE_STEP_TIMEOUT 环境变量 (全局兜底, 覆盖硬编码)
|
# 2. PIPELINE_STEP_TIMEOUT 环境变量 (全局兜底, 覆盖硬编码)
|
||||||
# 3. STEP_TIMEOUTS 硬编码字典 (代码内默认值)
|
# 3. STEP_TIMEOUTS 硬编码字典 (代码内默认值)
|
||||||
# 4. 1800s (最终兜底)
|
# 4. 1800s (最终兜底)
|
||||||
import os
|
|
||||||
specific_key = f"TIMEOUT_{name.upper()}"
|
specific_key = f"TIMEOUT_{name.upper()}"
|
||||||
if specific_key in os.environ:
|
timeout_raw = env_raw(specific_key) or env_raw("PIPELINE_STEP_TIMEOUT")
|
||||||
timeout = int(os.environ[specific_key])
|
default_timeout = STEP_TIMEOUTS.get(name, 1800)
|
||||||
elif "PIPELINE_STEP_TIMEOUT" in os.environ:
|
try:
|
||||||
timeout = int(os.environ["PIPELINE_STEP_TIMEOUT"])
|
timeout = int(timeout_raw) if timeout_raw else default_timeout
|
||||||
else:
|
except ValueError:
|
||||||
timeout = STEP_TIMEOUTS.get(name, 1800)
|
logger.warning("超时配置 {!r} 非法,回退默认 {}s", timeout_raw, default_timeout)
|
||||||
|
timeout = default_timeout
|
||||||
started = datetime.now()
|
started = datetime.now()
|
||||||
logger.info("步骤 {} 开始: {}", name, " ".join(full_cmd))
|
logger.info("步骤 {} 开始: {}", name, " ".join(full_cmd))
|
||||||
|
|
||||||
|
|||||||
+119
-46
@@ -10,7 +10,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os as _os
|
|
||||||
import re as _re
|
import re as _re
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
@@ -22,14 +21,11 @@ from typing import TYPE_CHECKING, Any
|
|||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from llm.client import LLMConfig
|
from llm.client import LLMConfig
|
||||||
|
|
||||||
from dotenv import load_dotenv
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from configs.runtime_env import env_get
|
||||||
from report_db.models import EventRow, ReportData # noqa: F401 - 供 _build_report_data 注解使用
|
from report_db.models import EventRow, ReportData # noqa: F401 - 供 _build_report_data 注解使用
|
||||||
|
|
||||||
# 确保 .env 已加载(模块级常量依赖环境变量)
|
|
||||||
load_dotenv()
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# 配置
|
# 配置
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -37,14 +33,64 @@ load_dotenv()
|
|||||||
UPLOAD_HOST = "simon@doorcome.cn"
|
UPLOAD_HOST = "simon@doorcome.cn"
|
||||||
UPLOAD_BASE = "/var/www/html/echart/research"
|
UPLOAD_BASE = "/var/www/html/echart/research"
|
||||||
|
|
||||||
CNINFO_DAYS_BACK = int(_os.environ.get("STOCK_REPORT_DAYS", "15")) # 与个股日报共用参数, 默认值保持一致
|
|
||||||
NEWS_DAYS_BACK = 1 # 新闻回溯天数
|
NEWS_DAYS_BACK = 1 # 新闻回溯天数
|
||||||
|
|
||||||
_MAX_HIGH_EVENTS = 20
|
_MAX_HIGH_EVENTS = 20
|
||||||
|
|
||||||
# LLM 摘要调用重试参数(环境变量可覆盖)
|
# AI 摘要输出预算(token)。
|
||||||
_LLM_RETRY_TIMES = int(_os.environ.get("LLM_RETRY_TIMES", "3"))
|
# 推理模型(deepseek-v4.1-flash 等)的 reasoning token 与正文共用 max_tokens:
|
||||||
_LLM_RETRY_BACKOFF_SEC = float(_os.environ.get("LLM_RETRY_BACKOFF_SEC", "2.0"))
|
# 预算过小时"思考"会占满配额,正文为空(finish_reason=length、0 字符),
|
||||||
|
# 日报就会没有 AI 摘要。默认值需为 reasoning 预留余量。
|
||||||
|
DEFAULT_SUMMARY_MAX_TOKENS = 4000 # 单块 / 合并摘要默认预算
|
||||||
|
DEFAULT_SUMMARY_CHUNK_MAX_TOKENS = 2000 # 分块摘要默认预算
|
||||||
|
MAX_SUMMARY_MAX_TOKENS = 16000 # 正文为空时预算升级上限
|
||||||
|
_MAX_BUDGET_ESCALATIONS = 2 # 正文为空时最多升级预算次数
|
||||||
|
|
||||||
|
|
||||||
|
def _env_int(key: str, default: int) -> int:
|
||||||
|
"""读取整数环境变量(热加载 .env);非法值回退默认。"""
|
||||||
|
try:
|
||||||
|
return int(env_get(key, str(default)) or default)
|
||||||
|
except ValueError:
|
||||||
|
logger.warning("环境变量 {} 不是整数, 回退默认 {}", key, default)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def _env_float(key: str, default: float) -> float:
|
||||||
|
"""读取浮点环境变量(热加载 .env);非法值回退默认。"""
|
||||||
|
try:
|
||||||
|
return float(env_get(key, str(default)) or default)
|
||||||
|
except ValueError:
|
||||||
|
logger.warning("环境变量 {} 不是数字, 回退默认 {}", key, default)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def _cninfo_days_back() -> int:
|
||||||
|
"""公告/调研回溯天数(与个股日报共用 STOCK_REPORT_DAYS)。"""
|
||||||
|
return _env_int("STOCK_REPORT_DAYS", 15)
|
||||||
|
|
||||||
|
|
||||||
|
def _llm_retry_times() -> int:
|
||||||
|
return _env_int("LLM_RETRY_TIMES", 3)
|
||||||
|
|
||||||
|
|
||||||
|
def _llm_retry_backoff_sec() -> float:
|
||||||
|
return _env_float("LLM_RETRY_BACKOFF_SEC", 2.0)
|
||||||
|
|
||||||
|
|
||||||
|
def _summary_max_tokens(config: LLMConfig | None = None) -> int:
|
||||||
|
"""摘要输出预算:场景配置 max_tokens(热更新)> 代码内置默认。"""
|
||||||
|
value = getattr(config, "max_tokens", None)
|
||||||
|
try:
|
||||||
|
return int(value) if value else DEFAULT_SUMMARY_MAX_TOKENS
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
logger.warning("场景 max_tokens 非法({!r}), 回退默认 {}", value, DEFAULT_SUMMARY_MAX_TOKENS)
|
||||||
|
return DEFAULT_SUMMARY_MAX_TOKENS
|
||||||
|
|
||||||
|
|
||||||
|
def _chunk_max_tokens(config: LLMConfig | None = None) -> int:
|
||||||
|
"""分块摘要预算:不超过单块/合并预算,避免小块调用过度消耗。"""
|
||||||
|
return min(_summary_max_tokens(config), DEFAULT_SUMMARY_CHUNK_MAX_TOKENS)
|
||||||
|
|
||||||
# 日报新闻回溯窗口(小时):07:00 生成当日日报时覆盖昨日全天至今晨的新闻
|
# 日报新闻回溯窗口(小时):07:00 生成当日日报时覆盖昨日全天至今晨的新闻
|
||||||
_NEWS_LOOKBACK_HOURS = 30
|
_NEWS_LOOKBACK_HOURS = 30
|
||||||
@@ -181,12 +227,14 @@ def _collect_news_events(day_str: str) -> dict[str, Any]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _collect_cninfo_events(today_str: str, days_back: int = CNINFO_DAYS_BACK) -> dict[str, Any]:
|
def _collect_cninfo_events(today_str: str, days_back: int | None = None) -> dict[str, Any]:
|
||||||
"""收集近 N 日 cninfo 公告/调研/互动(直接从 processed 数据读取,不依赖 M4 事件抽取)。
|
"""收集近 N 日 cninfo 公告/调研/互动(直接从 processed 数据读取,不依赖 M4 事件抽取)。
|
||||||
|
|
||||||
cninfo 公告/调研数据已结构化(stock_code/name/title/time/type),
|
cninfo 公告/调研数据已结构化(stock_code/name/title/time/type),
|
||||||
无需经过 LLM 事件抽取即可直接用于日报。
|
无需经过 LLM 事件抽取即可直接用于日报。
|
||||||
"""
|
"""
|
||||||
|
if days_back is None:
|
||||||
|
days_back = _cninfo_days_back()
|
||||||
today = datetime.strptime(today_str, "%Y%m%d")
|
today = datetime.strptime(today_str, "%Y%m%d")
|
||||||
since_str = (today - timedelta(days=days_back)).strftime("%Y-%m-%d")
|
since_str = (today - timedelta(days=days_back)).strftime("%Y-%m-%d")
|
||||||
wl_codes = _load_watchlist_codes()
|
wl_codes = _load_watchlist_codes()
|
||||||
@@ -514,7 +562,7 @@ def _generate_ai_summary(news: dict, cninfo: dict, day_str: str,
|
|||||||
|
|
||||||
# 公告/调研
|
# 公告/调研
|
||||||
if cninfo["high"]:
|
if cninfo["high"]:
|
||||||
lines.append(f"## 近 {CNINFO_DAYS_BACK} 日重要公告/调研 ({len(cninfo['high'])} 条)")
|
lines.append(f"## 近 {_cninfo_days_back()} 日重要公告/调研 ({len(cninfo['high'])} 条)")
|
||||||
for e in cninfo["high"][:8]:
|
for e in cninfo["high"][:8]:
|
||||||
ev = e.get("event", {})
|
ev = e.get("event", {})
|
||||||
lines.append(f"- [{ev.get('event_type', '公司公告')}] {e['title']}")
|
lines.append(f"- [{ev.get('event_type', '公司公告')}] {e['title']}")
|
||||||
@@ -570,7 +618,7 @@ def _llm_summarize(client, config: LLMConfig, lines: list[str], day_str: str) ->
|
|||||||
{chr(10).join(chunk)}
|
{chr(10).join(chunk)}
|
||||||
|
|
||||||
直接输出要点列表:"""
|
直接输出要点列表:"""
|
||||||
result = _llm_call(client, config, prompt, max_tokens=800)
|
result = _llm_call(client, config, prompt, max_tokens=_chunk_max_tokens(config))
|
||||||
if result:
|
if result:
|
||||||
partials.append(result)
|
partials.append(result)
|
||||||
logger.info("AI 摘要: 分块 {}/{} 完成 ({} 字)", i, len(chunks), len(result))
|
logger.info("AI 摘要: 分块 {}/{} 完成 ({} 字)", i, len(chunks), len(result))
|
||||||
@@ -589,14 +637,14 @@ def _llm_summarize(client, config: LLMConfig, lines: list[str], day_str: str) ->
|
|||||||
|
|
||||||
请合并为要点总结,每条一行以 "- " 开头,要求:
|
请合并为要点总结,每条一行以 "- " 开头,要求:
|
||||||
1. 前 3 条为影响最大的事件,说明为什么重要
|
1. 前 3 条为影响最大的事件,说明为什么重要
|
||||||
2. 汇总近 {CNINFO_DAYS_BACK} 日公司公告/调研核心信息
|
2. 汇总近 {_cninfo_days_back()} 日公司公告/调研核心信息
|
||||||
3. 市场情绪基调(利好/利空/中性)
|
3. 市场情绪基调(利好/利空/中性)
|
||||||
4. 值得持续关注的行业或主题
|
4. 值得持续关注的行业或主题
|
||||||
5. 纯要点,不要开场白/结束语
|
5. 纯要点,不要开场白/结束语
|
||||||
6. 总字数 500 字以内
|
6. 总字数 500 字以内
|
||||||
|
|
||||||
直接输出要点列表:"""
|
直接输出要点列表:"""
|
||||||
return _llm_call(client, config, merge_prompt, max_tokens=1500)
|
return _llm_call(client, config, merge_prompt, max_tokens=_summary_max_tokens(config))
|
||||||
|
|
||||||
|
|
||||||
def _build_prompt(lines: list[str], day_str: str) -> str:
|
def _build_prompt(lines: list[str], day_str: str) -> str:
|
||||||
@@ -607,7 +655,7 @@ def _build_prompt(lines: list[str], day_str: str) -> str:
|
|||||||
|
|
||||||
请用要点总结,每条一行,以 "- " 开头,要求:
|
请用要点总结,每条一行,以 "- " 开头,要求:
|
||||||
1. 前 3 条为过去 24 小时影响最大的事件(优先参考新闻联播中的重大政策信号),说明为什么重要
|
1. 前 3 条为过去 24 小时影响最大的事件(优先参考新闻联播中的重大政策信号),说明为什么重要
|
||||||
2. 汇总近 {CNINFO_DAYS_BACK} 日重要公司公告/调研的核心信息
|
2. 汇总近 {_cninfo_days_back()} 日重要公司公告/调研的核心信息
|
||||||
3. 市场情绪基调(利好/利空/中性)
|
3. 市场情绪基调(利好/利空/中性)
|
||||||
4. 值得持续关注的行业或主题
|
4. 值得持续关注的行业或主题
|
||||||
5. 纯要点,不要开场白/结束语/标题
|
5. 纯要点,不要开场白/结束语/标题
|
||||||
@@ -616,45 +664,70 @@ def _build_prompt(lines: list[str], day_str: str) -> str:
|
|||||||
直接输出要点列表:"""
|
直接输出要点列表:"""
|
||||||
|
|
||||||
|
|
||||||
def _llm_call(client, config: LLMConfig, prompt: str, max_tokens: int = 1500) -> str:
|
def _call_once(client, config: LLMConfig, prompt: str, max_tokens: int) -> tuple[str, str | None]:
|
||||||
"""单次 LLM 调用(带重试),返回 strip 后的文本。
|
"""单次 LLM 调用,返回 (正文, finish_reason)。异常由调用方处理。"""
|
||||||
|
resp = client.chat.completions.create(
|
||||||
|
model=config.model,
|
||||||
|
messages=[
|
||||||
|
{"role": "system", "content": "你是 A 股日报撰写助手,输出简洁、有洞察的新闻摘要。"},
|
||||||
|
{"role": "user", "content": prompt},
|
||||||
|
],
|
||||||
|
temperature=config.temperature,
|
||||||
|
max_tokens=max_tokens,
|
||||||
|
)
|
||||||
|
choice = resp.choices[0]
|
||||||
|
content = (choice.message.content or "").strip()
|
||||||
|
return content, getattr(choice, "finish_reason", None)
|
||||||
|
|
||||||
config 为 llm.client.LLMConfig(daily_report 场景),提供 model / temperature。
|
|
||||||
失败按指数退避重试 `_LLM_RETRY_TIMES` 次(默认 3),全部失败则抛出最后一次异常。
|
def _llm_call(client, config: LLMConfig, prompt: str, max_tokens: int | None = None) -> str:
|
||||||
若 finish_reason 为 'length' 则说明达到 max_tokens 上限被截断。
|
"""单次 LLM 调用(带重试 + 空正文预算升级),返回 strip 后的文本。
|
||||||
|
|
||||||
|
config 为 llm.client.LLMConfig(daily_report 场景),提供 model / temperature /
|
||||||
|
max_tokens(未配置 max_tokens 时用 DEFAULT_SUMMARY_MAX_TOKENS)。
|
||||||
|
网络等异常按指数退避重试 `LLM_RETRY_TIMES` 次(默认 3),全部失败则抛出最后一次异常。
|
||||||
|
若 finish_reason 为 'length' 且正文为空(推理模型 reasoning 占满预算),
|
||||||
|
自动加倍预算重试,最多 `_MAX_BUDGET_ESCALATIONS` 次、上限 MAX_SUMMARY_MAX_TOKENS;
|
||||||
|
此时返回空字符串而非抛异常,由调用方降级。
|
||||||
"""
|
"""
|
||||||
|
retry_times = _llm_retry_times()
|
||||||
|
backoff_sec = _llm_retry_backoff_sec()
|
||||||
|
budget = max_tokens or _summary_max_tokens(config)
|
||||||
last_exc: Exception | None = None
|
last_exc: Exception | None = None
|
||||||
for attempt in range(_LLM_RETRY_TIMES):
|
escalations = 0
|
||||||
|
|
||||||
|
for attempt in range(retry_times):
|
||||||
try:
|
try:
|
||||||
resp = client.chat.completions.create(
|
content, finish = _call_once(client, config, prompt, budget)
|
||||||
model=config.model,
|
|
||||||
messages=[
|
|
||||||
{"role": "system", "content": "你是 A 股日报撰写助手,输出简洁、有洞察的新闻摘要。"},
|
|
||||||
{"role": "user", "content": prompt},
|
|
||||||
],
|
|
||||||
temperature=config.temperature,
|
|
||||||
max_tokens=max_tokens,
|
|
||||||
)
|
|
||||||
content = (resp.choices[0].message.content or "").strip()
|
|
||||||
finish = getattr(resp.choices[0], "finish_reason", None)
|
|
||||||
if finish == "length":
|
|
||||||
logger.warning(
|
|
||||||
"AI 摘要可能被截断: max_tokens={} finish_reason=length 实际输出 {} 字符",
|
|
||||||
max_tokens, len(content),
|
|
||||||
)
|
|
||||||
return content
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
last_exc = e
|
last_exc = e
|
||||||
if attempt < _LLM_RETRY_TIMES - 1:
|
if attempt < retry_times - 1:
|
||||||
wait = _LLM_RETRY_BACKOFF_SEC * (2 ** attempt)
|
wait = backoff_sec * (2 ** attempt)
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"AI 摘要 LLM 调用失败(第 {}/{} 次): {}; {} 秒后重试",
|
"AI 摘要 LLM 调用失败(第 {}/{} 次): {}; {} 秒后重试",
|
||||||
attempt + 1, _LLM_RETRY_TIMES, e, round(wait, 2),
|
attempt + 1, retry_times, e, round(wait, 2),
|
||||||
)
|
)
|
||||||
time.sleep(wait)
|
time.sleep(wait)
|
||||||
logger.error("AI 摘要 LLM 调用重试 {} 次仍失败: {}", _LLM_RETRY_TIMES, last_exc)
|
continue
|
||||||
assert last_exc is not None
|
|
||||||
raise last_exc
|
if finish == "length":
|
||||||
|
logger.warning(
|
||||||
|
"AI 摘要可能被截断: max_tokens={} finish_reason=length 实际输出 {} 字符",
|
||||||
|
budget, len(content),
|
||||||
|
)
|
||||||
|
# 正文为空 = 推理占满预算;加倍预算重试(不改动调用方传入的显式预算以外逻辑)
|
||||||
|
if not content and escalations < _MAX_BUDGET_ESCALATIONS:
|
||||||
|
budget = min(budget * 2, MAX_SUMMARY_MAX_TOKENS)
|
||||||
|
escalations += 1
|
||||||
|
logger.warning("AI 摘要正文为空(推理占满预算), 提升 max_tokens 至 {} 重试", budget)
|
||||||
|
continue
|
||||||
|
return content
|
||||||
|
|
||||||
|
if last_exc is not None:
|
||||||
|
logger.error("AI 摘要 LLM 调用重试 {} 次仍失败: {}", retry_times, last_exc)
|
||||||
|
raise last_exc
|
||||||
|
logger.warning("AI 摘要 {} 次尝试仍未获得正文, 返回空(日报降级为无 AI 摘要)", retry_times)
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -964,7 +1037,7 @@ def _render_html(news: dict, cninfo: dict, pipeline: dict,
|
|||||||
news_table=news_table,
|
news_table=news_table,
|
||||||
cninfo_high_count=len(cninfo["high"]),
|
cninfo_high_count=len(cninfo["high"]),
|
||||||
cninfo_threshold=cninfo.get("hi_threshold", 4),
|
cninfo_threshold=cninfo.get("hi_threshold", 4),
|
||||||
cninfo_days=CNINFO_DAYS_BACK,
|
cninfo_days=_cninfo_days_back(),
|
||||||
cninfo_table=cninfo_table,
|
cninfo_table=cninfo_table,
|
||||||
raw_total=pipeline["raw_total"],
|
raw_total=pipeline["raw_total"],
|
||||||
raw_total_24h=pipeline.get("raw_total_24h", pipeline["raw_total"]),
|
raw_total_24h=pipeline.get("raw_total_24h", pipeline["raw_total"]),
|
||||||
@@ -1065,7 +1138,7 @@ def generate_report(day_str: str | None = None, *, upload: bool = True) -> int |
|
|||||||
# 收集数据
|
# 收集数据
|
||||||
try:
|
try:
|
||||||
news = _collect_news_events(day_str)
|
news = _collect_news_events(day_str)
|
||||||
cninfo = _collect_cninfo_events(day_str, days_back=CNINFO_DAYS_BACK)
|
cninfo = _collect_cninfo_events(day_str, days_back=_cninfo_days_back())
|
||||||
pipeline = _collect_pipeline_stats(day_str)
|
pipeline = _collect_pipeline_stats(day_str)
|
||||||
xwlb = _collect_xwlb(day_str)
|
xwlb = _collect_xwlb(day_str)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
+147
-65
@@ -14,14 +14,15 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import json
|
||||||
import signal
|
import signal
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from dotenv import load_dotenv
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from configs.runtime_env import ensure_env_loaded, env_raw, start_env_watcher
|
||||||
from scheduler import DEFAULT_NEWS_STEPS, run_pipeline
|
from scheduler import DEFAULT_NEWS_STEPS, run_pipeline
|
||||||
from scheduler.stock_reporter import generate_all_stock_reports
|
from scheduler.stock_reporter import generate_all_stock_reports
|
||||||
from scheduler.timeutil import now as tz_now
|
from scheduler.timeutil import now as tz_now
|
||||||
@@ -59,6 +60,131 @@ def _parse_schedule_times(raw: str) -> list[tuple[int, int]]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_hhmm(raw: str, default: tuple[int, int]) -> tuple[int, int]:
|
||||||
|
"""解析 "HH:MM";非法或越界时记警告并返回 default。"""
|
||||||
|
parts = raw.split(":")
|
||||||
|
try:
|
||||||
|
hour = int(parts[0])
|
||||||
|
minute = int(parts[1]) if len(parts) > 1 else 0
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
logger.warning("时间格式错误: {!r},改用默认 {:02d}:{:02d}", raw, *default)
|
||||||
|
return default
|
||||||
|
if not (0 <= hour <= 23 and 0 <= minute <= 59):
|
||||||
|
logger.warning("时间越界: {!r},改用默认 {:02d}:{:02d}", raw, *default)
|
||||||
|
return default
|
||||||
|
return hour, minute
|
||||||
|
|
||||||
|
|
||||||
|
def _scheduled_pipeline(steps: list[str] | None = None) -> None:
|
||||||
|
"""定时触发的全链路:每次触发时按调度时区重新计算日期。"""
|
||||||
|
run_pipeline(today_str(), steps=steps)
|
||||||
|
|
||||||
|
|
||||||
|
def _desired_jobs() -> dict[str, dict[str, Any]]:
|
||||||
|
"""按当前环境变量算出「期望的」定时任务集合。
|
||||||
|
|
||||||
|
每次调用都通过 env_get 读取,所以改 .env 后无需重启即可反映到 _sync_jobs。
|
||||||
|
"""
|
||||||
|
jobs: dict[str, dict[str, Any]] = {}
|
||||||
|
|
||||||
|
times = _parse_schedule_times(
|
||||||
|
env_raw("SCHEDULE_TIMES", "07:00,12:00,18:00,22:00") or ""
|
||||||
|
)
|
||||||
|
if times:
|
||||||
|
first = min(times)
|
||||||
|
for hour, minute in times:
|
||||||
|
with_report = (hour, minute) == first
|
||||||
|
jobs[f"pipeline_{hour:02d}{minute:02d}"] = {
|
||||||
|
"kind": "pipeline",
|
||||||
|
"hour": hour,
|
||||||
|
"minute": minute,
|
||||||
|
"steps": list(DEFAULT_NEWS_STEPS) + (["report"] if with_report else []),
|
||||||
|
"name": f"全链路 {'+日报' if with_report else ''} {hour:02d}:{minute:02d}",
|
||||||
|
}
|
||||||
|
|
||||||
|
cninfo_h, cninfo_m = _parse_hhmm(
|
||||||
|
env_raw("CNINFO_SCHEDULE_TIME", "06:30") or "06:30", (6, 30)
|
||||||
|
)
|
||||||
|
jobs["pipeline_cninfo"] = {
|
||||||
|
"kind": "pipeline",
|
||||||
|
"hour": cninfo_h,
|
||||||
|
"minute": cninfo_m,
|
||||||
|
"steps": ["cninfo_crawl", "cninfo_extract", "cninfo_pdf",
|
||||||
|
"dedup", "llm", "embedding", "qdrant"],
|
||||||
|
"name": f"cninfo 公告管道 {cninfo_h:02d}:{cninfo_m:02d}",
|
||||||
|
}
|
||||||
|
|
||||||
|
# STOCK_REPORT_TIME 缺省 07:30;显式留空表示禁用
|
||||||
|
stock_raw = env_raw("STOCK_REPORT_TIME")
|
||||||
|
if stock_raw is None:
|
||||||
|
stock_raw = "07:30"
|
||||||
|
if stock_raw:
|
||||||
|
stock_h, stock_m = _parse_hhmm(stock_raw, (7, 30))
|
||||||
|
jobs["stock_report"] = {
|
||||||
|
"kind": "stock",
|
||||||
|
"hour": stock_h,
|
||||||
|
"minute": stock_m,
|
||||||
|
"name": f"个股日报 {stock_h:02d}:{stock_m:02d}",
|
||||||
|
}
|
||||||
|
return jobs
|
||||||
|
|
||||||
|
|
||||||
|
_jobs_sig: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_jobs(scheduler: Any) -> bool:
|
||||||
|
"""把「期望任务」同步到 APScheduler;只在配置变化时增删。返回是否变更。
|
||||||
|
|
||||||
|
以 ``_`` 开头的内部任务(如配置热同步自身)不参与增删。
|
||||||
|
"""
|
||||||
|
global _jobs_sig
|
||||||
|
|
||||||
|
from apscheduler.triggers.cron import CronTrigger # noqa: E402
|
||||||
|
|
||||||
|
desired = _desired_jobs()
|
||||||
|
if not any(jid.startswith("pipeline_") for jid in desired):
|
||||||
|
logger.error("SCHEDULE_TIMES 为空或全部非法,保留现有定时任务不改动")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sig = json.dumps(desired, sort_keys=True, ensure_ascii=False)
|
||||||
|
if sig == _jobs_sig:
|
||||||
|
return False
|
||||||
|
|
||||||
|
for jid, spec in desired.items():
|
||||||
|
trigger = CronTrigger(
|
||||||
|
hour=spec["hour"], minute=spec["minute"], timezone=str(schedule_tz())
|
||||||
|
)
|
||||||
|
if spec["kind"] == "pipeline":
|
||||||
|
scheduler.add_job(
|
||||||
|
_scheduled_pipeline,
|
||||||
|
trigger=trigger,
|
||||||
|
kwargs={"steps": spec["steps"]},
|
||||||
|
id=jid,
|
||||||
|
name=spec["name"],
|
||||||
|
replace_existing=True,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
scheduler.add_job(
|
||||||
|
generate_all_stock_reports,
|
||||||
|
trigger=trigger,
|
||||||
|
id=jid,
|
||||||
|
name=spec["name"],
|
||||||
|
replace_existing=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
for job in scheduler.get_jobs():
|
||||||
|
if job.id.startswith("_") or job.id in desired:
|
||||||
|
continue
|
||||||
|
scheduler.remove_job(job.id)
|
||||||
|
|
||||||
|
_jobs_sig = sig
|
||||||
|
logger.info(
|
||||||
|
"定时任务已同步: {}",
|
||||||
|
", ".join(f"{s['name']}" for s in desired.values()),
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def _once(args: argparse.Namespace) -> int:
|
def _once(args: argparse.Namespace) -> int:
|
||||||
"""单次执行模式。
|
"""单次执行模式。
|
||||||
|
|
||||||
@@ -76,77 +202,20 @@ def _once(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
def _daemon(args: argparse.Namespace) -> int:
|
def _daemon(args: argparse.Namespace) -> int:
|
||||||
"""守护进程模式(APScheduler)。"""
|
"""守护进程模式(APScheduler)。"""
|
||||||
import os
|
|
||||||
|
|
||||||
from apscheduler.schedulers.background import BackgroundScheduler # noqa: E402
|
from apscheduler.schedulers.background import BackgroundScheduler # noqa: E402
|
||||||
from apscheduler.triggers.cron import CronTrigger # noqa: E402
|
|
||||||
|
|
||||||
times_raw = os.environ.get("SCHEDULE_TIMES", "07:00,12:00,18:00,22:00")
|
times = _parse_schedule_times(
|
||||||
times = _parse_schedule_times(times_raw)
|
env_raw("SCHEDULE_TIMES", "07:00,12:00,18:00,22:00") or ""
|
||||||
|
)
|
||||||
if not times:
|
if not times:
|
||||||
logger.error("SCHEDULE_TIMES 为空或全部非法,无法启动定时任务")
|
logger.error("SCHEDULE_TIMES 为空或全部非法,无法启动定时任务")
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
# 找出最早的时间(当天首次运行),仅该次追加日报步骤
|
|
||||||
sorted_times = sorted(times)
|
sorted_times = sorted(times)
|
||||||
first_hour, first_minute = sorted_times[0] if sorted_times else (0, 0)
|
|
||||||
|
|
||||||
scheduler = BackgroundScheduler()
|
scheduler = BackgroundScheduler()
|
||||||
|
# 首次注册;此后由 _config_watch 每 30s 热同步,改 .env 无需重启
|
||||||
# 包装函数:每次触发时按调度时区重新计算日期(P1-3),
|
_sync_jobs(scheduler)
|
||||||
# 避免 date.today() 在注册时冻结或与 cron 时区不一致。
|
|
||||||
def _scheduled_pipeline(steps: list[str] | None = None) -> None:
|
|
||||||
run_pipeline(today_str(), steps=steps)
|
|
||||||
|
|
||||||
for hour, minute in times:
|
|
||||||
trigger = CronTrigger(hour=hour, minute=minute, timezone=str(schedule_tz()))
|
|
||||||
is_first = (hour == first_hour and minute == first_minute)
|
|
||||||
job_kwargs: dict | None = None
|
|
||||||
if is_first:
|
|
||||||
job_kwargs = {
|
|
||||||
"steps": list(DEFAULT_NEWS_STEPS) + ["report"]
|
|
||||||
}
|
|
||||||
scheduler.add_job(
|
|
||||||
_scheduled_pipeline,
|
|
||||||
trigger=trigger,
|
|
||||||
kwargs=job_kwargs,
|
|
||||||
id=f"pipeline_{hour:02d}{minute:02d}",
|
|
||||||
name=f"全链路 {'+日报' if is_first else ''} {hour:02d}:{minute:02d}",
|
|
||||||
)
|
|
||||||
logger.info("已注册定时任务: {}每天 {:02d}:{:02d}{}", trigger, hour, minute,
|
|
||||||
" (含日报)" if is_first else "")
|
|
||||||
|
|
||||||
# cninfo 公告管道(可配置,默认 06:30)
|
|
||||||
cninfo_raw = os.environ.get("CNINFO_SCHEDULE_TIME", "06:30")
|
|
||||||
cninfo_parts = cninfo_raw.split(":")
|
|
||||||
cninfo_h, cninfo_m = int(cninfo_parts[0]), int(cninfo_parts[1]) if len(cninfo_parts) > 1 else 0
|
|
||||||
cninfo_trigger = CronTrigger(hour=cninfo_h, minute=cninfo_m, timezone=str(schedule_tz()))
|
|
||||||
cninfo_steps = ["cninfo_crawl", "cninfo_extract", "cninfo_pdf",
|
|
||||||
"dedup", "llm", "embedding", "qdrant"]
|
|
||||||
scheduler.add_job(
|
|
||||||
_scheduled_pipeline,
|
|
||||||
trigger=cninfo_trigger,
|
|
||||||
kwargs={"steps": cninfo_steps},
|
|
||||||
id="pipeline_cninfo",
|
|
||||||
name=f"cninfo 公告管道 {cninfo_h:02d}:{cninfo_m:02d}",
|
|
||||||
)
|
|
||||||
logger.info("已注册定时任务: cninfo 公告管道 每天 {:02d}:{:02d}", cninfo_h, cninfo_m)
|
|
||||||
|
|
||||||
# 个股日报(可配置,默认 07:30, 设为空可禁用)
|
|
||||||
stock_raw = os.environ.get("STOCK_REPORT_TIME", "07:30")
|
|
||||||
if stock_raw:
|
|
||||||
stock_parts = stock_raw.split(":")
|
|
||||||
stock_h, stock_m = int(stock_parts[0]), int(stock_parts[1]) if len(stock_parts) > 1 else 0
|
|
||||||
stock_trigger = CronTrigger(hour=stock_h, minute=stock_m, timezone=str(schedule_tz()))
|
|
||||||
scheduler.add_job(
|
|
||||||
generate_all_stock_reports,
|
|
||||||
trigger=stock_trigger,
|
|
||||||
id="stock_report",
|
|
||||||
name=f"个股日报 {stock_h:02d}:{stock_m:02d}",
|
|
||||||
)
|
|
||||||
logger.info("已注册定时任务: 个股日报 每天 {:02d}:{:02d}", stock_h, stock_m)
|
|
||||||
else:
|
|
||||||
logger.info("STOCK_REPORT_TIME 为空, 已禁用个股日报")
|
|
||||||
|
|
||||||
# 优雅退出
|
# 优雅退出
|
||||||
def _shutdown(signum: int, frame: Any) -> None:
|
def _shutdown(signum: int, frame: Any) -> None:
|
||||||
@@ -158,7 +227,19 @@ def _daemon(args: argparse.Namespace) -> int:
|
|||||||
signal.signal(signal.SIGTERM, _shutdown)
|
signal.signal(signal.SIGTERM, _shutdown)
|
||||||
|
|
||||||
scheduler.start()
|
scheduler.start()
|
||||||
logger.info("调度器已启动,等待触发... (按 Ctrl+C 退出)")
|
# 配置热同步:每 30s 重新计算 SCHEDULE_TIMES / CNINFO_SCHEDULE_TIME / STOCK_REPORT_TIME
|
||||||
|
scheduler.add_job(
|
||||||
|
_sync_jobs,
|
||||||
|
"interval",
|
||||||
|
seconds=30,
|
||||||
|
args=[scheduler],
|
||||||
|
id="_config_watch",
|
||||||
|
name="配置热同步",
|
||||||
|
replace_existing=True,
|
||||||
|
)
|
||||||
|
# .env 热加载监听:provider / model / key / base_url 等改动无需重启
|
||||||
|
start_env_watcher()
|
||||||
|
logger.info("调度器已启动,等待触发... (按 Ctrl+C 退出;改 .env 无需重启)")
|
||||||
|
|
||||||
# 启动时检查是否有因重启/宕机错过的定时任务,30 分钟内补跑
|
# 启动时检查是否有因重启/宕机错过的定时任务,30 分钟内补跑
|
||||||
now = tz_now()
|
now = tz_now()
|
||||||
@@ -203,7 +284,8 @@ def main() -> int:
|
|||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
_setup_logger(args.log_level)
|
_setup_logger(args.log_level)
|
||||||
load_dotenv()
|
# .env 热加载(改文件后常驻进程无需重启;--once 也会即时读取最新配置)
|
||||||
|
ensure_env_loaded()
|
||||||
|
|
||||||
if args.once:
|
if args.once:
|
||||||
return _once(args)
|
return _once(args)
|
||||||
|
|||||||
@@ -0,0 +1,217 @@
|
|||||||
|
"""配置热加载测试:改 .env / llm_models.yaml 后无需重启即生效。
|
||||||
|
|
||||||
|
覆盖三层:
|
||||||
|
- configs/loader.py YAML 场景((mtime, size) 失效缓存)
|
||||||
|
- configs/runtime_env .env(热更新 / 删除语义 / 外部显式覆盖优先)
|
||||||
|
- scripts/run_scheduler 定时任务热同步(改 SCHEDULE_TIMES 等)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from configs import loader, runtime_env
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean_caches() -> None:
|
||||||
|
"""每个用例前后都清干净,避免托管的环境变量污染其它测试。"""
|
||||||
|
loader.clear_cache()
|
||||||
|
runtime_env.reset_cache()
|
||||||
|
yield
|
||||||
|
loader.clear_cache()
|
||||||
|
runtime_env.reset_cache()
|
||||||
|
|
||||||
|
|
||||||
|
def _use_env_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, content: str) -> Path:
|
||||||
|
"""把 runtime_env 指向临时 .env,返回其路径。"""
|
||||||
|
env = tmp_path / ".env"
|
||||||
|
env.write_text(content, encoding="utf-8")
|
||||||
|
monkeypatch.setenv(runtime_env.ENV_FILE_OVERRIDE, str(env))
|
||||||
|
runtime_env.reset_cache()
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# YAML 场景热加载
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_yaml_scene_hot_reload(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""改 llm_models.yaml 后,下一次 load_scene_config 立即读到新值。"""
|
||||||
|
cfg = tmp_path / "llm_models.yaml"
|
||||||
|
cfg.write_text(
|
||||||
|
"scenes:\n daily_report:\n provider: qwen\n model: qwen3.6-flash\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
monkeypatch.setenv(loader.MODELS_CONFIG_OVERRIDE, str(cfg))
|
||||||
|
|
||||||
|
assert loader.load_scene_config("daily_report")["model"] == "qwen3.6-flash"
|
||||||
|
|
||||||
|
# 内容长度不同,保证 (mtime, size) 签名一定变化
|
||||||
|
cfg.write_text(
|
||||||
|
"scenes:\n daily_report:\n provider: qwen\n model: deepseek-v4.1-flash\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
# 不重启、不 clear_cache 也应生效
|
||||||
|
assert loader.load_scene_config("daily_report")["model"] == "deepseek-v4.1-flash"
|
||||||
|
|
||||||
|
|
||||||
|
def test_yaml_scene_removed_falls_back(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""场景被删掉后返回空 dict(由调用方回退 .env)。"""
|
||||||
|
cfg = tmp_path / "llm_models.yaml"
|
||||||
|
cfg.write_text("scenes:\n stock_report:\n model: x\n", encoding="utf-8")
|
||||||
|
monkeypatch.setenv(loader.MODELS_CONFIG_OVERRIDE, str(cfg))
|
||||||
|
assert loader.load_scene_config("stock_report")["model"] == "x"
|
||||||
|
|
||||||
|
cfg.write_text("scenes: {}\n", encoding="utf-8")
|
||||||
|
assert loader.load_scene_config("stock_report") == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_yaml_missing_file_returns_empty(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
monkeypatch.setenv(loader.MODELS_CONFIG_OVERRIDE, str(tmp_path / "nope.yaml"))
|
||||||
|
assert loader.load_scene_config("daily_report") == {}
|
||||||
|
assert loader.load_defaults() == {}
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# .env 热加载
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_env_hot_reload(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""改 .env 后 env_get 立即读到新值。"""
|
||||||
|
env = _use_env_file(tmp_path, monkeypatch, "HOT_RELOAD_KEY=one\n")
|
||||||
|
monkeypatch.delenv("HOT_RELOAD_KEY", raising=False)
|
||||||
|
|
||||||
|
assert runtime_env.env_get("HOT_RELOAD_KEY") == "one"
|
||||||
|
|
||||||
|
env.write_text("HOT_RELOAD_KEY=two-longer-value\n", encoding="utf-8")
|
||||||
|
assert runtime_env.env_get("HOT_RELOAD_KEY") == "two-longer-value"
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_removed_key_is_unset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""从 .env 删掉的托管键要同步从 os.environ 移除。"""
|
||||||
|
env = _use_env_file(tmp_path, monkeypatch, "HOT_REMOVE_KEY=abc\n")
|
||||||
|
monkeypatch.delenv("HOT_REMOVE_KEY", raising=False)
|
||||||
|
assert runtime_env.env_get("HOT_REMOVE_KEY") == "abc"
|
||||||
|
assert os.environ["HOT_REMOVE_KEY"] == "abc"
|
||||||
|
|
||||||
|
env.write_text("OTHER_KEY=1\n", encoding="utf-8")
|
||||||
|
assert runtime_env.env_get("HOT_REMOVE_KEY") is None
|
||||||
|
assert "HOT_REMOVE_KEY" not in os.environ
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_empty_value_is_unset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
_use_env_file(tmp_path, monkeypatch, "HOT_EMPTY_KEY=\n")
|
||||||
|
monkeypatch.delenv("HOT_EMPTY_KEY", raising=False)
|
||||||
|
assert runtime_env.env_get("HOT_EMPTY_KEY", "fallback") == "fallback"
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_process_env_wins(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""进程环境显式设置且与文件不同 → 不被 .env 覆盖(一次性覆盖语义)。"""
|
||||||
|
env = _use_env_file(tmp_path, monkeypatch, "HOT_OVERRIDE_KEY=from-file\n")
|
||||||
|
monkeypatch.setenv("HOT_OVERRIDE_KEY", "from-shell")
|
||||||
|
|
||||||
|
assert runtime_env.env_get("HOT_OVERRIDE_KEY") == "from-shell"
|
||||||
|
|
||||||
|
env.write_text("HOT_OVERRIDE_KEY=from-file-changed\n", encoding="utf-8")
|
||||||
|
assert runtime_env.env_get("HOT_OVERRIDE_KEY") == "from-shell"
|
||||||
|
|
||||||
|
|
||||||
|
def test_llm_config_follows_env_hot_reload(
|
||||||
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""端到端:改 .env 里的 QWEN_MODEL,load_llm_config 立即用新模型。"""
|
||||||
|
from llm.client import load_llm_config
|
||||||
|
|
||||||
|
contents = (
|
||||||
|
"LLM_PROVIDER=qwen\n"
|
||||||
|
"QWEN_API_KEY=sk-test\n"
|
||||||
|
"QWEN_BASE_URL=https://token-plan.example/compatible-mode/v1\n"
|
||||||
|
"QWEN_MODEL=qwen3.6-flash\n"
|
||||||
|
)
|
||||||
|
env = _use_env_file(tmp_path, monkeypatch, contents)
|
||||||
|
for key in ("LLM_PROVIDER", "QWEN_API_KEY", "QWEN_BASE_URL", "QWEN_MODEL"):
|
||||||
|
monkeypatch.delenv(key, raising=False)
|
||||||
|
|
||||||
|
cfg = load_llm_config(scene="")
|
||||||
|
assert cfg.provider == "qwen"
|
||||||
|
assert cfg.model == "qwen3.6-flash"
|
||||||
|
|
||||||
|
env.write_text(contents.replace("qwen3.6-flash", "deepseek-v4.1-flash"), encoding="utf-8")
|
||||||
|
assert load_llm_config(scene="").model == "deepseek-v4.1-flash"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# 调度任务热同步
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_desired_jobs_tracks_env(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""SCHEDULE_TIMES / CNINFO_SCHEDULE_TIME / STOCK_REPORT_TIME 改完立即反映。"""
|
||||||
|
from scripts.run_scheduler import _desired_jobs
|
||||||
|
|
||||||
|
_use_env_file(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
"SCHEDULE_TIMES=09:15\nCNINFO_SCHEDULE_TIME=05:45\nSTOCK_REPORT_TIME=\n",
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs = _desired_jobs()
|
||||||
|
assert set(jobs) == {"pipeline_0915", "pipeline_cninfo"}
|
||||||
|
# 当天唯一次数 → 附带日报步骤
|
||||||
|
assert jobs["pipeline_0915"]["steps"][-1] == "report"
|
||||||
|
assert jobs["pipeline_cninfo"]["hour"] == 5
|
||||||
|
assert jobs["pipeline_cninfo"]["minute"] == 45
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_jobs_adds_and_removes(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""_sync_jobs 只按需增删:加时间点、启停个股日报、幂等。"""
|
||||||
|
from apscheduler.schedulers.background import BackgroundScheduler
|
||||||
|
|
||||||
|
import scripts.run_scheduler as rs
|
||||||
|
|
||||||
|
env = _use_env_file(
|
||||||
|
tmp_path,
|
||||||
|
monkeypatch,
|
||||||
|
"SCHEDULE_TIMES=09:15\nCNINFO_SCHEDULE_TIME=05:45\nSTOCK_REPORT_TIME=\n",
|
||||||
|
)
|
||||||
|
rs._jobs_sig = None
|
||||||
|
|
||||||
|
sched = BackgroundScheduler()
|
||||||
|
assert rs._sync_jobs(sched) is True
|
||||||
|
assert {j.id for j in sched.get_jobs()} == {"pipeline_0915", "pipeline_cninfo"}
|
||||||
|
|
||||||
|
# 新增一个时间点 + 启用个股日报
|
||||||
|
env.write_text(
|
||||||
|
"SCHEDULE_TIMES=09:15,16:40\nCNINFO_SCHEDULE_TIME=05:45\nSTOCK_REPORT_TIME=07:30\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
assert rs._sync_jobs(sched) is True
|
||||||
|
ids = {j.id for j in sched.get_jobs()}
|
||||||
|
assert ids == {"pipeline_0915", "pipeline_1640", "pipeline_cninfo", "stock_report"}
|
||||||
|
|
||||||
|
# 配置未变 → 不重复变更
|
||||||
|
assert rs._sync_jobs(sched) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_hhmm_falls_back_on_bad_input() -> None:
|
||||||
|
from scripts.run_scheduler import _parse_hhmm
|
||||||
|
|
||||||
|
assert _parse_hhmm("06:30", (0, 0)) == (6, 30)
|
||||||
|
assert _parse_hhmm("99:99", (6, 30)) == (6, 30)
|
||||||
|
assert _parse_hhmm("garbage", (7, 0)) == (7, 0)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# 其它模块复用热读取
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_reporter_days_back_is_lazy(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""日报的 STOCK_REPORT_DAYS 走懒读取,不再冻结在模块常量里。"""
|
||||||
|
from scheduler import reporter
|
||||||
|
|
||||||
|
_use_env_file(tmp_path, monkeypatch, "STOCK_REPORT_DAYS=15\n")
|
||||||
|
monkeypatch.delenv("STOCK_REPORT_DAYS", raising=False)
|
||||||
|
assert reporter._cninfo_days_back() == 15
|
||||||
+58
-1
@@ -356,7 +356,23 @@ def test_load_llm_config_deepseek_from_env(monkeypatch: pytest.MonkeyPatch) -> N
|
|||||||
assert "deepseek" in cfg.base_url
|
assert "deepseek" in cfg.base_url
|
||||||
|
|
||||||
|
|
||||||
def test_load_llm_config_qwen_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_load_llm_config_qwen_from_env(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path
|
||||||
|
) -> None:
|
||||||
|
"""qwen 的 key 兜底链:QWEN_API_KEY 缺失时回退 DASHSCOPE_API_KEY。
|
||||||
|
|
||||||
|
真实 .env 里带有 QWEN_API_KEY,会盖过 DASHSCOPE_API_KEY,所以本用例先把
|
||||||
|
热加载指向一个空 .env,测完再切回真实 .env。
|
||||||
|
"""
|
||||||
|
from configs import runtime_env
|
||||||
|
|
||||||
|
empty = tmp_path / ".env"
|
||||||
|
empty.write_text("", encoding="utf-8")
|
||||||
|
monkeypatch.setenv(runtime_env.ENV_FILE_OVERRIDE, str(empty))
|
||||||
|
runtime_env.reset_cache()
|
||||||
|
monkeypatch.delenv("QWEN_API_KEY", raising=False)
|
||||||
|
monkeypatch.delenv("QWEN_BASE_URL", raising=False)
|
||||||
|
|
||||||
monkeypatch.setenv("LLM_PROVIDER", "qwen")
|
monkeypatch.setenv("LLM_PROVIDER", "qwen")
|
||||||
monkeypatch.setenv("DASHSCOPE_API_KEY", "sk-test-qwen")
|
monkeypatch.setenv("DASHSCOPE_API_KEY", "sk-test-qwen")
|
||||||
monkeypatch.setenv("QWEN_MODEL", "qwen-plus")
|
monkeypatch.setenv("QWEN_MODEL", "qwen-plus")
|
||||||
@@ -367,6 +383,11 @@ def test_load_llm_config_qwen_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
|||||||
assert cfg.model == "qwen-plus"
|
assert cfg.model == "qwen-plus"
|
||||||
assert "dashscope" in cfg.base_url or "aliyuncs" in cfg.base_url
|
assert "dashscope" in cfg.base_url or "aliyuncs" in cfg.base_url
|
||||||
|
|
||||||
|
# 切回真实 .env,避免影响后续用例(DB / Qdrant 等直接读 os.environ 的测试)
|
||||||
|
monkeypatch.delenv(runtime_env.ENV_FILE_OVERRIDE, raising=False)
|
||||||
|
runtime_env.reset_cache()
|
||||||
|
runtime_env.ensure_env_loaded(force=True)
|
||||||
|
|
||||||
|
|
||||||
def test_load_llm_config_unknown_provider_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_load_llm_config_unknown_provider_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
@@ -498,3 +519,39 @@ def test_load_llm_config_scene_max_attempts_zero() -> None:
|
|||||||
assert _pick_int({"max_attempts": 0}, "max_attempts", 3) == 0
|
assert _pick_int({"max_attempts": 0}, "max_attempts", 3) == 0
|
||||||
assert _pick_int({"max_attempts": ""}, "max_attempts", 3) == 3
|
assert _pick_int({"max_attempts": ""}, "max_attempts", 3) == 3
|
||||||
assert _pick_int({}, "max_attempts", 3) == 3
|
assert _pick_int({}, "max_attempts", 3) == 3
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# max_tokens 场景配置(推理模型 reasoning 与正文共用预算,回归 9-25 无 AI 摘要)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_load_llm_config_scene_max_tokens(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""scenes.<scene>.max_tokens 应写入 LLMConfig;未配置则为 None(调用方走默认)。"""
|
||||||
|
monkeypatch.setenv("QWEN_API_KEY", "sk-qwen")
|
||||||
|
_patch_scene(monkeypatch, {"provider": "qwen", "model": "deepseek-v4.1-flash",
|
||||||
|
"max_tokens": 4000})
|
||||||
|
cfg = load_llm_config(scene="daily_report")
|
||||||
|
assert cfg.max_tokens == 4000
|
||||||
|
|
||||||
|
_patch_scene(monkeypatch, {"provider": "qwen", "model": "deepseek-v4.1-flash"})
|
||||||
|
assert load_llm_config(scene="daily_report").max_tokens is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_pick_optional_int() -> None:
|
||||||
|
"""_pick_optional_int:未配置/空值/非法 → None;数字字符串可解析。"""
|
||||||
|
from llm.client import _pick_optional_int
|
||||||
|
|
||||||
|
assert _pick_optional_int({"max_tokens": 4000}, "max_tokens") == 4000
|
||||||
|
assert _pick_optional_int({"max_tokens": "4000"}, "max_tokens") == 4000
|
||||||
|
assert _pick_optional_int({"max_tokens": ""}, "max_tokens") is None
|
||||||
|
assert _pick_optional_int({"max_tokens": "abc"}, "max_tokens") is None
|
||||||
|
assert _pick_optional_int({}, "max_tokens") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_yaml_daily_report_max_tokens_has_reasoning_headroom() -> None:
|
||||||
|
"""真实配置的日报预算必须大于旧的 1500(为 reasoning token 留余量)。"""
|
||||||
|
from configs.loader import load_scene_config
|
||||||
|
|
||||||
|
scene = load_scene_config("daily_report")
|
||||||
|
max_tokens = int(scene.get("max_tokens") or 0)
|
||||||
|
assert max_tokens >= 4000, f"daily_report.max_tokens 过小: {max_tokens}"
|
||||||
|
|||||||
@@ -132,8 +132,8 @@ class TestLlmCallRetry:
|
|||||||
|
|
||||||
def test_retry_then_success(self, monkeypatch) -> None:
|
def test_retry_then_success(self, monkeypatch) -> None:
|
||||||
import scheduler.reporter as rep
|
import scheduler.reporter as rep
|
||||||
monkeypatch.setattr(rep, "_LLM_RETRY_TIMES", 3)
|
monkeypatch.setattr(rep, "_llm_retry_times", lambda: 3)
|
||||||
monkeypatch.setattr(rep, "_LLM_RETRY_BACKOFF_SEC", 0.01)
|
monkeypatch.setattr(rep, "_llm_retry_backoff_sec", lambda: 0.01)
|
||||||
client, n = self._fake_client(2) # 前 2 次失败,第 3 次成功
|
client, n = self._fake_client(2) # 前 2 次失败,第 3 次成功
|
||||||
out = rep._llm_call(client, self._cfg(), "p")
|
out = rep._llm_call(client, self._cfg(), "p")
|
||||||
assert out == "今日要点摘要"
|
assert out == "今日要点摘要"
|
||||||
@@ -141,8 +141,8 @@ class TestLlmCallRetry:
|
|||||||
|
|
||||||
def test_exhausts_retries_raises(self, monkeypatch) -> None:
|
def test_exhausts_retries_raises(self, monkeypatch) -> None:
|
||||||
import scheduler.reporter as rep
|
import scheduler.reporter as rep
|
||||||
monkeypatch.setattr(rep, "_LLM_RETRY_TIMES", 2)
|
monkeypatch.setattr(rep, "_llm_retry_times", lambda: 2)
|
||||||
monkeypatch.setattr(rep, "_LLM_RETRY_BACKOFF_SEC", 0.01)
|
monkeypatch.setattr(rep, "_llm_retry_backoff_sec", lambda: 0.01)
|
||||||
client, n = self._fake_client(99) # 一直失败
|
client, n = self._fake_client(99) # 一直失败
|
||||||
with pytest.raises(ConnectionError):
|
with pytest.raises(ConnectionError):
|
||||||
rep._llm_call(client, self._cfg(), "p")
|
rep._llm_call(client, self._cfg(), "p")
|
||||||
@@ -171,6 +171,84 @@ class TestLlmCallRetry:
|
|||||||
assert n["count"] == 3 # 2 块 + 1 次合并
|
assert n["count"] == 3 # 2 块 + 1 次合并
|
||||||
|
|
||||||
|
|
||||||
|
class TestReasoningBudgetEscalation:
|
||||||
|
"""推理模型占满 max_tokens 导致正文为空时的预算升级(回归 9-25 无 AI 摘要)。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _empty_then_ok_client(empty_times: int):
|
||||||
|
"""前 empty_times 次返回空正文 + finish_reason=length,之后返回正常摘要。"""
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
seen: list[int] = []
|
||||||
|
|
||||||
|
class Completions:
|
||||||
|
def create(self, **kwargs):
|
||||||
|
seen.append(kwargs.get("max_tokens"))
|
||||||
|
if len(seen) <= empty_times:
|
||||||
|
return SimpleNamespace(
|
||||||
|
choices=[SimpleNamespace(
|
||||||
|
message=SimpleNamespace(content=""),
|
||||||
|
finish_reason="length",
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
return SimpleNamespace(
|
||||||
|
choices=[SimpleNamespace(
|
||||||
|
message=SimpleNamespace(content="恢复后的摘要"),
|
||||||
|
finish_reason="stop",
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
|
||||||
|
return SimpleNamespace(chat=SimpleNamespace(completions=Completions())), seen
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _cfg(max_tokens: int | None = None):
|
||||||
|
from llm.client import LLMConfig
|
||||||
|
|
||||||
|
return LLMConfig(
|
||||||
|
provider="qwen", model="deepseek-v4.1-flash",
|
||||||
|
api_key="sk-test", base_url="https://example.invalid/v1",
|
||||||
|
temperature=0.3, max_tokens=max_tokens,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_empty_content_escalates_and_recovers(self, monkeypatch) -> None:
|
||||||
|
"""正文为空时自动加倍预算并最终拿到摘要(不再静默返回空)。"""
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
monkeypatch.setattr(rep, "_llm_retry_times", lambda: 3)
|
||||||
|
monkeypatch.setattr(rep, "_llm_retry_backoff_sec", lambda: 0.01)
|
||||||
|
client, seen = self._empty_then_ok_client(1)
|
||||||
|
out = rep._llm_call(client, self._cfg(4000), "p")
|
||||||
|
assert out == "恢复后的摘要"
|
||||||
|
assert seen == [4000, 8000] # 首次失败后预算翻倍
|
||||||
|
|
||||||
|
def test_scene_max_tokens_wins_over_default(self, monkeypatch) -> None:
|
||||||
|
"""场景 max_tokens 生效;未配置时回退代码默认 4000。"""
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
monkeypatch.setattr(rep, "_llm_retry_times", lambda: 1)
|
||||||
|
client, seen = self._empty_then_ok_client(0)
|
||||||
|
rep._llm_call(client, self._cfg(6000), "p")
|
||||||
|
assert seen == [6000]
|
||||||
|
client, seen = self._empty_then_ok_client(0)
|
||||||
|
rep._llm_call(client, self._cfg(), "p")
|
||||||
|
assert seen == [rep.DEFAULT_SUMMARY_MAX_TOKENS]
|
||||||
|
|
||||||
|
def test_all_empty_returns_blank_without_raising(self, monkeypatch) -> None:
|
||||||
|
"""预算升级用尽仍为空时降级返回空串(日报仍可入库,不抛异常)。"""
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
monkeypatch.setattr(rep, "_llm_retry_times", lambda: 3)
|
||||||
|
monkeypatch.setattr(rep, "_llm_retry_backoff_sec", lambda: 0.01)
|
||||||
|
client, seen = self._empty_then_ok_client(99)
|
||||||
|
out = rep._llm_call(client, self._cfg(4000), "p")
|
||||||
|
assert out == ""
|
||||||
|
# 4000 → 8000 → 16000(受 MAX_SUMMARY_MAX_TOKENS 上限约束)
|
||||||
|
assert seen == [4000, 8000, 16000]
|
||||||
|
|
||||||
|
def test_budget_never_exceeds_cap(self) -> None:
|
||||||
|
"""升级预算不超过 MAX_SUMMARY_MAX_TOKENS,避免无限放大。"""
|
||||||
|
import scheduler.reporter as rep
|
||||||
|
assert rep.MAX_SUMMARY_MAX_TOKENS == 16000
|
||||||
|
assert rep.DEFAULT_SUMMARY_MAX_TOKENS > rep.DEFAULT_SUMMARY_CHUNK_MAX_TOKENS
|
||||||
|
|
||||||
|
|
||||||
class TestCollectXwlb:
|
class TestCollectXwlb:
|
||||||
"""_collect_xwlb 取数逻辑:应查询日报前一日(已播出的联播),并跳过内容提要。"""
|
"""_collect_xwlb 取数逻辑:应查询日报前一日(已播出的联播),并跳过内容提要。"""
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
|
||||||
import uuid
|
import uuid
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -30,6 +29,8 @@ from qdrant_client.http.models import (
|
|||||||
VectorParams,
|
VectorParams,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
from configs.runtime_env import env_get
|
||||||
|
|
||||||
from .models import CollectionInfo, SearchFilter, SearchResult
|
from .models import CollectionInfo, SearchFilter, SearchResult
|
||||||
|
|
||||||
# 默认配置
|
# 默认配置
|
||||||
@@ -52,10 +53,8 @@ def url_hash_to_uuid(url_hash: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _read_env(key: str, default: str | None = None) -> str | None:
|
def _read_env(key: str, default: str | None = None) -> str | None:
|
||||||
val = os.environ.get(key)
|
"""读取环境变量(先热加载 .env,改文件后无需重启进程)。"""
|
||||||
if val is None or val.strip() == "":
|
return env_get(key, default)
|
||||||
return default
|
|
||||||
return val.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def make_qdrant_client(
|
def make_qdrant_client(
|
||||||
|
|||||||
Reference in New Issue
Block a user