simon/qlib
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 股息率案例口径 + 策略库与图表统一 + 回测存档完整化

Browse Source 
汇总三轮未提交的开发（每轮均在本机 MariaDB + 真实浏览器上验证）：

1) 股息率案例（全市场股息率最高 n 只，默认 20，每 m 月择股）
   - 新增日频估值表 daily_basic + 迁移；股息率因子（dv_ratio / dividend_yield / TTM）
   - 名称历史表 stock_name_history：剔除 ST 按**择股日当时名称**判定，消除
     「曾高股息后 ST」的股息陷阱（实测 3.70pp 偏差）
   - 区间择股/调仓双周期（m 择股 / y 调仓）、指数成分与白名单、停牌近似剔除
   - 复权因子口径核对（4,164,742 行、缺失 0.0%）、收盘价成交与涨跌停拦单
   - 案例实测：2020-01-01~2026-09-04 总收益 +24.86%（年化 3.52%、回撤 -28.58%）

2) 策略库与前端统一
   - strategy 表 + CRUD/PUT 原地更新 + `describe_strategy` 按 spec 真实推导
     「一句话说明 + 计算公式 + 执行步骤 + 注意事项」（与引擎实执行规则同源）
   - 任何出现股票代码处都成对显示名称且可点击进个股页
   - 全站图表基座统一 TradingView Lightweight Charts（ECharts 依赖、
     锁文件、组件与文档标注一并清除），买卖点标记只落在真实交易日上

3) 回测存档完整化（可往复查看）
   - 同步端点（POST /api/backtests、/api/factor-tests）此前完全不落库 → 现在同样归档，
     归档 id 经响应头 X-Experiment-Id 返回（不破坏 response_model）
   - data_version 首次真实写入（数据快照指纹：最新交易日 + 各表规模）
   - 个股收益曲线默认**全量保存**（此前硬截断 60 只）；超出体积预算才裁剪，
     并写 archive_meta（机器可读）+ unimplemented（人可读）如实标注
   - 列表 kind/q 过滤 + X-Total-Count（此前 limit=50 静默截断）、DELETE 归档
   - 只读归档页 /experiments/{id}（Server Component，SSR 直出**选股条件**与
     **交易执行依据**）；结果视图按 kind 分发（backtest/factor_test/selection），
     非回测归档不套用回测口径
   - 新增 CLI：prune_experiments（保留策略，默认 dry-run）、
     restore_experiment_from_job（从 Job 副本按原 id 重建被删的历史归档，默认 dry-run）

门禁：pytest 388 passed、ruff All checks passed、tsc 0 错误、图表单测 7 passed、
next build 成功、契约脚本 verify_strategy_workspace 59/59（含按 kind 逐类验证归档页）。
This commit is contained in:
Simon
2026-09-20 07:31:04 +08:00
parent 7e15b7251e
commit 23972e7063
112 changed files with 17908 additions and 3893 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/application/services/job_executor.py
+95 -48
View File
@@ -18,15 +18,12 @@ import os
import subprocess
import sys
import threading
import uuid
from collections.abc import Callable
from datetime import datetime
from pathlib import Path
from app.application.services.experiment_archive import archive_experiment, new_id
from app.domain.entities.research import (
BacktestResult,
ExperimentRecord,
FactorTestReport,
JobRecord,
JobStatus,
ResearchSpec,
@@ -42,38 +39,6 @@ _active_jobs: dict[str, subprocess.Popen | None] = {}
_active_lock = threading.Lock()
def new_id(prefix: str) -> str:
return f"{prefix}-{uuid.uuid4().hex[:8].upper()}"
def _git_short_rev() -> str | None:
try:
out = subprocess.run(
["git", "rev-parse", "--short", "HEAD"],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=5,
check=False,
)
return out.stdout.strip() or None
except Exception: # noqa: BLE001
return None
def _summary_text(kind: str, result) -> str | None:
from app.domain.entities.selection import SelectionResult
if kind == "backtest" and isinstance(result, BacktestResult):
s = result.summary
return f"总收益 {s.total_return_pct:.2f}% · 年化 {s.annual_return_pct:.2f}% · 回撤 {s.max_drawdown_pct:.2f}%"
if isinstance(result, FactorTestReport):
return f"IC {result.ic_mean:.4f} · RankIC {result.rank_ic_mean:.4f} · 样本 {result.sample_days} 日"
if kind == "selection" and isinstance(result, SelectionResult):
return f"as_of {result.as_of_date} · 选出 {result.statistics.selected} / 评估 {result.statistics.evaluated}"
return None
def _execute_inner(
job_id: str,
*,
@@ -83,7 +48,22 @@ def _execute_inner(
stock_repo_factory: Callable,
daily_repo_factory: Callable,
engine,
basic_repo_factory: Callable | None = None,
financial_repo_factory: Callable | None = None,
index_repo_factory: Callable | None = None,
name_repo_factory: Callable | None = None,
) -> None:
"""执行 Job。
可选工厂的语义（AGENT.md §24：不做静默降级）：
- `basic_repo_factory`：daily_basic（dv_ratio 等每日指标）—— 因子/条件引用时必需
- 复权（qfq/hfq）折算在行情仓储 SQL 内完成，无需 adjust_factor 工厂
- `financial_repo_factory`：财务表 —— 条件引用 fundamental.* 时必需
- `index_repo_factory`：指数成分 —— universe.index_code 时必需
- `name_repo_factory`：名称变更历史 —— universe.exclude_st 时点口径；未注入则
回退最新名称（旧行为），结果如实标注残余偏差
未注入且 spec 需要时，由 Service 抛出明确错误（而非返回空结果）。
"""
with session_factory() as session:
job_repo = job_repo_factory(session)
experiment_repo = experiment_repo_factory(session)
@@ -96,18 +76,34 @@ def _execute_inner(
session.commit()
try:
is_selection = job.kind == "selection"
basic_repo = basic_repo_factory(session) if basic_repo_factory else None
index_repo = index_repo_factory(session) if index_repo_factory else None
name_repo = name_repo_factory(session) if name_repo_factory else None
if is_selection:
from app.application.services.selection_service import SelectionService
from app.domain.entities.selection import SelectionQuery
spec = SelectionQuery.model_validate_json(job.spec_json)
service = SelectionService(
stock_repo_factory(session), daily_repo_factory(session)
stock_repo_factory(session),
daily_repo_factory(session),
financial_repo_factory(session) if financial_repo_factory else None,
index_repo=index_repo,
basic_repo=basic_repo,
name_repo=name_repo,
)
else:
spec = ResearchSpec.model_validate_json(job.spec_json)
service = ResearchService(
stock_repo_factory(session), daily_repo_factory(session), engine
stock_repo_factory(session),
daily_repo_factory(session),
engine,
index_repo=index_repo,
basic_repo=basic_repo,
financial_repo=(
financial_repo_factory(session) if financial_repo_factory else None
),
name_repo=name_repo,
)
def _set_stage(name: str) -> None:
@@ -130,20 +126,20 @@ def _execute_inner(
else:
result = service.run_factor_test(spec, on_stage=_set_stage)
result_json = json.dumps(result.model_dump(mode="json"), ensure_ascii=False)
experiment = ExperimentRecord(
id=new_id("EXP"),
experiment = archive_experiment(
session=session,
kind=job.kind,
spec_json=job.spec_json,
result_json=result_json,
summary_text=_summary_text(job.kind, result),
code_version=_git_short_rev(),
result=result,
job_id=job.id,
created_at=datetime.now(),
experiment_repo=experiment_repo,
)
experiment_repo.save(experiment)
job.result_json = result_json
# 完整结果**只在 experiment 存一份**（此前 job.result_json 与
# experiment.result_json 各存一份相同内容，完整存档后 2× 浪费）。
# GET /api/jobs/{id} 经 job.experiment_id 回读 experiment；
# 老记录（result_json 有值、experiment_id 为空）仍走 job 回退解码。
job.result_json = None
job.experiment_id = experiment.id
job.status = JobStatus.SUCCESS
job.error = None
@@ -173,6 +169,10 @@ def execute_job(
stock_repo_factory: Callable,
daily_repo_factory: Callable,
engine,
basic_repo_factory: Callable | None = None,
financial_repo_factory: Callable | None = None,
index_repo_factory: Callable | None = None,
name_repo_factory: Callable | None = None,
) -> None:
"""入口包装：任何未预期异常都将 Job 标记 failed（防止卡在 queued/running）。"""
try:
@@ -184,6 +184,10 @@ def execute_job(
stock_repo_factory=stock_repo_factory,
daily_repo_factory=daily_repo_factory,
engine=engine,
basic_repo_factory=basic_repo_factory,
financial_repo_factory=financial_repo_factory,
index_repo_factory=index_repo_factory,
name_repo_factory=name_repo_factory,
)
except Exception as exc: # noqa: BLE001
try:
@@ -202,12 +206,18 @@ def execute_job(
def default_factories() -> dict:
"""后台执行所需的独立 Session / Repository / 引擎装配（跨请求生命周期）。"""
from app.infrastructure.persistence.sqlalchemy.repositories.index_impl import (
SqlAlchemyIndexConstituentRepository,
)
from app.infrastructure.persistence.sqlalchemy.repositories.jobs_impl import (
SqlAlchemyExperimentRepository,
SqlAlchemyJobRepository,
)
from app.infrastructure.persistence.sqlalchemy.repositories.market_impl import (
SqlAlchemyDailyBarRepository,
SqlAlchemyDailyBasicRepository,
SqlAlchemyFinancialRepository,
SqlAlchemyStockNameHistoryRepository,
SqlAlchemyStockRepository,
)
from app.infrastructure.persistence.sqlalchemy.session import SessionLocal
@@ -219,6 +229,13 @@ def default_factories() -> dict:
"experiment_repo_factory": lambda s: SqlAlchemyExperimentRepository(s),
"stock_repo_factory": lambda s: SqlAlchemyStockRepository(s),
"daily_repo_factory": lambda s: SqlAlchemyDailyBarRepository(s),
# 新增装配（daily_basic / adjust_factor / 财务 / 指数成分）：
# 缺失时对应能力（dv_ratio 因子、qfq/hfq 复权、fundamental 条件、指数成份池）
# 会由 Service 明确报错，绝不静默降级
"basic_repo_factory": lambda s: SqlAlchemyDailyBasicRepository(s),
"financial_repo_factory": lambda s: SqlAlchemyFinancialRepository(s),
"index_repo_factory": lambda s: SqlAlchemyIndexConstituentRepository(s),
"name_repo_factory": lambda s: SqlAlchemyStockNameHistoryRepository(s),
"engine": LocalEngine(),
}
@@ -252,9 +269,21 @@ def submit_and_run(spec: ResearchSpec, *, factories: dict | None = None) -> JobR
stock_repo_factory=facts["stock_repo_factory"],
daily_repo_factory=facts["daily_repo_factory"],
engine=facts["engine"],
basic_repo_factory=facts.get("basic_repo_factory"),
financial_repo_factory=facts.get("financial_repo_factory"),
index_repo_factory=facts.get("index_repo_factory"),
name_repo_factory=facts.get("name_repo_factory"),
)
with session_factory() as session:
done = job_repo(session).get(job.id)
# **仅内存**读透（不落库）：完整结果只存 experiment 一份，但 submit_and_run 的
# 既有调用方（scripts/run_dividend_case.py、agent 工具）习惯从 job.result_json
# 取结果，这里按 experiment_id 回读一次填进返回对象，避免调用方静默拿到空结果。
# 数据库中的 job.result_json 仍然保持 NULL（P1：不重复存第二份）。
if done is not None and done.experiment_id and done.result_json is None:
exp = facts["experiment_repo_factory"](session).get(done.experiment_id)
if exp is not None:
done.result_json = exp.result_json
assert done is not None
return done
@@ -341,6 +370,21 @@ def _mark_failed(
pass
def _subprocess_log_target():
"""研究子进程 stderr 的落盘目标。
原先父进程用 DEVNULL，会吞掉子进程全部诊断（含 run_job 打印的内存上限设置失败
警告），使「如实提示未实现项」落空。改为追加写入 <项目根>/logs/job-subprocess.log。
记日志失败（无权限/磁盘满）绝不能导致 Job 失败 —— 此时退回 DEVNULL。
"""
try:
log_dir = BACKEND_ROOT.parent / "logs"
log_dir.mkdir(parents=True, exist_ok=True)
return (log_dir / "job-subprocess.log").open("a", encoding="utf-8")
except OSError:
return subprocess.DEVNULL
def _run_in_subprocess(
job_id: str,
*,
@@ -361,13 +405,14 @@ def _run_in_subprocess(
env["QLIB_JOB_MEM_LIMIT_GB"] = str(_job_memory_limit_gb())
rc = -1
already_marked = False
errlog = _subprocess_log_target()
try:
proc = subprocess.Popen(
_job_worker_cmd(job_id),
cwd=BACKEND_ROOT,
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
stderr=errlog,
)
with _active_lock:
_active_jobs[job_id] = proc
@@ -381,6 +426,8 @@ def _run_in_subprocess(
)
already_marked = True
finally:
if errlog is not subprocess.DEVNULL:
errlog.close()
_release_slot(job_id)
if rc != 0 and not already_marked:
_mark_failed(