功能：每日动态股票池回测（--mode daily）+ 每日增量同步 + PIT 批量取数层
说明：本提交是工作区中此前的未提交工作（在 14ec0c6 之后产生），**非本次会话所写**，
按用户要求整理并推送。已做安全检查（无明文凭据、无大文件、.env/logs/output 仍被忽略），
并完成可执行范围内的测试验证（见「测试」一节）。
## 新增能力
1) `hdiv backtest --mode daily --start <日期>`
- src/hdiv/backtest/daily.py：两趟式（先逐日选股，再复用既有引擎模拟）
- 每个交易日按当日可见数据重建股票池（PIT），每个交易日判断买卖点
- `pool_exit_action`：hold（只减不加、不因掉出池子而清仓）/ sell（掉出即清仓）
- `profile_on_trade`：买卖决策发生时计算并留痕个股画像，**不区分是否在当日池内**
（卖出/减仓同样留痕，否则「为什么卖」缺证据）
- 与 walkforward 的分工：daily 是一条连续路径的推演，不是过拟合检验；
因此不使用训练段、不冻结分布，阈值口径一律 rolling
- 拒绝 `--universe-run`（daily 的定义就是逐日重筛，冻结池与之矛盾）
2) PIT 批量取数层 src/hdiv/universe/pit.py
- PitRepo 继承 Repo，**只重写取数**（按区块批量预载 + 逐日内存切片），
派生逻辑（最新一期财报合并、单位归一化、支付率口径等）一行不重写
—— 以保证与逐日单点查询**结果等价**
- 候选集预剪枝：用「不可能通过」的边界条件提前排除，文档论证为精确等价而非近似
- src/hdiv/universe/daily.py：每日动态筛选器（仍然调用既有 selector 与四个 Filter）
3) 每日增量同步 `hdiv sync daily`
- src/hdiv/data/sync/daily.py：只抓「库里还没有的那几天」，
按「当日股票数 ≥ 当年规模阈值」判定缺口，不重拉历史、不覆盖既有行；
支持 `--dry-run` 先看待抓清单
- deploy/daily-sync.sh、deploy/install-sync-schedule.sh、
deploy/com.hddiv.sync.plist.example（launchd 每天 17:00）
- 新表 hd_daily_universe（逐日入选成员留痕）+ sql/hd_daily_universe.sql + schema.py
（该表已存在于库中，`ddl plan` 返回 0 个待执行动作）
4) Web 与文档
- 前端支持 daily 模式记录下钻（web/app.js、web/app.css、web/index.html、
web/favicon.svg）
- README / docs/user-guide.md / docs/implementation-status.md 同步更新：
三种回测模式的取舍、daily 的成本说明（6.7 年约 1.5 小时）与调优手段
## 测试
tests/ 共 500 项（新增 tests/test_daily.py 43 项、tests/test_sync_daily.py 36 项）。
已验证通过：
- 排除上述两个新文件的 **421 项：全部通过（pytest 退出码 0）**
- 两个新文件的**非 DB 单元测试 60 项：全部通过**
未能在合理时间内跑完：
- 两个新文件中 **19 项 DB 标记的重型测试**。实测瓶颈是一条**无界全表扫描**：
`SELECT ... FROM hd_cashflow WHERE ann_date <= :asof ORDER BY symbol, end_date, ann_date`
（31 万行，无 symbol/报告期下限）。全量套件跑到 161 项时已耗时 20 分钟、
0 失败，按该速率预计需 3 小时以上，因此改为分档验证。
- 旁证：库中存在 3 次成功的 daily 端到端运行（2026-10-05 10:05 / 10:32 / 11:03，
区间 2024-03-01~03-15），说明该路径可正常完成。
## 已知待改进
- 上述 `hd_cashflow`（及同类「按 ann_date 上界取全历史」）的查询缺
symbol / 报告期下限，是 daily 模式的主要性能瓶颈，建议下一轮优化。
This commit is contained in:
@@ -0,0 +1,557 @@
|
||||
"""每日动态股票池回测（``backtest --mode daily``）。
|
||||
|
||||
**它回答的问题**：给定一个起点（如 2020-01-05），如果从那天起按这套规则
|
||||
**每个交易日**重新选股、每个交易日判断买卖点，实际会发生什么。
|
||||
|
||||
**与 walk-forward 的区别（不是同一件事，也不互相替代）**：
|
||||
|
||||
============================ ========================================== ==========================================
|
||||
``--mode walkforward`` ``--mode daily``
|
||||
============================ ========================================== ==========================================
|
||||
回答的问题 参数在样本外能否复现（过拟合检验） 从某天起连续实盘推演会怎样
|
||||
时间结构 切多个 (train, test) 窗口 一条连续的 [start, latest]
|
||||
阈值口径 测试段**冻结**训练段分布 一律 rolling（PIT 滚动窗口）
|
||||
股票池 每个窗口/调仓日按 asof 重筛 **每个交易日**按 asof 重筛
|
||||
持仓掉出股票池 ——（每个窗口独立重来） pool_exit_action：只减不加 / 清仓
|
||||
产出 多窗口样本外统计 一条净值 + 逐日选股 + 逐笔信号（含画像留痕）
|
||||
============================ ========================================== ==========================================
|
||||
|
||||
daily 模式**没有训练段**，因此不存在「冻结分布」；它检验的不是参数稳定性，
|
||||
而是「动态股票池下这套规则的连续表现」。两者应当**都看**，不要只采信一个。
|
||||
|
||||
**实现要点**：筛选与模拟分成两趟，共用一个 :class:`~hdiv.universe.pit.PitRepo`：
|
||||
|
||||
1. **选股趟**：行情/每日指标按区块（默认一年）批量预载，逐交易日在内存里
|
||||
跑**原有的** ``UniverseSelector`` 与四个 ``Filter``（口径一行未改），
|
||||
记录每天的入选成员后释放区块 —— 内存峰值是一个区块，不随区间长度增长；
|
||||
2. **模拟趟**：把「交易日 → 股票池」交给原有的
|
||||
:class:`~hdiv.backtest.engine.BacktestEngine`，由它完成撮合、成本、分红、
|
||||
公司行为、盯市与绩效。引擎在 daily 模式下只改了四件事：
|
||||
股票池按交易日切换、信号按交易日评估、池外持仓只减不加、买卖决策附画像留痕。
|
||||
|
||||
**没有未来函数**：每个决策日的股票池只用 ``<= 该日`` 的行情、财报（
|
||||
``ann_date <= 该日``）与已实施分红（``imp_ann_date <= 该日`` 且 ``ex_date <= 该日``）。
|
||||
这一点由既有 PIT 纪律与滤网实现保证，daily 模式只是把它的**调用频率**提到每日。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from datetime import date, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from hdiv.core.config import BacktestConfig, DailyConfig, load_config
|
||||
from hdiv.core.errors import DataGapError, HdivError
|
||||
from hdiv.data import db
|
||||
from hdiv.data.repo import data_version
|
||||
from hdiv.data.sync.base import stable_id
|
||||
from hdiv.strategy.registry import StrategyRegistry
|
||||
from hdiv.universe.daily import DailyUniverseScreener, ScreenDay
|
||||
from hdiv.universe.pit import PitRepo
|
||||
|
||||
__all__ = ["DailyRunner", "DailyProgress"]
|
||||
|
||||
#: 实测速率（本机、本数据集），仅用于「预计耗时」提示，不参与任何业务判定。
|
||||
#:
|
||||
#: 2026-10-05 用**同一区间**实测两次（2026-08-01 起 43 个交易日，
|
||||
#: 池内 30~40 只、面板 41 只）：
|
||||
#:
|
||||
#: 选股每 1 日 / 信号每 1 日 → 选股 114 秒 + 模拟 131 秒
|
||||
#: 选股每 5 日 / 信号每 5 日 → 选股 43 秒 + 模拟 约 28 秒
|
||||
#:
|
||||
#: 由此把模拟拆成两项 —— **只有第二项随信号频率下降**：
|
||||
#: ``BASE`` 每个交易日都要做的盯市 / 分红 / 持仓记账（很小）
|
||||
#: ``SIGNAL`` **每次「评估买卖」**的开销（收益率序列 + 逐笔决策画像），
|
||||
#: 与信号频率成反比
|
||||
#: 早先把整段模拟都写成「随交易日数增长」，会把粗粒度方案的耗时**高估**约 1.6 倍。
|
||||
#: 注意这组常数来自 2026 年窗口；全区间面板是 123 只（此处 41 只），实际会更慢。
|
||||
_SEC_PER_SCREEN_DAY = 2.6
|
||||
_SEC_PER_SIM_DAY_BASE = 0.05
|
||||
_SEC_PER_SIM_SIGNAL = 3.0
|
||||
_SEC_PER_CHUNK = 25.0
|
||||
|
||||
|
||||
class DailyProgress:
|
||||
"""逐日选股的进度打印（默认每 N 个交易日一行）。
|
||||
|
||||
为什么必须打印：每日全市场筛选是分钟级到小时级的操作，静默运行会让人
|
||||
无法区分「在算」和「卡死」；而不打印时长的估计，用户也无从判断该不该
|
||||
用更粗的 ``universe_refresh_days``。
|
||||
"""
|
||||
|
||||
def __init__(self, total: int, every: int, *, enabled: bool = True) -> None:
|
||||
self.total = total
|
||||
self.every = max(1, int(every))
|
||||
self.enabled = enabled
|
||||
self.t0 = time.time()
|
||||
self.done = 0
|
||||
|
||||
def tick(self, label: str = "") -> None:
|
||||
self.done += 1
|
||||
if not self.enabled:
|
||||
return
|
||||
if self.done % self.every and self.done != self.total:
|
||||
return
|
||||
el = time.time() - self.t0
|
||||
rate = el / max(self.done, 1)
|
||||
eta = rate * max(self.total - self.done, 0)
|
||||
print(
|
||||
f" 选股 {self.done}/{self.total} {el:,.0f}s"
|
||||
f"（{rate:.2f}s/日，预计剩余 {eta / 60:,.1f} 分钟）{label}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
class DailyRunner:
|
||||
"""每日动态股票池回测的执行器。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
strategy_path: str | Path = "config/strategy/high_dividend_v1.yml",
|
||||
) -> None:
|
||||
db.load_dotenv_once()
|
||||
self.registry = StrategyRegistry()
|
||||
self.strategy = self.registry.load(strategy_path)
|
||||
self.bt: BacktestConfig = load_config("backtest")
|
||||
self.daily: DailyConfig = self.bt.daily
|
||||
|
||||
@classmethod
|
||||
def from_strategy(cls, path: str | Path) -> DailyRunner:
|
||||
return cls(path)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 主流程
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def run(
|
||||
self,
|
||||
*,
|
||||
start: date | None = None,
|
||||
end: date | None = None,
|
||||
persist: bool = True,
|
||||
verbose: bool = True,
|
||||
refresh_pools: bool = False,
|
||||
every_n_days: int | None = None,
|
||||
signal_every_n_days: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""执行每日（或每 N 日）动态股票池回测。
|
||||
|
||||
两个**频率**参数是仅有的速度旋钮，语义不同、代价不同：
|
||||
|
||||
- ``every_n_days``：股票池**重建**频率（交易日）。1 = 每个交易日重筛。
|
||||
只影响「选股」那一趟的耗时（近似线性下降）；两次重建之间池子不变。
|
||||
- ``signal_every_n_days``：买卖**判断**频率（交易日）。1 = 每个交易日判断。
|
||||
影响「模拟」那一趟（信号评估 + 逐笔决策画像），也直接影响交易机会数量。
|
||||
|
||||
两者都**不改变判定规则**，只改变「多久看一次」。所以调大它们得到的是
|
||||
「粗粒度版本」，不是同一策略的加速版 —— 交易机会与换手都会下降，
|
||||
结果不可与逐日口径直接比较。详见用户手册 §5.7b 的对照表。
|
||||
"""
|
||||
from hdiv.backtest.engine import BacktestEngine
|
||||
|
||||
repo = PitRepo()
|
||||
days_all = self._trading_days(repo, start, end)
|
||||
start, end = days_all[0], days_all[-1]
|
||||
if len(days_all) < 2:
|
||||
raise DataGapError(f"{start} ~ {end} 交易日不足，无法回测")
|
||||
|
||||
step = max(1, int(every_n_days or self.daily.universe_refresh_days))
|
||||
sig_step = max(
|
||||
1, int(signal_every_n_days or self.daily.signal_frequency_days)
|
||||
)
|
||||
screen_days = days_all[::step]
|
||||
|
||||
if verbose:
|
||||
print(
|
||||
f"每日动态股票池回测 {self.strategy.strategy.id} "
|
||||
f"v{self.strategy.strategy.version}："
|
||||
f"{start} ~ {end}（{len(days_all)} 个交易日）",
|
||||
flush=True,
|
||||
)
|
||||
print(
|
||||
f" 选股频率：每 {step} 个交易日"
|
||||
f"（共 {len(screen_days)} 次筛选）；"
|
||||
f"信号频率：每 {sig_step} 个交易日；"
|
||||
f"池外持仓：{self.daily.pool_exit_action}",
|
||||
flush=True,
|
||||
)
|
||||
if step > 1 or sig_step > 1:
|
||||
print(
|
||||
f" 注意：粗粒度口径 —— 股票池每 {step} 个交易日才重建、"
|
||||
f"买卖每 {sig_step} 个交易日才判断。\n"
|
||||
f" 判定规则未变，但机会数量与换手低于逐日口径，"
|
||||
f"结果不可与逐日口径直接比较。",
|
||||
flush=True,
|
||||
)
|
||||
if len(screen_days) > 60:
|
||||
# 成本要**先说清楚**：全市场筛选是分钟级到小时级操作，
|
||||
# 让人先看到预计时长，再决定是否继续（或改用更粗的频率／更短区间）。
|
||||
n_chunks = max(
|
||||
1,
|
||||
len({d.year for d in screen_days})
|
||||
// max(1, int(self.daily.chunk_years)),
|
||||
)
|
||||
est = (
|
||||
len(screen_days) * _SEC_PER_SCREEN_DAY
|
||||
+ len(days_all) * _SEC_PER_SIM_DAY_BASE
|
||||
+ len(days_all) * _SEC_PER_SIM_SIGNAL / sig_step
|
||||
+ n_chunks * _SEC_PER_CHUNK
|
||||
+ 20.0
|
||||
)
|
||||
print(
|
||||
f" 预计耗时约 {est / 60:,.1f} 分钟"
|
||||
f"（按实测 选股 {_SEC_PER_SCREEN_DAY:.1f}s/次、"
|
||||
f"模拟 {_SEC_PER_SIM_DAY_BASE:.1f}s/交易日 + "
|
||||
f"{_SEC_PER_SIM_SIGNAL:.1f}s/次信号评估 估算；"
|
||||
f"机器与数据量不同会有出入）",
|
||||
flush=True,
|
||||
)
|
||||
print(
|
||||
" ⚠ 区间较长。可按需选择速度方案（详见手册 §5.7b）：\n"
|
||||
" --every-n-days 5 股票池每 5 个交易日重建一次（选股耗时 ~÷5）\n"
|
||||
" --signal-every-n-days 5 买卖每 5 个交易日判断一次（模拟耗时大幅下降）\n"
|
||||
" 两者可叠加；也可缩短 --start/--end，或改 config/backtest.yml 的 daily 段。",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
# --- 参照数据（财务/分红/日历）一次载入，全程常驻 ---
|
||||
t0 = time.time()
|
||||
repo.load_reference(end=end)
|
||||
if verbose:
|
||||
print(f" 已载入参照数据（财报/分红/交易日历）：{time.time() - t0:,.1f}s",
|
||||
flush=True)
|
||||
|
||||
# --- 第一趟：逐区块选股（可按可复现的指纹复用缓存）---
|
||||
#
|
||||
# **为什么必须有缓存**：选股这一趟在 6.7 年区间上要 1 小时以上，而它完全
|
||||
# 由（策略 + 区间 + 筛选配置 + 刷新频率）唯一决定。第二趟模拟若因任何原因
|
||||
# 失败或需要重跑，重新筛一遍纯属浪费 —— 实测一次失败就白烧掉一小时。
|
||||
# 缓存键只含**输入**（不含时间戳），所以「同样的输入 ⇒ 同样的选股结果」，
|
||||
# 复用它不引入任何未来信息。
|
||||
cache_key = self._pools_cache_key(start, end, step)
|
||||
cached = None if refresh_pools else self._read_cache(cache_key)
|
||||
if cached is not None:
|
||||
pools, member_rows, prune_stats, screen_dates = cached
|
||||
screens = []
|
||||
if verbose:
|
||||
print(
|
||||
f" 复用已缓存的选股结果：{len(pools)} 个决策时点"
|
||||
f"（省去逐日筛选；加 --refresh-pools 可强制重筛）",
|
||||
flush=True,
|
||||
)
|
||||
else:
|
||||
# --- 保守预剪枝 ---
|
||||
screener = DailyUniverseScreener.from_strategy(
|
||||
self.registry, self.strategy, repo, verbose=False
|
||||
)
|
||||
allowed = screener.build_prune_set(start, end)
|
||||
if verbose:
|
||||
p = screener.prune.as_dict()
|
||||
print(
|
||||
f" 候选集预剪枝：{p['total']} → {p['kept']} 只"
|
||||
f"（剔除 交易所 {p['pruned_exchange']}、板块 {p['pruned_board']}、"
|
||||
f"上市年限 {p['pruned_listing']}、市值 {p['pruned_market_cap']}）"
|
||||
f"；被剔除者在本区间内不可能通过市场滤网，不影响最终入选",
|
||||
flush=True,
|
||||
)
|
||||
screens = self._screen_all(repo, screener, screen_days, verbose=verbose)
|
||||
pools = {s.trade_date: set(s.symbols) for s in screens}
|
||||
prune_stats = screener.prune.as_dict()
|
||||
screen_dates = sorted(pools)
|
||||
member_rows = DailyUniverseScreener.member_rows(
|
||||
"", screens, created_at=datetime.now()
|
||||
)
|
||||
self._write_cache(cache_key, pools, member_rows, prune_stats, verbose=verbose)
|
||||
|
||||
nonempty = sum(1 for v in pools.values() if v)
|
||||
if verbose:
|
||||
sizes = [len(v) for v in pools.values()]
|
||||
print(
|
||||
f" 选股完成：{nonempty}/{len(pools)} 个决策日选出非空股票池，"
|
||||
f"成员数 {min(sizes) if sizes else 0}~{max(sizes) if sizes else 0}，"
|
||||
f"累计出现过的股票 {len(set().union(*pools.values())) if pools else 0} 只",
|
||||
flush=True,
|
||||
)
|
||||
if not nonempty:
|
||||
raise DataGapError(
|
||||
"每日选股未产出任何非空股票池：请检查筛选条件与数据覆盖。"
|
||||
)
|
||||
|
||||
# --- 第二趟：模拟（复用既有引擎）---
|
||||
engine = BacktestEngine(
|
||||
self.strategy,
|
||||
backtest=self.bt,
|
||||
repo=repo,
|
||||
universe_by_refresh=pools,
|
||||
signal_frequency_days=sig_step,
|
||||
pool_exit_action=self.daily.pool_exit_action,
|
||||
profile_on_trade=bool(self.daily.profile_on_trade),
|
||||
)
|
||||
result = engine.run(
|
||||
start=start, end=end, persist=persist, mode="daily", verbose=verbose
|
||||
)
|
||||
|
||||
# --- 每日选股留痕（行内容来自选股趟，run_id 来自本次模拟）---
|
||||
result["daily_screening"] = {
|
||||
"screen_count": len(pools),
|
||||
"screen_days": [str(d) for d in screen_dates],
|
||||
"pool_size_min": min(len(v) for v in pools.values()),
|
||||
"pool_size_max": max(len(v) for v in pools.values()),
|
||||
"pool_size_mean": sum(len(v) for v in pools.values()) / len(pools),
|
||||
"distinct_symbols": len(set().union(*pools.values())),
|
||||
"prune": prune_stats,
|
||||
"refresh_days": step,
|
||||
"signal_every_days": sig_step,
|
||||
"pool_exit_action": self.daily.pool_exit_action,
|
||||
"profile_on_trade": bool(self.daily.profile_on_trade),
|
||||
"repo": repo.stats(),
|
||||
"pools_from_cache": cached is not None,
|
||||
}
|
||||
if persist and self.daily.persist_daily_universe:
|
||||
result["written_daily_universe"] = self._persist_members(
|
||||
result["run_id"], member_rows, verbose=verbose
|
||||
)
|
||||
if verbose:
|
||||
self._print_changes(pools)
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 选股
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _screen_all(
|
||||
self,
|
||||
repo: PitRepo,
|
||||
screener: DailyUniverseScreener,
|
||||
screen_days: list[date],
|
||||
*,
|
||||
verbose: bool,
|
||||
) -> list[ScreenDay]:
|
||||
"""按年分块预载行情 → 区内逐日筛选 → 释放区块。"""
|
||||
chunks = _chunks(screen_days, int(self.daily.chunk_years))
|
||||
progress = DailyProgress(
|
||||
len(screen_days), int(self.daily.progress_every_days), enabled=verbose
|
||||
)
|
||||
out: list[ScreenDay] = []
|
||||
for ci, chunk in enumerate(chunks, start=1):
|
||||
t0 = time.time()
|
||||
repo.load_range(chunk[0], chunk[-1])
|
||||
if verbose:
|
||||
print(
|
||||
f" 区块 {ci}/{len(chunks)} {chunk[0]} ~ {chunk[-1]}"
|
||||
f"（{len(chunk)} 个交易日，已载入行情 "
|
||||
f"{repo.stats()['market_rows']:,} 行 / "
|
||||
f"{time.time() - t0:,.1f}s）",
|
||||
flush=True,
|
||||
)
|
||||
try:
|
||||
for d in chunk:
|
||||
sd = screener.screen_day(d)
|
||||
out.append(sd)
|
||||
progress.tick(f"{d} 池内 {sd.member_count} 只")
|
||||
finally:
|
||||
repo.release_range()
|
||||
return out
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 选股结果缓存（让「选股一小时的成果」不会因模拟失败而丢失）
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _cache_path(key: str) -> Path:
|
||||
from hdiv.core.paths import output_dir
|
||||
|
||||
return output_dir("output") / "cache" / f"daily_pools_{key}.json"
|
||||
|
||||
def _pools_cache_key(self, start: date, end: date, step: int) -> str:
|
||||
"""选股缓存的指纹：**只由输入决定**，不含时间戳。
|
||||
|
||||
``step``（股票池重建频率）必须在键里：用 1 日/5 日筛出的池子是不同的输入，
|
||||
共用一份缓存会静默给出错的股票池。信号频率（``--signal-every-n-days``）
|
||||
**不在**键里 —— 它只影响模拟，不影响选股结果。
|
||||
"""
|
||||
return stable_id(
|
||||
"dailypools",
|
||||
self.strategy.strategy.id,
|
||||
self.strategy.strategy.version,
|
||||
self.registry.hash_of(self.strategy),
|
||||
str(start), str(end), str(int(step)),
|
||||
)
|
||||
|
||||
def _read_cache(self, key: str) -> tuple[Any, list[dict], dict, list[date]] | None:
|
||||
p = self._cache_path(key)
|
||||
if not p.is_file():
|
||||
return None
|
||||
try:
|
||||
blob = json.loads(p.read_text(encoding="utf-8"))
|
||||
pools = {
|
||||
date.fromisoformat(d): set(v)
|
||||
for d, v in (blob.get("pools") or {}).items()
|
||||
}
|
||||
if not pools:
|
||||
return None
|
||||
return (
|
||||
pools,
|
||||
list(blob.get("member_rows") or []),
|
||||
dict(blob.get("prune") or {}),
|
||||
[date.fromisoformat(x) for x in (blob.get("screen_days") or [])],
|
||||
)
|
||||
except Exception:
|
||||
# 缓存损坏一律视为「没有缓存」并重新筛选 —— 绝不因为一个坏文件而
|
||||
# 用错的股票池回测（那会静默改变结果）。
|
||||
return None
|
||||
|
||||
def _write_cache(
|
||||
self, key: str, pools: dict[date, set[str]], member_rows: list[dict],
|
||||
prune: dict, *, verbose: bool,
|
||||
) -> None:
|
||||
p = self._cache_path(key)
|
||||
try:
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
rows = []
|
||||
for r in member_rows:
|
||||
r = dict(r)
|
||||
r.pop("created_at", None) # 由落库时统一填充
|
||||
r["trade_date"] = str(r["trade_date"])
|
||||
rows.append(r)
|
||||
p.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"pools": {str(d): sorted(v) for d, v in pools.items()},
|
||||
"member_rows": rows,
|
||||
"prune": prune,
|
||||
"screen_days": [str(d) for d in sorted(pools)],
|
||||
},
|
||||
ensure_ascii=False, default=str,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if verbose:
|
||||
mb = p.stat().st_size / 1e6
|
||||
print(f" 选股结果已缓存：{p}（{mb:,.1f} MB，供重跑复用）", flush=True)
|
||||
except Exception as exc: # pragma: no cover - 缓存失败不该影响回测
|
||||
if verbose:
|
||||
print(f" ⚠ 选股结果缓存写入失败（不影响本次运行）：{exc}", flush=True)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 落库
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _persist_members(
|
||||
self, run_id: str, member_rows: list[dict], *, verbose: bool
|
||||
) -> int:
|
||||
cfg = load_config("datasource")
|
||||
if not db.table_exists("hd_daily_universe", cfg):
|
||||
if verbose:
|
||||
print(
|
||||
" ⚠ hd_daily_universe 不存在，跳过每日选股留痕。"
|
||||
"请先执行 `python -m hdiv ddl apply`。",
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
if not member_rows:
|
||||
return 0
|
||||
now = datetime.now()
|
||||
rows = []
|
||||
for r in member_rows:
|
||||
r = dict(r)
|
||||
r["run_id"] = run_id
|
||||
r["created_at"] = now
|
||||
td = r.get("trade_date")
|
||||
r["trade_date"] = (
|
||||
td if isinstance(td, date) else date.fromisoformat(str(td))
|
||||
)
|
||||
rows.append(r)
|
||||
n = 0
|
||||
for i in range(0, len(rows), 2000):
|
||||
n += db.upsert_dataframe(
|
||||
"hd_daily_universe",
|
||||
pd.DataFrame(rows[i : i + 2000]),
|
||||
cfg=cfg,
|
||||
update_columns=[
|
||||
"name", "industry", "dividend_yield", "total_mv", "roe_avg",
|
||||
"listed_count", "candidate_count", "values_json",
|
||||
],
|
||||
)
|
||||
if verbose:
|
||||
print(f" 每日选股留痕：hd_daily_universe {n} 行"
|
||||
f"（{len({r['trade_date'] for r in rows})} 个时点）", flush=True)
|
||||
return n
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 辅助
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _trading_days(
|
||||
self, repo: PitRepo, start: date | None, end: date | None
|
||||
) -> list[date]:
|
||||
"""确定回测区间（含 ``end=latest`` 与「起点对齐到交易日」）。"""
|
||||
from hdiv.data.repo import Repo
|
||||
|
||||
plain = Repo()
|
||||
s = start or self.bt.period.start
|
||||
raw_end = end or (
|
||||
plain.trading_day(None) if self.bt.period.end == "latest"
|
||||
else self.bt.period.end
|
||||
)
|
||||
# 用直连 Repo 定位端点（PitRepo 此时尚未载入日历）
|
||||
s = plain.trading_day(s)
|
||||
e = plain.trading_day(raw_end)
|
||||
if e <= s:
|
||||
raise HdivError(f"回测区间非法：{s} ~ {e}")
|
||||
days = plain.trading_days(s, e)
|
||||
if len(days) < 2:
|
||||
raise DataGapError(f"{s} ~ {e} 交易日不足，无法回测")
|
||||
return days
|
||||
|
||||
def _print_changes(self, pools: dict[date, set[str]]) -> None:
|
||||
"""打印股票池的进出（动态池最值得看的东西）。"""
|
||||
prev: set[str] | None = None
|
||||
enter = exit_ = 0
|
||||
events: list[str] = []
|
||||
for day in sorted(pools):
|
||||
cur = pools[day]
|
||||
if prev is not None:
|
||||
add, drop = cur - prev, prev - cur
|
||||
enter += len(add)
|
||||
exit_ += len(drop)
|
||||
if (add or drop) and len(events) < 8:
|
||||
events.append(
|
||||
f" {day}: +{len(add)} -{len(drop)}"
|
||||
+ (f" 进 {'、'.join(sorted(add)[:4])}" if add else "")
|
||||
+ (f" 出 {'、'.join(sorted(drop)[:4])}" if drop else "")
|
||||
)
|
||||
prev = cur
|
||||
if not pools:
|
||||
return
|
||||
n = max(len(pools) - 1, 1)
|
||||
print(
|
||||
f" 股票池变动：累计进入 {enter} 次 / 移出 {exit_} 次"
|
||||
f"（平均每日 {enter / n:.1f} 进 {exit_ / n:.1f} 出）",
|
||||
flush=True,
|
||||
)
|
||||
for line in events:
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def _chunks(days: list[date], chunk_years: int) -> list[list[date]]:
|
||||
"""按自然年（或 chunk_years 年）切块。"""
|
||||
if not days:
|
||||
return []
|
||||
n = max(1, int(chunk_years))
|
||||
out: list[list[date]] = []
|
||||
cur: list[date] = []
|
||||
bucket = days[0].year
|
||||
for d in days:
|
||||
if d.year >= bucket + n:
|
||||
out.append(cur)
|
||||
cur = []
|
||||
bucket = d.year
|
||||
cur.append(d)
|
||||
if cur:
|
||||
out.append(cur)
|
||||
return out
|
||||
+396
-108
@@ -146,6 +146,50 @@ class CostModel:
|
||||
return float(dt.rates.get("gt1y", 0.0))
|
||||
|
||||
|
||||
def dividend_handling_notes(bt_cfg: BacktestConfig) -> list[str]:
|
||||
"""声明 ``backtest.yml`` 里「写了但引擎没实现」的分红/公司行为配置。
|
||||
|
||||
口径必须与 :meth:`BacktestEngine._apply_dividends` 与 :meth:`_execute`
|
||||
的实际行为逐条对应 —— 配置承诺与实际行为不一致是本项目反复记录的一类缺陷：
|
||||
run 记录看起来「一切正常」，使用者却以为某项规则生效了。
|
||||
|
||||
当前**已实现**的组合（不产生声明）：
|
||||
|
||||
- ``cash_mode: reinvest`` + ``reinvest_rule: portfolio_rebalance``
|
||||
—— 分红现金回落到**可投资现金池**，下次调仓按目标权重再配置；
|
||||
这笔钱与初始资金同一个 ``cash`` 变量，可以直接用于买入（不会被隔离）。
|
||||
- ``apply_dividend_tax``（总闸）+ ``cost.yml`` 的分档税率。
|
||||
- ``handle_stock_dividend`` —— 送转股按 ``stk_div`` 调整股数、总成本不变。
|
||||
|
||||
其余取值都会逐条写入 ``hd_backtest_run.unimplemented_json``。
|
||||
"""
|
||||
d = bt_cfg.dividend
|
||||
notes: list[str] = []
|
||||
mode = str(d.cash_mode)
|
||||
rule = str(d.reinvest_rule)
|
||||
if mode != "reinvest":
|
||||
notes.append(
|
||||
f"未实现 cash_mode={mode}：引擎一律把分红现金回落到**可投资现金池**，"
|
||||
f"在下次调仓按目标权重再配置（行为等价 reinvest + portfolio_rebalance）"
|
||||
)
|
||||
elif rule != "portfolio_rebalance":
|
||||
notes.append(
|
||||
f"未实现 reinvest_rule={rule}（按同一只股票再投资）：实际行为等价 "
|
||||
f"portfolio_rebalance —— 分红现金回落到可投资现金池，"
|
||||
f"在下次调仓按目标权重再配置"
|
||||
)
|
||||
if not d.handle_stock_dividend:
|
||||
notes.append(
|
||||
"未处理送转股（handle_stock_dividend=false）：股数不调整，"
|
||||
"而价格是不复权价、除权日照常下跌 → 送转被记成虚假亏损"
|
||||
)
|
||||
if d.handle_rights_issue:
|
||||
notes.append(
|
||||
"未实现配股处理（handle_rights_issue）：配股缴款/股数变动不入账"
|
||||
)
|
||||
return notes
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 引擎
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -158,16 +202,26 @@ class BacktestEngine:
|
||||
*,
|
||||
cost: CostConfig | None = None,
|
||||
backtest: BacktestConfig | None = None,
|
||||
repo: Repo | None = None,
|
||||
frozen_reference: tuple[date, date] | None = None,
|
||||
universe_run_id: str | None = None,
|
||||
allow_lookahead_universe: bool = False,
|
||||
universe_by_refresh: dict[date, set[str]] | None = None,
|
||||
universe_refresh_days: int | None = None,
|
||||
signal_frequency_days: int | None = None,
|
||||
pool_exit_action: str = "hold",
|
||||
profile_on_trade: bool = False,
|
||||
profile_window_years: int | None = None,
|
||||
) -> None:
|
||||
db.load_dotenv_once()
|
||||
self.strategy = strategy
|
||||
self.cost_cfg = cost or load_config("cost")
|
||||
self.bt_cfg = backtest or load_config("backtest")
|
||||
self.cost = CostModel(self.cost_cfg)
|
||||
self.repo = Repo()
|
||||
#: 取数出口。默认直连数据库；daily 模式注入已批量预载的 PitRepo，
|
||||
#: 这样实时画像的逐 asof 财报面板来自内存而不是每天重查 4 张财务表
|
||||
#: （后者约 5~6 秒/时点，几千个决策时点就废掉了）。
|
||||
self.repo = repo if repo is not None else Repo()
|
||||
self.registry = StrategyRegistry()
|
||||
# frozen_reference 非空时，分位分布冻结在该区间（walk-forward 测试段必须）
|
||||
self.frozen_reference = frozen_reference
|
||||
@@ -183,6 +237,22 @@ class BacktestEngine:
|
||||
self.gate_cfg = strategy.entry.profile_gate
|
||||
self.pit: Any = None
|
||||
|
||||
# --- 每日动态股票池模式（--mode daily）的注入点，默认全部关闭 ---
|
||||
#: 外部预先算好的「交易日 → 股票池」映射。给定时引擎**不再自行筛选**：
|
||||
#: 每日选股由 DailyRunner 独立完成（它需要按区块预载行情才能跑得动），
|
||||
#: 引擎只负责模拟。默认 None，行为与改造前逐字一致。
|
||||
self.universe_by_refresh = universe_by_refresh
|
||||
#: 股票池重建频率改以**交易日**计（1 = 每个交易日）。
|
||||
#: 默认 None → 沿用 backtest.yml 的 universe_refresh_months（月）。
|
||||
self.universe_refresh_days = universe_refresh_days
|
||||
#: 信号评估频率改以**交易日**计（1 = 每个交易日）。
|
||||
self.signal_frequency_days = signal_frequency_days
|
||||
#: 持仓掉出当日股票池后的处置：hold = 只减不加（默认）；sell = 清仓
|
||||
self.pool_exit_action = pool_exit_action
|
||||
#: 买卖决策发生时计算并留痕个股画像（不区分是否在当日池内）
|
||||
self.profile_on_trade = profile_on_trade
|
||||
self.profile_window_years = profile_window_years
|
||||
|
||||
@classmethod
|
||||
def from_strategy(cls, path: str | Path, **kw: Any) -> BacktestEngine:
|
||||
reg = StrategyRegistry()
|
||||
@@ -293,8 +363,10 @@ class BacktestEngine:
|
||||
rc = result["reconciliation"]
|
||||
print(
|
||||
f" 期初 {result['initial_capital']:,.0f} → 期末 {result['final_capital']:,.0f}"
|
||||
f" | 总收益 {result['total_return']:.2%} | CAGR {result['cagr']:.2%}"
|
||||
f" | 最大回撤 {result['max_drawdown']:.2%} | Sharpe {result['sharpe']:.2f}"
|
||||
f" | 总收益 {_pct(result['total_return'])}"
|
||||
f" | CAGR {_pct(result['cagr'])}"
|
||||
f" | 最大回撤 {_pct(result['max_drawdown'])}"
|
||||
f" | Sharpe {_num(result['sharpe'])}"
|
||||
f" | 成交 {result['trade_count']} 笔",
|
||||
flush=True,
|
||||
)
|
||||
@@ -373,55 +445,91 @@ class BacktestEngine:
|
||||
if self.frozen_reference
|
||||
else bt.percentile_reference.lookback_years
|
||||
)
|
||||
data_start = date(max(days[0].year - max_years - 1, 2000), 1, 1)
|
||||
# 取数起点见下面 P6 处：它要与实时画像的最长窗口取并集，
|
||||
# 因此这里不再单独定义 data_start（避免出现两个"起点"口径）。
|
||||
|
||||
# --- 股票池：按 universe_refresh_months 周期重建（PIT）---
|
||||
refresh_dates: list[date] = []
|
||||
step = bt.schedule.universe_refresh_months
|
||||
cur = days[0]
|
||||
for d in days:
|
||||
if not refresh_dates or _months_between(refresh_dates[-1], d) >= step:
|
||||
refresh_dates.append(d)
|
||||
del cur
|
||||
universe_by_refresh: dict[date, set[str]] = {}
|
||||
if self.universe_run_id:
|
||||
# 未来函数守卫：股票池自带 asof。若它晚于回测起点，名单里就含有
|
||||
# 「当时不可能知道」的信息（哪些公司此后仍满足分红/质量条件），
|
||||
# 把它套到更早的年份上就是用未来信息选股 —— 与 walk-forward
|
||||
# 拒绝 --universe-run 是同一条理由，这里必须同样拒绝。
|
||||
self._check_universe_asof(days[0])
|
||||
# 冻结股票池：直接取该次筛选的入选成员，所有调仓日复用同一份清单。
|
||||
# 好处是可复现（同一 run_id 永远对应同一股票池），并建立双向关联。
|
||||
dfu = db.read_sql(
|
||||
"SELECT symbol FROM hd_universe_member "
|
||||
"WHERE run_id = :r AND passed = 1 ORDER BY symbol",
|
||||
{"r": self.universe_run_id}, cfg=load_config("datasource"),
|
||||
)
|
||||
frozen = set(dfu["symbol"].tolist()) if not dfu.empty else set()
|
||||
if not frozen:
|
||||
raise DataGapError(
|
||||
f"股票池 {self.universe_run_id} 没有任何入选成员，无法回测"
|
||||
)
|
||||
for rd in refresh_dates:
|
||||
universe_by_refresh[rd] = frozen
|
||||
if self.universe_by_refresh is not None:
|
||||
# 每日动态股票池：名单已由 DailyRunner 逐交易日算好（它必须先在
|
||||
# 区块内批量预载行情才跑得动，因此不能在这里临时筛）。引擎不再筛选，
|
||||
# 只按这些「决策时点」切换池子 —— 每个时点的名单仍是 PIT 的。
|
||||
universe_by_refresh = {
|
||||
d: set(v) for d, v in self.universe_by_refresh.items() if v
|
||||
}
|
||||
refresh_dates = sorted(universe_by_refresh)
|
||||
if not refresh_dates:
|
||||
raise DataGapError("每日选股未产出任何非空股票池，无法回测")
|
||||
if verbose:
|
||||
print(
|
||||
f" 冻结股票池 {self.universe_run_id}：{len(frozen)} 只"
|
||||
f"（不随调仓日重新筛选）",
|
||||
f" 每日动态股票池：{len(refresh_dates)} 个决策时点，"
|
||||
f"成员数 {min(len(v) for v in universe_by_refresh.values())}"
|
||||
f"~{max(len(v) for v in universe_by_refresh.values())} 只",
|
||||
flush=True,
|
||||
)
|
||||
else:
|
||||
selector = self.registry.selector(s)
|
||||
for rd in refresh_dates:
|
||||
res = selector.run(asof=rd, persist=False, verbose=False)
|
||||
universe_by_refresh[rd] = set(res["selected"]["symbol"].tolist())
|
||||
step_days = self.universe_refresh_days
|
||||
step = bt.schedule.universe_refresh_months
|
||||
if step_days:
|
||||
# 按交易日步进（daily 模式）：每 N 个交易日重建一次
|
||||
refresh_dates = days[:: max(1, int(step_days))]
|
||||
else:
|
||||
refresh_dates = []
|
||||
for d in days:
|
||||
if not refresh_dates or _months_between(refresh_dates[-1], d) >= step:
|
||||
refresh_dates.append(d)
|
||||
if self.universe_run_id:
|
||||
# 未来函数守卫：股票池自带 asof。若它晚于回测起点，名单里就含有
|
||||
# 「当时不可能知道」的信息（哪些公司此后仍满足分红/质量条件），
|
||||
# 把它套到更早的年份上就是用未来信息选股 —— 与 walk-forward
|
||||
# 拒绝 --universe-run 是同一条理由，这里必须同样拒绝。
|
||||
self._check_universe_asof(days[0])
|
||||
# 冻结股票池：直接取该次筛选的入选成员，所有调仓日复用同一份清单。
|
||||
# 好处是可复现（同一 run_id 永远对应同一股票池），并建立双向关联。
|
||||
dfu = db.read_sql(
|
||||
"SELECT symbol FROM hd_universe_member "
|
||||
"WHERE run_id = :r AND passed = 1 ORDER BY symbol",
|
||||
{"r": self.universe_run_id}, cfg=load_config("datasource"),
|
||||
)
|
||||
frozen = set(dfu["symbol"].tolist()) if not dfu.empty else set()
|
||||
if not frozen:
|
||||
raise DataGapError(
|
||||
f"股票池 {self.universe_run_id} 没有任何入选成员，无法回测"
|
||||
)
|
||||
for rd in refresh_dates:
|
||||
universe_by_refresh[rd] = frozen
|
||||
if verbose:
|
||||
print(f" 股票池 {rd}: {len(universe_by_refresh[rd])} 只", flush=True)
|
||||
print(
|
||||
f" 冻结股票池 {self.universe_run_id}：{len(frozen)} 只"
|
||||
f"（不随调仓日重新筛选）",
|
||||
flush=True,
|
||||
)
|
||||
else:
|
||||
selector = self.registry.selector(s)
|
||||
for rd in refresh_dates:
|
||||
res = selector.run(asof=rd, persist=False, verbose=False)
|
||||
universe_by_refresh[rd] = set(res["selected"]["symbol"].tolist())
|
||||
if verbose:
|
||||
print(f" 股票池 {rd}: {len(universe_by_refresh[rd])} 只", flush=True)
|
||||
|
||||
all_syms = sorted(set().union(*universe_by_refresh.values())) if universe_by_refresh else []
|
||||
|
||||
# --- 价格（不复权）与股息率序列 ---
|
||||
price = self.repo.price_history(all_syms, data_start, days[-1], adjust="none")
|
||||
# --- P6：价格只取一次，覆盖「引擎需要的分位窗口」与「画像需要的最长窗口」的并集 ---
|
||||
#
|
||||
# 两处窗口本来是分开取的：引擎按 backtest.yml 的 lookback_years（如 5 年），
|
||||
# 实时画像按 profile.yml 的 windows_years 最大值（如 10 年）。于是同一批股票
|
||||
# 的价格会被查两遍，而且画像那遍起点更早。这里先把两遍并起来算起点，
|
||||
# 取一次，再把同一份帧交给画像服务复用（见下面的 `price=` 参数）。
|
||||
need_pit = self.gate_cfg.enabled or self.profile_on_trade
|
||||
pit_years = 0
|
||||
if need_pit:
|
||||
from hdiv.profile.builder import ProfileBuilder
|
||||
|
||||
pit_years = max(ProfileBuilder.from_config().config.windows_years)
|
||||
price_start = date(max(days[0].year - max(max_years, pit_years) - 1, 2000), 1, 1)
|
||||
|
||||
price = self.repo.price_history(all_syms, price_start, days[-1], adjust="none")
|
||||
dividends = self.repo.dividend_records(days[-1], years_back=max_years + 3)
|
||||
dividends = dividends[dividends["symbol"].isin(set(all_syms))]
|
||||
events = build_dps_events(dividends)
|
||||
@@ -434,6 +542,22 @@ class BacktestEngine:
|
||||
g["trade_date"] = pd.to_datetime(g["trade_date"])
|
||||
px_by_sym[sym] = g.set_index("trade_date")[["open", "close"]]
|
||||
|
||||
# --- P2：股息率序列**每只股票只算一次** ---
|
||||
#
|
||||
# 原实现每个评估日都对每只股票从零重算一遍 TTM 股息率序列
|
||||
# （`ttm_dps_series(idx, events)`，idx 是从取数起点到当天的**前缀**），
|
||||
# 于是同一段历史被反复计算 —— 典型 O(交易日数²)：1600 天 × 池内几十只，
|
||||
# 越到后期窗口越长、越慢。
|
||||
#
|
||||
# `ttm_dps_series` 的每一天取值只依赖「该日期 + 分红事件」，与传入的日期
|
||||
# 序列里还有哪些其它日期无关（`ttm_dps_at` 与序列右端点一致就是这条性质
|
||||
# 的现成证据，见 tests/test_dividend_smoothing.py）。因此可以先按股票算
|
||||
# 整段序列，之后每天只是切片 —— 结果逐值相同。
|
||||
_w, _g, _sm = ttm_params()
|
||||
yield_by_sym: dict[str, pd.Series] = build_yield_series(
|
||||
px_by_sym, events, ttm_days=_w, grace_days=_g, smooth_spikes=_sm
|
||||
)
|
||||
|
||||
# --- 分红事件（含送转），用于持仓期间的现金与股数调整 ---
|
||||
div_events = self.repo.dividend_events(days[0], days[-1])
|
||||
div_events = div_events[div_events["symbol"].isin(set(all_syms))]
|
||||
@@ -447,19 +571,29 @@ class BacktestEngine:
|
||||
limits = self._load_limits(all_syms, days[0], days[-1])
|
||||
|
||||
# --- 实时画像闸门：预载跨决策日共享的面板（仅在启用时）---
|
||||
if self.gate_cfg.enabled:
|
||||
# profile_on_trade（daily 模式）也会用到画像：语义是「买卖决策发生时
|
||||
# 计算并留痕」，不区分是否在当日池内，也不要求闸门开启。
|
||||
if need_pit:
|
||||
from hdiv.profile.pit import PitProfileService
|
||||
|
||||
self.pit = PitProfileService(window_years=self.gate_cfg.window_years)
|
||||
# 画像取数起点必须覆盖最长窗口（profile.yml 的 windows_years），
|
||||
# 与分位参照窗口（backtest.yml 的 lookback_years）是两个独立的量。
|
||||
pit_start = date(max(days[0].year - self.pit.max_years - 1, 2000), 1, 1)
|
||||
self.pit.prepare(all_syms, pit_start, days[-1])
|
||||
self.pit.configure({r.metric for r in self.gate_cfg.rules})
|
||||
window_years = (
|
||||
self.profile_window_years
|
||||
if self.profile_window_years is not None
|
||||
else self.gate_cfg.window_years
|
||||
)
|
||||
self.pit = PitProfileService(window_years=window_years, repo=self.repo)
|
||||
# 画像取数起点必须覆盖最长窗口（profile.yml 的 windows_years）；
|
||||
# price_start 已经把这个窗口考虑进来了，这里只需把**同一份价格帧**
|
||||
# 交给它复用（P6），避免同一批股票被查两遍。
|
||||
pit_start = date(max(days[0].year - pit_years - 1, 2000), 1, 1)
|
||||
self.pit.prepare(all_syms, pit_start, days[-1], price=price)
|
||||
if self.gate_cfg.enabled:
|
||||
# 成本控制：闸门规则用不到的指标不必载入财报
|
||||
self.pit.configure({r.metric for r in self.gate_cfg.rules})
|
||||
if verbose:
|
||||
print(
|
||||
f" 实时画像闸门已启用：窗口 {self.gate_cfg.window_years} 年，"
|
||||
f"{len(self.gate_cfg.rules)} 条规则，"
|
||||
f" 实时画像：窗口 {window_years} 年，"
|
||||
f"{'闸门 %d 条规则' % len(self.gate_cfg.rules) if self.gate_cfg.enabled else '仅留痕（闸门关闭）'}，"
|
||||
f"面板自 {pit_start} 起载入（{len(all_syms)} 只）",
|
||||
flush=True,
|
||||
)
|
||||
@@ -474,7 +608,8 @@ class BacktestEngine:
|
||||
"div_by_date": div_by_date,
|
||||
"suspend": suspend,
|
||||
"limits": limits,
|
||||
"data_start": data_start,
|
||||
"data_start": price_start,
|
||||
"yield_by_sym": yield_by_sym,
|
||||
}
|
||||
|
||||
def _load_suspend(self, syms: list[str], start: date, end: date) -> set[tuple[str, date]]:
|
||||
@@ -553,12 +688,14 @@ class BacktestEngine:
|
||||
dividend_ledger: list[dict[str, Any]] = []
|
||||
|
||||
freq = bt.schedule.signal_frequency_months
|
||||
# daily 模式：信号频率改以**交易日**计（1 = 每个交易日）
|
||||
freq_days = self.signal_frequency_days
|
||||
last_signal_month: tuple[int, int] | None = None
|
||||
bench = self._benchmark(days)
|
||||
current_universe: set[str] = set()
|
||||
last_refresh: date | None = None
|
||||
|
||||
for day in days:
|
||||
for day_index, day in enumerate(days):
|
||||
# --- (0) 股票池切换 ---
|
||||
if last_refresh is None or day in ctx["universe_by_refresh"]:
|
||||
current_universe = ctx["universe_by_refresh"].get(
|
||||
@@ -586,14 +723,22 @@ class BacktestEngine:
|
||||
cash = self._apply_dividends(day, positions, ctx, cash, dividend_ledger)
|
||||
|
||||
# --- (3) 收盘：评估信号 ---
|
||||
if (day.month, day.year) != last_signal_month:
|
||||
if last_signal_month is None or _months_between(
|
||||
date(last_signal_month[1], last_signal_month[0], 1), day
|
||||
) >= freq:
|
||||
new_signals = self._evaluate(day, cash, positions, current_universe, ctx)
|
||||
pending = [x for x in new_signals if x.kind in {"BUY", "ADD", "SELL", "TRIM"}]
|
||||
signals.extend([x for x in new_signals if x.kind not in {"BUY", "ADD", "SELL", "TRIM"}])
|
||||
last_signal_month = (day.month, day.year)
|
||||
if freq_days:
|
||||
# 按交易日步进（daily 模式）。原「按月」判定的语义完全保留在
|
||||
# else 分支里，两者互斥，freq_days=None 时行为与改造前一致。
|
||||
do_signal = day_index % max(1, int(freq_days)) == 0
|
||||
else:
|
||||
do_signal = (day.month, day.year) != last_signal_month and (
|
||||
last_signal_month is None
|
||||
or _months_between(
|
||||
date(last_signal_month[1], last_signal_month[0], 1), day
|
||||
) >= freq
|
||||
)
|
||||
if do_signal:
|
||||
new_signals = self._evaluate(day, cash, positions, current_universe, ctx)
|
||||
pending = [x for x in new_signals if x.kind in {"BUY", "ADD", "SELL", "TRIM"}]
|
||||
signals.extend([x for x in new_signals if x.kind not in {"BUY", "ADD", "SELL", "TRIM"}])
|
||||
last_signal_month = (day.month, day.year)
|
||||
|
||||
# --- (4) 收盘：盯市（总市值 = 现金 + 持仓）---
|
||||
pos_value = self._mark_to_market(day, positions, ctx)
|
||||
@@ -643,7 +788,7 @@ class BacktestEngine:
|
||||
unimplemented.add("涨跌停约束未生效（hd_limit 在回测区间内无数据，成交按可达价格近似）")
|
||||
if not self._has_constraint_rows("hd_suspend", days[0], days[-1]):
|
||||
unimplemented.add("停牌约束未生效（hd_suspend 在回测区间内无数据）")
|
||||
# 以下三项**配置写了但引擎没实现**，必须如实声明 —— 否则 run 记录看起来
|
||||
# 以下各项**配置写了但引擎没实现**，必须如实声明 —— 否则 run 记录看起来
|
||||
# 「一切正常」，而使用者以为 backtest.yml 的 defer / reinvest 生效了。
|
||||
# （配置承诺与实际行为不一致，是本项目反复记录的一类缺陷。）
|
||||
# 注意字段归属：fill/dividend 在 backtest.yml；execution/risk 在策略 yml。
|
||||
@@ -654,21 +799,15 @@ class BacktestEngine:
|
||||
"未实现停牌/涨跌停顺延（suspended_rule / limit_up_down_rule 的 "
|
||||
"defer 分支）：未成交信号在当日被**丢弃**，不会顺延到下一个可成交日"
|
||||
)
|
||||
if str(bt.dividend.cash_mode) != "hold" or bt.dividend.reinvest_rule:
|
||||
unimplemented.add(
|
||||
"未实现分红再投资规则（cash_mode=reinvest / reinvest_rule）："
|
||||
"现金分红按除权日入账后**留存为现金**，在下次调仓时按目标权重重新配置"
|
||||
)
|
||||
# 分红模式/送转/配股的声明口径集中在 dividend_handling_notes，
|
||||
# 与 _apply_dividends 的实际行为一一对应（unit test 覆盖每一档取值）。
|
||||
unimplemented.update(dividend_handling_notes(bt))
|
||||
if s.execution.signal_to_execution != "next_open" or str(bt.fill.price) != "next_open":
|
||||
unimplemented.add(
|
||||
f"未实现 signal_to_execution/fill.price 的 "
|
||||
f"{s.execution.signal_to_execution}/{bt.fill.price} 分支："
|
||||
f"成交固定按信号次日开盘价"
|
||||
)
|
||||
if bt.dividend.handle_rights_issue:
|
||||
unimplemented.add(
|
||||
"未实现配股处理（handle_rights_issue）：配股缴款/股数变动不入账"
|
||||
)
|
||||
if not bt.fill.partial_fill:
|
||||
unimplemented.add("未启用部分成交（按信号全额成交，但受资金与权重上限约束）")
|
||||
if bt.fill.max_volume_pct is not None:
|
||||
@@ -719,21 +858,14 @@ class BacktestEngine:
|
||||
px_hist = ctx["px_by_sym"].get(sym)
|
||||
if px_hist is None or px_hist.empty:
|
||||
continue
|
||||
close_hist = px_hist["close"].loc[: pd.Timestamp(day)]
|
||||
if close_hist.empty:
|
||||
# P2：整段股息率序列在 _prepare 里已按股票算好，这里只做切片。
|
||||
# 每一天的取值只依赖「该日期 + 分红事件」，与前缀里还有哪些日期无关，
|
||||
# 所以「先算整段再切前缀」与「每次按前缀重算」逐值相同 ——
|
||||
# 早先的实现是后者，代价是 O(交易日数²)。
|
||||
ser_all = ctx["yield_by_sym"].get(sym)
|
||||
if ser_all is None or ser_all.empty:
|
||||
continue
|
||||
idx = pd.DatetimeIndex(close_hist.index)
|
||||
# 参数从因子层的统一来源取，不再硬编码 ——
|
||||
# 否则改了 profile.yml 的回测也不会变（曾如此）。
|
||||
_w, _g, _sm = ttm_params()
|
||||
dps = ttm_dps_series(
|
||||
idx, ctx["events"].get(sym, pd.DataFrame()),
|
||||
ttm_days=_w, grace_days=_g, smooth_spikes=_sm,
|
||||
)
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
y = np.where(close_hist.to_numpy(dtype="float64") > 0,
|
||||
dps / close_hist.to_numpy(dtype="float64"), np.nan)
|
||||
ser = pd.Series(y, index=idx).dropna()
|
||||
ser = ser_all.loc[: pd.Timestamp(day)]
|
||||
if ser.empty:
|
||||
continue
|
||||
current = float(ser.iloc[-1])
|
||||
@@ -752,7 +884,17 @@ class BacktestEngine:
|
||||
pct = float((ref_ser <= current).sum() / ref_ser.size * 100.0)
|
||||
|
||||
held = sym in positions
|
||||
# 成交/信号记的是**当日该股最后一个可得收盘价**（停牌时就是最近一次收盘）。
|
||||
# 这与股息率序列的最后一点不是一回事：序列里只保留收益率非缺失的日期，
|
||||
# 价格则必须有值才能下单，所以这里单独取一次（O(log n) 切片）。
|
||||
close_hist = px_hist["close"].loc[: pd.Timestamp(day)]
|
||||
if close_hist.empty:
|
||||
continue
|
||||
price = float(close_hist.iloc[-1])
|
||||
# 动态股票池的语义核心：池子回答「今天能**买**什么」，
|
||||
# 不自动回答「必须卖什么」。掉出池子的持仓在 pool_exit_action=hold
|
||||
# 时只被禁止加仓，仍按股息率分位规则决定减仓/卖出。
|
||||
in_pool = sym in universe
|
||||
common = {
|
||||
"dividend_yield": round(current, 6),
|
||||
"yield_percentile": round(pct, 2),
|
||||
@@ -764,6 +906,8 @@ class BacktestEngine:
|
||||
"min_observations": int(
|
||||
self.bt_cfg.percentile_reference.min_observations),
|
||||
"close": price,
|
||||
"in_universe": bool(in_pool),
|
||||
"universe_size": len(universe),
|
||||
}
|
||||
|
||||
# 统一阶梯：先算目标仓位，再决定动作。
|
||||
@@ -779,48 +923,74 @@ class BacktestEngine:
|
||||
sym, day, current, pct, price, common, gate,
|
||||
))
|
||||
continue
|
||||
out.append(Signal(
|
||||
sym, day, "BUY", target, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"股息率历史分位 {pct:.1f}% >= P{s.entry.yield_percentile:g}，"
|
||||
f"目标仓位 {target:.0%}",
|
||||
"reason_cn": "股息率进入历史高位区间，达到买入阈值",
|
||||
**({"profile_gate": gate} if gate else {})},
|
||||
out.append(self._trade_signal(
|
||||
Signal(
|
||||
sym, day, "BUY", target, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"股息率历史分位 {pct:.1f}% >= P{s.entry.yield_percentile:g}，"
|
||||
f"目标仓位 {target:.0%}",
|
||||
"reason_cn": "股息率进入历史高位区间，达到买入阈值",
|
||||
**({"profile_gate": gate} if gate else {})},
|
||||
), day,
|
||||
))
|
||||
else:
|
||||
# --- 持仓掉出当日股票池 ---
|
||||
if not in_pool and self.pool_exit_action == "sell":
|
||||
out.append(self._trade_signal(Signal(
|
||||
sym, day, "SELL", 0.0, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"掉出当日股票池（池内 {len(universe)} 只），"
|
||||
f"pool_exit_action=sell → 清仓",
|
||||
"reason_cn": "动态股票池移出，按配置清仓"},
|
||||
), day))
|
||||
continue
|
||||
if target is None:
|
||||
continue # 死区：保持仓位
|
||||
if abs(target) <= 1e-9:
|
||||
out.append(Signal(
|
||||
out.append(self._trade_signal(Signal(
|
||||
sym, day, "SELL", 0.0, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"股息率历史分位 {pct:.1f}% <= P{s.exit.yield_percentile:g}",
|
||||
"reason_cn": "股息率回落至历史低位区间，达到卖出阈值，清仓"},
|
||||
))
|
||||
), day))
|
||||
elif target < 1.0:
|
||||
out.append(Signal(
|
||||
out.append(self._trade_signal(Signal(
|
||||
sym, day, "TRIM", target, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"分位 {pct:.1f}% 对应目标仓位 {target:.0%}",
|
||||
"reason_cn": "股息率分位变动，按阶梯规则调整仓位"},
|
||||
))
|
||||
), day))
|
||||
else:
|
||||
# ADD 也是买入 —— 同样要过实时画像闸门。
|
||||
# 被拒时**不动已有仓位**（REJECT 不进入待成交队列），
|
||||
# 因为闸门的语义是「不值得买」，不是「该卖」。
|
||||
if not in_pool and self.pool_exit_action == "hold":
|
||||
# 只减不加：留下一条 HOLD 记录，说明「想加但被池子挡住」。
|
||||
# 不进入待成交队列（kind 不在 BUY/ADD/SELL/TRIM 里）。
|
||||
out.append(self._trade_signal(Signal(
|
||||
sym, day, "HOLD", 0.0, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"分位 {pct:.1f}% 本应加仓至 {target:.0%}，"
|
||||
f"但该股已掉出当日股票池（pool_exit_action=hold）",
|
||||
"reason_cn": "已掉出动态股票池，停止加仓（不清仓）",
|
||||
"skip_reason": "OUT_OF_UNIVERSE",
|
||||
"executed": False,
|
||||
"rule_hit": "pool_exit_hold"},
|
||||
), day))
|
||||
continue
|
||||
gate = self._gate(sym, day)
|
||||
if gate is not None and gate["verdict"] != "PASS":
|
||||
out.append(self._reject_signal(
|
||||
sym, day, current, pct, price, common, gate,
|
||||
))
|
||||
continue
|
||||
out.append(Signal(
|
||||
out.append(self._trade_signal(Signal(
|
||||
sym, day, "ADD", target, current, pct, price,
|
||||
{**common,
|
||||
"rule": f"分位 {pct:.1f}% 对应目标仓位 {target:.0%}",
|
||||
"reason_cn": "股息率分位变动，按阶梯规则调整仓位",
|
||||
**({"profile_gate": gate} if gate else {})},
|
||||
))
|
||||
), day))
|
||||
return out
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -855,6 +1025,40 @@ class BacktestEngine:
|
||||
min_window_coverage=self.gate_cfg.min_window_coverage,
|
||||
)
|
||||
|
||||
def _trade_signal(self, sig: Signal, day: date) -> Signal:
|
||||
"""买卖信号的统一出口：在这里附加**个股画像留痕**。
|
||||
|
||||
``profile_on_trade``（daily 模式的默认）语义是「买卖决策发生时计算并留痕」：
|
||||
买入/加仓、卖出/减仓都会附上该股在决策日的实时画像，**不区分它是否在
|
||||
当日股票池内** —— 掉出池子的持仓被卖出时，更需要「当时它长什么样」的证据。
|
||||
|
||||
闸门（``profile_gate``）与此是两件事：闸门决定「买不买得到」，
|
||||
留痕只记录「当时看到了什么」。闸门关闭时留痕照常工作。
|
||||
"""
|
||||
note = self._profile_note(sig.symbol, day)
|
||||
if note is not None:
|
||||
sig.reason["profile"] = note
|
||||
return sig
|
||||
|
||||
def _profile_note(self, sym: str, day: date) -> dict[str, Any] | None:
|
||||
"""该股在 ``day`` 的实时画像快照（供 reason_json 留痕）。"""
|
||||
if not self.profile_on_trade or self.pit is None:
|
||||
return None
|
||||
snap = self.pit.snapshot(sym, day)
|
||||
if snap is None:
|
||||
return {"asof": str(day), "status": "NO_SNAPSHOT"}
|
||||
return {
|
||||
"asof": str(snap.asof),
|
||||
"window_years": snap.window_years,
|
||||
"values": {k: round(v, 8) for k, v in snap.values.items()
|
||||
if isinstance(v, (int, float))},
|
||||
"percentiles": {k: round(v, 4) for k, v in snap.percentiles.items()},
|
||||
"status": dict(snap.status),
|
||||
"n_obs": dict(snap.n_obs),
|
||||
"coverage": {k: round(v, 4) for k, v in snap.coverage.items()},
|
||||
"scores": {k: round(v, 4) for k, v in snap.scores.items()},
|
||||
}
|
||||
|
||||
def _reject_signal(
|
||||
self, sym: str, day: date, current: float, pct: float, price: float,
|
||||
common: dict[str, Any], gate: dict[str, Any],
|
||||
@@ -1068,38 +1272,62 @@ class BacktestEngine:
|
||||
这是「不复权价 + 独立分红现金流」的关键实现 ——
|
||||
与复权价配合会造成重复计算，因此价格一律用不复权。
|
||||
|
||||
**现金与送转必须各自独立判断**。纯送转（如 10 送 10：股价腰斩、
|
||||
股数翻倍）的 ``cash_div_tax`` 是 NULL/0 而 ``stk_div`` > 0；早期实现
|
||||
在算送股**之前**就按 ``per_share <= 0`` 整行 ``continue``，等于把纯送转
|
||||
丢掉 —— 而价格是不复权价、除权日仍会下跌，于是凭空记出一笔亏损。
|
||||
实测本库 2015-2026 区间内高股息池成员有 824 笔纯送转（0.3~1.2 股/股）。
|
||||
|
||||
分红**不计入成交流水**（``hd_backtest_trade`` 只记录买卖），
|
||||
它直接改变现金与持股数量，并通过净值曲线体现。
|
||||
"""
|
||||
stock_div_on = bool(self.bt_cfg.dividend.handle_stock_dividend)
|
||||
# 红利税总闸：backtest.yml 的 apply_dividend_tax 与 cost.yml 的
|
||||
# dividend_tax.enabled 必须同时为真（前者曾是死字段，改了不起作用）。
|
||||
tax_on = bool(self.bt_cfg.dividend.apply_dividend_tax)
|
||||
for r in ctx["div_by_date"].get(day, []):
|
||||
sym = r["symbol"]
|
||||
pos = positions.get(sym)
|
||||
if pos is None or pos.quantity <= 0:
|
||||
continue
|
||||
holding = pos.holding_days(day)
|
||||
rate = self.cost.dividend_tax_rate(holding)
|
||||
# 必须用 NaN 感知的转换：`float(x or 0.0)` 在 x=NaN 时会返回 NaN
|
||||
# （NaN 是真值），导致后续 gross/tax/net 全为 NaN，
|
||||
# 对账时 `NaN <= x` 为 False，表现为「税后大于税前」的假象。
|
||||
per_share = _fnum(r.get("cash_div_tax"))
|
||||
# 只处理**正的**现金分红：跳过 NULL / 0 / 负数（纯送转或数据异常）。
|
||||
if per_share is None or per_share <= 0:
|
||||
continue
|
||||
gross = pos.quantity * per_share
|
||||
tax = gross * rate
|
||||
net = gross - tax
|
||||
if net > 0:
|
||||
cash += net
|
||||
# 送转股：股数增加，成本不变（成本不变更符合税务口径）
|
||||
stk = _fnum(r.get("stk_div")) or 0.0
|
||||
has_cash = per_share is not None and per_share > 0
|
||||
has_stock = stk > 0
|
||||
# 既无现金也无送转：NULL / 0 / 负数（数据异常）—— 这才是该跳过的行
|
||||
if not has_cash and not has_stock:
|
||||
continue
|
||||
holding = pos.holding_days(day)
|
||||
rate = self.cost.dividend_tax_rate(holding) if tax_on else 0.0
|
||||
gross = tax = net = 0.0
|
||||
if has_cash:
|
||||
gross = pos.quantity * per_share
|
||||
tax = gross * rate
|
||||
net = gross - tax
|
||||
if net > 0:
|
||||
cash += net
|
||||
# 送转股：股数增加，**总成本不变** → 每股成本随之下降，与「不复权价
|
||||
# 在除权日下跌、股数补上」严格相抵，市值不变。
|
||||
# 不重算 avg_cost 会让后续卖出的 realized_pnl 按除权前的旧每股成本
|
||||
# 多扣成本（1000 股 @10 送 0.3 后全卖 @10 会记成 0 而非 +3000）。
|
||||
shares_added = 0.0
|
||||
if stk > 0 and self.bt_cfg.dividend.handle_stock_dividend:
|
||||
if has_stock and stock_div_on:
|
||||
shares_added = pos.quantity * stk
|
||||
pos.quantity += shares_added
|
||||
if pos.quantity > 0:
|
||||
pos.avg_cost = pos.cost_basis / pos.quantity
|
||||
ledger.append({
|
||||
"ex_date": day, "symbol": sym, "quantity": pos.quantity,
|
||||
"cash_div_tax": per_share, "gross": gross, "tax": tax, "net": net,
|
||||
"cash_div_tax": per_share if has_cash else 0.0,
|
||||
"stk_div": stk,
|
||||
"stk_bo_rate": _fnum(r.get("stk_bo_rate")) or 0.0,
|
||||
"stk_co_rate": _fnum(r.get("stk_co_rate")) or 0.0,
|
||||
"gross": gross, "tax": tax, "net": net,
|
||||
"holding_days": holding, "tax_rate": rate, "shares_added": shares_added,
|
||||
"stock_div_applied": bool(has_stock and stock_div_on),
|
||||
"cash_mode": self.bt_cfg.dividend.cash_mode,
|
||||
})
|
||||
return cash
|
||||
@@ -1350,6 +1578,47 @@ def _write(table: str, df: pd.DataFrame, cfg: Any, updates: list[str]) -> int:
|
||||
return n
|
||||
|
||||
|
||||
def build_yield_series(
|
||||
px_by_sym: dict[str, pd.DataFrame],
|
||||
events: dict[str, pd.DataFrame],
|
||||
*,
|
||||
ttm_days: int | None = None,
|
||||
grace_days: int | None = None,
|
||||
smooth_spikes: bool | None = None,
|
||||
) -> dict[str, pd.Series]:
|
||||
"""按股票预计算**整段**股息率序列（P2）。
|
||||
|
||||
引擎原先在每个评估日对每只股票从零重算一遍「到当天为止」的前缀序列，
|
||||
代价是 O(交易日数²)。``ttm_dps_series`` 的每一天取值只依赖「该日期 + 分红事件」，
|
||||
与传入的日期序列里还有哪些其它日期无关，所以「整段算一次再切片」与
|
||||
「每天按前缀重算」逐值相同（由
|
||||
``tests/test_daily.py::test_ttm_dps_series_prefix_equals_full`` 锁定）。
|
||||
|
||||
返回 ``{symbol: Series[close>0 的日期 -> 股息率]}``，索引为 ``DatetimeIndex``。
|
||||
"""
|
||||
w, g, sm = ttm_params()
|
||||
ttm_days = w if ttm_days is None else ttm_days
|
||||
grace_days = g if grace_days is None else grace_days
|
||||
smooth_spikes = sm if smooth_spikes is None else smooth_spikes
|
||||
|
||||
out: dict[str, pd.Series] = {}
|
||||
for sym, frame in px_by_sym.items():
|
||||
close = frame["close"]
|
||||
if close.empty:
|
||||
continue
|
||||
dps = ttm_dps_series(
|
||||
pd.DatetimeIndex(close.index), events.get(sym, pd.DataFrame()),
|
||||
ttm_days=ttm_days, grace_days=grace_days, smooth_spikes=smooth_spikes,
|
||||
)
|
||||
c = close.to_numpy(dtype="float64")
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
y = np.where(c > 0, dps / c, np.nan)
|
||||
s = pd.Series(y, index=close.index).dropna()
|
||||
if not s.empty:
|
||||
out[sym] = s
|
||||
return out
|
||||
|
||||
|
||||
def _fnum(v: Any) -> float | None:
|
||||
"""把值转成 float，None / NaN / 非数 一律返回 None。
|
||||
|
||||
@@ -1365,6 +1634,25 @@ def _fnum(v: Any) -> float | None:
|
||||
return None if (f != f or np.isinf(f)) else f
|
||||
|
||||
|
||||
def _pct(v: Any) -> str:
|
||||
"""百分比格式化；不可计算时显示「—」。
|
||||
|
||||
**短区间下 Sharpe / CAGR 会不可计算**（样本不足，见
|
||||
``analysis/performance.py``），而它们此前被直接 ``:.2%`` 格式化 ——
|
||||
实测：15 个交易日的回测会抛 ``TypeError: unsupported format string
|
||||
passed to NoneType`` 并以完整 traceback 结束。那不是「用户可理解的错误」，
|
||||
而是把「指标不可计算」这一个正常状态说成了程序缺陷。
|
||||
"""
|
||||
f = _fnum(v)
|
||||
return "—" if f is None else f"{f:.2%}"
|
||||
|
||||
|
||||
def _num(v: Any, digits: int = 2) -> str:
|
||||
"""数值格式化；不可计算时显示「—」。见 :func:`_pct`。"""
|
||||
f = _fnum(v)
|
||||
return "—" if f is None else f"{f:,.{digits}f}"
|
||||
|
||||
|
||||
def _round_lot(qty: float, lot: int = 100) -> float:
|
||||
"""A 股按手（100 股）取整。"""
|
||||
if qty <= 0:
|
||||
|
||||
@@ -163,16 +163,20 @@ class WalkForwardRunner:
|
||||
)
|
||||
|
||||
if verbose:
|
||||
# 用容忍 None 的格式化：短窗口下 CAGR / Sharpe 会不可计算，
|
||||
# 直接 :.2% 会抛 TypeError 并以 traceback 结束（见 engine._pct）。
|
||||
from hdiv.backtest.engine import _pct
|
||||
|
||||
print(
|
||||
f" 训练: 收益 {train_res['total_return']:>8.2%} "
|
||||
f"回撤 {train_res['max_drawdown']:>8.2%} "
|
||||
f"CAGR {train_res['cagr']:>7.2%} 成交 {train_res['trade_count']:>3d}",
|
||||
f" 训练: 收益 {_pct(train_res['total_return']):>8} "
|
||||
f"回撤 {_pct(train_res['max_drawdown']):>8} "
|
||||
f"CAGR {_pct(train_res['cagr']):>7} 成交 {train_res['trade_count']:>3d}",
|
||||
flush=True,
|
||||
)
|
||||
print(
|
||||
f" 测试: 收益 {test_res['total_return']:>8.2%} "
|
||||
f"回撤 {test_res['max_drawdown']:>8.2%} "
|
||||
f"CAGR {test_res['cagr']:>7.2%} 成交 {test_res['trade_count']:>3d}",
|
||||
f" 测试: 收益 {_pct(test_res['total_return']):>8} "
|
||||
f"回撤 {_pct(test_res['max_drawdown']):>8} "
|
||||
f"CAGR {_pct(test_res['cagr']):>7} 成交 {test_res['trade_count']:>3d}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
+154
-8
@@ -15,13 +15,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from hdiv.core.errors import HdivError
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from datetime import date, datetime
|
||||
from datetime import date
|
||||
|
||||
from hdiv import __version__
|
||||
from hdiv.core.errors import HdivError
|
||||
|
||||
|
||||
def _setup_logging(level: str) -> None:
|
||||
@@ -63,6 +63,21 @@ def cmd_ddl(args: argparse.Namespace) -> int:
|
||||
|
||||
def cmd_sync(args: argparse.Namespace) -> int:
|
||||
target = args.target
|
||||
if target == "daily":
|
||||
from hdiv.data.sync import daily
|
||||
|
||||
summary = daily.run(
|
||||
asof=date.fromisoformat(args.asof) if args.asof else None,
|
||||
lookback_days=args.lookback_days,
|
||||
only=args.only,
|
||||
include_financial=not args.no_financial,
|
||||
financial_limit=args.financial_limit,
|
||||
dry_run=args.dry_run,
|
||||
verbose=not args.json,
|
||||
)
|
||||
if args.json:
|
||||
print(json.dumps(summary, ensure_ascii=False, default=str, indent=2))
|
||||
return 0 if summary.get("ok", True) else 1
|
||||
if target == "dividend":
|
||||
from hdiv.data.sync import dividend
|
||||
|
||||
@@ -289,6 +304,64 @@ def cmd_backtest(args: argparse.Namespace) -> int:
|
||||
from hdiv.backtest.engine import BacktestEngine
|
||||
from hdiv.backtest.walk_forward import WalkForwardRunner
|
||||
|
||||
if args.mode == "daily":
|
||||
# 每日动态股票池：与 walk-forward 是两种不同的检验（见
|
||||
# hdiv/backtest/daily.py 的模块文档）。同样拒绝 --universe-run ——
|
||||
# 固定股票池与「每日动态」在定义上互斥，而且冻结名单自带未来信息。
|
||||
if args.universe_run:
|
||||
raise HdivError(
|
||||
"daily 模式不支持 --universe-run。\n"
|
||||
" 原因：daily 的定义就是「每个交易日按当时可见的数据重新选股」，\n"
|
||||
" 而固定股票池自带一个 asof（例如 2025-01-21），把它套到\n"
|
||||
" 更早的年份就是用未来信息选股（名单里含有回测起点时不可能\n"
|
||||
" 知道的信息）。\n"
|
||||
" 正确做法：去掉 --universe-run，让引擎每日重新筛选：\n"
|
||||
" python -m hdiv backtest --mode daily --start 2020-01-05\n"
|
||||
" 若确实要检验「固定股票池 + 月频调仓」，请用普通回测：\n"
|
||||
" python -m hdiv backtest --universe-run <run_id>"
|
||||
)
|
||||
_reject_no_persist_with_html(args, "hdiv backtest --mode daily")
|
||||
from hdiv.backtest.daily import DailyRunner
|
||||
|
||||
runner = DailyRunner.from_strategy(args.strategy)
|
||||
res = runner.run(
|
||||
start=date.fromisoformat(args.start) if args.start else None,
|
||||
end=date.fromisoformat(args.end) if args.end else None,
|
||||
persist=not args.no_persist,
|
||||
refresh_pools=bool(args.refresh_pools),
|
||||
every_n_days=args.every_n_days,
|
||||
signal_every_n_days=args.signal_every_n_days,
|
||||
)
|
||||
if args.no_persist:
|
||||
print()
|
||||
print("注意：--no-persist 已启用，本次回测与每日选股**未写入数据库**，"
|
||||
"不会出现在 Web 前端。")
|
||||
from hdiv.backtest.engine import _num as _fnum, _pct as _fpct
|
||||
|
||||
ds = res.get("daily_screening") or {}
|
||||
cadence = ""
|
||||
if int(ds.get("refresh_days") or 1) > 1 or int(ds.get("signal_every_days") or 1) > 1:
|
||||
cadence = (f"（粗粒度：选股每 {ds.get('refresh_days')} 日、"
|
||||
f"信号每 {ds.get('signal_every_days')} 日，"
|
||||
f"不可与逐日口径比较）")
|
||||
print(
|
||||
f"每日动态股票池回测 {res['run_id']} "
|
||||
f"{res['start_date']} ~ {res['end_date']}\n"
|
||||
f" 选股 {ds.get('screen_count')} 次（每 {ds.get('refresh_days')} 个交易日），"
|
||||
f"池内 {ds.get('pool_size_min')}~{ds.get('pool_size_max')} 只，"
|
||||
f"累计个股 {ds.get('distinct_symbols')} 只\n"
|
||||
f" 期初 {res['initial_capital']:,.0f} → 期末 {res['final_capital']:,.0f}\n"
|
||||
f" 总收益 {_fpct(res['total_return'])} CAGR {_fpct(res['cagr'])} "
|
||||
f"最大回撤 {_fpct(res['max_drawdown'])} Sharpe {_fnum(res['sharpe'])}\n"
|
||||
f" 成交 {res['trade_count']} 笔{cadence}"
|
||||
)
|
||||
_print_frontend_hint(res["run_id"])
|
||||
if args.html:
|
||||
from hdiv.report.build import build_backtest_report
|
||||
|
||||
print("HTML：", build_backtest_report(res["run_id"]))
|
||||
return 0
|
||||
|
||||
if args.mode == "walkforward":
|
||||
# 冻结股票池与 walk-forward 在时序上不兼容：
|
||||
# 股票池有其自身的 asof（如 2025-01-21），而 walk-forward 的窗口从
|
||||
@@ -326,13 +399,16 @@ def cmd_backtest(args: argparse.Namespace) -> int:
|
||||
if args.no_persist:
|
||||
print()
|
||||
print("注意：--no-persist 已启用，本次回测**未写入数据库**，不会出现在 Web 前端。")
|
||||
from hdiv.backtest.engine import _num as _fnum, _pct as _fpct
|
||||
|
||||
print(
|
||||
f"回测 {res['run_id']} {res['start_date']} ~ {res['end_date']}\n"
|
||||
f" 期初 {res['initial_capital']:,.0f} → 期末 {res['final_capital']:,.0f}\n"
|
||||
f" 总收益 {res['total_return']:.2%} CAGR {res['cagr']:.2%} "
|
||||
f"最大回撤 {res['max_drawdown']:.2%} Sharpe {res['sharpe']:.2f}\n"
|
||||
f" 总收益 {_fpct(res['total_return'])} CAGR {_fpct(res['cagr'])} "
|
||||
f"最大回撤 {_fpct(res['max_drawdown'])} Sharpe {_fnum(res['sharpe'])}\n"
|
||||
f" 成交 {res['trade_count']} 笔"
|
||||
)
|
||||
_print_frontend_hint(res["run_id"])
|
||||
if args.html:
|
||||
from hdiv.report.build import build_backtest_report
|
||||
|
||||
@@ -340,6 +416,25 @@ def cmd_backtest(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _print_frontend_hint(run_id: str) -> None:
|
||||
"""告诉用户「去哪里看明细」——结果已经入库，不需要再跑第二条命令。
|
||||
|
||||
这一步是刻意的：``--mode daily`` 一次运行会把完整明细写进
|
||||
``hd_backtest_run/_equity/_position/_trade/_signal/_metric`` 与
|
||||
``hd_daily_universe``，前端「回测记录」页能直接下钻。曾把「导出结果」
|
||||
写成需要另跑一个脚本并手工传 run_id，那等于让用户猜 run_id 从哪来 ——
|
||||
而它本来就在上面这一行里。
|
||||
"""
|
||||
print(
|
||||
f"\n明细已入库，直接在前端查看（无需再执行其它命令）：\n"
|
||||
f" 打开 Web 前端 → 「回测记录」→ 选中 run_id "
|
||||
f"{run_id[:12]}…（模式 daily）\n"
|
||||
f" 页面内可看：净值曲线与基准、持仓明细、逐笔成交与理由、\n"
|
||||
f" 成交个股的实时画像、每日动态股票池（逐日选股留痕）、\n"
|
||||
f" 点任一成交个股 → 趋势与买卖点 + 决策时点实时画像"
|
||||
)
|
||||
|
||||
|
||||
def cmd_sensitivity(args: argparse.Namespace) -> int:
|
||||
from hdiv.analysis.sensitivity import SensitivityRunner
|
||||
|
||||
@@ -384,7 +479,7 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
)
|
||||
s.add_argument(
|
||||
"target",
|
||||
choices=["dividend", "financial", "index", "price", "trading", "backfill"],
|
||||
choices=["daily", "dividend", "financial", "index", "price", "trading", "backfill"],
|
||||
)
|
||||
s.add_argument("--symbols", nargs="*")
|
||||
s.add_argument("--only-missing", action="store_true")
|
||||
@@ -405,6 +500,33 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
s.add_argument("--basic-end", default="2019-12-31", help="daily_basic 回补终点")
|
||||
s.add_argument("--no-resume", action="store_true")
|
||||
s.add_argument("--no-weight", action="store_true")
|
||||
# ---- sync daily：按缺口增量补齐（缺几天抓几天）----
|
||||
s.add_argument(
|
||||
"--asof", default=None,
|
||||
help="sync daily：以哪天为「今天」（YYYY-MM-DD，默认系统当天）",
|
||||
)
|
||||
s.add_argument(
|
||||
"--lookback-days", type=int, default=45,
|
||||
help="sync daily：向前回溯多少自然日找缺口（默认 45；更早的空洞属于回补）",
|
||||
)
|
||||
s.add_argument(
|
||||
"--only", nargs="*", default=None,
|
||||
help="sync daily：只同步指定目标（price/trading/index/dividend/financial，"
|
||||
"或单个目标 daily/adj_factor/daily_basic/suspend/limit）",
|
||||
)
|
||||
s.add_argument(
|
||||
"--no-financial", action="store_true",
|
||||
help="sync daily：跳过财报四表（它们是按股票拉取，最耗时）",
|
||||
)
|
||||
s.add_argument(
|
||||
"--financial-limit", type=int, default=500,
|
||||
help="sync daily：单次最多重拉多少只股票的财报（按市值降序，默认 500）",
|
||||
)
|
||||
s.add_argument(
|
||||
"--dry-run", action="store_true",
|
||||
help="sync daily：只打印待抓清单，不调用接口、不写库",
|
||||
)
|
||||
s.add_argument("--json", action="store_true", help="sync daily：以 JSON 输出结果")
|
||||
s.set_defaults(func=cmd_sync)
|
||||
|
||||
# audit
|
||||
@@ -505,10 +627,18 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
# backtest
|
||||
b = sub.add_parser(
|
||||
"backtest", help="回测与 Walk-forward",
|
||||
description="执行历史回测；--mode walkforward 执行滚动样本外验证。",
|
||||
description=(
|
||||
"执行历史回测。--mode walkforward 执行滚动样本外验证（过拟合检验）；"
|
||||
"--mode daily 执行「每日动态股票池」连续推演：从 --start 起每个交易日"
|
||||
"重新选股、每个交易日判断买卖点。"
|
||||
),
|
||||
)
|
||||
b.add_argument("-s", "--strategy", default="config/strategy/high_dividend_v1.yml")
|
||||
b.add_argument("--mode", choices=["single", "walkforward"], default="single")
|
||||
b.add_argument(
|
||||
"--mode", choices=["single", "walkforward", "daily"], default="single",
|
||||
help="single=单条路径回测；walkforward=滚动样本外；"
|
||||
"daily=每日动态股票池（每个交易日重新选股）",
|
||||
)
|
||||
b.add_argument("--start", default=None)
|
||||
b.add_argument("--end", default=None)
|
||||
b.add_argument(
|
||||
@@ -522,6 +652,22 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
"会如实写入 hd_backtest_run.unimplemented_json）",
|
||||
)
|
||||
b.add_argument("--no-persist", action="store_true")
|
||||
b.add_argument(
|
||||
"--refresh-pools", action="store_true",
|
||||
help="daily 模式：忽略已缓存的每日选股结果，强制重新逐日筛选（默认复用缓存）",
|
||||
)
|
||||
b.add_argument(
|
||||
"--every-n-days", type=int, default=None, metavar="N",
|
||||
help="daily 模式：股票池每 N 个交易日重建一次（默认 1 = 每个交易日）。"
|
||||
"N 越大选股耗时越少（近似 ÷N），代价是池子更新变粗；"
|
||||
"判定规则不变，但结果不可与逐日口径直接比较",
|
||||
)
|
||||
b.add_argument(
|
||||
"--signal-every-n-days", type=int, default=None, metavar="N",
|
||||
help="daily 模式：买卖每 N 个交易日判断一次（默认 1 = 每个交易日）。"
|
||||
"这是**模拟阶段**的主要速度旋钮（信号评估 + 逐笔决策画像都按 N 摊薄），"
|
||||
"但交易机会与换手会随之下降",
|
||||
)
|
||||
# HTML 报告已降级为「导出件」：默认不生成，需要时显式 --html。
|
||||
# --no-html 保留为空操作，避免历史命令与脚本报错。
|
||||
b.add_argument("--no-html", action="store_true",
|
||||
|
||||
@@ -515,6 +515,51 @@ class PercentileReferenceConfig(StrictModel):
|
||||
return self
|
||||
|
||||
|
||||
class DailyConfig(StrictModel):
|
||||
"""每日动态股票池模式（``backtest --mode daily``）。
|
||||
|
||||
与 walk-forward 的区别在**时间结构与目的**：walk-forward 切多个
|
||||
(train, test) 窗口来检验过拟合；daily 只跑**一条连续路径**
|
||||
``[start, latest]``，目的回答「从某天起按这套规则每天重新选股、每天判断
|
||||
买卖，实际会怎样」。其中「股票池每日变化」是本模式的核心，因此：
|
||||
|
||||
- ``universe_refresh_days=1``：**每个交易日**按当日可见数据重建股票池
|
||||
（PIT），而不是 walk-forward 的「每个窗口一次」；
|
||||
- ``signal_frequency_days=1``：每个交易日评估买卖点；
|
||||
- ``pool_exit_action``：持仓掉出当日股票池后的处置 ——
|
||||
``hold`` = **只减不加**（默认，符合「池子决定能买什么，不决定必须卖」），
|
||||
``sell`` = 掉出即清仓。
|
||||
|
||||
**刻意没有 ``enabled`` 开关**：模式由命令行 ``--mode daily`` 显式选择。
|
||||
加一个「配置写了但没有任何代码路径会读」的字段，正是本项目反复记录的
|
||||
「配置承诺与实际行为不一致」那一类缺陷。
|
||||
"""
|
||||
|
||||
#: 股票池重建频率（交易日）。1 = 每个交易日重新筛选
|
||||
universe_refresh_days: int = 1
|
||||
#: 信号评估频率（交易日）。1 = 每个交易日评估
|
||||
signal_frequency_days: int = 1
|
||||
#: 持仓掉出当日股票池后的处置：hold = 只减不加；sell = 清仓
|
||||
pool_exit_action: Literal["hold", "sell"] = "hold"
|
||||
#: 买卖决策发生时计算并留痕个股画像（无论该股是否在当日池内）
|
||||
profile_on_trade: bool = True
|
||||
#: 是否把每日入选成员写入 hd_daily_universe
|
||||
persist_daily_universe: bool = True
|
||||
#: 批量预载的分块年数（内存控制）
|
||||
chunk_years: int = 1
|
||||
#: 每日股票池重建时打印进度的间隔（交易日）；1 = 每天都打印
|
||||
progress_every_days: int = 20
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _check(self) -> DailyConfig:
|
||||
for name in ("universe_refresh_days", "signal_frequency_days", "chunk_years"):
|
||||
if getattr(self, name) <= 0:
|
||||
raise SchemaValidationError(f"daily.{name} 必须为正")
|
||||
if self.progress_every_days <= 0:
|
||||
raise SchemaValidationError("daily.progress_every_days 必须为正")
|
||||
return self
|
||||
|
||||
|
||||
class BacktestConfig(StrictModel):
|
||||
version: int = 1
|
||||
capital: CapitalConfig = Field(default_factory=CapitalConfig)
|
||||
@@ -524,6 +569,7 @@ class BacktestConfig(StrictModel):
|
||||
default_factory=PercentileReferenceConfig
|
||||
)
|
||||
walk_forward: WalkForwardConfig = Field(default_factory=WalkForwardConfig)
|
||||
daily: DailyConfig = Field(default_factory=DailyConfig)
|
||||
dividend: DividendHandlingConfig = Field(default_factory=DividendHandlingConfig)
|
||||
benchmark: list[BenchmarkConfig] = Field(default_factory=list)
|
||||
risk_free_rate: float = 0.02
|
||||
|
||||
@@ -497,6 +497,37 @@ CREATE TABLE IF NOT EXISTS `hd_universe_member` (
|
||||
""",
|
||||
)
|
||||
|
||||
T_DAILY_UNIVERSE = Table(
|
||||
name="hd_daily_universe",
|
||||
comment="每日动态股票池成员（daily 模式「每日选股」的留痕）",
|
||||
ddl=f"""
|
||||
CREATE TABLE IF NOT EXISTS `hd_daily_universe` (
|
||||
`id` BIGINT NOT NULL AUTO_INCREMENT,
|
||||
`run_id` VARCHAR(32) NOT NULL COMMENT '所属回测 run_id',
|
||||
`trade_date` DATE NOT NULL COMMENT '该交易日的选股结果',
|
||||
`symbol` VARCHAR(12) NOT NULL,
|
||||
`name` VARCHAR(64) NULL,
|
||||
`industry` VARCHAR(64) NULL,
|
||||
`dividend_yield` DECIMAL(18,8) NULL COMMENT '入选当日股息率（池内排序口径）',
|
||||
`total_mv` DECIMAL(24,4) NULL,
|
||||
`roe_avg` DECIMAL(18,8) NULL,
|
||||
`listed_count` INT NULL COMMENT '当日市场候选数（未预剪枝）',
|
||||
`candidate_count` INT NULL COMMENT '当日实际参与筛选的候选数（已预剪枝）',
|
||||
`values_json` TEXT NULL COMMENT '入选时的关键因子快照',
|
||||
`created_at` DATETIME NOT NULL,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_hd_daily_uni` (`run_id`,`trade_date`,`symbol`),
|
||||
KEY `ix_hd_daily_uni_date` (`trade_date`),
|
||||
KEY `ix_hd_daily_uni_sym` (`symbol`,`trade_date`)
|
||||
) {CHARSET} COMMENT='每日动态股票池成员'
|
||||
""",
|
||||
# 建表时已含上述列；这里的 added_columns 是为了让**先建表、后加列**的库
|
||||
# 也能幂等补上（CREATE TABLE IF NOT EXISTS 不会修改已存在的表）。
|
||||
added_columns={
|
||||
"listed_count": "INT NULL COMMENT '当日市场候选数（未预剪枝）'",
|
||||
},
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# C. 因子与画像层
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1034,6 +1065,7 @@ ALL_TABLES: tuple[Table, ...] = (
|
||||
# B. 股票池层
|
||||
T_UNIVERSE_RUN,
|
||||
T_UNIVERSE_MEMBER,
|
||||
T_DAILY_UNIVERSE,
|
||||
# C. 因子与画像层
|
||||
T_FACTOR_SNAPSHOT,
|
||||
T_PROFILE_RUN,
|
||||
|
||||
@@ -0,0 +1,729 @@
|
||||
"""每日增量同步 —— 「缺几天就抓几天」。
|
||||
|
||||
本模块是 ``hdiv sync daily`` 的实现，用途只有一个：
|
||||
**在每天收盘后（默认 17:00）把外部数据源里「本地还没有的那几天」补下来**，
|
||||
不重拉已经完整的历史，也不碰已经存在的行。
|
||||
|
||||
外部数据源与本地表的对应关系（只有这些表是「抓来的」）::
|
||||
|
||||
本地表 数据源接口 分区方式 缺口判定
|
||||
------------------ --------------- -------------------- ------------------------------
|
||||
stock_daily daily trade_date 当日股票数 ≥ 当年规模阈值
|
||||
adjust_factor adj_factor trade_date 同上
|
||||
daily_basic daily_basic trade_date 同上
|
||||
hd_suspend suspend_d trade_date 有行即视为已同步
|
||||
hd_limit stk_limit trade_date 当日股票数 ≥ 当年规模阈值
|
||||
hd_index_daily index_daily (指数, trade_date) 每个指数各自的最后一天
|
||||
hd_dividend dividend ann/imp_ann/ex/record 四个日期列都查过才算同步
|
||||
hd_fina_indicator fina_indicator ts_code（接口强制） 缺股票 / 报告期滞后
|
||||
hd_cashflow cashflow ts_code（接口强制） 同上
|
||||
hd_balancesheet balancesheet ts_code（接口强制） 同上
|
||||
hd_income income ts_code（接口强制） 同上
|
||||
index_weight index_weight 月度区间 最后一个权重日之后
|
||||
|
||||
**为什么财报四表不能按天**：实测 ``fina_indicator`` / ``income`` / ``balancesheet`` /
|
||||
``cashflow`` 传 ``period`` / ``ann_date`` / ``start_date`` 而不传 ``ts_code`` 一律返回
|
||||
``50101 必填参数, ts_code`` —— 服务端强制按股票拉取。所以这四张表只能按
|
||||
「股票 × 报告期」补：先补完全没数据的股票，再按报告期水位补滞后的股票，
|
||||
每次运行有上限（``--financial-limit``），积压会在随后的每天里自动排空。
|
||||
|
||||
**为什么分红可以按天**：``dividend`` 接口支持 ``ann_date`` / ``imp_ann_date`` /
|
||||
``ex_date`` / ``record_date`` 四种日期参数（实测可用），因此不必像早期实现那样
|
||||
逐只股票重拉全历史（5,900 次调用）。这里改为「缺哪天查哪天」——
|
||||
每天最多 4 次调用，且四个日期列都查，避免漏掉「预案日已过、除权日未到」的记录。
|
||||
|
||||
**写入门槛**：``stock_daily`` / ``adjust_factor`` / ``daily_basic`` 是 qlib 的
|
||||
**只读表**，写入需要 ``HDIV_ALLOW_BACKFILL=1``（StatementGuard 的安全开关）。
|
||||
:func:`run` 会在进程内自动打开它 —— 这是本项目**唯一**被授权写入这些表的日常通道，
|
||||
且一律使用 ``INSERT IGNORE``，冲突行完全不改动（见 ``price.sync_days``）。
|
||||
``index_weight`` 已在 ``allow_write_tables`` 中，不受该开关限制。
|
||||
|
||||
**重复查询的取舍**：分红表里「一条记录都没有」的股票（实测 23 只，全是刚上市、
|
||||
尚未分红的次新股）每天都会被重查一次 —— 这正是想要的：它们第一次分红公告要能
|
||||
当天入库。财报四表则相反，已退市多年的标的必须排除（见 :data:`DELIST_GRACE_DAYS`），
|
||||
否则每天白拉 800 次调用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from hdiv.core.config import DataSourceConfig, load_config
|
||||
from hdiv.data import db
|
||||
from hdiv.data.sync import dividend as dividend_sync
|
||||
from hdiv.data.sync import financial as financial_sync
|
||||
from hdiv.data.sync import index as index_sync
|
||||
from hdiv.data.sync import price as price_sync
|
||||
from hdiv.data.sync import trading as trading_sync
|
||||
from hdiv.data.sync.base import symbols_by_priority, sync_job, to_date, upsert
|
||||
from hdiv.data.tushare_client import TushareClient
|
||||
|
||||
#: 默认回溯窗口（自然日）。日频表只在这个窗口内找缺口 ——
|
||||
#: 更早的历史空洞属于「回补」而不是「每日增量」，由 ``hdiv audit`` 报告、
|
||||
#: 由 ``hdiv sync backfill`` 处理。窗口存在是为了「昨夜失败今晨自愈」。
|
||||
DEFAULT_LOOKBACK_DAYS = 45
|
||||
|
||||
#: 单次运行最多重拉多少只股票的财报（按市值降序）。
|
||||
DEFAULT_FINANCIAL_LIMIT = 500
|
||||
|
||||
#: 分红按日期补时向前多查几天：公告日与除权日之间常有几天差，
|
||||
#: 单日查询会漏掉「昨天公告、今天才入库」的记录。
|
||||
DIVIDEND_OVERLAP_DAYS = 7
|
||||
|
||||
#: 已退市超过这么多天的标的，财报不会再更新，不再进入每日队列。
|
||||
#:
|
||||
#: 为什么必须有这条规则：实测「报告期滞后」的股票里 **200+ 只已退市**
|
||||
#: （最后一份财报停在退市前，如 1996-12-31）。不退市过滤的话，它们每天都会被
|
||||
#: 重拉一次（4 个接口 × 200 只 = 800 次调用/天），而结果永远是同一批旧数据 ——
|
||||
#: 实测一次全量重拉耗时约 2 分钟、返回 6.5 万行，全是无功而返。
|
||||
#: 留下 90 天宽限期，是为了兜住「刚退市、最终财报还没入库」的情况。
|
||||
#: 更早的历史缺口属于**一次性回补**：hdiv sync financial --only-missing
|
||||
DELIST_GRACE_DAYS = 90
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 目标描述
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DayTarget:
|
||||
"""按交易日分区的同步目标。"""
|
||||
|
||||
key: str
|
||||
table: str
|
||||
api: str
|
||||
label: str
|
||||
which: str | None = None # price.SPECS 的键；仅行情三表有
|
||||
min_symbols: int | None = None # 显式覆盖完整性阈值（停牌表用 1）
|
||||
group: str = "price"
|
||||
|
||||
|
||||
DAY_TARGETS: tuple[DayTarget, ...] = (
|
||||
DayTarget("daily", "stock_daily", "daily", "日线行情", which="daily"),
|
||||
DayTarget("adj_factor", "adjust_factor", "adj_factor", "复权因子", which="adj_factor"),
|
||||
DayTarget("daily_basic", "daily_basic", "daily_basic", "每日指标", which="daily_basic"),
|
||||
DayTarget("suspend", "hd_suspend", "suspend_d", "停牌记录", min_symbols=1, group="trading"),
|
||||
DayTarget("limit", "hd_limit", "stk_limit", "涨跌停价", group="trading"),
|
||||
)
|
||||
|
||||
DAY_TARGET_BY_KEY = {t.key: t for t in DAY_TARGETS}
|
||||
|
||||
#: 财报四表（接口强制按 ts_code 拉取）
|
||||
FINANCIAL_TABLES = ("hd_fina_indicator", "hd_cashflow", "hd_balancesheet", "hd_income")
|
||||
|
||||
#: 分红可用的日期参数：四个都查，避免漏记录
|
||||
DIVIDEND_DATE_PARAMS = ("ann_date", "imp_ann_date", "ex_date", "record_date")
|
||||
|
||||
#: ``--only`` 允许的组名
|
||||
GROUPS = ("price", "trading", "index", "dividend", "financial")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 计划（纯读，不写库）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyPlan:
|
||||
"""一次每日增量同步的「待抓清单」。"""
|
||||
|
||||
asof: date
|
||||
window_start: date
|
||||
day_gaps: dict[str, list[date]] = field(default_factory=dict)
|
||||
index_days: dict[str, list[date]] = field(default_factory=dict)
|
||||
dividend_days: list[date] = field(default_factory=list)
|
||||
dividend_symbols: list[str] = field(default_factory=list)
|
||||
financial_symbols: list[str] = field(default_factory=list)
|
||||
financial_missing: dict[str, list[str]] = field(default_factory=dict)
|
||||
financial_stale: dict[str, list[str]] = field(default_factory=dict)
|
||||
financial_watermark: date | None = None
|
||||
index_weight_window: tuple[date, date] | None = None
|
||||
notes: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def total_days(self) -> int:
|
||||
"""所有按日分区的缺口天数合计（不含分红，分红按「四个日期列」计）。"""
|
||||
return sum(len(v) for v in self.day_gaps.values()) + sum(
|
||||
len(v) for v in self.index_days.values()
|
||||
)
|
||||
|
||||
@property
|
||||
def empty(self) -> bool:
|
||||
return not (
|
||||
any(self.day_gaps.values())
|
||||
or any(self.index_days.values())
|
||||
or self.dividend_days
|
||||
or self.dividend_symbols
|
||||
or self.financial_symbols
|
||||
or self.index_weight_window
|
||||
)
|
||||
|
||||
|
||||
def _trading_days(start: date, end: date, cfg: DataSourceConfig) -> list[date]:
|
||||
"""交易日列表；区间内没有交易日（长假）时返回空而不是抛错。"""
|
||||
if start > end:
|
||||
return []
|
||||
try:
|
||||
return list(price_sync.open_days(start, end, cfg))
|
||||
except RuntimeError:
|
||||
return []
|
||||
|
||||
|
||||
def missing_days(
|
||||
table: str,
|
||||
start: date,
|
||||
end: date,
|
||||
cfg: DataSourceConfig,
|
||||
*,
|
||||
min_symbols: int | None = None,
|
||||
) -> list[date]:
|
||||
"""``[start, end]`` 内**数据不完整**的交易日。
|
||||
|
||||
「不完整」的口径与断点续传完全一致（``price.fetched_days``）：
|
||||
当日股票数 ≥ ``max(下限, 比例 × 当年应有上市股票数)``。
|
||||
这样半成品日期（只填了几百只）也会被重抓，而不是被当成已完成跳过。
|
||||
"""
|
||||
days = _trading_days(start, end, cfg)
|
||||
if not days:
|
||||
return []
|
||||
done = price_sync.fetched_days(table, start, end, cfg, min_symbols=min_symbols)
|
||||
return [d for d in days if d not in done]
|
||||
|
||||
|
||||
def index_daily_gaps(
|
||||
start: date, end: date, cfg: DataSourceConfig
|
||||
) -> dict[str, list[date]]:
|
||||
"""每个指数各自缺失的交易日（指数接口一次调用返回整段区间）。"""
|
||||
days = _trading_days(start, end, cfg)
|
||||
if not days:
|
||||
return {}
|
||||
df = db.read_sql(
|
||||
"SELECT index_code AS c, MAX(trade_date) AS m FROM hd_index_daily GROUP BY index_code",
|
||||
cfg=cfg,
|
||||
)
|
||||
have = {str(r.c): to_date(r.m) for r in df.itertuples()}
|
||||
out: dict[str, list[date]] = {}
|
||||
for item in index_sync.default_indices(cfg):
|
||||
code = item["code"]
|
||||
last = have.get(code)
|
||||
if last is None:
|
||||
# 从未同步过的指数：只补窗口内的（全历史回补请用 hdiv sync index --start）
|
||||
out[code] = list(days)
|
||||
continue
|
||||
gap = [d for d in days if d > last]
|
||||
if gap:
|
||||
out[code] = gap
|
||||
return out
|
||||
|
||||
|
||||
def dividend_gaps(
|
||||
start: date, end: date, asof: date, cfg: DataSourceConfig
|
||||
) -> tuple[list[date], date]:
|
||||
"""分红表缺失的交易日。
|
||||
|
||||
分红没有「每天应有 N 条」的规模口径，因此退化为与 ``hd_suspend`` 相同的
|
||||
「查过即算」：四个日期列里任何一列在该交易日有行，就认为那天已经查过。
|
||||
代价是**真正没有分红记录的交易日会被重复查询**（每天 4 次调用、返回空），
|
||||
收益是绝不漏记录 —— 与 ``trading.py`` 对停牌表的取舍一致。
|
||||
|
||||
返回 ``(待查交易日, 锚点日期)``；锚点用于把窗口收窄到「上次数据附近」，
|
||||
避免每天把整个回溯窗口重查一遍。
|
||||
"""
|
||||
df = db.read_sql(
|
||||
"SELECT MAX(d) AS m FROM ("
|
||||
" SELECT MAX(ann_date) AS d FROM hd_dividend"
|
||||
" UNION ALL SELECT MAX(imp_ann_date) FROM hd_dividend"
|
||||
" UNION ALL SELECT MAX(ex_date) FROM hd_dividend"
|
||||
" UNION ALL SELECT MAX(record_date) FROM hd_dividend"
|
||||
") t",
|
||||
cfg=cfg,
|
||||
)
|
||||
anchor = to_date(df["m"].iloc[0]) if not df.empty else None
|
||||
# 未来日期（已公告但尚未除权）不能当锚点，否则窗口会落在未来
|
||||
if anchor is None or anchor > asof:
|
||||
anchor = asof
|
||||
since = max(start, anchor - timedelta(days=DIVIDEND_OVERLAP_DAYS))
|
||||
|
||||
days = _trading_days(since, end, cfg)
|
||||
if not days:
|
||||
return [], anchor
|
||||
have = db.read_sql(
|
||||
"SELECT DISTINCT d FROM ("
|
||||
" SELECT ann_date AS d FROM hd_dividend WHERE ann_date BETWEEN :s AND :e"
|
||||
" UNION SELECT imp_ann_date FROM hd_dividend WHERE imp_ann_date BETWEEN :s AND :e"
|
||||
" UNION SELECT ex_date FROM hd_dividend WHERE ex_date BETWEEN :s AND :e"
|
||||
" UNION SELECT record_date FROM hd_dividend WHERE record_date BETWEEN :s AND :e"
|
||||
") t",
|
||||
{"s": since, "e": end},
|
||||
cfg=cfg,
|
||||
)
|
||||
seen = {to_date(x) for x in have["d"].tolist()} if not have.empty else set()
|
||||
return [d for d in days if d not in seen], anchor
|
||||
|
||||
|
||||
def _active_since(asof: date) -> date:
|
||||
"""退市宽限期的截止日：早于它退市的标的不再进入每日队列。"""
|
||||
return asof - timedelta(days=DELIST_GRACE_DAYS)
|
||||
|
||||
|
||||
def _symbols_without_rows(
|
||||
table: str, cfg: DataSourceConfig, *, asof: date | None = None
|
||||
) -> list[str]:
|
||||
"""``stock`` 里有、目标表里一条记录都没有、且**未长期退市**的股票。"""
|
||||
cutoff = _active_since(asof or datetime.now().date())
|
||||
df = db.read_sql(
|
||||
f"SELECT s.symbol AS symbol FROM stock s "
|
||||
f"WHERE (s.delist_date IS NULL OR s.delist_date > :cutoff) "
|
||||
f"AND NOT EXISTS (SELECT 1 FROM `{table}` x WHERE x.symbol = s.symbol)",
|
||||
{"cutoff": cutoff},
|
||||
cfg=cfg,
|
||||
)
|
||||
return [str(x) for x in df["symbol"].tolist()] if not df.empty else []
|
||||
|
||||
|
||||
def _symbols_lagging(
|
||||
table: str, watermark: date, cfg: DataSourceConfig, *, asof: date | None = None
|
||||
) -> list[str]:
|
||||
"""有数据但最新报告期早于水位、且仍可能出新财报的股票（按市值降序）。"""
|
||||
cutoff = _active_since(asof or datetime.now().date())
|
||||
df = db.read_sql(
|
||||
f"SELECT s.symbol AS symbol, t.mx AS mx FROM stock s "
|
||||
f"LEFT JOIN (SELECT symbol, MAX(end_date) AS mx FROM `{table}` GROUP BY symbol) t "
|
||||
f"ON t.symbol = s.symbol "
|
||||
f"WHERE s.delist_date IS NULL OR s.delist_date > :cutoff",
|
||||
{"cutoff": cutoff},
|
||||
cfg=cfg,
|
||||
)
|
||||
out: list[str] = []
|
||||
for r in df.itertuples():
|
||||
mx = to_date(r.mx)
|
||||
if mx is None or mx < watermark:
|
||||
out.append(str(r.symbol))
|
||||
return symbols_by_priority(cfg, out)
|
||||
|
||||
|
||||
def financial_watermark(asof: date) -> date:
|
||||
"""按 A 股披露截止日推算「此刻理应已披露的最新报告期」。
|
||||
|
||||
年报 4/30、一季报 4/30、半年报 8/31、三季报 10/31。取**已过截止日**的
|
||||
最近一期作为水位：最新报告期晚于水位的股票才算「滞后」，
|
||||
这样财报季里刚披露的公司会立刻退出待抓队列，不必重拉全市场。
|
||||
"""
|
||||
y = asof.year
|
||||
if asof >= date(y, 11, 1):
|
||||
return date(y, 9, 30)
|
||||
if asof >= date(y, 9, 1):
|
||||
return date(y, 6, 30)
|
||||
if asof >= date(y, 5, 1):
|
||||
return date(y, 3, 31)
|
||||
# 1~4 月是年报季（上一年 12-31 的年报 4/30 前披露）
|
||||
return date(y - 1, 12, 31)
|
||||
|
||||
|
||||
def _financial_candidates(
|
||||
watermark: date, asof: date, cfg: DataSourceConfig
|
||||
) -> tuple[dict[str, list[str]], dict[str, list[str]]]:
|
||||
missing = {t: _symbols_without_rows(t, cfg, asof=asof) for t in FINANCIAL_TABLES}
|
||||
stale = {t: _symbols_lagging(t, watermark, cfg, asof=asof) for t in FINANCIAL_TABLES}
|
||||
return missing, stale
|
||||
|
||||
|
||||
def plan(
|
||||
cfg: DataSourceConfig | None = None,
|
||||
*,
|
||||
asof: date | None = None,
|
||||
lookback_days: int = DEFAULT_LOOKBACK_DAYS,
|
||||
only: Sequence[str] | None = None,
|
||||
include_financial: bool = True,
|
||||
financial_limit: int = DEFAULT_FINANCIAL_LIMIT,
|
||||
) -> DailyPlan:
|
||||
"""算出待抓清单（只读，不调用 Tushare、不写库）。"""
|
||||
if cfg is None:
|
||||
db.load_dotenv_once()
|
||||
cfg = load_config("datasource")
|
||||
asof = asof or datetime.now().date()
|
||||
if isinstance(asof, datetime):
|
||||
asof = asof.date()
|
||||
window_start = asof - timedelta(days=max(1, lookback_days))
|
||||
targets = _resolve_only(only)
|
||||
|
||||
out = DailyPlan(asof=asof, window_start=window_start)
|
||||
|
||||
if "price" in targets or "trading" in targets:
|
||||
for t in DAY_TARGETS:
|
||||
if t.group not in targets:
|
||||
continue
|
||||
out.day_gaps[t.key] = missing_days(
|
||||
t.table, window_start, asof, cfg, min_symbols=t.min_symbols
|
||||
)
|
||||
|
||||
if "index" in targets:
|
||||
out.index_days = index_daily_gaps(window_start, asof, cfg)
|
||||
out.index_weight_window = _index_weight_window(window_start, asof, cfg)
|
||||
if out.index_weight_window and db.row_count("index_weight", cfg) == 0:
|
||||
out.notes.append(
|
||||
"index_weight 为空：本次只补最近窗口。全历史成分股权重是一次性回补，"
|
||||
"请执行 hdiv sync index --start 20150101"
|
||||
)
|
||||
|
||||
if "dividend" in targets:
|
||||
out.dividend_days, _ = dividend_gaps(window_start, asof, asof, cfg)
|
||||
out.dividend_symbols = _symbols_without_rows("hd_dividend", cfg, asof=asof)
|
||||
if out.dividend_symbols:
|
||||
out.notes.append(
|
||||
f"{len(out.dividend_symbols)} 只股票在分红表里仍无任何记录"
|
||||
"（多为新上市公司，尚未分红；首次公告后会自动入库）"
|
||||
)
|
||||
|
||||
if "financial" in targets and include_financial:
|
||||
wm = financial_watermark(asof)
|
||||
out.financial_watermark = wm
|
||||
missing, stale = _financial_candidates(wm, asof, cfg)
|
||||
out.financial_missing = {k: v for k, v in missing.items() if v}
|
||||
out.financial_stale = {k: v for k, v in stale.items() if v}
|
||||
# 完全缺数据的股票优先（它们连一行都没有），再按市值补滞后股票
|
||||
priority: list[str] = []
|
||||
for table in FINANCIAL_TABLES:
|
||||
for s in missing[table]:
|
||||
if s not in priority:
|
||||
priority.append(s)
|
||||
rest: list[str] = []
|
||||
for table in FINANCIAL_TABLES:
|
||||
for s in stale[table]:
|
||||
if s not in priority and s not in rest:
|
||||
rest.append(s)
|
||||
rest = symbols_by_priority(cfg, rest)
|
||||
out.financial_symbols = (priority + rest)[: max(0, financial_limit)]
|
||||
if len(priority) + len(rest) > len(out.financial_symbols):
|
||||
out.notes.append(
|
||||
f"财报待补 {len(priority) + len(rest)} 只，本次按上限只处理 "
|
||||
f"{len(out.financial_symbols)} 只，其余在后续运行中自动排空"
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _index_weight_window(
|
||||
start: date, end: date, cfg: DataSourceConfig
|
||||
) -> tuple[date, date] | None:
|
||||
"""成分股权重的待补区间（月度接口，按最后一个权重日之后起算）。
|
||||
|
||||
右端点收到**最近一个交易日**：否则长假期间（如国庆 10-01 ~ 10-08）
|
||||
每天都会重查一段没有任何数据的「未来」区间，白白消耗调用次数。
|
||||
"""
|
||||
days = _trading_days(start, end, cfg)
|
||||
if not days:
|
||||
return None
|
||||
end = days[-1]
|
||||
df = db.read_sql("SELECT MAX(trade_date) AS m FROM index_weight", cfg=cfg)
|
||||
last = to_date(df["m"].iloc[0]) if not df.empty else None
|
||||
if last is not None and last >= end:
|
||||
return None
|
||||
since = start if last is None else max(start, last + timedelta(days=1))
|
||||
if since > end:
|
||||
return None
|
||||
return since, end
|
||||
|
||||
|
||||
def _resolve_only(only: Sequence[str] | None) -> set[str]:
|
||||
if not only:
|
||||
return set(GROUPS)
|
||||
out: set[str] = set()
|
||||
for item in only:
|
||||
key = str(item).strip()
|
||||
if not key:
|
||||
continue
|
||||
if key in GROUPS:
|
||||
out.add(key)
|
||||
elif key in DAY_TARGET_BY_KEY:
|
||||
out.add(DAY_TARGET_BY_KEY[key].group)
|
||||
elif key == "index_weight":
|
||||
out.add("index")
|
||||
else:
|
||||
raise ValueError(
|
||||
f"未知的同步目标：{key}（可用：{'/'.join(GROUPS)}"
|
||||
f" 或单个目标 {'/'.join(DAY_TARGET_BY_KEY)}）"
|
||||
)
|
||||
return out or set(GROUPS)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 执行
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def authorize_qlib_writes() -> bool:
|
||||
"""打开「写入 qlib 既有表」的进程内开关。
|
||||
|
||||
返回 True 表示本次由本函数打开（调用方应据此在结束时不回滚 —— 进程退出即失效）。
|
||||
StatementGuard 在**引擎创建时**读取该开关，所以这里必须同时丢弃已缓存的引擎，
|
||||
否则在一个已经建过引擎的进程里调用会拿到旧 guard。
|
||||
"""
|
||||
already = os.environ.get("HDIV_ALLOW_BACKFILL", "").strip().lower() in {"1", "true", "yes"}
|
||||
if already:
|
||||
return False
|
||||
os.environ["HDIV_ALLOW_BACKFILL"] = "1"
|
||||
db.reset_engine_cache()
|
||||
return True
|
||||
|
||||
|
||||
def sync_dividend_days(days: Sequence[date], cfg: DataSourceConfig) -> int:
|
||||
"""按日期补分红：每个交易日查四个日期参数，幂等写入 ``hd_dividend``。"""
|
||||
if not days:
|
||||
return 0
|
||||
written = 0
|
||||
with TushareClient(cfg.tushare) as client, sync_job(
|
||||
"dividend:dates",
|
||||
api="dividend",
|
||||
params={"days": [str(d) for d in days]},
|
||||
table="hd_dividend",
|
||||
cfg=cfg,
|
||||
) as ctx:
|
||||
ctx["data_start"], ctx["data_end"] = days[0], days[-1]
|
||||
for i, d in enumerate(days, 1):
|
||||
stamp = d.strftime("%Y%m%d")
|
||||
for param in DIVIDEND_DATE_PARAMS:
|
||||
rows = client.query("dividend", {param: stamp}, dividend_sync.FIELDS)
|
||||
if not rows:
|
||||
continue
|
||||
df = dividend_sync.rows_to_frame(rows)
|
||||
if df.empty:
|
||||
continue
|
||||
upsert("hd_dividend", df, cfg=cfg, ctx=ctx)
|
||||
if i % 10 == 0:
|
||||
print(f" [dividend] [{i}/{len(days)}] 累计入库 {ctx['rows_written']} 行", flush=True)
|
||||
written = ctx["rows_written"]
|
||||
print(
|
||||
f"[dividend] 完成：{len(days)} 个交易日（每校 {len(DIVIDEND_DATE_PARAMS)} 个日期列），"
|
||||
f"写入 {written} 行",
|
||||
flush=True,
|
||||
)
|
||||
return written
|
||||
|
||||
|
||||
def _print_plan(p: DailyPlan, targets: set[str]) -> None:
|
||||
print(f"=== 每日增量同步计划（asof={p.asof}，窗口自 {p.window_start}）===", flush=True)
|
||||
for t in DAY_TARGETS:
|
||||
if t.group not in targets:
|
||||
continue
|
||||
days = p.day_gaps.get(t.key, [])
|
||||
head = ", ".join(str(d) for d in days[:6])
|
||||
tail = " …" if len(days) > 6 else ""
|
||||
print(f" [{t.label:8s}] {t.table:16s} 缺 {len(days):>3d} 个交易日 {head}{tail}", flush=True)
|
||||
if "index" in targets:
|
||||
total = sum(len(v) for v in p.index_days.values())
|
||||
print(f" [指数行情 ] hd_index_daily 缺 {total:>3d} 个交易日·指数", flush=True)
|
||||
for code, days in p.index_days.items():
|
||||
print(f" {code} 缺 {len(days)} 天（{days[0]} ~ {days[-1]}）", flush=True)
|
||||
if p.index_weight_window:
|
||||
print(
|
||||
f" [成分权重 ] index_weight 待补 {p.index_weight_window[0]} ~ "
|
||||
f"{p.index_weight_window[1]}",
|
||||
flush=True,
|
||||
)
|
||||
if "dividend" in targets:
|
||||
print(
|
||||
f" [分红明细 ] hd_dividend 缺 {len(p.dividend_days)} 个交易日"
|
||||
f"（{len(DIVIDEND_DATE_PARAMS)} 个日期列/天）；"
|
||||
f"完全无记录的股票 {len(p.dividend_symbols)} 只",
|
||||
flush=True,
|
||||
)
|
||||
if "financial" in targets:
|
||||
if p.financial_watermark is None:
|
||||
print(" [财报四表 ] 已跳过", flush=True)
|
||||
else:
|
||||
print(
|
||||
f" [财报四表 ] 报告期水位 {p.financial_watermark}，"
|
||||
f"本次待补 {len(p.financial_symbols)} 只（"
|
||||
f"完全缺失 {sum(len(v) for v in p.financial_missing.values())} 只·表）",
|
||||
flush=True,
|
||||
)
|
||||
for n in p.notes:
|
||||
print(f" 注意：{n}", flush=True)
|
||||
|
||||
|
||||
def run(
|
||||
cfg: DataSourceConfig | None = None,
|
||||
*,
|
||||
asof: date | None = None,
|
||||
lookback_days: int = DEFAULT_LOOKBACK_DAYS,
|
||||
only: Sequence[str] | None = None,
|
||||
include_financial: bool = True,
|
||||
financial_limit: int = DEFAULT_FINANCIAL_LIMIT,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""执行一次每日增量同步。
|
||||
|
||||
``dry_run=True`` 只打印计划、不调用 Tushare、不写库。
|
||||
``verbose=False`` 不打任何进度文字，只返回结果（供 ``--json`` 使用）——
|
||||
否则 stdout 会混进人类可读文本，机器解析不了。
|
||||
"""
|
||||
if cfg is None:
|
||||
db.load_dotenv_once()
|
||||
cfg = load_config("datasource")
|
||||
targets = _resolve_only(only)
|
||||
p = plan(
|
||||
cfg,
|
||||
asof=asof,
|
||||
lookback_days=lookback_days,
|
||||
only=only,
|
||||
include_financial=include_financial,
|
||||
financial_limit=financial_limit,
|
||||
)
|
||||
if verbose:
|
||||
_print_plan(p, targets)
|
||||
|
||||
summary: dict[str, Any] = {
|
||||
"asof": str(p.asof),
|
||||
"window_start": str(p.window_start),
|
||||
"dry_run": dry_run,
|
||||
"targets": {},
|
||||
"errors": [],
|
||||
"ok": True,
|
||||
}
|
||||
|
||||
if dry_run:
|
||||
summary["planned"] = {
|
||||
"day_gaps": {k: len(v) for k, v in p.day_gaps.items()},
|
||||
"index_days": {k: len(v) for k, v in p.index_days.items()},
|
||||
"dividend_days": len(p.dividend_days),
|
||||
"dividend_symbols": len(p.dividend_symbols),
|
||||
"financial_symbols": len(p.financial_symbols),
|
||||
}
|
||||
if verbose:
|
||||
print("（--dry-run：未调用任何接口，也未写库）", flush=True)
|
||||
return summary
|
||||
|
||||
if p.empty:
|
||||
if verbose:
|
||||
print("全部目标均已是最新，无需抓取。", flush=True)
|
||||
return summary
|
||||
|
||||
# 行情三表落在 qlib 既有表上，写入需要显式授权（进程内生效）
|
||||
needs_backfill = any(t.which for t in DAY_TARGETS if p.day_gaps.get(t.key))
|
||||
if needs_backfill or p.index_weight_window:
|
||||
authorize_qlib_writes()
|
||||
|
||||
def _record(key: str, label: str, fn) -> None: # type: ignore[no-untyped-def]
|
||||
try:
|
||||
result = fn()
|
||||
summary["targets"][key] = {"label": label, "status": "ok", "result": result}
|
||||
except Exception as exc: # 单个目标失败不应中断其余目标
|
||||
summary["ok"] = False
|
||||
summary["errors"].append(f"{key}: {type(exc).__name__}: {exc}")
|
||||
summary["targets"][key] = {
|
||||
"label": label,
|
||||
"status": "fail",
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
}
|
||||
if verbose:
|
||||
print(f" [失败] {label}：{type(exc).__name__}: {exc}", flush=True)
|
||||
|
||||
# 1) 行情三表（内部按 fetched_days 再筛一次，真正抓的就是缺口那几天）
|
||||
for t in DAY_TARGETS:
|
||||
days = p.day_gaps.get(t.key)
|
||||
if not days or not t.which:
|
||||
continue
|
||||
_record(
|
||||
t.key,
|
||||
t.label,
|
||||
lambda t=t, days=days: price_sync.sync_days(
|
||||
t.which, days[0], days[-1], resume=True, insert_ignore=True, cfg=cfg
|
||||
),
|
||||
)
|
||||
|
||||
# 2) 停牌 / 涨跌停（两者共用一次窗口计算，各自按自己的阈值判缺口）
|
||||
trading_days = (p.day_gaps.get("suspend") or []) + (p.day_gaps.get("limit") or [])
|
||||
if trading_days:
|
||||
_record(
|
||||
"trading",
|
||||
"停牌/涨跌停",
|
||||
lambda: trading_sync.sync_trading_constraints(
|
||||
min(trading_days), max(trading_days), resume=True, cfg=cfg
|
||||
),
|
||||
)
|
||||
|
||||
# 3) 指数行情：逐个指数按自己的缺口起点拉，已最新的指数一次调用都不发
|
||||
if "index" in targets:
|
||||
for code, days in p.index_days.items():
|
||||
item = next(
|
||||
(i for i in index_sync.default_indices(cfg) if i["code"] == code),
|
||||
{"code": code, "name": None},
|
||||
)
|
||||
_record(
|
||||
f"index_daily:{code}",
|
||||
f"指数行情 {code}",
|
||||
lambda item=item, days=days: index_sync.sync_index_daily(
|
||||
indices=[item],
|
||||
start_date=days[0].strftime("%Y%m%d"),
|
||||
end_date=days[-1].strftime("%Y%m%d"),
|
||||
cfg=cfg,
|
||||
),
|
||||
)
|
||||
if p.index_weight_window:
|
||||
s, e = p.index_weight_window
|
||||
_record(
|
||||
"index_weight",
|
||||
"指数成分权重",
|
||||
lambda s=s, e=e: index_sync.sync_index_weight(
|
||||
start_date=s.strftime("%Y%m%d"), end_date=e.strftime("%Y%m%d"), cfg=cfg
|
||||
),
|
||||
)
|
||||
|
||||
# 4) 分红：按日期补（新记录）+ 补齐完全无记录的股票
|
||||
if "dividend" in targets:
|
||||
if p.dividend_days:
|
||||
_record(
|
||||
"dividend:dates",
|
||||
"分红明细（按日）",
|
||||
lambda: {"written": sync_dividend_days(p.dividend_days, cfg)},
|
||||
)
|
||||
if p.dividend_symbols:
|
||||
_record(
|
||||
"dividend:symbols",
|
||||
"分红明细（缺股票）",
|
||||
lambda: dividend_sync.run(symbols=p.dividend_symbols, cfg=cfg),
|
||||
)
|
||||
|
||||
# 5) 财报四表：按股票交错拉，一次把四张表补齐
|
||||
if p.financial_symbols:
|
||||
_record(
|
||||
"financial",
|
||||
"财报四表",
|
||||
lambda: financial_sync.run_interleaved(
|
||||
apis=list(financial_sync.SPECS),
|
||||
symbols=p.financial_symbols,
|
||||
only_missing=False,
|
||||
cfg=cfg,
|
||||
),
|
||||
)
|
||||
|
||||
if verbose:
|
||||
print("\n=== 本次增量同步结果 ===", flush=True)
|
||||
for info in summary["targets"].values():
|
||||
print(f" [{info['status']:4s}] {info['label']}", flush=True)
|
||||
if summary["errors"]:
|
||||
print(f" 失败 {len(summary['errors'])} 项：", flush=True)
|
||||
for e in summary["errors"]:
|
||||
print(f" - {e}", flush=True)
|
||||
else:
|
||||
print(" 全部成功。", flush=True)
|
||||
return summary
|
||||
|
||||
|
||||
def format_summary(summary: dict[str, Any]) -> str:
|
||||
"""把 :func:`run` 的结果压成一行摘要（给 launchd 日志 / 告警用）。"""
|
||||
if summary.get("dry_run"):
|
||||
planned = summary.get("planned", {})
|
||||
days = sum(planned.get("day_gaps", {}).values()) + sum(
|
||||
planned.get("index_days", {}).values()
|
||||
)
|
||||
return f"计划：{days} 个交易日·指数缺口，分红 {planned.get('dividend_days', 0)} 天"
|
||||
if summary.get("errors"):
|
||||
return f"失败 {len(summary['errors'])} 项：" + "；".join(summary["errors"])
|
||||
return "全部目标同步成功"
|
||||
@@ -32,9 +32,14 @@ def ttm_params() -> tuple[int, int, bool]:
|
||||
walk-forward 与 Web 用函数默认值 —— 同一个「股息率」在不同环节定义不同，
|
||||
改了配置只有画像会变。现在统一从这里取。
|
||||
"""
|
||||
from hdiv.core.config import load_config
|
||||
# 必须走**带缓存**的 get_config：本函数在筛选器里是**逐股**调用的，
|
||||
# 而 load_config 每次都重新读盘 + YAML 解析 + pydantic 校验（实测约 16ms）。
|
||||
# 逐日全市场筛选时这会变成每天约 2.8 秒的纯开销 —— cProfile 实测：
|
||||
# 6 个交易日里 load_config 被调用 1008 次、共 16.6 秒，占整个筛选时间的 22%，
|
||||
# 而它每次返回的都是完全相同的一份配置。
|
||||
from hdiv.core.config import get_config
|
||||
|
||||
c = load_config("profile").ttm_dividend
|
||||
c = get_config("profile").ttm_dividend
|
||||
return int(c.window_days), int(c.grace_days), bool(
|
||||
getattr(c, "smooth_spikes", True)
|
||||
)
|
||||
|
||||
+134
-21
@@ -25,6 +25,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import date, timedelta
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from hdiv.core.errors import HdivError
|
||||
@@ -55,6 +56,21 @@ __all__ = [
|
||||
# 配置校验（core.config）与画像实现（profile.pit）不允许出现两个清单。
|
||||
|
||||
|
||||
def _with_dt(df: pd.DataFrame, col: str) -> pd.DataFrame:
|
||||
"""把日期列一次性转成 ``datetime64``（P1）。
|
||||
|
||||
内部一律用 datetime64 比较、排序、切片；``date`` 对象只在对外返回时出现。
|
||||
原因：object 的 ``datetime.date`` 列在 pandas 里做比较/排序会退化到 Python
|
||||
逐元素循环 —— 实测这些转换与比较占模拟阶段约 28%（``DatetimeArray.__iter__``
|
||||
一百万次调用、``pd.to_datetime`` 累计 7.4 秒）。
|
||||
"""
|
||||
if df.empty or col not in df.columns:
|
||||
return df
|
||||
if not pd.api.types.is_datetime64_any_dtype(df[col]):
|
||||
df[col] = pd.to_datetime(df[col])
|
||||
return df
|
||||
|
||||
|
||||
def metrics_needing_financials(metrics: set[str]) -> bool:
|
||||
"""是否需要财报面板。
|
||||
|
||||
@@ -162,6 +178,21 @@ class _AsOfContext:
|
||||
expected_obs: dict[int, int] = field(default_factory=dict)
|
||||
|
||||
|
||||
#: 时点面板缓存上限（``_ctx`` 的条目数）。
|
||||
#:
|
||||
#: **为什么必须设上限**：引擎逐日推进，**旧时点不会再被查询**；而每个时点面板
|
||||
#: 持有该 asof 可见的**分红超集**（约 12 年 × 全市场，实测约 4 MB）、财务历史与
|
||||
#: 财年表。不设上限时，一次 6.7 年（约 1600 个决策日）的每日回测会把 1600 份
|
||||
#: 面板全部留在内存里 —— 约 6 GB，**必然 OOM**。
|
||||
#: 保留 2 个（当前 + 上一个）是为了容忍调用方偶尔回看一天，代价可以忽略。
|
||||
_MAX_ASOF_CONTEXTS = 2
|
||||
|
||||
#: ``(symbol, asof)`` 画像快照的缓存上限（``_snapshots`` 的条目数）。
|
||||
#: 画像快照的复用只发生在**同一个 asof 内**（闸门算过一次、信号留痕再取一次），
|
||||
#: 跨日必然 miss。按 asof 清空即可保住这份收益，同时把内存限制在「一天」的量级。
|
||||
_SNAPSHOTS_PER_ASOF = 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 服务
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -207,6 +238,10 @@ class PitProfileService:
|
||||
self._all_dividends = pd.DataFrame()
|
||||
self._ctx: dict[date, _AsOfContext] = {}
|
||||
self._snapshots: dict[tuple[str, date], ProfileSnapshot] = {}
|
||||
#: 当前快照缓存所属的 asof（换日即清空，见 _SNAPSHOTS_PER_ASOF）
|
||||
self._snapshot_asof: date | None = None
|
||||
#: 累计构建过的时点面板数（与「当前缓存了几个」是两件事）
|
||||
self._asof_built = 0
|
||||
#: 闸门规则用到的指标集合；None = 未知，按「全都可能需要」处理（保守）
|
||||
self._needed: set[str] | None = None
|
||||
self._financial_required: bool | None = None
|
||||
@@ -244,24 +279,51 @@ class PitProfileService:
|
||||
# 批量预载（跨越整个回测区间、与 asof 无关的部分）
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def prepare(self, symbols: list[str], start: date, end: date) -> None:
|
||||
def prepare(
|
||||
self,
|
||||
symbols: list[str],
|
||||
start: date,
|
||||
end: date,
|
||||
*,
|
||||
price: pd.DataFrame | None = None,
|
||||
) -> None:
|
||||
"""载入跨决策日共享的面板。
|
||||
|
||||
只有 ``trade_date`` 范围过滤，没有 PIT 语义 —— 真正的 PIT 剪裁发生在
|
||||
:meth:`snapshot` 里逐 asof 进行（与 ``ProfileBuilder.run`` 的取数起点
|
||||
规则完全一致：``asof.year - max_years - 1`` 的 1 月 1 日）。
|
||||
|
||||
``price`` 允许调用方传入**已经取好的同一批不复权行情**（引擎就是这么做的）——
|
||||
回测引擎本来就要取这段价格来算分位，画像再取一遍是纯粹的重复查询。
|
||||
传入时必须已经覆盖 ``[start, end]``（调用方把两者的起点取并集）。
|
||||
"""
|
||||
self._symbols = sorted(set(symbols))
|
||||
if not self._symbols:
|
||||
self._prepared = True
|
||||
return
|
||||
self._price = self.repo.price_history(self._symbols, start, end, adjust="none")
|
||||
if price is not None and not price.empty:
|
||||
# 只保留面板里的股票，并**统一为 datetime64**（P1）
|
||||
self._price = _with_dt(
|
||||
price[price["symbol"].isin(set(self._symbols))].copy(), "trade_date"
|
||||
)
|
||||
else:
|
||||
self._price = _with_dt(
|
||||
self.repo.price_history(self._symbols, start, end, adjust="none"),
|
||||
"trade_date",
|
||||
)
|
||||
self._counters["price_loaded"] += 1
|
||||
if self.load_basics:
|
||||
self._basics = self.builder._load_daily_basic(self._symbols, start, end)
|
||||
# P1：日期列一次性转成 datetime64。原先每个 asof、每只股票都要
|
||||
# 把 object 的 datetime.date 列再 `pd.to_datetime` / `.dt.date` 一遍，
|
||||
# 实测这一族转换占模拟阶段约 28%（DatetimeArray.__iter__ 100 万次调用）。
|
||||
self._basics = _with_dt(
|
||||
self.builder._load_daily_basic(self._symbols, start, end), "trade_date"
|
||||
)
|
||||
self._counters["basics_loaded"] += 1
|
||||
if self.load_index:
|
||||
self._index = self.repo.index_history("000300.SH", start, end)
|
||||
self._index = _with_dt(
|
||||
self.repo.index_history("000300.SH", start, end), "trade_date"
|
||||
)
|
||||
# 分红：为**整个回测区间**取一次超集，逐 asof 再用与 repo.dividend_records
|
||||
# 完全相同的三重 PIT 条件（imp_ann_date / ex_date / 回看窗口）在 pandas 里剪裁。
|
||||
#
|
||||
@@ -274,7 +336,16 @@ class PitProfileService:
|
||||
if not self._all_dividends.empty:
|
||||
self._all_dividends = self._all_dividends[
|
||||
self._all_dividends["symbol"].isin(set(self._symbols))
|
||||
]
|
||||
].copy()
|
||||
# P1：预存 datetime64 的公告日/除权日，逐 asof 的 PIT 剪裁直接用它们比较
|
||||
# （原来每次都把 2 万行×2 列 `pd.to_datetime(...).dt.date` 转成 object
|
||||
# 再逐元素比较 —— object 比较会退化到 Python 循环）。
|
||||
self._all_dividends["_imp_dt"] = pd.to_datetime(
|
||||
self._all_dividends["imp_ann_date"]
|
||||
).to_numpy(dtype="datetime64[ns]")
|
||||
self._all_dividends["_ex_dt"] = pd.to_datetime(
|
||||
self._all_dividends["ex_date"]
|
||||
).to_numpy(dtype="datetime64[ns]")
|
||||
self._prepared = True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -285,6 +356,11 @@ class PitProfileService:
|
||||
"""计算（或取缓存）``symbol`` 在 ``asof`` 的实时画像。"""
|
||||
if not self._prepared:
|
||||
raise HdivError("PitProfileService 必须先 prepare(symbols, start, end)")
|
||||
if self._snapshot_asof is not None and asof != self._snapshot_asof:
|
||||
# 换日：跨日的 (symbol, asof) 键不可能再命中，直接释放。
|
||||
# 见 _SNAPSHOTS_PER_ASOF 对「为什么这样不改变任何结果」的说明。
|
||||
self._snapshots.clear()
|
||||
self._snapshot_asof = asof
|
||||
key = (symbol, asof)
|
||||
hit = self._snapshots.get(key)
|
||||
if hit is not None:
|
||||
@@ -299,7 +375,9 @@ class PitProfileService:
|
||||
|
||||
def stats(self) -> dict[str, int]:
|
||||
out = dict(self._counters)
|
||||
out["distinct_asof"] = len(self._ctx)
|
||||
# distinct_asof = **累计**构建过的时点数（缓存里当前只剩最近几个，
|
||||
# 用 len(self._ctx) 会把「涉及多少决策时点」报成 1~2，直接误导）。
|
||||
out["distinct_asof"] = self._asof_built
|
||||
out["cached_symbols"] = len(self._snapshots)
|
||||
return out
|
||||
|
||||
@@ -317,9 +395,17 @@ class PitProfileService:
|
||||
div = d
|
||||
else:
|
||||
since = asof - timedelta(days=int((self.max_years + 2) * 365.25))
|
||||
imp = pd.to_datetime(d["imp_ann_date"]).dt.date
|
||||
ex = pd.to_datetime(d["ex_date"]).dt.date
|
||||
div = d[(imp <= asof) & (ex <= asof) & (ex >= since)]
|
||||
# P1：用 prepare 里预存的 datetime64 列直接比较（原来每次都对 2 万行
|
||||
# ×2 列做 `pd.to_datetime(...).dt.date`，再用 object 比较 —— 后者会
|
||||
# 退化到 Python 逐元素循环）。日期都在零点，闭区间端点与 object 版一致。
|
||||
a = np.datetime64(asof, "ns")
|
||||
sn = np.datetime64(since, "ns")
|
||||
if "_imp_dt" in d.columns:
|
||||
imp, ex = d["_imp_dt"], d["_ex_dt"]
|
||||
else: # 兼容：未走 prepare 的调用路径
|
||||
imp = pd.to_datetime(d["imp_ann_date"]).to_numpy(dtype="datetime64[ns]")
|
||||
ex = pd.to_datetime(d["ex_date"]).to_numpy(dtype="datetime64[ns]")
|
||||
div = d[(imp <= a) & (ex <= a) & (ex >= sn)]
|
||||
div_by_symbol: dict[str, list[dict[str, Any]]] = {}
|
||||
if not div.empty:
|
||||
for rec in div.to_dict("records"):
|
||||
@@ -351,6 +437,14 @@ class PitProfileService:
|
||||
}
|
||||
self._counters["financial_loads"] += 1
|
||||
self._ctx[asof] = ctx
|
||||
self._asof_built += 1
|
||||
# 只保留最近 _MAX_ASOF_CONTEXTS 个时点（按插入顺序淘汰最旧）。
|
||||
# 见该常量的说明：不设上限会让长区间每日回测 OOM。
|
||||
while len(self._ctx) > _MAX_ASOF_CONTEXTS:
|
||||
oldest = next(iter(self._ctx))
|
||||
if oldest == asof:
|
||||
break
|
||||
self._ctx.pop(oldest, None)
|
||||
self._counters["asof_contexts"] += 1
|
||||
return ctx
|
||||
|
||||
@@ -360,21 +454,40 @@ class PitProfileService:
|
||||
"""调用 ``ProfileBuilder._profile_one`` —— **指标定义的单一口径来源**。"""
|
||||
start = date(asof.year - self.max_years - 1, 1, 1)
|
||||
|
||||
def _slice(df: pd.DataFrame) -> pd.DataFrame:
|
||||
if df.empty:
|
||||
return df
|
||||
td = pd.to_datetime(df["trade_date"]).dt.date
|
||||
return df[(td >= start) & (td <= asof)]
|
||||
def _slice(d: pd.DataFrame) -> pd.DataFrame:
|
||||
"""按 ``[start, asof]`` 剪裁（**必须已先按 symbol 过滤**）。
|
||||
|
||||
price = _slice(self._price) if not self._price.empty else self._price
|
||||
price = price[price["symbol"] == symbol] if not price.empty else price
|
||||
用 ``datetime64`` 比较而不是 ``.dt.date`` 的 object 比较：后者在
|
||||
pandas 里逐元素装箱，实测 50 万行约 150 毫秒，前者毫秒级。
|
||||
两者语义相同（日期都在零点，闭区间端点一致）。
|
||||
"""
|
||||
if d.empty:
|
||||
return d
|
||||
td = d["trade_date"]
|
||||
if not pd.api.types.is_datetime64_any_dtype(td):
|
||||
td = pd.to_datetime(td)
|
||||
return d[(td >= pd.Timestamp(start)) & (td <= pd.Timestamp(asof))]
|
||||
|
||||
def _of_symbol(d: pd.DataFrame) -> pd.DataFrame:
|
||||
"""先取该股票的行，再剪裁日期区间。
|
||||
|
||||
**顺序不能反**：``_price`` / ``_basics`` 是**全部股票**的面板
|
||||
（每日回测里 123 只 × 17 年 ≈ 50 万行）。原先「先剪裁整表、再筛 symbol」
|
||||
让每次画像快照都要处理 50 万行 —— 实测 174 毫秒，而「先筛 symbol
|
||||
再剪裁」只要 11 毫秒（**16 倍**）。两个过滤条件互相独立，交换顺序
|
||||
不改变任何结果（``ProfileBuilder._profile_one`` 内部还会
|
||||
``sort_values("trade_date")``，所以行序也不敏感）。
|
||||
"""
|
||||
if d.empty:
|
||||
return d
|
||||
hit = d[d["symbol"] == symbol]
|
||||
return _slice(hit)
|
||||
|
||||
price = _of_symbol(self._price)
|
||||
if price.empty:
|
||||
return None
|
||||
basics = _slice(self._basics) if not self._basics.empty else self._basics
|
||||
if not basics.empty:
|
||||
basics = basics[basics["symbol"] == symbol]
|
||||
basics = _with_columns(basics, _EMPTY_BASICS)
|
||||
index = _slice(self._index) if not self._index.empty else self._index
|
||||
basics = _with_columns(_of_symbol(self._basics), _EMPTY_BASICS)
|
||||
index = _slice(self._index)
|
||||
sym_div = ctx.div_by_symbol.get(symbol, [])
|
||||
# ProfileBuilder 期望的 events 是 {symbol: DataFrame}
|
||||
events = build_dps_events(pd.DataFrame(sym_div)) if sym_div else {}
|
||||
|
||||
@@ -0,0 +1,347 @@
|
||||
"""每日动态股票池筛选器（``backtest --mode daily`` 的选股环节）。
|
||||
|
||||
**它做什么**：从 ``start`` 起，对**每一个交易日**按当日可见数据重建股票池。
|
||||
判定逻辑一行不改 —— 仍然调用 :class:`~hdiv.universe.selector.UniverseSelector`
|
||||
与四个 ``Filter``；本模块只负责两件工程上的事：
|
||||
|
||||
1. **取数**：用 :class:`~hdiv.universe.pit.PitRepo` 按区块批量预载，
|
||||
逐日切片在内存完成（单次筛选从 10~18 秒降到秒级）；
|
||||
2. **候选集预剪枝**：见下。
|
||||
|
||||
------------------------------------------------------------
|
||||
候选集预剪枝：为什么是「精确」的，而不是「近似」
|
||||
------------------------------------------------------------
|
||||
|
||||
市场滤网要逐行判断 5000 余只股票的交易所/板块/上市年限/市值/流动性，
|
||||
这一段的 Python 开销与**候选数**成正比，是每日循环里最大的一项。
|
||||
|
||||
预剪枝只剔除「在整个回测区间内**不可能**通过市场滤网」的股票，
|
||||
判据都是**上界**：
|
||||
|
||||
- **交易所 / 板块**：与日期无关，不在配置名单里的股票永远不可能通过；
|
||||
- **上市年限**：当 ``list_date + min_listing_years`` 晚于区间**最后一天**时，
|
||||
该股在区间内任何一天都不满足 ``listed_years >= min_listing_years``；
|
||||
- **市值**：当该股在区间内的 ``MAX(total_mv)``（换算为元）仍低于
|
||||
``min_market_cap`` 时，任何一天都不满足市值下限。取不到市值（NULL）时
|
||||
**保留**，不剪。
|
||||
|
||||
被剪掉的股票在原流程里**必然**在第一个滤网（market）就被淘汰，因此：
|
||||
最终入选集合逐只相同，`hd_daily_universe` 的内容也相同。
|
||||
差别只在于「被剪掉的股票没有留下逐滤网的原因」—— 而每日选股模式
|
||||
**不落库逐股淘汰原因**（只落库每日入选成员），所以这个差别不可观测。
|
||||
|
||||
正确性由 ``tests/test_daily.py::test_prune_does_not_change_selection`` 锁定：
|
||||
同一批交易日，开/关剪枝必须选出**完全相同**的成员。
|
||||
|
||||
**流动性没有做预剪枝**：``stock_daily`` 的量价单位在 2015-2019 是「手/千元」、
|
||||
2020 起是「股/元」，用 ``MAX(amount)`` 做上界会在早年低估 1000 倍，
|
||||
误剪掉本该通过的股票。宁可少一项优化，也不接受一个会改变结果的上界。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, timedelta
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from hdiv.core.config import UniverseConfig
|
||||
from hdiv.core.errors import HdivError
|
||||
from hdiv.data import db
|
||||
from hdiv.universe.pit import PitRepo
|
||||
from hdiv.universe.selector import UniverseSelector
|
||||
|
||||
__all__ = ["DailyUniverseScreener", "PruneReport", "ScreenDay"]
|
||||
|
||||
#: daily_basic 的市值列以**万元**存放（见 data/units.py）
|
||||
_WAN = 1e4
|
||||
|
||||
|
||||
@dataclass
|
||||
class PruneReport:
|
||||
"""预剪枝的规模，用于回答「为什么候选从 5000 变成 1000」。"""
|
||||
|
||||
total: int = 0
|
||||
pruned_exchange: int = 0
|
||||
pruned_board: int = 0
|
||||
pruned_listing: int = 0
|
||||
pruned_market_cap: int = 0
|
||||
kept: int = 0
|
||||
|
||||
def as_dict(self) -> dict[str, int]:
|
||||
return dict(self.__dict__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScreenDay:
|
||||
"""某一天的选股结果。"""
|
||||
|
||||
trade_date: date
|
||||
candidate_count: int
|
||||
member_count: int
|
||||
symbols: list[str]
|
||||
members: pd.DataFrame
|
||||
stats: dict[str, int] = field(default_factory=dict)
|
||||
#: 当日**市场候选数**（未预剪枝）。与 ``candidate_count``（已预剪枝）分开记录 ——
|
||||
#: 预剪枝是纯性能开关，不该让页面上的「候选」含义随开关变化。
|
||||
listed_count: int = 0
|
||||
#: 入选股票在**决策日的因子取值**（来自 ``UniverseSelector.run`` 的 ``values``）。
|
||||
#: 必须带上它：``selected`` 里没有 ``dividend_yield`` 列（它在滤网的 values 里），
|
||||
#: 只从 ``selected`` 取列会让落库的股息率**整列为 NULL**。
|
||||
values: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
|
||||
|
||||
class DailyUniverseScreener:
|
||||
"""逐日重建股票池（PIT），复用既有滤网，不改判定口径。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: UniverseConfig,
|
||||
repo: PitRepo,
|
||||
*,
|
||||
verbose: bool = True,
|
||||
) -> None:
|
||||
self.config = config
|
||||
self.repo = repo
|
||||
self.selector = UniverseSelector(config, repo=repo)
|
||||
self.verbose = verbose
|
||||
self.prune = PruneReport()
|
||||
self._allowed: set[str] | None = None
|
||||
|
||||
@classmethod
|
||||
def from_strategy(cls, registry: Any, strategy: Any, repo: PitRepo,
|
||||
*, verbose: bool = True) -> DailyUniverseScreener:
|
||||
cfg = registry.resolved_universe(strategy)
|
||||
return cls(cfg, repo, verbose=verbose)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 预剪枝
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def build_prune_set(
|
||||
self, window_start: date, window_end: date, *, use_market_cap: bool = True
|
||||
) -> set[str]:
|
||||
"""计算「在 ``[window_start, window_end]`` 内不可能通过市场滤网」的补集。
|
||||
|
||||
返回**允许保留**的 symbol 集合。
|
||||
"""
|
||||
cfg = self.config.market
|
||||
master = self.repo.stock_master().copy()
|
||||
rep = PruneReport(total=len(master))
|
||||
keep = pd.Series(True, index=master.index)
|
||||
|
||||
def _drop(mask: pd.Series, counter: str) -> None:
|
||||
nonlocal keep
|
||||
hit = keep & mask
|
||||
setattr(rep, counter, getattr(rep, counter) + int(hit.sum()))
|
||||
keep = keep & ~mask
|
||||
|
||||
# --- 交易所 / 板块（与日期无关）---
|
||||
if cfg.exchanges:
|
||||
allowed = {str(x) for x in cfg.exchanges}
|
||||
_drop(~master["exchange"].astype(str).isin(allowed), "pruned_exchange")
|
||||
if cfg.markets:
|
||||
allowed_m = {str(x) for x in cfg.markets}
|
||||
_drop(~master["market"].astype(str).isin(allowed_m), "pruned_board")
|
||||
|
||||
# --- 上市年限：区间最后一天仍不足，则区间内永远不足 ---
|
||||
if cfg.min_listing_years and cfg.min_listing_years > 0:
|
||||
ld = pd.to_datetime(master["list_date"], errors="coerce")
|
||||
need_days = cfg.min_listing_years * 365.25
|
||||
can_pass = (window_end - ld.dt.date).apply(
|
||||
lambda x: x.days if pd.notna(x) else -1
|
||||
) >= need_days
|
||||
_drop(~can_pass, "pruned_listing")
|
||||
|
||||
# --- 市值：区间内 MAX(total_mv) 仍低于下限 ---
|
||||
if use_market_cap:
|
||||
cap = self._max_market_cap(window_start, window_end)
|
||||
if cap:
|
||||
for col, limit, counter in (
|
||||
("total_mv", cfg.min_market_cap, "pruned_market_cap"),
|
||||
("circ_mv", cfg.min_float_market_cap, "pruned_market_cap"),
|
||||
):
|
||||
if limit is None:
|
||||
continue
|
||||
mx = master["symbol"].map(cap.get(col, {}))
|
||||
# 取不到市值时不剪（保守）：只有**确知**上限低于阈值才剔除
|
||||
too_small = mx.notna() & (mx < float(limit))
|
||||
_drop(too_small, counter)
|
||||
|
||||
allowed = set(master.loc[keep, "symbol"].astype(str).tolist())
|
||||
rep.kept = len(allowed)
|
||||
self._allowed = allowed
|
||||
self.prune = rep
|
||||
self.repo.set_candidate_scope(allowed)
|
||||
return allowed
|
||||
|
||||
def _max_market_cap(
|
||||
self, start: date, end: date
|
||||
) -> dict[str, dict[str, float]]:
|
||||
"""区间内逐股 ``MAX(total_mv)`` / ``MAX(circ_mv)``，单位为**元**。
|
||||
|
||||
返回 ``{"total_mv": {symbol: 元}, "circ_mv": {symbol: 元}}``。
|
||||
|
||||
单位：``daily_basic.total_mv`` / ``circ_mv`` 以**万元**存放（Tushare 口径），
|
||||
因此换算为元后再与配置里的元阈值比较。与 ``normalize_market_panel``
|
||||
的 ×1e4 是同一件事。
|
||||
"""
|
||||
cfg = self.repo.cfg
|
||||
if not db.table_exists("daily_basic", cfg):
|
||||
return {}
|
||||
try:
|
||||
df = db.read_sql(
|
||||
"SELECT symbol, MAX(total_mv) AS mx_total_mv, "
|
||||
" MAX(circ_mv) AS mx_circ_mv "
|
||||
"FROM daily_basic WHERE trade_date BETWEEN :s AND :e "
|
||||
"GROUP BY symbol",
|
||||
{"s": start, "e": end}, cfg=cfg,
|
||||
)
|
||||
except Exception: # pragma: no cover - 取不到就退化为不剪枝
|
||||
return {}
|
||||
if df.empty:
|
||||
return {}
|
||||
out: dict[str, dict[str, float]] = {"total_mv": {}, "circ_mv": {}}
|
||||
syms = df["symbol"].astype(str).tolist()
|
||||
total = pd.to_numeric(df["mx_total_mv"], errors="coerce") * _WAN
|
||||
circ = pd.to_numeric(df["mx_circ_mv"], errors="coerce") * _WAN
|
||||
for sym, t, c in zip(syms, total, circ, strict=False):
|
||||
if pd.notna(t):
|
||||
out["total_mv"][sym] = float(t)
|
||||
if pd.notna(c):
|
||||
out["circ_mv"][sym] = float(c)
|
||||
return out
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 逐日筛选
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def screen_day(self, day: date) -> ScreenDay:
|
||||
"""筛选单个交易日。"""
|
||||
|
||||
def _hook(stage: str, live: pd.DataFrame) -> None:
|
||||
# 把取数范围收窄到本阶段真正要评估的股票（纯性能开关）
|
||||
self.repo.restrict_to(live["symbol"].tolist())
|
||||
|
||||
try:
|
||||
res = self.selector.run(
|
||||
asof=day, persist=False, verbose=False, on_stage=_hook
|
||||
)
|
||||
finally:
|
||||
self.repo.restrict_to(None)
|
||||
members = res["selected"]
|
||||
symbols = [str(s) for s in members["symbol"].tolist()]
|
||||
listed_total, screened = self.repo.listed_counts(res["asof_date"])
|
||||
vals = res.get("values") or {}
|
||||
return ScreenDay(
|
||||
trade_date=res["asof_date"],
|
||||
candidate_count=screened,
|
||||
member_count=int(res["member_count"]),
|
||||
symbols=symbols,
|
||||
members=members,
|
||||
stats=dict(res["stats"]),
|
||||
listed_count=listed_total,
|
||||
# 只为入选股票保留因子取值（全市场 4000 余只 × 1600 天会白占内存）
|
||||
values={s: dict(vals.get(s) or {}) for s in symbols},
|
||||
)
|
||||
|
||||
def screen(self, days: list[date]) -> dict[date, set[str]]:
|
||||
"""对 ``days`` 逐日筛选，返回 ``{交易日: 入选代码集合}``。"""
|
||||
if self._allowed is None:
|
||||
raise HdivError(
|
||||
"DailyUniverseScreener 必须先 build_prune_set(...) 再 screen(...)。\n"
|
||||
" 预剪枝是可选的性能优化；若不想剪枝，请显式传 use_market_cap=False\n"
|
||||
" 并把候选范围设为「全部上市股票」。"
|
||||
)
|
||||
out: dict[date, set[str]] = {}
|
||||
for day in days:
|
||||
sd = self.screen_day(day)
|
||||
out[sd.trade_date] = set(sd.symbols)
|
||||
return out
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 落库行
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def member_rows(
|
||||
run_id: str, screens: list[ScreenDay], *, created_at: Any
|
||||
) -> list[dict[str, Any]]:
|
||||
"""把逐日选股结果摊平成 ``hd_daily_universe`` 的行。
|
||||
|
||||
取值优先级：**滤网的 ``values`` → ``selected`` 的列**。
|
||||
股息率、支付率、FCF 覆盖等只在 ``values`` 里（它们是滤网算出来的），
|
||||
``selected`` 只有行情/年报均值那几列；反过来 ``total_mv``/``roe_avg``
|
||||
只在列里。只取其中一边都会让某些列整列为 NULL。
|
||||
"""
|
||||
rows: list[dict[str, Any]] = []
|
||||
for sd in screens:
|
||||
m = sd.members
|
||||
if m is None or m.empty:
|
||||
continue
|
||||
for rec in m.to_dict("records"):
|
||||
sym = str(rec.get("symbol"))
|
||||
vals = dict(sd.values.get(sym) or {})
|
||||
merged = {**{k: rec.get(k) for k in rec}, **vals}
|
||||
|
||||
def pick(key: str) -> Any:
|
||||
v = vals.get(key)
|
||||
if v is None or (isinstance(v, float) and v != v):
|
||||
v = rec.get(key)
|
||||
return v
|
||||
|
||||
rows.append({
|
||||
"run_id": run_id,
|
||||
"trade_date": sd.trade_date,
|
||||
"symbol": sym,
|
||||
"name": _s(rec.get("name")),
|
||||
"industry": _s(rec.get("industry")),
|
||||
# 股息率：筛选口径（自算优先）在 values 里；dv_ttm 在列里
|
||||
"dividend_yield": _f(pick("dividend_yield")),
|
||||
"total_mv": _f(pick("total_mv")),
|
||||
"roe_avg": _f(pick("roe_avg")),
|
||||
"listed_count": int(sd.listed_count) if sd.listed_count else None,
|
||||
"candidate_count": int(sd.candidate_count),
|
||||
"values_json": _values_json(merged),
|
||||
"created_at": created_at,
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def _s(v: Any) -> str | None:
|
||||
if v is None or (isinstance(v, float) and v != v):
|
||||
return None
|
||||
return str(v)[:64]
|
||||
|
||||
|
||||
def _f(v: Any) -> float | None:
|
||||
if v is None:
|
||||
return None
|
||||
try:
|
||||
x = float(v)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return None if (x != x or np.isinf(x)) else x
|
||||
|
||||
|
||||
def _values_json(rec: dict[str, Any]) -> str:
|
||||
"""入选时的关键因子快照（供「为什么是这只」复核）。"""
|
||||
import json
|
||||
|
||||
keys = (
|
||||
"dividend_yield", "dividend_yield_computed", "dv_ttm", "ttm_dps",
|
||||
"pe_ttm", "pb", "ps_ttm", "total_mv", "circ_mv", "avg_amount_20d",
|
||||
"dividend_continuity_years", "dividend_years_in_window", "payout_ratio",
|
||||
"fcf_dividend_cover", "dps_cagr_5y", "roe", "roe_avg", "roic", "roic_avg",
|
||||
"debt_ratio", "ocf_to_netprofit", "ocf_to_profit_avg", "fin_years_count",
|
||||
)
|
||||
out: dict[str, Any] = {}
|
||||
for k in keys:
|
||||
if k not in rec:
|
||||
continue
|
||||
v = _f(rec.get(k))
|
||||
if v is not None:
|
||||
out[k] = v
|
||||
return json.dumps(out, ensure_ascii=False)
|
||||
@@ -0,0 +1,884 @@
|
||||
"""Point-in-Time 批量取数层（每日动态股票池的性能基座）。
|
||||
|
||||
**为什么需要它**：``--mode daily`` 要在**每个交易日**按当时可见数据重建股票池，
|
||||
而 :class:`~hdiv.universe.selector.UniverseSelector` 单次运行约 10~18 秒 ——
|
||||
其中 ``financial_panel`` 独占约 6 秒。这个开销是**表量级**的（与查询哪一天无关），
|
||||
于是 1600 个交易日 × 12 秒 ≈ 5~8 小时，逐日筛选根本跑不完。
|
||||
|
||||
本模块把「取数」与「派生」拆开：
|
||||
|
||||
- **取数**改为**按区间批量预载一次**，之后逐日切片在内存里完成；
|
||||
- **派生**（最新一期财报的合并、ROE 年化、单位归一化、支付率口径……）
|
||||
**一行都不重写** —— :class:`PitRepo` 继承 :class:`~hdiv.data.repo.Repo`，
|
||||
只覆盖**最底层的那几个取数方法**。上层的
|
||||
``financial_panel`` / ``annual_financial_averages`` 等一律沿用父类实现，
|
||||
因此它们调用到的都是被覆盖后的底层方法。
|
||||
|
||||
这样做的直接后果：**筛选口径只有一份**。``UniverseSelector`` 与四个 ``Filter``
|
||||
在 daily 模式下逐字未改，它们看到的 DataFrame 与直连数据库时逐值相同。
|
||||
这一点有回归测试锁定（``tests/test_daily.py::test_pit_repo_matches_direct_repo``），
|
||||
与 ``tests/test_profile_pit.py`` 锁定实时画像／批量画像一致性的做法相同。
|
||||
|
||||
**内存**：行情与每日指标按区块（默认一年）载入，用完即 ``release_range()`` 释放；
|
||||
财务四表与分红明细体量小（合计约 120 万行），一次性常驻。
|
||||
|
||||
**越界即报错**：``market_panel`` / ``avg_amount`` 只对已载入区间内的时点有定义。
|
||||
查询落在区间外时**抛错而不是返回空表** —— 空表会被上层当成「当天没有股票」，
|
||||
静默产出一个空股票池，属于最难发现的一类错误。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from hdiv.core.config import DataSourceConfig, load_config
|
||||
from hdiv.core.errors import DataGapError, HdivError
|
||||
from hdiv.data import db
|
||||
from hdiv.data.repo import Repo
|
||||
from hdiv.data.units import (
|
||||
normalize_financial_panel,
|
||||
normalize_market_panel,
|
||||
normalize_ohlcv_units,
|
||||
)
|
||||
|
||||
__all__ = ["PitRepo"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 取数列清单（与 Repo 里的 SQL 逐列一致，多一列少一列都会破坏等价性）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MARKET_COLS = (
|
||||
"symbol, trade_date, close, turnover_rate, pe, pe_ttm, pb, ps, ps_ttm, "
|
||||
"dv_ratio, dv_ttm, total_share, float_share, free_share, total_mv, circ_mv"
|
||||
)
|
||||
|
||||
_DAILY_COLS = "symbol, close, volume, amount, trade_date"
|
||||
|
||||
#: 财务表 → 需要的列。``report_type`` 只用于过滤，**不进入返回值**
|
||||
#: （与 ``Repo._latest_financial`` 的 SELECT 列表保持一致）。
|
||||
_FIN_COLS: dict[str, tuple[str, ...]] = {
|
||||
"hd_fina_indicator": (
|
||||
"roe", "roic", "debt_to_assets", "grossprofit_margin",
|
||||
"netprofit_margin", "ocf_to_profit",
|
||||
),
|
||||
"hd_cashflow": (
|
||||
"report_type", "n_cashflow_act", "free_cashflow", "c_pay_dist_dpcp_int_exp",
|
||||
),
|
||||
"hd_balancesheet": (
|
||||
"report_type", "total_assets", "total_liab", "total_hldr_eqy_exc_min_int",
|
||||
"money_cap", "goodwill",
|
||||
),
|
||||
"hd_income": (
|
||||
"report_type", "total_revenue", "revenue", "n_income", "n_income_attr_p",
|
||||
),
|
||||
}
|
||||
|
||||
#: 需要 ``report_type = '1'``（合并报表）过滤的表 —— 与 Repo 中的判断一致
|
||||
_REPORT_TYPE_TABLES = frozenset({"hd_cashflow", "hd_balancesheet", "hd_income"})
|
||||
|
||||
_DIVIDEND_COLS = (
|
||||
"symbol, end_date, ann_date, imp_ann_date, div_proc, cash_div_tax, cash_div, "
|
||||
"stk_div, stk_bo_rate, stk_co_rate, record_date, ex_date, pay_date, base_share"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 诊断计数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class PitRepoStats:
|
||||
"""取数与切片次数（用于回答「到底慢在哪」，而不是靠猜）。"""
|
||||
|
||||
ranges_loaded: int = 0
|
||||
rows_loaded: int = 0
|
||||
reference_loaded: int = 0
|
||||
reference_rows: int = 0
|
||||
market_slices: int = 0
|
||||
amount_slices: int = 0
|
||||
financial_slices: int = 0
|
||||
dividend_slices: int = 0
|
||||
|
||||
def as_dict(self) -> dict[str, int]:
|
||||
return dict(self.__dict__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PitRepo
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class PitRepo(Repo):
|
||||
"""``Repo`` 的批量预载版本：同样的接口、同样的口径，不同的取数方式。
|
||||
|
||||
用法::
|
||||
|
||||
repo = PitRepo()
|
||||
repo.load_reference(end=date(2026, 9, 4)) # 财务/分红/日历（一次）
|
||||
repo.load_range(date(2020, 1, 1), date(2020, 12, 31))
|
||||
sel = UniverseSelector(cfg, repo=repo)
|
||||
sel.run(asof=date(2020, 6, 3), persist=False)
|
||||
repo.release_range() # 释放行情，换下一个区块
|
||||
"""
|
||||
|
||||
#: 预载时向前多取的自然日数，保证区块首日也能算 20 日均额 / 5 日回看
|
||||
OVERLAP_DAYS = 120
|
||||
|
||||
def __init__(self, cfg: DataSourceConfig | None = None) -> None:
|
||||
super().__init__(cfg)
|
||||
self.stats_ = PitRepoStats()
|
||||
self._loaded = False
|
||||
# 交易日历（常驻）
|
||||
self._cal: np.ndarray = np.array([], dtype="datetime64[ns]")
|
||||
self._cal_days: list[date] = []
|
||||
self._today = date.today()
|
||||
# 常驻参照数据
|
||||
self._fin: dict[str, pd.DataFrame] = {}
|
||||
self._dividend = pd.DataFrame()
|
||||
# 区块数据（可释放）
|
||||
self._market = pd.DataFrame()
|
||||
self._daily = pd.DataFrame()
|
||||
self._daily_pos: dict[str, np.ndarray] = {}
|
||||
self._suspend = pd.DataFrame()
|
||||
self._range: tuple[date, date] | None = None
|
||||
#: 参照数据（财务/分红）的终点。常驻帧只覆盖到这一天，
|
||||
#: 任何超出它的区间查询都必须退回直连，否则会静默少行。
|
||||
self._ref_end: date | None = None
|
||||
# 可见性指纹（每张财务表的已公告日集合）—— 缓存的键
|
||||
self._vis_dates: dict[str, np.ndarray] = {}
|
||||
self._memo: dict[tuple, Any] = {}
|
||||
#: 只对**指定股票**回答 ``annual_financials``（见 :meth:`restrict_to`）
|
||||
self._scope: set[str] | None = None
|
||||
#: 候选集预剪枝（见 :meth:`set_candidate_scope`）；None = 与 Repo 口径一致
|
||||
self._candidate_scope: set[str] | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 载入
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def load_reference(self, end: date) -> None:
|
||||
"""一次性载入跨区块共享的数据：交易日历、财务四表、分红明细。
|
||||
|
||||
财务表按 ``ann_date <= end`` 取全量（**不加 end_date 下界**）——
|
||||
``Repo._latest_financial`` 本身没有报告期下界，若在这里加了，
|
||||
「最新一期财报很旧」的股票会被静默漏掉，与直连口径不一致。
|
||||
"""
|
||||
cfg = self.cfg
|
||||
# --- 交易日历：从 2000 年起，保证任何 PIT 回看窗口都有交易日 ---
|
||||
cal = db.read_sql(
|
||||
"SELECT calendar_date FROM trading_calendar "
|
||||
"WHERE is_open = 1 AND calendar_date <= :e ORDER BY calendar_date",
|
||||
{"e": end}, cfg=cfg,
|
||||
)
|
||||
if cal.empty:
|
||||
raise DataGapError("交易日历为空，无法进行每日选股")
|
||||
self._cal = pd.to_datetime(cal["calendar_date"]).to_numpy(dtype="datetime64[ns]")
|
||||
self._cal_days = [pd.Timestamp(x).date() for x in self._cal]
|
||||
self.stats_.reference_rows += len(cal)
|
||||
|
||||
# --- 财务四表 ---
|
||||
for table, cols in _FIN_COLS.items():
|
||||
if not db.table_exists(table, cfg):
|
||||
self._fin[table] = pd.DataFrame(
|
||||
columns=["symbol", "end_date", "ann_date", *cols]
|
||||
)
|
||||
continue
|
||||
sel = ", ".join(f"`{c}`" for c in ("symbol", "end_date", "ann_date", *cols))
|
||||
df = db.read_sql(
|
||||
f"SELECT {sel} FROM `{table}` WHERE ann_date <= :e "
|
||||
"ORDER BY symbol, end_date, ann_date",
|
||||
{"e": end}, cfg=cfg,
|
||||
)
|
||||
for c in ("end_date", "ann_date"):
|
||||
if not df.empty:
|
||||
df[c] = pd.to_datetime(df[c]).dt.date
|
||||
# 内部再存一份 datetime64 版本：**判定与排序一律走它**。
|
||||
# object dtype 的 datetime.date 列在 pandas 里做比较/排序会退化到
|
||||
# Python 循环，实测一个 groupby(...).max() 就要 2.7 秒/次；
|
||||
# 换成 datetime64 后是 Cython 路径，快两个数量级。
|
||||
# 对外的 ann_date/end_date 仍是 date 对象，与 Repo 的返回类型一致。
|
||||
df["_ann"] = pd.to_datetime(df["ann_date"]).to_numpy(dtype="datetime64[ns]")
|
||||
df["_end"] = pd.to_datetime(df["end_date"]).to_numpy(dtype="datetime64[ns]")
|
||||
df["_end_month"] = pd.to_datetime(df["end_date"]).dt.month.to_numpy()
|
||||
if "report_type" in df.columns:
|
||||
df["_is1"] = df["report_type"].astype(str).to_numpy() == "1"
|
||||
else:
|
||||
df["_is1"] = True
|
||||
for c in cols:
|
||||
if c in df.columns:
|
||||
df[c] = pd.to_numeric(df[c], errors="coerce")
|
||||
# **一次性预排序**（symbol, 报告期, 公告日）升序。
|
||||
# 这一步让每个查询都不必再对 30 万行排序：窗口函数要的
|
||||
# 「每个 symbol 取 end_date 最大、并列时 ann_date 最大」恰好等价于
|
||||
# 「在已按 (symbol, end_date, ann_date) 升序的帧上按 symbol 取最后一行」，
|
||||
# 于是 drop_duplicates(keep="last") 就是 SQL 里
|
||||
# ROW_NUMBER() OVER (PARTITION BY symbol ORDER BY end_date DESC,
|
||||
# ann_date DESC) = 1。
|
||||
# 实测：每次查询省掉一次约 0.3~0.5 秒的全表排序；年报季缓存每天都会
|
||||
# 失效（新公告改变了可见集合），这一项就是模拟阶段的主要开销。
|
||||
df = df.sort_values(
|
||||
["symbol", "_end", "_ann"], kind="mergesort", ignore_index=True
|
||||
)
|
||||
self._fin[table] = df
|
||||
self.stats_.reference_rows += len(df)
|
||||
if not df.empty:
|
||||
self._vis_dates[table] = np.unique(df["_ann"])
|
||||
|
||||
# --- 分红（PIT 过滤留给查询时，因为窗口随 asof 滑动）---
|
||||
#
|
||||
# 预载条件必须是**超集**：`imp_ann_date <= :e OR ex_date <= :e`。
|
||||
# 若只按 `imp_ann_date <= :e` 取，会漏掉两类行，而 `dividend_events`
|
||||
# （持仓期间分红入账）需要它们：
|
||||
# - 74 行 `imp_ann_date IS NULL` 但有 `ex_date`（Repo.dividend_events
|
||||
# 本身没有 imp_ann_date 条件，是包含这些行的）；
|
||||
# - 9 行 `imp_ann_date > ex_date`（数据源瑕疵）。
|
||||
# 漏掉就是**静默少算现金分红**，所以宁可取宽。
|
||||
# `dividend_records` 的 PIT 语义不受影响：它的 `imp_ann_date <= asof`
|
||||
# 会把这两类行排除（NaT 比较为 False）。
|
||||
if db.table_exists("hd_dividend", cfg):
|
||||
div = db.read_sql(
|
||||
f"SELECT {_DIVIDEND_COLS} FROM hd_dividend "
|
||||
"WHERE (imp_ann_date <= :e OR ex_date <= :e)",
|
||||
{"e": end}, cfg=cfg,
|
||||
)
|
||||
for c in ("end_date", "ann_date", "imp_ann_date", "record_date",
|
||||
"ex_date", "pay_date"):
|
||||
if c in div.columns and not div.empty:
|
||||
div[c] = pd.to_datetime(div[c]).dt.date
|
||||
for c in ("cash_div_tax", "cash_div", "stk_div", "stk_bo_rate",
|
||||
"stk_co_rate", "base_share"):
|
||||
if c in div.columns:
|
||||
div[c] = pd.to_numeric(div[c], errors="coerce")
|
||||
else:
|
||||
div = pd.DataFrame(columns=[c.strip() for c in _DIVIDEND_COLS.split(",")])
|
||||
self._dividend = div
|
||||
self.stats_.reference_rows += len(div)
|
||||
self.stats_.reference_loaded += 1
|
||||
self._ref_end = end
|
||||
self._loaded = True
|
||||
|
||||
def load_range(self, start: date, end: date) -> None:
|
||||
"""载入 ``[start, end]`` 的行情/每日指标/停牌数据（替换上一个区块）。
|
||||
|
||||
实际取数区间会向前扩 :attr:`OVERLAP_DAYS` 个自然日，使区块首日的
|
||||
「5 日回看」「20 日均额」仍能算全。
|
||||
"""
|
||||
if not self._loaded:
|
||||
raise HdivError("PitRepo 必须先 load_reference(end) 再 load_range(...)")
|
||||
cfg = self.cfg
|
||||
lo = start - timedelta(days=self.OVERLAP_DAYS)
|
||||
|
||||
mk = db.read_sql(
|
||||
f"SELECT {_MARKET_COLS} FROM daily_basic "
|
||||
"WHERE trade_date BETWEEN :s AND :e",
|
||||
{"s": lo, "e": end}, cfg=cfg,
|
||||
)
|
||||
if not mk.empty:
|
||||
mk["trade_date"] = pd.to_datetime(mk["trade_date"])
|
||||
# 排序在 pandas 里做（约 0.5 秒），不用 SQL 的 ORDER BY ——
|
||||
# 实测 MySQL 对 170 万行结果集做 filesort 要多花约 60 秒，
|
||||
# 而后续的 searchsorted 切片只要求「按 trade_date 升序」。
|
||||
mk = mk.sort_values(["trade_date", "symbol"], ignore_index=True)
|
||||
for c in ("close", "turnover_rate", "pe", "pe_ttm", "pb", "ps", "ps_ttm",
|
||||
"dv_ratio", "dv_ttm", "total_share", "float_share", "free_share",
|
||||
"total_mv", "circ_mv"):
|
||||
if c in mk.columns:
|
||||
mk[c] = pd.to_numeric(mk[c], errors="coerce")
|
||||
self._market = mk.reset_index(drop=True)
|
||||
self.stats_.rows_loaded += len(mk)
|
||||
|
||||
dl = db.read_sql(
|
||||
f"SELECT {_DAILY_COLS} FROM stock_daily "
|
||||
"WHERE trade_date BETWEEN :s AND :e",
|
||||
{"s": lo, "e": end}, cfg=cfg,
|
||||
)
|
||||
if not dl.empty:
|
||||
dl["trade_date"] = pd.to_datetime(dl["trade_date"])
|
||||
dl = dl.sort_values(["trade_date", "symbol"], ignore_index=True)
|
||||
for c in ("close", "volume", "amount"):
|
||||
if c in dl.columns:
|
||||
dl[c] = pd.to_numeric(dl[c], errors="coerce")
|
||||
self._daily = dl.reset_index(drop=True)
|
||||
self._daily_pos = self._build_symbol_index(self._daily)
|
||||
self.stats_.rows_loaded += len(dl)
|
||||
|
||||
if db.table_exists("hd_suspend", cfg):
|
||||
sp = db.read_sql(
|
||||
"SELECT symbol, trade_date FROM hd_suspend "
|
||||
"WHERE trade_date BETWEEN :s AND :e AND suspend_type = 'S'",
|
||||
{"s": lo, "e": end}, cfg=cfg,
|
||||
)
|
||||
if not sp.empty:
|
||||
sp["trade_date"] = pd.to_datetime(sp["trade_date"]).dt.date
|
||||
else:
|
||||
sp = pd.DataFrame(columns=["symbol", "trade_date"])
|
||||
self._suspend = sp
|
||||
self.stats_.rows_loaded += len(sp)
|
||||
|
||||
self._range = (lo, end)
|
||||
self.stats_.ranges_loaded += 1
|
||||
|
||||
def release_range(self) -> None:
|
||||
"""释放区块数据（内存随区块数保持常数，而不是随回测长度增长）。"""
|
||||
self._market = pd.DataFrame()
|
||||
self._daily = pd.DataFrame()
|
||||
self._daily_pos = {}
|
||||
self._suspend = pd.DataFrame()
|
||||
self._range = None
|
||||
|
||||
@staticmethod
|
||||
def _build_symbol_index(df: pd.DataFrame) -> dict[str, np.ndarray]:
|
||||
"""``symbol → 行位置数组``（行内保持日期升序）。
|
||||
|
||||
只存整数位置（每行 8 字节），不复制数据。财务/画像会对**单只股票**做
|
||||
上万次 ``avg_amount`` 查询，逐次 ``df[df.symbol == s]`` 是全表扫描，
|
||||
实测会占掉每日循环的大头。
|
||||
"""
|
||||
if df.empty:
|
||||
return {}
|
||||
codes, labels = pd.factorize(df["symbol"])
|
||||
order = np.argsort(codes, kind="stable")
|
||||
sorted_codes = codes[order]
|
||||
bounds = np.searchsorted(sorted_codes, np.arange(len(labels)), side="left")
|
||||
ends = np.searchsorted(sorted_codes, np.arange(len(labels)), side="right")
|
||||
return {
|
||||
str(label): order[bounds[i]:ends[i]] for i, label in enumerate(labels)
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 交易日历（覆盖父类的逐次 SQL 查询）
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _require_loaded(self) -> None:
|
||||
if not self._loaded or self._cal.size == 0:
|
||||
raise HdivError("PitRepo 未载入参照数据，请先调用 load_reference(end)")
|
||||
|
||||
def trading_day(self, asof: date | None = None) -> date:
|
||||
self._require_loaded()
|
||||
if asof is None:
|
||||
asof = min(self._today, self._cal_days[-1])
|
||||
i = int(np.searchsorted(self._cal, np.datetime64(asof, "ns"), side="right")) - 1
|
||||
if i < 0:
|
||||
raise DataGapError(f"交易日历中找不到 <= {asof} 的交易日")
|
||||
return self._cal_days[i]
|
||||
|
||||
def prev_trading_day(self, d: date) -> date:
|
||||
self._require_loaded()
|
||||
i = int(np.searchsorted(self._cal, np.datetime64(d, "ns"), side="left")) - 1
|
||||
if i < 0:
|
||||
raise DataGapError(f"交易日历中找不到 < {d} 的交易日")
|
||||
return self._cal_days[i]
|
||||
|
||||
def trading_days(self, start: date, end: date) -> list[date]:
|
||||
self._require_loaded()
|
||||
i0 = int(np.searchsorted(self._cal, np.datetime64(start, "ns"), side="left"))
|
||||
i1 = int(np.searchsorted(self._cal, np.datetime64(end, "ns"), side="right"))
|
||||
return self._cal_days[i0:i1]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 区块内切片助手
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _assert_in_range(self, asof: date, what: str) -> None:
|
||||
if self._range is None:
|
||||
raise HdivError(f"PitRepo 尚未 load_range，无法查询 {what}")
|
||||
lo, hi = self._range
|
||||
if not (lo <= asof <= hi):
|
||||
raise HdivError(
|
||||
f"{what} 的时点 {asof} 落在已载入区间 [{lo}, {hi}] 之外。\n"
|
||||
f" 这是保护性报错：返回空表会被上层当成「该日没有股票」，\n"
|
||||
f" 静默产出一个空股票池。请调整 load_range 的区间。"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _slice_by_date(df: pd.DataFrame, lo: date, hi: date) -> pd.DataFrame:
|
||||
"""按 ``trade_date``（datetime64）取闭区间切片（对数复杂度）。"""
|
||||
if df.empty:
|
||||
return df
|
||||
col = df["trade_date"].to_numpy(dtype="datetime64[ns]")
|
||||
i0 = int(np.searchsorted(col, np.datetime64(lo, "ns"), side="left"))
|
||||
i1 = int(np.searchsorted(col, np.datetime64(hi, "ns"), side="right"))
|
||||
return df.iloc[i0:i1]
|
||||
|
||||
def _window_days(self, d0: date, window: int) -> list[date]:
|
||||
"""``<= d0`` 的最近 ``window`` 个交易日（与 Repo 的取值方式一致）。"""
|
||||
i1 = int(np.searchsorted(self._cal, np.datetime64(d0, "ns"), side="right"))
|
||||
# Repo 先按自然日 [d0 - 3*window, d0] 取全部交易日，再取末尾 window 个
|
||||
lo = d0 - timedelta(days=window * 3)
|
||||
i0 = int(np.searchsorted(self._cal, np.datetime64(lo, "ns"), side="left"))
|
||||
return self._cal_days[i0:i1][-window:]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 行情 / 每日指标
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def set_candidate_scope(self, allowed: list[str] | set[str] | None) -> None:
|
||||
"""把 :meth:`listed_universe` 的候选集限制在 ``allowed``（``None`` = 不限制）。
|
||||
|
||||
仅供每日选股器的**保守预剪枝**使用：``allowed`` 必须是「在整个回测区间内
|
||||
不可能通过市场滤网」的补集（见 :mod:`hdiv.universe.daily`）。
|
||||
设了范围之后 :meth:`listed_universe` 不再与 :class:`Repo` 逐值一致 ——
|
||||
这是**故意的**，因此默认 ``None``，等价性测试在默认状态下进行。
|
||||
"""
|
||||
self._candidate_scope = None if allowed is None else {str(s) for s in allowed}
|
||||
|
||||
def listed_universe(self, asof: date) -> pd.DataFrame:
|
||||
"""与 :meth:`Repo.listed_universe` 一致，另可叠加候选集预剪枝。"""
|
||||
out = super().listed_universe(asof)
|
||||
if self._candidate_scope is None or out.empty:
|
||||
return out
|
||||
keep = out["symbol"].astype(str).isin(self._candidate_scope)
|
||||
return out[keep].reset_index(drop=True)
|
||||
|
||||
def listed_counts(self, asof: date) -> tuple[int, int]:
|
||||
"""``(当日市场候选数, 预剪枝后候选数)``。
|
||||
|
||||
两个数都要留痕：只记一个会让页面上的「候选」在开/关预剪枝时含义不同，
|
||||
而预剪枝是**纯性能开关**，不该改变任何可见数字的语义。
|
||||
额外的这次 ``listed_universe`` 约 1 毫秒（``stock_master`` 已缓存）。
|
||||
"""
|
||||
out = Repo.listed_universe(self, asof)
|
||||
total = int(len(out))
|
||||
if self._candidate_scope is None or out.empty:
|
||||
return total, total
|
||||
kept = int(out["symbol"].astype(str).isin(self._candidate_scope).sum())
|
||||
return total, kept
|
||||
|
||||
def market_panel(self, asof: date, *, lookback_days: int = 0) -> pd.DataFrame:
|
||||
"""与 :meth:`Repo.market_panel` 逐值一致，只是数据来自内存区块。"""
|
||||
self._assert_in_range(asof, "market_panel")
|
||||
self.stats_.market_slices += 1
|
||||
d0 = self.trading_day(asof)
|
||||
if lookback_days <= 0:
|
||||
sub = self._slice_by_date(self._market, d0, d0).copy()
|
||||
if not sub.empty:
|
||||
sub["trade_date"] = sub["trade_date"].dt.date
|
||||
sub["is_fresh"] = True
|
||||
sub["asof_trade_date"] = d0
|
||||
return normalize_market_panel(sub)
|
||||
|
||||
days = self._window_days(d0, lookback_days)
|
||||
if not days:
|
||||
out = self._market.iloc[0:0].copy()
|
||||
out["is_fresh"] = pd.Series(dtype=bool)
|
||||
out["asof_trade_date"] = pd.Series(dtype="object")
|
||||
return out
|
||||
sub = self._slice_by_date(self._market, days[0], d0).copy()
|
||||
if sub.empty:
|
||||
sub["is_fresh"] = pd.Series(dtype=bool)
|
||||
sub["asof_trade_date"] = pd.Series(dtype="object")
|
||||
return sub
|
||||
# Repo 把 trade_date 转成 datetime.date 后再排序取 last，此处保持一致
|
||||
sub["trade_date"] = sub["trade_date"].dt.date
|
||||
sub = sub.sort_values(["symbol", "trade_date"])
|
||||
last = sub.groupby("symbol", as_index=False).last()
|
||||
last["is_fresh"] = last["trade_date"].eq(d0)
|
||||
last["asof_trade_date"] = d0
|
||||
return normalize_market_panel(last.reset_index(drop=True))
|
||||
|
||||
def avg_amount(
|
||||
self, asof: date, window: int = 20, *, symbols: list[str] | None = None
|
||||
) -> pd.DataFrame:
|
||||
"""与 :meth:`Repo.avg_amount` 逐值一致。
|
||||
|
||||
``symbols`` 非空时走**预建的 symbol→行位置索引**：画像会对单只股票
|
||||
反复查询，逐次全表过滤会让每日循环退化成「天数 × 全表扫描」。
|
||||
|
||||
**区块外**：带 ``symbols`` 的窄查询回退到直连数据库 —— 它走
|
||||
``symbol IN (...)`` 索引，是毫秒级；而全市场查询必须落在已载入区块内，
|
||||
否则会返回空表（被上层当成「当天没有股票」）。画像路径在区块释放后
|
||||
正是靠这条回退工作的。
|
||||
"""
|
||||
if self._range is None or not (self._range[0] <= asof <= self._range[1]):
|
||||
if symbols:
|
||||
return Repo.avg_amount(self, asof, window=window, symbols=symbols)
|
||||
self._assert_in_range(asof, "avg_amount（全市场）")
|
||||
self.stats_.amount_slices += 1
|
||||
d0 = self.trading_day(asof)
|
||||
days = self._window_days(d0, window)
|
||||
if not days:
|
||||
return pd.DataFrame(columns=["symbol", "avg_amount", "n"])
|
||||
lo, hi = days[0], days[-1]
|
||||
if symbols:
|
||||
parts: list[pd.DataFrame] = []
|
||||
for sym in symbols:
|
||||
pos = self._daily_pos.get(str(sym))
|
||||
if pos is None or pos.size == 0:
|
||||
continue
|
||||
sub = self._daily.take(pos)
|
||||
sub = self._slice_by_date(sub, lo, hi)
|
||||
if not sub.empty:
|
||||
parts.append(sub)
|
||||
df = pd.concat(parts, ignore_index=True) if parts else self._daily.iloc[0:0]
|
||||
else:
|
||||
df = self._slice_by_date(self._daily, lo, hi)
|
||||
|
||||
if df.empty:
|
||||
return pd.DataFrame(columns=["symbol", "avg_amount", "n"])
|
||||
df = df[["symbol", "close", "volume", "amount"]].copy()
|
||||
for c in ("close", "volume", "amount"):
|
||||
df[c] = pd.to_numeric(df[c], errors="coerce")
|
||||
# 必须**逐切片**归一化：单位判定是逐行的，切片与全表的判定结果相同，
|
||||
# 但如果在预载时统一换算，区块边界处会与 Repo 的窗口口径分叉。
|
||||
df, _diag = normalize_ohlcv_units(df)
|
||||
g = df.groupby("symbol", as_index=False).agg(
|
||||
avg_amount=("amount", "mean"), n=("amount", "size")
|
||||
)
|
||||
return g
|
||||
|
||||
def suspended_on(self, asof: date) -> set[str]:
|
||||
self._assert_in_range(asof, "suspended_on")
|
||||
if self._suspend.empty:
|
||||
return set()
|
||||
d0 = self.trading_day(asof)
|
||||
hit = self._suspend[self._suspend["trade_date"] == d0]
|
||||
return set(hit["symbol"].tolist())
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 财务（只覆盖底层取数；上层派生一律沿用父类）
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _fin_table(self, table: str) -> pd.DataFrame:
|
||||
df = self._fin.get(table)
|
||||
if df is None:
|
||||
return pd.DataFrame(columns=["symbol", "end_date", "ann_date"])
|
||||
return df
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 可见性缓存：同一批「已公告财报」→ 同一个结果
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _visible_key(self, table: str, asof: date) -> int:
|
||||
"""``asof`` 之前已公告的财报**条数**（该表已公告日的排序插入位置）。
|
||||
|
||||
这是「哪些财报当时可见」的**充分统计量**：条数相同，可见集合就相同，
|
||||
因此以它为缓存键是**精确**的，不是近似 —— 与「按 asof 缓存」不同，
|
||||
后者在年报季几乎每天都不命中，等于没缓存。
|
||||
"""
|
||||
arr = self._vis_dates.get(table)
|
||||
if arr is None or arr.size == 0:
|
||||
return 0
|
||||
return int(np.searchsorted(arr, np.datetime64(asof, "ns"), side="right"))
|
||||
|
||||
def _memo_get(self, key: tuple, build: Any) -> Any:
|
||||
hit = self._memo.get(key)
|
||||
if hit is None:
|
||||
hit = build()
|
||||
# 键随 asof 单调变化，历史条目不会再被命中 —— 小容量即可
|
||||
if len(self._memo) > 16:
|
||||
self._memo.clear()
|
||||
self._memo[key] = hit
|
||||
return hit
|
||||
|
||||
def restrict_to(self, symbols: list[str] | None) -> None:
|
||||
"""把 ``annual_financials`` 的回答范围收窄到 ``symbols``（``None`` = 全市场）。
|
||||
|
||||
为什么可以收窄：``annual_financials`` 唯一的消费者是分红滤网的
|
||||
``_fy_table``，它把结果建成 ``{(symbol, 财年): 行}`` 后只按**候选股**查表，
|
||||
非候选股的行永远不会被读取。而它默认会对全市场 5000 余只 × 10 个财年
|
||||
构造约 5 万行再逐行 ``iterrows()`` —— 实测 2.8 秒/天，是每日选股里
|
||||
最大的单项开销；收窄到市场/风险滤网的存活者（约 170 只）后降到 0.1 秒级。
|
||||
|
||||
这不是「近似」：被剔除的行在调用方从未被访问。收窄只在**筛选路径**上生效，
|
||||
调用方必须在用完后显式 ``restrict_to(None)`` 复位（每日选股器就是这么做的），
|
||||
否则画像路径的 ``annual_financials`` 会被误伤。
|
||||
"""
|
||||
self._scope = None if symbols is None else {str(s) for s in symbols}
|
||||
|
||||
def _scoped(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
if self._scope is None or df.empty or "symbol" not in df.columns:
|
||||
return df
|
||||
return df[df["symbol"].isin(self._scope)]
|
||||
|
||||
@staticmethod
|
||||
def _np(d: date) -> np.datetime64:
|
||||
return np.datetime64(d, "ns")
|
||||
|
||||
def _latest_financial(
|
||||
self, asof: date, table: str, cols: list[str], *,
|
||||
symbols: list[str] | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo._latest_financial`` 的窗口函数：
|
||||
|
||||
``ROW_NUMBER() OVER (PARTITION BY symbol ORDER BY end_date DESC, ann_date DESC)``
|
||||
，条件 ``ann_date <= asof AND ann_date >= end_date``（三张表另加
|
||||
``report_type = '1'``）。父类的 :meth:`Repo.financial_panel` 会继续
|
||||
调用本方法，因此合并与衍生指标的代码原封不动。
|
||||
"""
|
||||
self.stats_.financial_slices += 1
|
||||
empty = pd.DataFrame(columns=["symbol", "end_date", "ann_date", *cols])
|
||||
df = self._fin_table(table)
|
||||
if df.empty:
|
||||
return empty
|
||||
|
||||
def _build() -> pd.DataFrame:
|
||||
a = self._np(asof)
|
||||
m = (df["_ann"] <= a) & (df["_ann"] >= df["_end"])
|
||||
if table in _REPORT_TYPE_TABLES:
|
||||
m &= df["_is1"].to_numpy()
|
||||
sub = df[m]
|
||||
if sub.empty:
|
||||
return empty
|
||||
# 存储已按 (symbol, _end, _ann) 升序预排 → 取每个 symbol 的最后一行
|
||||
# 就是 SQL 的 ROW_NUMBER(... ORDER BY end_date DESC, ann_date DESC) = 1
|
||||
first = sub.drop_duplicates(subset=["symbol"], keep="last")
|
||||
out = first[["symbol", "end_date", "ann_date", *cols]].reset_index(drop=True)
|
||||
return normalize_financial_panel(out)
|
||||
|
||||
# 全市场结果按「可见财报条数」缓存；symbols 只是它的子集，不能反过来
|
||||
# 用子集覆盖全量缓存（否则下一次全市场查询会拿到残缺的面板）。
|
||||
key = ("latest", table, tuple(cols), self._visible_key(table, asof))
|
||||
full = self._memo_get(key, _build)
|
||||
if symbols:
|
||||
return full[full["symbol"].isin(set(symbols))].reset_index(drop=True)
|
||||
return full
|
||||
|
||||
def annual_financial_history(
|
||||
self, asof: date, *, years: int = 6, symbols: list[str] | None = None
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo.annual_financial_history``：只取年报，按
|
||||
``(symbol, end_date)`` 取最大公告日那一行，再做百分数归一化。"""
|
||||
df = self._fin_table("hd_fina_indicator")
|
||||
if df.empty:
|
||||
return pd.DataFrame(columns=["symbol", "year", "roe", "roic"])
|
||||
since = date(asof.year - years - 1, 12, 31)
|
||||
|
||||
def _build() -> pd.DataFrame:
|
||||
a, sn = self._np(asof), self._np(since)
|
||||
m = (
|
||||
(df["_ann"] <= a)
|
||||
& (df["_ann"] >= df["_end"])
|
||||
& (df["_end_month"] == 12)
|
||||
& (df["_end"] >= sn)
|
||||
)
|
||||
sub = df[m]
|
||||
if sub.empty:
|
||||
return pd.DataFrame(columns=["symbol", "year", "roe", "roic"])
|
||||
# 存储已按 (symbol, _end, _ann) 升序预排 → 每个 (symbol, 报告期)
|
||||
# 取最后一行即「最大 ann_date」那一行（等价于 SQL 的 MAX(ann_date) 自连接；
|
||||
# 前提是 (symbol, end_date, ann_date) 无重复，已由实测确认）。
|
||||
fin = sub.drop_duplicates(subset=["symbol", "_end"], keep="last").copy()
|
||||
fin["year"] = fin["_end"].dt.year
|
||||
# 只保留 Repo.annual_financial_history 的 SELECT 列表 ——
|
||||
# 多带一列（如 debt_to_assets）会让下游列集合与直连口径分叉。
|
||||
keep = ["symbol", "end_date", "ann_date", "roe", "roic",
|
||||
"grossprofit_margin", "netprofit_margin", "ocf_to_profit", "year"]
|
||||
for c in ("roe", "roic", "grossprofit_margin", "netprofit_margin",
|
||||
"ocf_to_profit"):
|
||||
if c in fin.columns:
|
||||
fin[c] = fin[c] / 100.0
|
||||
fin = fin[keep]
|
||||
ocf = self._annual_ocf_ratio(asof, since, symbols=None)
|
||||
if not ocf.empty:
|
||||
fin = fin.merge(ocf, on=["symbol", "year"], how="left")
|
||||
return fin
|
||||
|
||||
key = ("afh", self._visible_key("hd_fina_indicator", asof), years, str(since))
|
||||
full = self._memo_get(key, _build)
|
||||
if symbols:
|
||||
return full[full["symbol"].isin(set(symbols))].reset_index(drop=True)
|
||||
return full
|
||||
|
||||
def annual_financial_averages(
|
||||
self, asof: date, *, years: int = 5, min_years: int = 3,
|
||||
symbols: list[str] | None = None, hist: pd.DataFrame | None = None,
|
||||
) -> pd.DataFrame:
|
||||
"""缓存版本：全市场结果按可见财报条数复用，``symbols`` 只做子集。
|
||||
|
||||
逐股聚合与「是否只算这些股票」无关（每个 symbol 独立求均值），
|
||||
因此「先算全市场再取子集」与直连口径逐值相同。
|
||||
"""
|
||||
if hist is not None:
|
||||
return Repo.annual_financial_averages(
|
||||
self, asof, years=years, min_years=min_years, symbols=symbols, hist=hist,
|
||||
)
|
||||
key = ("afa", self._visible_key("hd_fina_indicator", asof), years, min_years)
|
||||
full = self._memo_get(
|
||||
key,
|
||||
lambda: Repo.annual_financial_averages(
|
||||
self, asof, years=years, min_years=min_years, symbols=None,
|
||||
),
|
||||
)
|
||||
if symbols:
|
||||
return full[full["symbol"].isin(set(symbols))].reset_index(drop=True)
|
||||
return full
|
||||
|
||||
def _annual_ocf_ratio(
|
||||
self, asof: date, since: date, *, symbols: list[str] | None = None
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo._annual_ocf_ratio``：现金流量表 ⋈ 利润表（同年报期同公告日）。"""
|
||||
cf, inc = self._fin_table("hd_cashflow"), self._fin_table("hd_income")
|
||||
if cf.empty or inc.empty:
|
||||
return pd.DataFrame(columns=["symbol", "year", "ocf_to_netprofit_calc"])
|
||||
|
||||
def _build() -> pd.DataFrame:
|
||||
a, sn = self._np(asof), self._np(since)
|
||||
c = cf[cf["_is1"].to_numpy()]
|
||||
i = inc[inc["_is1"].to_numpy()]
|
||||
c = c[(c["_end_month"] == 12) & (c["_end"] >= sn) & (c["_ann"] <= a)]
|
||||
if c.empty:
|
||||
return pd.DataFrame(columns=["symbol", "year", "ocf_to_netprofit_calc"])
|
||||
joined = c.merge(
|
||||
i[["symbol", "_end", "_ann", "n_income_attr_p"]],
|
||||
on=["symbol", "_end", "_ann"], how="inner",
|
||||
suffixes=("", "_i"),
|
||||
)
|
||||
if joined.empty:
|
||||
return pd.DataFrame(columns=["symbol", "year", "ocf_to_netprofit_calc"])
|
||||
joined["year"] = joined["_end"].dt.year
|
||||
ocf = pd.to_numeric(joined["n_cashflow_act"], errors="coerce")
|
||||
ni = pd.to_numeric(joined["n_income_attr_p"], errors="coerce")
|
||||
joined["ocf_to_netprofit_calc"] = ocf / ni.replace(0, pd.NA)
|
||||
return joined[["symbol", "year", "ocf_to_netprofit_calc"]]
|
||||
|
||||
key = ("ocf", self._visible_key("hd_cashflow", asof),
|
||||
self._visible_key("hd_income", asof), str(since))
|
||||
full = self._memo_get(key, _build)
|
||||
if symbols:
|
||||
return full[full["symbol"].isin(set(symbols))].reset_index(drop=True)
|
||||
return full
|
||||
|
||||
def annual_financials(
|
||||
self, asof: date, *, years: int = 12, symbols: list[str] | None = None
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo.annual_financials``（年报口径、按财年对齐）。
|
||||
|
||||
``symbols`` 为空时若已通过 :meth:`restrict_to` 设定了范围，则只返回
|
||||
范围内的股票 —— 见 :meth:`restrict_to` 对「为什么这是精确的」的说明。
|
||||
"""
|
||||
inc, cf = self._fin_table("hd_income"), self._fin_table("hd_cashflow")
|
||||
empty = pd.DataFrame(
|
||||
columns=["symbol", "year", "n_income", "n_income_attr_p",
|
||||
"n_cashflow_act", "free_cashflow", "c_pay_dist_dpcp_int_exp"]
|
||||
)
|
||||
if inc.empty:
|
||||
return empty
|
||||
since = date(asof.year - years - 1, 12, 31)
|
||||
a, sn = self._np(asof), self._np(since)
|
||||
want = set(symbols) if symbols else self._scope
|
||||
n12 = inc["_end_month"].to_numpy() == 12
|
||||
i = inc[inc["_is1"].to_numpy() & n12].copy()
|
||||
i = i[(i["_end"] >= sn) & (i["_ann"] <= a) & (i["_ann"] >= i["_end"])]
|
||||
if want:
|
||||
i = i[i["symbol"].isin(want)]
|
||||
if i.empty:
|
||||
return empty
|
||||
if cf.empty:
|
||||
joined = i.copy()
|
||||
for c in ("n_cashflow_act", "free_cashflow", "c_pay_dist_dpcp_int_exp"):
|
||||
joined[c] = pd.NA
|
||||
else:
|
||||
cn12 = cf["_end_month"].to_numpy() == 12
|
||||
f = cf[cf["_is1"].to_numpy() & cn12 & (cf["_end"].to_numpy() >= sn)][
|
||||
["symbol", "_end", "_ann", "n_cashflow_act", "free_cashflow",
|
||||
"c_pay_dist_dpcp_int_exp"]
|
||||
]
|
||||
joined = i.merge(f, on=["symbol", "_end", "_ann"], how="left")
|
||||
# 利润表与现金流量表可能各有多行（重复公告），LEFT JOIN 会放大行数 ——
|
||||
# 与 SQL 的 LEFT JOIN 行为一致，故此处不额外去重。
|
||||
joined["year"] = joined["_end"].dt.year
|
||||
for c in ("n_income", "n_income_attr_p", "n_cashflow_act", "free_cashflow",
|
||||
"c_pay_dist_dpcp_int_exp"):
|
||||
if c in joined.columns:
|
||||
joined[c] = pd.to_numeric(joined[c], errors="coerce")
|
||||
return joined[["symbol", "year", "n_income", "n_income_attr_p", "n_cashflow_act",
|
||||
"free_cashflow", "c_pay_dist_dpcp_int_exp"]]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 分红
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def dividend_records(
|
||||
self, asof: date, *, years_back: int = 12, implemented_only: bool = True
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo.dividend_records`` 的 PIT 三重条件（内存过滤）。
|
||||
|
||||
**P4：若已 ``restrict_to`` 设定范围，只返回范围内的股票。**
|
||||
分红滤网拿到记录后先 ``_group()`` 建成 ``{symbol: [记录]}``，再**只按候选股**
|
||||
查表；为全市场建表的那部分永远不会被读到。逐日筛选时这一步是每天对
|
||||
2 万余行做一次 ``to_dict("records")``，而候选通常只有一两百只 ——
|
||||
收窄范围把它降到百分之几，且不改变任何候选股的判定输入。
|
||||
|
||||
注意：画像路径（``PitProfileService._context`` 里的 ``self._all_dividends``）
|
||||
调用本方法时范围必须是 ``None``；每日选股器用完后会显式复位。
|
||||
"""
|
||||
self.stats_.dividend_slices += 1
|
||||
d = self._dividend
|
||||
if d.empty:
|
||||
return d
|
||||
if self._scope is not None:
|
||||
d = d[d["symbol"].isin(self._scope)]
|
||||
if d.empty:
|
||||
return d
|
||||
since = asof - timedelta(days=int(years_back * 365.25))
|
||||
imp = pd.to_datetime(d["imp_ann_date"]).dt.date
|
||||
ex = pd.to_datetime(d["ex_date"], errors="coerce").dt.date
|
||||
m = (imp <= asof) & ex.notna() & (ex <= asof) & (ex >= since)
|
||||
if implemented_only:
|
||||
m &= d["div_proc"].astype(str) == "实施"
|
||||
out = d[m].sort_values(["symbol", "ex_date"])
|
||||
return out.reset_index(drop=True)
|
||||
|
||||
def dividend_events(
|
||||
self, start: date, end: date, *, implemented_only: bool = True
|
||||
) -> pd.DataFrame:
|
||||
"""等价于 ``Repo.dividend_events``，但直接从常驻分红明细里筛（P6）。
|
||||
|
||||
引擎在 ``_prepare`` 里为「持仓期间的分红入账」取一次区间事件，
|
||||
原先这会再查一次库；而参照数据阶段已经把**全部分红明细**常驻在内存里了。
|
||||
过滤条件与 ``Repo.dividend_events`` 逐条对齐（区间按 ex_date、
|
||||
现金或送转为正、可选只取已实施）。
|
||||
"""
|
||||
cols = ["symbol", "end_date", "imp_ann_date", "cash_div_tax", "cash_div",
|
||||
"stk_div", "stk_bo_rate", "stk_co_rate", "record_date", "ex_date",
|
||||
"pay_date"]
|
||||
if self._ref_end is not None and end > self._ref_end:
|
||||
# 区间超出参照数据的终点：常驻帧里可能缺行（超集只覆盖到 _ref_end）。
|
||||
# 这种情况宁可退回直连查询，也不能返回一个**看起来正常但少了几行**的表
|
||||
# —— 少一行就是少一笔现金分红。
|
||||
return Repo.dividend_events(
|
||||
self, start, end, implemented_only=implemented_only
|
||||
)
|
||||
d = self._dividend
|
||||
if d.empty:
|
||||
return pd.DataFrame(columns=cols)
|
||||
ex = pd.to_datetime(d["ex_date"], errors="coerce")
|
||||
cash = pd.to_numeric(d["cash_div_tax"], errors="coerce")
|
||||
stk = pd.to_numeric(d["stk_div"], errors="coerce")
|
||||
m = (
|
||||
ex.notna()
|
||||
& (ex >= pd.Timestamp(start))
|
||||
& (ex <= pd.Timestamp(end))
|
||||
& ((cash > 0) | (stk > 0))
|
||||
)
|
||||
if implemented_only:
|
||||
m &= d["div_proc"].astype(str) == "实施"
|
||||
out = d[m].sort_values(["ex_date", "symbol"])[cols].reset_index(drop=True)
|
||||
# 可空日期列统一成「object，缺值为 None」—— 与 `Repo.dividend_events`
|
||||
# （直接读 SQL）的形态一致，而不是 NaT。`NaT != None` 这种差别虽然不影响
|
||||
# 当前调用方（引擎只用 cash_div_tax / stk_div），但会让逐值比对失败，
|
||||
# 也会给后续消费者埋一个「判空写法依赖列类型」的坑。
|
||||
for c in ("end_date", "imp_ann_date", "record_date", "ex_date", "pay_date"):
|
||||
if c in out.columns:
|
||||
s = pd.to_datetime(out[c], errors="coerce")
|
||||
out[c] = s.dt.date.where(s.notna(), None).astype(object)
|
||||
return out
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 诊断
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def stats(self) -> dict[str, int]:
|
||||
d = self.stats_.as_dict()
|
||||
d["market_rows"] = int(len(self._market))
|
||||
d["daily_rows"] = int(len(self._daily))
|
||||
d["fin_rows"] = int(sum(len(v) for v in self._fin.values()))
|
||||
d["dividend_rows"] = int(len(self._dividend))
|
||||
return d
|
||||
@@ -41,9 +41,12 @@ FILTER_ORDER: tuple[str, ...] = ("market", "risk", "dividend", "quality")
|
||||
|
||||
|
||||
class UniverseSelector:
|
||||
def __init__(self, config: UniverseConfig) -> None:
|
||||
def __init__(self, config: UniverseConfig, repo: Any | None = None) -> None:
|
||||
self.config = config
|
||||
self.repo = Repo()
|
||||
#: 取数出口。默认直连数据库；``--mode daily`` 会注入
|
||||
#: :class:`~hdiv.universe.pit.PitRepo`（批量预载版，口径相同）。
|
||||
#: 注入点放在这里，是为了让滤网代码与口径**完全不需要改**。
|
||||
self.repo = repo if repo is not None else Repo()
|
||||
self._filters: dict[str, Filter] = {}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -51,14 +54,14 @@ class UniverseSelector:
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, path: str | Path) -> UniverseSelector:
|
||||
def from_config(cls, path: str | Path, repo: Any | None = None) -> UniverseSelector:
|
||||
"""从 YAML 加载筛选配置（支持 ``strategy`` 段落里的 override）。"""
|
||||
raw = _read_yaml(path)
|
||||
return cls(UniverseConfig.model_validate(raw))
|
||||
return cls(UniverseConfig.model_validate(raw), repo=repo)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, raw: dict[str, Any]) -> UniverseSelector:
|
||||
return cls(UniverseConfig.model_validate(raw))
|
||||
def from_dict(cls, raw: dict[str, Any], repo: Any | None = None) -> UniverseSelector:
|
||||
return cls(UniverseConfig.model_validate(raw), repo=repo)
|
||||
|
||||
@classmethod
|
||||
def with_override(
|
||||
@@ -97,6 +100,7 @@ class UniverseSelector:
|
||||
*,
|
||||
persist: bool = True,
|
||||
verbose: bool = True,
|
||||
on_stage: Any | None = None,
|
||||
) -> dict[str, Any]:
|
||||
cfg = load_config("datasource")
|
||||
filters = self._build_filters()
|
||||
@@ -126,6 +130,11 @@ class UniverseSelector:
|
||||
if live.empty:
|
||||
stats[fname] = 0
|
||||
continue
|
||||
# 可选钩子：把「本阶段将要评估的股票」告诉调用方。
|
||||
# 每日选股用它把取数范围收窄到存活者（``PitRepo.restrict_to``），
|
||||
# 这是纯性能开关，不改变任何判定 —— 默认 None 时行为与改造前一致。
|
||||
if on_stage is not None:
|
||||
on_stage(fname, live)
|
||||
outcome = flt.compute(live, self.repo, effective)
|
||||
# 注意：outcome.passed 的索引是 live 的 DataFrame 索引（不是 symbol），
|
||||
# 必须经 live.at[i, "symbol"] 映射，否则会把索引当代码用。
|
||||
|
||||
+231
-8
@@ -20,7 +20,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import date, timedelta
|
||||
from datetime import date, datetime, timedelta
|
||||
from decimal import Decimal
|
||||
from typing import Any
|
||||
|
||||
@@ -48,6 +48,10 @@ SERIES_KEYS = ("close", "dv_yield", "pe_ttm", "roe", "pb", "drawdown")
|
||||
#: 只影响画图，不影响买卖点（买卖点单独返回且不降采样）。
|
||||
MAX_POINTS = 3200
|
||||
|
||||
#: 缺省区间往前至少回看的年数：股息率分位是买入判据，
|
||||
#: 成交之前那几年的历史正是「凭什么买」的依据。
|
||||
MIN_LOOKBACK_YEARS = 5
|
||||
|
||||
|
||||
def _v(x: Any) -> Any:
|
||||
if x is None:
|
||||
@@ -278,6 +282,146 @@ def run_stocks(run_id: str) -> list[dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
def _reason_of(raw: Any) -> dict[str, Any]:
|
||||
"""reason_json → dict；坏数据不该让整个接口挂掉。"""
|
||||
if not raw:
|
||||
return {}
|
||||
try:
|
||||
out = json.loads(raw)
|
||||
except (TypeError, ValueError):
|
||||
return {}
|
||||
return out if isinstance(out, dict) else {}
|
||||
|
||||
|
||||
def closed_positions(run_id: str) -> dict[str, Any]:
|
||||
"""已清仓（期末不再持有）的个股清单。
|
||||
|
||||
「已清仓」以**持仓表**为准：在该回测中持有过、但最后一个持仓日已不在其中。
|
||||
只按成交净额判断会漏掉「卖了又买回、期末仍持有」的股票。
|
||||
|
||||
每只票带上「卖出后至今」涨跌：清仓复盘真正要回答的是
|
||||
「这笔卖对了没有」，只看成交明细是答不了的。
|
||||
"""
|
||||
cfg = load_config("datasource")
|
||||
if db.read_sql("SELECT 1 AS x FROM hd_backtest_run WHERE run_id = :r LIMIT 1",
|
||||
{"r": run_id}, cfg=cfg).empty:
|
||||
raise HdivError(f"回测不存在：{run_id}")
|
||||
pos = db.read_sql(
|
||||
"SELECT symbol, COUNT(*) AS hold_days, MIN(trade_date) AS first_hold, "
|
||||
" MAX(trade_date) AS last_hold "
|
||||
"FROM hd_backtest_position WHERE run_id = :r GROUP BY symbol",
|
||||
{"r": run_id}, cfg=cfg,
|
||||
)
|
||||
last_day = _v(db.read_sql(
|
||||
"SELECT MAX(trade_date) AS d FROM hd_backtest_position WHERE run_id = :r",
|
||||
{"r": run_id}, cfg=cfg)["d"].iloc[0])
|
||||
empty: dict[str, Any] = {
|
||||
"items": [],
|
||||
"summary": {"count": 0, "realized_pnl": 0.0, "since_sell_up": 0,
|
||||
"since_sell_down": 0, "asof": None},
|
||||
"asof": last_day,
|
||||
}
|
||||
if last_day is None or pos.empty:
|
||||
return empty
|
||||
held = set(db.read_sql(
|
||||
"SELECT DISTINCT symbol FROM hd_backtest_position "
|
||||
"WHERE run_id = :r AND trade_date = :d",
|
||||
{"r": run_id, "d": last_day}, cfg=cfg)["symbol"])
|
||||
closed = [s for s in pos["symbol"] if s not in held]
|
||||
if not closed:
|
||||
return empty
|
||||
|
||||
# 成交汇总：买入/卖出金额、已实现盈亏、最后一笔卖出的日期与理由
|
||||
tr = db.read_sql(
|
||||
"SELECT symbol, side, execution_date, amount, realized_pnl, reason_json "
|
||||
"FROM hd_backtest_trade WHERE run_id = :r ORDER BY execution_date, trade_id",
|
||||
{"r": run_id}, cfg=cfg,
|
||||
)
|
||||
agg: dict[str, dict[str, Any]] = {}
|
||||
for _, r in tr.iterrows():
|
||||
a = agg.setdefault(r["symbol"], {"buy": 0.0, "sell": 0.0, "pnl": 0.0,
|
||||
"last_sell": None, "reason": None})
|
||||
if r["side"] == "BUY":
|
||||
a["buy"] += _fnum(r["amount"]) or 0.0
|
||||
else:
|
||||
a["sell"] += _fnum(r["amount"]) or 0.0
|
||||
a["pnl"] += _fnum(r["realized_pnl"]) or 0.0
|
||||
a["last_sell"] = _v(r["execution_date"])
|
||||
a["reason"] = _reason_text(_reason_of(r["reason_json"]))
|
||||
|
||||
ph = ", ".join(f":c{i}" for i in range(len(closed)))
|
||||
cparams = {f"c{i}": s for i, s in enumerate(closed)}
|
||||
meta = {r["symbol"]: r for _, r in db.read_sql(
|
||||
f"SELECT symbol, MAX(name) AS name, MAX(industry) AS industry FROM stock "
|
||||
f"WHERE symbol IN ({ph}) GROUP BY symbol", cparams, cfg=cfg).iterrows()}
|
||||
|
||||
# 清仓当日收盘（「卖出后至今」的基准）与最新收盘
|
||||
pairs = [(s, agg[s]["last_sell"]) for s in closed
|
||||
if agg.get(s, {}).get("last_sell")]
|
||||
at_sell: dict[str, float | None] = {}
|
||||
if pairs:
|
||||
cond = ", ".join(f"(:p{i}s, :p{i}d)" for i in range(len(pairs)))
|
||||
pparams: dict[str, Any] = {}
|
||||
for i, (s, d) in enumerate(pairs):
|
||||
pparams[f"p{i}s"], pparams[f"p{i}d"] = s, d
|
||||
for _, r in db.read_sql(
|
||||
f"SELECT symbol, close FROM daily_basic "
|
||||
f"WHERE (symbol, trade_date) IN ({cond})", pparams, cfg=cfg,
|
||||
).iterrows():
|
||||
at_sell[r["symbol"]] = _fnum(r["close"])
|
||||
|
||||
latest: dict[str, tuple[float | None, str | None]] = {}
|
||||
for _, r in db.read_sql(
|
||||
f"SELECT d.symbol, d.trade_date, d.close FROM daily_basic d "
|
||||
f"JOIN (SELECT symbol, MAX(trade_date) AS mx FROM daily_basic "
|
||||
f" WHERE symbol IN ({ph}) GROUP BY symbol) t "
|
||||
f" ON t.symbol = d.symbol AND t.mx = d.trade_date",
|
||||
cparams, cfg=cfg,
|
||||
).iterrows():
|
||||
latest[r["symbol"]] = (_fnum(r["close"]), _v(r["trade_date"]))
|
||||
|
||||
items = []
|
||||
for s in closed:
|
||||
a = agg.get(s, {})
|
||||
row = pos[pos["symbol"] == s].iloc[0]
|
||||
pnl = _fnum(a.get("pnl")) or 0.0
|
||||
buy = _fnum(a.get("buy")) or 0.0
|
||||
c0 = at_sell.get(s)
|
||||
c1, c1_date = latest.get(s, (None, None))
|
||||
since = (c1 / c0 - 1.0) if (c0 and c1 and c0 > 0) else None
|
||||
items.append({
|
||||
"symbol": s,
|
||||
"name": _v(meta[s]["name"]) if s in meta else None,
|
||||
"industry": _v(meta[s]["industry"]) if s in meta else None,
|
||||
"first_hold": _v(row["first_hold"]), "last_hold": _v(row["last_hold"]),
|
||||
"hold_days": int(row["hold_days"]),
|
||||
"last_sell": a.get("last_sell"),
|
||||
"buy_amount": buy or None, "sell_amount": _fnum(a.get("sell")),
|
||||
"realized_pnl": pnl,
|
||||
# 已清仓，所以「已实现盈亏 ÷ 买入金额」就是这笔投资的收益率
|
||||
"return_pct": (pnl / buy) if buy > 0 else None,
|
||||
"close_at_sell": c0, "close_latest": c1, "price_asof": c1_date,
|
||||
"since_sell_pct": since,
|
||||
"sell_reason": a.get("reason"),
|
||||
})
|
||||
items.sort(key=lambda x: (x["last_sell"] or "", x["symbol"]), reverse=True)
|
||||
|
||||
asof = max((x["price_asof"] for x in items if x["price_asof"]), default=None)
|
||||
return {
|
||||
"items": items,
|
||||
"summary": {
|
||||
"count": len(items),
|
||||
"realized_pnl": sum(x["realized_pnl"] or 0.0 for x in items),
|
||||
"since_sell_up": sum(1 for x in items if (x["since_sell_pct"] or 0) > 0),
|
||||
"since_sell_down": sum(1 for x in items
|
||||
if x["since_sell_pct"] is not None
|
||||
and x["since_sell_pct"] <= 0),
|
||||
"asof": asof,
|
||||
},
|
||||
"asof": asof or last_day,
|
||||
}
|
||||
|
||||
|
||||
def _price_panel(symbol: str, start: date, end: date, cfg: Any) -> pd.DataFrame:
|
||||
"""不复权收盘价 + PE/PB（同一张 daily_basic，一次查询）。
|
||||
|
||||
@@ -346,6 +490,46 @@ def _dividend_yield_series(
|
||||
return out.astype(float)
|
||||
|
||||
|
||||
def _to_date(x: Any) -> date | None:
|
||||
"""DB 取出的日期/时间戳 → ``date``；空值或 NaN 返回 None。"""
|
||||
if x is None:
|
||||
return None
|
||||
if isinstance(x, float) and x != x:
|
||||
return None
|
||||
if isinstance(x, date) and not isinstance(x, datetime):
|
||||
return x
|
||||
try:
|
||||
return pd.Timestamp(x).date()
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _parse_date(s: str | None, field: str) -> date | None:
|
||||
"""用户传入的日期参数；格式不对要报可读的 400，而不是 500。"""
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return pd.Timestamp(str(s)).date()
|
||||
except (TypeError, ValueError):
|
||||
raise HdivError(f"{field} 不是合法日期：{s}（应为 YYYY-MM-DD）") from None
|
||||
|
||||
|
||||
def _lookback_years(run_row: pd.DataFrame) -> int:
|
||||
"""买入判据的回看年数 = 股息率滚动分位的窗口长度（config: percentile_reference）。
|
||||
|
||||
至少要 ``MIN_LOOKBACK_YEARS`` 年：判据数据本身就在成交之前，
|
||||
图上不带上它就没法回答「当时凭什么买」。
|
||||
"""
|
||||
years = MIN_LOOKBACK_YEARS
|
||||
try:
|
||||
cfg_json = run_row["backtest_config_json"].iloc[0]
|
||||
pcfg = (json.loads(cfg_json) if cfg_json else {}).get("percentile_reference") or {}
|
||||
years = max(MIN_LOOKBACK_YEARS, int(pcfg.get("lookback_years") or 0))
|
||||
except (KeyError, TypeError, ValueError, json.JSONDecodeError):
|
||||
pass
|
||||
return years
|
||||
|
||||
|
||||
def _downsample(n: int, target: int) -> np.ndarray:
|
||||
"""等间隔取索引，保留首尾。仅用于画图，买卖点不降采样。"""
|
||||
if n <= target:
|
||||
@@ -385,8 +569,14 @@ def stock_detail(
|
||||
"FROM hd_backtest_position WHERE run_id = :r AND symbol = :s",
|
||||
{"r": run_id, "s": symbol}, cfg=cfg,
|
||||
)
|
||||
first_trade = db.read_sql(
|
||||
"SELECT MIN(execution_date) AS a FROM hd_backtest_trade "
|
||||
"WHERE run_id = :r AND symbol = :s",
|
||||
{"r": run_id, "s": symbol}, cfg=cfg,
|
||||
)
|
||||
run = db.read_sql(
|
||||
"SELECT start_date, end_date FROM hd_backtest_run WHERE run_id = :r",
|
||||
"SELECT start_date, end_date, backtest_config_json "
|
||||
"FROM hd_backtest_run WHERE run_id = :r",
|
||||
{"r": run_id}, cfg=cfg,
|
||||
)
|
||||
if run.empty:
|
||||
@@ -394,16 +584,36 @@ def stock_detail(
|
||||
run_a, run_b = run["start_date"].iloc[0], run["end_date"].iloc[0]
|
||||
|
||||
has_hold = not hold.empty and hold["n"].iloc[0]
|
||||
d_a = pd.to_datetime(start).date() if start else (
|
||||
hold["a"].iloc[0] if has_hold else run_a)
|
||||
d_b = pd.to_datetime(end).date() if end else (
|
||||
hold["b"].iloc[0] if has_hold else run_b)
|
||||
|
||||
# 该股行情能覆盖到的范围（daily_basic 是本图唯一价格源）
|
||||
avail = db.read_sql(
|
||||
"SELECT MIN(trade_date) AS a, MAX(trade_date) AS b "
|
||||
"FROM daily_basic WHERE symbol = :s",
|
||||
{"s": symbol}, cfg=cfg,
|
||||
)
|
||||
avail_a = _to_date(avail["a"].iloc[0]) if not avail.empty else None
|
||||
avail_b = _to_date(avail["b"].iloc[0]) if not avail.empty else None
|
||||
|
||||
# 缺省区间：起点 = **首笔成交往前留够判据回看年数**（无成交则从回测起点往前留），
|
||||
# 终点 = 行情最新日期 —— 卖出当天之后曲线就断，等于把「卖飞了没有」这个问题
|
||||
# 从图上抹掉；而只画持仓期又把「当时凭什么买」的判据数据裁掉了。
|
||||
anchor = (_to_date(first_trade["a"].iloc[0]) if not first_trade.empty else None) \
|
||||
or (_to_date(hold["a"].iloc[0]) if has_hold else None) \
|
||||
or _to_date(run_a)
|
||||
def_a = (pd.Timestamp(anchor) - pd.DateOffset(years=_lookback_years(run))).date()
|
||||
if avail_a and def_a < avail_a: # 别超出该股行情，否则输入框会给出选不到的日期
|
||||
def_a = avail_a
|
||||
def_b = avail_b or _to_date(run_b)
|
||||
d_a = _parse_date(start, "start") or def_a
|
||||
d_b = _parse_date(end, "end") or def_b
|
||||
if d_a > d_b:
|
||||
raise HdivError(f"开始日期晚于结束日期：{d_a} > {d_b}")
|
||||
|
||||
panel = _price_panel(symbol, d_a, d_b, cfg)
|
||||
if panel.empty:
|
||||
span = f"{avail_a} ~ {avail_b}" if avail_a else "无"
|
||||
raise HdivError(
|
||||
f"{symbol} 在 {d_a} ~ {d_b} 没有行情数据。"
|
||||
f"该股行情覆盖见 stock_daily/daily_basic。"
|
||||
f"{symbol} 在 {d_a} ~ {d_b} 没有行情数据；该股行情覆盖 {span}。"
|
||||
)
|
||||
|
||||
dates = panel["trade_date"]
|
||||
@@ -463,6 +673,14 @@ def stock_detail(
|
||||
})
|
||||
|
||||
idx = _downsample(len(dates), MAX_POINTS)
|
||||
# 降采样必须保留成交日：前端是按日期把买卖点对到横轴上的，
|
||||
# 漏掉那一天，这笔成交就会从图上凭空消失（还会被误报成「不在所选区间内」）。
|
||||
# 实测：区间放宽到 5 年判据 + 至今之后，13 只降采样股票里有 7 只会丢成交日。
|
||||
keep = {_v(dates.iloc[i]): i for i in range(len(dates))}
|
||||
hits = sorted(keep[t["execution_date"]] for t in trades
|
||||
if t["execution_date"] in keep)
|
||||
if hits:
|
||||
idx = np.unique(np.concatenate([idx, np.asarray(hits, dtype=idx.dtype)]))
|
||||
dates_out = [_v(dates.iloc[i]) for i in idx]
|
||||
series_out = {k: [v[i] for i in idx] for k, v in series_out.items()}
|
||||
|
||||
@@ -476,6 +694,11 @@ def stock_detail(
|
||||
"info": {k: _v(v) for k, v in info.iloc[0].items()},
|
||||
"range": {"start": _v(dates.iloc[0]), "end": _v(dates.iloc[-1]),
|
||||
"requested_start": d_a.isoformat(), "requested_end": d_b.isoformat(),
|
||||
# 供前端日期选择器使用：缺省区间用于「重置」，
|
||||
# available_* 是该股行情边界，用作输入框的 min/max
|
||||
"default_start": def_a.isoformat(), "default_end": def_b.isoformat(),
|
||||
"available_start": avail_a.isoformat() if avail_a else None,
|
||||
"available_end": avail_b.isoformat() if avail_b else None,
|
||||
"points": len(dates), "downsampled": len(idx) < len(dates)},
|
||||
"available_series": list(SERIES_KEYS),
|
||||
"series": series_out,
|
||||
|
||||
@@ -246,12 +246,26 @@ def _run_stock_detail(run_id: str, symbol: str, q: dict[str, list[str]],
|
||||
)
|
||||
|
||||
|
||||
@route("GET", r"/api/backtests/(?P<run_id>[\w-]+)/closed-positions")
|
||||
def _closed_positions(run_id: str, **_: Any) -> dict[str, Any]:
|
||||
"""已清仓（期末不再持有）的个股清单，含清仓后至今涨跌。"""
|
||||
return analysis.closed_positions(run_id)
|
||||
|
||||
|
||||
@route("GET", r"/api/backtests/(?P<run_id>[\w-]+)/signals")
|
||||
def _backtest_signals(run_id: str, q: dict[str, list[str]], **_: Any) -> dict[str, Any]:
|
||||
return {"items": service.list_backtest_signals(
|
||||
run_id, only_skipped=_one(q, "only_skipped") != "0")}
|
||||
|
||||
|
||||
@route("GET", r"/api/backtests/(?P<run_id>[\w-]+)/daily-universe")
|
||||
def _backtest_daily_universe(
|
||||
run_id: str, q: dict[str, list[str]], **_: Any
|
||||
) -> dict[str, Any]:
|
||||
"""每日动态股票池（``--mode daily``）：时间线或某日成员明细。"""
|
||||
return service.get_daily_universe(run_id, trade_date=_one(q, "date") or None)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 请求辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
+88
-1
@@ -150,10 +150,38 @@ def describe_strategy(cfg: dict[str, Any]) -> dict[str, Any]:
|
||||
"status": st.get("status"),
|
||||
"description": st.get("description", "").strip(),
|
||||
"conditions": lines,
|
||||
# 个股画像闸门：触发买入后按**当日可见数据**重算画像再筛一遍。
|
||||
# 只回结构化规则，展示名/单位由前端既有的 LABEL/UNIT 表渲染
|
||||
# （profile.builder.METRIC_META 少 14 个指标，前端那张表反而更全）。
|
||||
"profile_gate": _profile_gate(entry.get("profile_gate")),
|
||||
}
|
||||
except Exception as exc: # 不因说明生成失败而让接口 500
|
||||
return {"id": None, "name": None, "version": None, "status": None,
|
||||
"description": f"（条件说明生成失败：{exc}）", "conditions": []}
|
||||
"description": f"（条件说明生成失败：{exc}）", "conditions": [],
|
||||
"profile_gate": None}
|
||||
|
||||
|
||||
def _profile_gate(gate: Any) -> dict[str, Any] | None:
|
||||
"""把 entry.profile_gate 规整成前端可直接渲染的结构。"""
|
||||
if not isinstance(gate, dict):
|
||||
return None
|
||||
rules = []
|
||||
for r in gate.get("rules") or []:
|
||||
if not isinstance(r, dict) or not r.get("metric"):
|
||||
continue
|
||||
rules.append({
|
||||
"metric": r.get("metric"),
|
||||
"stat": r.get("stat") or "current_value",
|
||||
"op": r.get("op"),
|
||||
"value": _num(r.get("value")),
|
||||
})
|
||||
return {
|
||||
"enabled": bool(gate.get("enabled")),
|
||||
"window_years": _num(gate.get("window_years")),
|
||||
"on_unverifiable": gate.get("on_unverifiable"),
|
||||
"min_window_coverage": _num(gate.get("min_window_coverage")),
|
||||
"rules": rules,
|
||||
}
|
||||
|
||||
|
||||
def _yi(v: Any) -> str:
|
||||
@@ -182,6 +210,11 @@ def summary() -> dict[str, Any]:
|
||||
WHERE deleted_at IS NULL AND mode = 'single') AS backtests,
|
||||
(SELECT COUNT(*) FROM hd_backtest_run
|
||||
WHERE deleted_at IS NULL AND mode = 'single' AND archived_at IS NULL) AS backtests_active,
|
||||
(SELECT COUNT(*) FROM hd_backtest_run
|
||||
WHERE deleted_at IS NULL AND mode = 'daily') AS daily_backtests,
|
||||
(SELECT COUNT(*) FROM hd_backtest_run
|
||||
WHERE deleted_at IS NULL AND mode = 'daily' AND archived_at IS NULL)
|
||||
AS daily_backtests_active,
|
||||
(SELECT COUNT(*) FROM hd_walkforward_run) AS walkforwards,
|
||||
(SELECT COUNT(*) FROM hd_sensitivity_run) AS sensitivities,
|
||||
(SELECT COUNT(*) FROM hd_profile_run) AS profiles,
|
||||
@@ -885,6 +918,57 @@ def list_backtest_signals(run_id: str, *, only_skipped: bool = True) -> list[dic
|
||||
return out
|
||||
|
||||
|
||||
def get_daily_universe(run_id: str, *, trade_date: str | None = None,
|
||||
limit: int = 5000) -> dict[str, Any]:
|
||||
"""每日动态股票池（``--mode daily`` 的逐日选股留痕）。
|
||||
|
||||
两个形态：
|
||||
- 不给 ``trade_date``：返回**时间线**（每个决策日的成员数），用于看池子如何变化；
|
||||
- 给 ``trade_date``：返回该日的成员明细（含入选时的因子快照）。
|
||||
"""
|
||||
cfg = load_config("datasource")
|
||||
if not db.table_exists("hd_daily_universe", cfg):
|
||||
return {"available": False, "timeline": [], "members": [], "trade_date": None}
|
||||
if trade_date:
|
||||
df = db.read_sql(
|
||||
"SELECT trade_date, symbol, name, industry, dividend_yield, total_mv, "
|
||||
" roe_avg, listed_count, candidate_count, values_json "
|
||||
"FROM hd_daily_universe "
|
||||
"WHERE run_id = :r AND trade_date = :d ORDER BY dividend_yield DESC "
|
||||
"LIMIT :lim",
|
||||
{"r": run_id, "d": trade_date, "lim": int(limit)}, cfg=cfg,
|
||||
)
|
||||
members = []
|
||||
for _, row in df.iterrows():
|
||||
m = _rec(row)
|
||||
m["values"] = _json_field(m.pop("values_json", None)) or {}
|
||||
members.append(m)
|
||||
return {"available": True, "trade_date": trade_date,
|
||||
"members": members, "timeline": []}
|
||||
|
||||
df = db.read_sql(
|
||||
"SELECT trade_date, COUNT(*) AS member_count, "
|
||||
" MAX(candidate_count) AS candidate_count, "
|
||||
" MAX(listed_count) AS listed_count "
|
||||
"FROM hd_daily_universe WHERE run_id = :r "
|
||||
"GROUP BY trade_date ORDER BY trade_date",
|
||||
{"r": run_id}, cfg=cfg,
|
||||
)
|
||||
timeline = [
|
||||
{
|
||||
"trade_date": str(_rec(r).get("trade_date")),
|
||||
"member_count": int(r["member_count"]),
|
||||
"candidate_count": int(r["candidate_count"])
|
||||
if pd.notna(r["candidate_count"]) else None,
|
||||
"listed_count": int(r["listed_count"])
|
||||
if pd.notna(r["listed_count"]) else None,
|
||||
}
|
||||
for _, r in df.iterrows()
|
||||
]
|
||||
return {"available": True, "timeline": timeline, "members": [],
|
||||
"trade_date": timeline[-1]["trade_date"] if timeline else None}
|
||||
|
||||
|
||||
_SKIP_LABELS = {
|
||||
"LIMIT_UP": "开盘涨停，无法买入",
|
||||
"LIMIT_DOWN": "开盘跌停，无法卖出",
|
||||
@@ -898,6 +982,9 @@ _SKIP_LABELS = {
|
||||
# 实时画像闸门剔除（信号类型 REJECT）：不是撮合失败，而是「按当日可见
|
||||
# 数据重算画像后判定不值得买」。详情在 reason_json.profile_gate.checks。
|
||||
"PROFILE_GATE": "实时画像未通过，主动放弃买入",
|
||||
# 每日动态股票池（--mode daily）：持仓已掉出当日股票池，
|
||||
# 按 pool_exit_action=hold 只停止加仓，不清仓。
|
||||
"OUT_OF_UNIVERSE": "已掉出当日动态股票池，停止加仓（不清仓）",
|
||||
}
|
||||
|
||||
|
||||
|
||||
+38
-1
@@ -168,6 +168,40 @@ def move_reports_to_subdir(*, verbose: bool = True) -> int:
|
||||
return n
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 权限：nginx worker 不是文件属主
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: 站点文件 / 目录的发布权限。
|
||||
#:
|
||||
#: nginx 的 master 以 root 运行、worker 以 nobody 运行（Homebrew 默认），所以
|
||||
#: **worker 不是站点文件的属主**：文件只要不是「所有人可读」就 403。
|
||||
#: 而 ``shutil.copy2`` 会连权限一起复制，用 umask 077 的编辑器/工具存下来的
|
||||
#: 600 文件会一路带进 output/ —— 症状是「HTML 打得开、CSS/JS 403、页面裸奔」，
|
||||
#: 而 nginx 错误日志里只写 ``failed (13: Permission denied)``。
|
||||
#: 这里在发布时统一收敛权限，别让单个文件的 umask 决定线上是否可用。
|
||||
SITE_FILE_MODE = 0o644
|
||||
SITE_DIR_MODE = 0o755
|
||||
|
||||
|
||||
def ensure_readable(root: Path, *, verbose: bool = False) -> int:
|
||||
"""把站点目录树收敛为「目录 755 / 文件 644」，返回被修正的条目数。"""
|
||||
if not root.is_dir():
|
||||
return 0
|
||||
fixed = 0
|
||||
for p in [root, *sorted(root.rglob("*"))]:
|
||||
want = SITE_DIR_MODE if p.is_dir() else SITE_FILE_MODE
|
||||
try:
|
||||
if (p.stat().st_mode & 0o777) != want:
|
||||
p.chmod(want)
|
||||
fixed += 1
|
||||
except OSError: # pragma: no cover - 权限不足/平台不支持
|
||||
pass
|
||||
if fixed and verbose:
|
||||
print(f" 已修正 {fixed} 个发布产物的权限（nginx worker 为 nobody，需要 world-readable）")
|
||||
return fixed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 前端同步
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -206,9 +240,12 @@ def sync_frontend(*, verbose: bool = True) -> dict[str, Any]:
|
||||
shutil.copy2(ec_src, assets_dst / "echarts.min.js")
|
||||
copied.append(str((assets_dst / "echarts.min.js").relative_to(project_root())))
|
||||
|
||||
# 整棵站点树（含 reports/ 与 archive/）统一权限，避免 copy2 把 600 带进来
|
||||
fixed = ensure_readable(out, verbose=verbose)
|
||||
|
||||
if verbose:
|
||||
print(f" 已同步前端 {len(copied)} 个文件到 output/")
|
||||
return {"copied": copied}
|
||||
return {"copied": copied, "perm_fixed": fixed}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user