修复:量价单位 / 未来函数守卫 / 实时画像闸门;行情回补到 2005;手册补全流程

本轮会话的三项正确性改造(均为「不报错、只让结果静默错」的类型):

1) 修复 stock_daily 量价单位前后不一致
   - 现象:2015-2019 存 Tushare 原始单位(手/千元),2020 起存(股/元),2019 同日混合;
     而流动性阈值按「元」配置 → 早年门槛实际是「日均成交额 ≥ 200 亿元」,
     把 2015-2019 的股票池整体清空(实测 2016/2017/2018 各选出 0 只)。
   - 修复:写入端 sync/price.py 统一换算;读取端 units.normalize_ohlcv_units
     按行判定并幂等换算(price_history / avg_amount 都走它);
     审计新增 UNIT-OHLCV 防回归。
   - 效果:2016/2017/2018 的股票池变为 7/11/13 只。

2) 未来函数守卫(单次回测)
   - 股票池自带 asof:若晚于回测起点即**拒绝执行**(原先静默冻结套用),
     与 walk-forward 已有的拒绝理由一致;确需复现加 --allow-lookahead-universe,
     偏差写入 unimplemented_json。

3) 新增实时(PIT)个股画像闸门
   - profile/pit.py:每个决策日按当时可见数据重算过去 5 年画像,
     惰性(仅买入条件已触发的标的)、面板按 asof 缓存、
     规则不含财务指标时不查财报表;被剔除时产出 REJECT + 逐规则留痕。
   - 指标定义复用 ProfileBuilder._profile_one(与批量画像逐值等价的回归测试)。
   - profile/coverage.py:窗口覆盖率(按交易日历的真实开市天数),
     策略新增 entry.profile_gate.min_window_coverage(默认 0,不改变既有行为)。
   - core/metrics.py:闸门可用指标的唯一定义(配置期即校验,避免写错指标名静默失效)。

4) 行情回补到 2005(使 5/8/10 年窗口真正完整)
   - stock_daily / adjust_factor / daily_basic 补到 2005-01-04;
     hd_suspend / hd_limit 补到 2010-01-04。
   - 5 年窗口覆盖率:2018-05-18 由 67.0% → 99.1%,2016-12-30 由 39.8% → 99.0%;
     残差经逐日与 hd_suspend 交叉核实为真实停牌(16/16 命中)。
   - 审计 G2/G3 与断点续传原先用固定阈值(2000 / 1500 只),
     会把 2005-2009 的正常数据误判为异常 —— 改为按「当年应有上市股票数」成比例判定。
   - 节流修正:daily/adj_factor/daily_basic 限频 480 → 170(实测该 token 约 196/min 即被拒)。

5) 自我声明如实化
   - 原先「约束未生效」由「过滤后集合为空」判定,会把「这批股票恰好没停牌」
     误报成「hd_suspend 无数据」;改为按表级判定。
   - 补齐此前静默的「配置承诺但未实现」项:suspended_rule/limit_up_down_rule 的 defer、
     cash_mode=reinvest/reinvest_rule、handle_rights_issue、signal_to_execution、
     max_volume_pct、liquidity_limit_pct_adv —— 全部写入 unimplemented_json。

6) 手册:新增 §0「全流程操作(选股 → 画像 → 回测)」置于最前
   - 逐步说明「命令做了什么、数据从哪来、落了哪些库、有哪些坑」;
     含实时画像闸门 9 问 9 答、未来函数守卫表、成交与成本口径、验证 SQL。
   - 修正旧 §2.4 漏传 --universe-run(选了池子却没用于回测);
     修正两处声称「停牌顺延」「分红再投资」已实现的相反表述。

测试:403 项全部通过(含新增 test_units.py、test_profile_pit.py、
未实现声明诚实性测试、行序无关性回归测试)。

注意:本提交中 docs/*、README.md、src/hdiv/web/service.py 除本轮修改外,
也含此前遗留的未提交改动(无法按文件切分)。
This commit is contained in:
2026-10-04 12:47:17 +08:00
parent fb6608193b
commit 14ec0c6c86
25 changed files with 4543 additions and 209 deletions
+233
View File
@@ -458,3 +458,236 @@ def test_metrics_do_not_invent_values() -> None:
})
m = compute_metrics(eq, [], load_config("backtest"), "r3")
assert m["sharpe"] is None, "1 个观测算不出波动率,Sharpe 必须是 None"
# ---------------------------------------------------------------------------
# 分位参照的最小样本量保护
# ---------------------------------------------------------------------------
def test_min_observations_config_exists_with_sane_default() -> None:
"""回归:分位参考必须设最小样本量,否则退化分布会伪造 100% 分位。
分位 = 「≤当前值的观测占比」。窗口里只有 1 个观测且恰好等于当前值时
占比 100%,击穿任何买入阈值 —— 实测 2015-01-06(行情数据首日)
8 只股票因此被「100% 分位」买入。
"""
from hdiv.core.config import load_config
ref = load_config("backtest").percentile_reference
assert hasattr(ref, "min_observations"), "缺少 min_observations 配置"
assert ref.min_observations >= 60, \
f"最小样本量过低({ref.min_observations}),至少应约一个季度"
def test_engine_guards_against_insufficient_reference_sample() -> None:
"""引擎必须在样本不足时跳过信号,而不是照常算分位。"""
import inspect
from hdiv.backtest import engine as eng
src = inspect.getsource(eng)
assert "min_observations" in src, "引擎未使用 min_observations"
# 保护必须在计算 pct 之前,且以 continue 跳过该股当日
i_guard = src.find("ref_ser.size < self.bt_cfg.percentile_reference.min_observations")
i_pct = src.find("pct = float((ref_ser <= current)")
assert i_guard != -1, "未找到最小样本量判断"
assert i_pct != -1 and i_guard < i_pct, "样本量判断必须早于分位计算"
assert "continue" in src[i_guard:i_pct], "样本不足应跳过(continue)而非降级计算"
@pytest.mark.db
def test_recent_backtests_have_no_weak_sample_trades() -> None:
"""按新配置跑出的回测不应存在弱样本成交(样本 < min_observations)。"""
from hdiv.core.config import load_config
from hdiv.data import db
db.load_dotenv_once()
cfg = load_config("datasource")
min_obs = load_config("backtest").percentile_reference.min_observations
df = db.read_sql(
"SELECT run_id, COUNT(*) AS n FROM hd_backtest_trade "
"WHERE JSON_EXTRACT(reason_json, '$.observation_count') IS NOT NULL "
" AND JSON_EXTRACT(reason_json, '$.observation_count') < :m "
" AND run_id IN (SELECT run_id FROM hd_backtest_run "
" WHERE created_at > '2026-10-03 14:30:00') "
"GROUP BY run_id",
{"m": min_obs}, cfg=cfg,
)
assert df.empty, (
f"存在弱样本成交的回测(应为 0):"
f"{[(r['run_id'][:10], int(r['n'])) for _, r in df.iterrows()]}"
)
# ---------------------------------------------------------------------------
# 实时画像闸门(entry.profile_gate)
# ---------------------------------------------------------------------------
def _trigger_ctx(sym: str = "000001.SZ", n: int = 280) -> dict:
"""构造一个「股息率处于历史最高分位」的最小上下文。
每股分红恒定 1 元、股价从 20 元跌到 10 元 → 股息率从 5% 升到 10%,
当前值即窗口最大值,分位 = 100% ≥ P75,必然触发买入条件。
``n`` 必须 **同时** 满足两个约束:
- ≥ ``backtest.yml: percentile_reference.min_observations``(250,否则引擎跳过);
- ≈ ≤ 410 个自然日(TTM 分红窗口 365 + 宽限 45),否则序列尾部 TTM 分红归零、
股息率变成 0、分位塌到 30% 以下,触发不了买入。280 个交易日 ≈ 392 天,两者都满足。
"""
days = pd.bdate_range("2015-01-05", periods=n)
close = np.concatenate([np.full(n - 50, 20.0), np.linspace(20.0, 10.0, 50)])
px = pd.DataFrame({"open": close, "close": close}, index=days)
ev = pd.DataFrame([{
"ex_date": days[0], "imp_ann_date": days[0], "cash_div_tax": 1.0,
}])
return {"px_by_sym": {sym: px}, "events": {sym: ev}, "_last_day": days[-1].date()}
def _pass_gate(*_a, **_k) -> dict:
return {"verdict": "PASS", "checks": [], "failed": [], "unverifiable": []}
def _reject_gate(*_a, **_k) -> dict:
return {
"verdict": "REJECT",
"checks": [{"metric": "payout_ratio", "stat": "current_value", "op": "<=",
"threshold": 1.0, "actual": 1.4, "status": "OK",
"window_years": 0, "passed": False}],
"failed": ["payout_ratio.current_value<=1"],
"unverifiable": [],
}
def test_gate_rejects_buy(engine, monkeypatch) -> None:
"""闸门不通过时必须改为 REJECT,且不产生 BUY。"""
ctx = _trigger_ctx()
day = ctx["_last_day"]
monkeypatch.setattr(engine, "_gate", _reject_gate)
sigs = engine._evaluate(day, 1e6, {}, {"000001.SZ"}, ctx)
kinds = [s.kind for s in sigs]
assert "BUY" not in kinds, f"闸门未拦住买入:{kinds}"
assert kinds == ["REJECT"]
rej = sigs[0]
assert rej.reason["skip_reason"] == "PROFILE_GATE"
assert rej.reason["executed"] is False
# 「为什么不买」必须可追溯:逐条规则的实际值与阈值都要留下
chk = rej.reason["profile_checks"]["payout_ratio"]
assert chk["actual"] == pytest.approx(1.4) and chk["threshold"] == 1.0
assert chk["passed"] is False
def test_gate_pass_keeps_buy(engine, monkeypatch) -> None:
"""闸门通过时买入必须照常发生(不能误杀)。"""
ctx = _trigger_ctx()
monkeypatch.setattr(engine, "_gate", _pass_gate)
sigs = engine._evaluate(ctx["_last_day"], 1e6, {}, {"000001.SZ"}, ctx)
kinds = [s.kind for s in sigs]
assert kinds == ["BUY"], kinds
def test_gate_reject_leaves_existing_position_untouched(engine, monkeypatch) -> None:
"""闸门语义是「不值得买」,不是「该卖」—— 被拒时不得动已有仓位。"""
ctx = _trigger_ctx()
pos = {"000001.SZ": Position(symbol="000001.SZ", quantity=1000.0, avg_cost=15.0,
first_buy_date=date(2015, 1, 5), last_buy_date=date(2015, 1, 5),
cost_basis=15000.0)}
monkeypatch.setattr(engine, "_gate", _reject_gate)
sigs = engine._evaluate(ctx["_last_day"], 1e6, pos, {"000001.SZ"}, ctx)
kinds = [s.kind for s in sigs]
assert "TRIM" not in kinds and "SELL" not in kinds and "ADD" not in kinds, kinds
assert kinds == ["REJECT"], "高仓位侧被拒时应只留痕,不调仓"
def test_gate_handles_unverifiable_conservatively(engine, monkeypatch) -> None:
"""无法验证(数据缺失/样本不足)时按配置保守处理,且理由要能区分。"""
def _unver(*_a, **_k):
return {"verdict": "REJECT", "checks": [
{"metric": "roe_avg", "stat": "current_value", "op": ">=", "threshold": 0.08,
"actual": None, "status": "INSUFFICIENT", "window_years": 0, "passed": None}],
"failed": [], "unverifiable": ["roe_avg.current_value"]}
ctx = _trigger_ctx()
monkeypatch.setattr(engine, "_gate", _unver)
sigs = engine._evaluate(ctx["_last_day"], 1e6, {}, {"000001.SZ"}, ctx)
assert [s.kind for s in sigs] == ["REJECT"]
assert "无法验证" in sigs[0].reason["rule"]
assert "样本不足" in sigs[0].reason["reason_cn"]
def test_gate_disabled_returns_none(engine) -> None:
"""闸门关闭时 _gate 必须返回 None —— 调用方不产生任何额外行为。"""
from hdiv.backtest.engine import BacktestEngine
eng = BacktestEngine.from_strategy("config/strategy/high_dividend_v1.yml")
eng.gate_cfg.enabled = False
eng.pit = object() # 即使被注入也不得被使用
assert eng._gate("000001.SZ", date(2018, 5, 18)) is None
def test_gate_enabled_but_uninitialized_fails_loudly() -> None:
"""启用但未初始化必须报错,不得静默放行(否则等于风控悄悄失效)。"""
from hdiv.backtest.engine import BacktestEngine
from hdiv.core.errors import HdivError
eng = BacktestEngine.from_strategy("config/strategy/high_dividend_v1.yml")
eng.gate_cfg.enabled = True
eng.pit = None
with pytest.raises(HdivError) as ei:
eng._gate("000001.SZ", date(2018, 5, 18))
assert "_prepare" in str(ei.value)
@pytest.mark.db
def test_gate_enabled_backtest_records_rejections() -> None:
"""端到端:启用闸门的短区间回测必须留下可追溯的 REJECT 记录且资金对账平衡。"""
from hdiv.backtest.engine import BacktestEngine
try:
eng = BacktestEngine.from_strategy("config/strategy/high_dividend_v1.yml")
assert eng.gate_cfg.enabled is True, "默认策略应已启用实时画像闸门"
res = eng.run(start=date(2016, 1, 1), end=date(2016, 12, 31),
persist=False, verbose=False)
except Exception as exc: # 数据不可用
pytest.skip(f"数据库不可用:{exc}")
assert res["reconciliation"]["balanced"] is True
stats = res["profile_gate"]
assert stats["asof_contexts"] > 0, "应产生实时画像时点"
rejects = [s for s in res["signals"] if s.kind == "REJECT"]
assert rejects, "该区间应存在被画像剔除的买入信号"
for s in rejects:
assert s.reason["skip_reason"] == "PROFILE_GATE"
gate = s.reason["profile_gate"]
assert gate["verdict"] in {"REJECT", "UNVERIFIABLE"}
assert gate["checks"], "每条 REJECT 都必须带逐规则留痕"
assert gate["failed"] or gate["unverifiable"]
for c in gate["checks"]:
assert set(c) >= {"metric", "op", "threshold", "actual", "status", "passed"}
@pytest.mark.db
def test_unimplemented_declarations_are_honest() -> None:
"""自我声明必须两头都准:既不能漏报「配置写了但没实现」,
也不能把「这批股票恰好没停牌」误报成「数据缺失」。
背景:2026-10-04 之前,``unimplemented`` 用「过滤后集合为空」判定约束失效,
于是 2026-08~09(hd_suspend 明明覆盖到 2026-09-30,只是这批股票没停牌)
被声明成「hd_suspend 无数据」—— 把自己的建模正常状态说成数据缺陷。
"""
from hdiv.backtest.engine import BacktestEngine
try:
eng = BacktestEngine.from_strategy("config/strategy/high_dividend_v1.yml")
res = eng.run(start=date(2026, 8, 3), end=date(2026, 9, 30),
persist=False, verbose=False)
except Exception as exc:
pytest.skip(f"数据库不可用:{exc}")
decl = " ".join(res["unimplemented"])
# ① 不得把「无停牌」误报成「无数据」(约束表在 2010 起有数据)
assert "无数据" not in decl, f"误报数据缺失:{decl}"
# ② 必须如实声明「配置承诺但未实现」的项
for must in ("defer", "分红再投资", "配股", "成交量占比"):
assert must in decl, f"漏报未实现项 {must}:{decl}"