功能:每日动态股票池回测(--mode daily)+ 每日增量同步 + PIT 批量取数层
说明:本提交是工作区中此前的未提交工作(在 14ec0c6 之后产生),**非本次会话所写**,
按用户要求整理并推送。已做安全检查(无明文凭据、无大文件、.env/logs/output 仍被忽略),
并完成可执行范围内的测试验证(见「测试」一节)。
## 新增能力
1) `hdiv backtest --mode daily --start <日期>`
- src/hdiv/backtest/daily.py:两趟式(先逐日选股,再复用既有引擎模拟)
- 每个交易日按当日可见数据重建股票池(PIT),每个交易日判断买卖点
- `pool_exit_action`:hold(只减不加、不因掉出池子而清仓)/ sell(掉出即清仓)
- `profile_on_trade`:买卖决策发生时计算并留痕个股画像,**不区分是否在当日池内**
(卖出/减仓同样留痕,否则「为什么卖」缺证据)
- 与 walkforward 的分工:daily 是一条连续路径的推演,不是过拟合检验;
因此不使用训练段、不冻结分布,阈值口径一律 rolling
- 拒绝 `--universe-run`(daily 的定义就是逐日重筛,冻结池与之矛盾)
2) PIT 批量取数层 src/hdiv/universe/pit.py
- PitRepo 继承 Repo,**只重写取数**(按区块批量预载 + 逐日内存切片),
派生逻辑(最新一期财报合并、单位归一化、支付率口径等)一行不重写
—— 以保证与逐日单点查询**结果等价**
- 候选集预剪枝:用「不可能通过」的边界条件提前排除,文档论证为精确等价而非近似
- src/hdiv/universe/daily.py:每日动态筛选器(仍然调用既有 selector 与四个 Filter)
3) 每日增量同步 `hdiv sync daily`
- src/hdiv/data/sync/daily.py:只抓「库里还没有的那几天」,
按「当日股票数 ≥ 当年规模阈值」判定缺口,不重拉历史、不覆盖既有行;
支持 `--dry-run` 先看待抓清单
- deploy/daily-sync.sh、deploy/install-sync-schedule.sh、
deploy/com.hddiv.sync.plist.example(launchd 每天 17:00)
- 新表 hd_daily_universe(逐日入选成员留痕)+ sql/hd_daily_universe.sql + schema.py
(该表已存在于库中,`ddl plan` 返回 0 个待执行动作)
4) Web 与文档
- 前端支持 daily 模式记录下钻(web/app.js、web/app.css、web/index.html、
web/favicon.svg)
- README / docs/user-guide.md / docs/implementation-status.md 同步更新:
三种回测模式的取舍、daily 的成本说明(6.7 年约 1.5 小时)与调优手段
## 测试
tests/ 共 500 项(新增 tests/test_daily.py 43 项、tests/test_sync_daily.py 36 项)。
已验证通过:
- 排除上述两个新文件的 **421 项:全部通过(pytest 退出码 0)**
- 两个新文件的**非 DB 单元测试 60 项:全部通过**
未能在合理时间内跑完:
- 两个新文件中 **19 项 DB 标记的重型测试**。实测瓶颈是一条**无界全表扫描**:
`SELECT ... FROM hd_cashflow WHERE ann_date <= :asof ORDER BY symbol, end_date, ann_date`
(31 万行,无 symbol/报告期下限)。全量套件跑到 161 项时已耗时 20 分钟、
0 失败,按该速率预计需 3 小时以上,因此改为分档验证。
- 旁证:库中存在 3 次成功的 daily 端到端运行(2026-10-05 10:05 / 10:32 / 11:03,
区间 2024-03-01~03-15),说明该路径可正常完成。
## 已知待改进
- 上述 `hd_cashflow`(及同类「按 ann_date 上界取全历史」)的查询缺
symbol / 报告期下限,是 daily 模式的主要性能瓶颈,建议下一轮优化。
This commit is contained in:
@@ -0,0 +1,223 @@
|
||||
"""股息率(TTM 每股分红)毛刺诊断脚本。
|
||||
|
||||
**用途**:定位「数据毛刺导致回测异常成交」这一类问题。
|
||||
它不改动任何数据,只读库、只打印。
|
||||
|
||||
用法::
|
||||
|
||||
export PYTHONPATH=src
|
||||
.venv/bin/python tools/diag_dividend_artifact.py --symbol 600690.SH \
|
||||
--start 2026-07-01 --end 2026-08-05
|
||||
|
||||
诊断三件事:
|
||||
|
||||
1. **同一除权日的重复分红记录**:``hd_dividend`` 写入侧刻意保留
|
||||
预案/股东大会通过/实施 全量记录(决策 D6),但查询侧把它们当成
|
||||
**多笔独立分红**,于是 ``cash_div_tax`` 被重复累加。
|
||||
2. **TTM 每股分红的时间线**:逐交易日打印去重前 / 去重后的取值,
|
||||
毛刺会表现为「无任何真实现金事件的一天突然跳变」。
|
||||
3. **异常成交反查**:给定回测 run_id,列出每一笔卖出当日 TTM 值的
|
||||
去重前/去重后差异,标出「去重后不再触发卖出」的笔数。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import date, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||
|
||||
from hdiv.data import db # noqa: E402
|
||||
from hdiv.factor.dividend_yield import ( # noqa: E402
|
||||
build_dps_events,
|
||||
ttm_dps_series,
|
||||
ttm_params,
|
||||
)
|
||||
|
||||
|
||||
def _dedup(events: pd.DataFrame) -> pd.DataFrame:
|
||||
"""同一除权日只保留一笔(取金额最大者)。
|
||||
|
||||
这是「写入口径全量保留、查询口径按经济事件聚合」的最小实现。
|
||||
"""
|
||||
if events.empty:
|
||||
return events
|
||||
return (
|
||||
events.sort_values("cash_div_tax")
|
||||
.drop_duplicates("ex_date", keep="last")
|
||||
.reset_index(drop=True)
|
||||
)
|
||||
|
||||
|
||||
def diagnose_symbol(symbol: str, start: date, end: date) -> None:
|
||||
div = db.read_sql(
|
||||
"SELECT symbol, end_date, ann_date, imp_ann_date, div_proc, "
|
||||
" cash_div_tax, ex_date "
|
||||
"FROM hd_dividend WHERE symbol = :s AND cash_div_tax > 0 "
|
||||
" AND ex_date IS NOT NULL AND div_proc = '实施' "
|
||||
"ORDER BY ex_date",
|
||||
{"s": symbol},
|
||||
)
|
||||
events = build_dps_events(div).get(symbol)
|
||||
if events is None or events.empty:
|
||||
print(f"{symbol}: 无已实施现金分红记录")
|
||||
return
|
||||
|
||||
dups = events.groupby("ex_date").size()
|
||||
dups = dups[dups > 1]
|
||||
print(f"=== {symbol} 分红记录 ===")
|
||||
print(f"已实施现金分红行数:{len(events)},唯一除权日:{events['ex_date'].nunique()}")
|
||||
print(f"**同一除权日重复 {len(dups)} 组**(这些金额被重复累加):")
|
||||
for ex, n in dups.items():
|
||||
rows = events[events["ex_date"] == ex]
|
||||
print(
|
||||
f" {ex.date()} x{n} 金额={rows['cash_div_tax'].tolist()} "
|
||||
f"实施日={[str(pd.Timestamp(x).date()) for x in rows['imp_ann_date']]}"
|
||||
)
|
||||
|
||||
px = db.read_sql(
|
||||
"SELECT trade_date, close FROM stock_daily "
|
||||
"WHERE symbol = :s AND trade_date BETWEEN :a AND :b ORDER BY trade_date",
|
||||
{"s": symbol, "a": start, "b": end},
|
||||
)
|
||||
if px.empty:
|
||||
print("区间内无行情")
|
||||
return
|
||||
px["trade_date"] = pd.to_datetime(px["trade_date"])
|
||||
idx = pd.DatetimeIndex(px["trade_date"])
|
||||
w, g, sm = ttm_params()
|
||||
raw = ttm_dps_series(idx, events, ttm_days=w, grace_days=g, smooth_spikes=sm)
|
||||
ded = ttm_dps_series(idx, _dedup(events), ttm_days=w, grace_days=g, smooth_spikes=sm)
|
||||
out = pd.DataFrame(
|
||||
{
|
||||
"trade_date": px["trade_date"].dt.date,
|
||||
"close": px["close"],
|
||||
"ttm_dps(去重前)": raw,
|
||||
"ttm_dps(去重后)": ded,
|
||||
}
|
||||
)
|
||||
out["股息率%(去重前)"] = (out["ttm_dps(去重前)"] / out["close"] * 100).round(3)
|
||||
out["股息率%(去重后)"] = (out["ttm_dps(去重后)"] / out["close"] * 100).round(3)
|
||||
print("\n=== 逐交易日 TTM 每股分红 ===")
|
||||
print(out.to_string(index=False))
|
||||
|
||||
jump = out[pd.Series(raw, index=range(len(out))).diff().abs() > 1e-9]
|
||||
print("\n出现跳变的交易日(应与真实分红除权日一一对应):")
|
||||
print(jump[["trade_date", "ttm_dps(去重前)", "ttm_dps(去重后)"]].to_string(index=False))
|
||||
|
||||
|
||||
def diagnose_run(run_id: str) -> None:
|
||||
"""反查某次回测的每一笔卖出:去重后是否仍会触发。"""
|
||||
from hdiv.data.repo import Repo
|
||||
|
||||
repo = Repo()
|
||||
trades = db.read_sql(
|
||||
"SELECT symbol, signal_date, execution_date, reason_json "
|
||||
"FROM hd_backtest_trade WHERE run_id = :r AND side = 'SELL' "
|
||||
"ORDER BY execution_date",
|
||||
{"r": run_id},
|
||||
)
|
||||
if trades.empty:
|
||||
print("该 run 无卖出成交")
|
||||
return
|
||||
w, g, sm = ttm_params()
|
||||
years = 5
|
||||
rows = []
|
||||
for _, t in trades.iterrows():
|
||||
sym = t["symbol"]
|
||||
d0 = pd.Timestamp(t["signal_date"]).date()
|
||||
div = repo.dividend_records(d0, years_back=years + 3)
|
||||
div = div[div["symbol"] == sym]
|
||||
ev = build_dps_events(div).get(sym)
|
||||
if ev is None or ev.empty:
|
||||
continue
|
||||
px = repo.price_history([sym], date(d0.year - years - 1, 1, 1), d0, adjust="none")
|
||||
px = px.sort_values("trade_date")
|
||||
px["trade_date"] = pd.to_datetime(px["trade_date"])
|
||||
s = px.set_index("trade_date")["close"]
|
||||
i = pd.DatetimeIndex(s.index)
|
||||
ref = (d0 - timedelta(days=int(365.25 * years)), d0)
|
||||
|
||||
def decision(events: pd.DataFrame) -> tuple[float, float, float]:
|
||||
dps = pd.Series(
|
||||
ttm_dps_series(i, events, ttm_days=w, grace_days=g, smooth_spikes=sm),
|
||||
index=i,
|
||||
)
|
||||
y = (dps / s).loc[: pd.Timestamp(d0)]
|
||||
cur = float(y.iloc[-1])
|
||||
rs = y.loc[pd.Timestamp(ref[0]) : pd.Timestamp(ref[1])]
|
||||
pct = float((rs <= cur).sum() / rs.size * 100) if rs.size else float("nan")
|
||||
return cur, pct, float(rs.quantile(0.25)) if rs.size else float("nan")
|
||||
|
||||
for label, evx in (("去重前", ev), ("去重后", _dedup(ev))):
|
||||
cur, pct, p25 = decision(evx)
|
||||
rows.append(
|
||||
{
|
||||
"symbol": sym,
|
||||
"signal_date": d0,
|
||||
"口径": label,
|
||||
"股息率%": round(cur * 100, 3),
|
||||
"分位%": round(pct, 2),
|
||||
"P25%": round(p25 * 100, 3),
|
||||
"触发卖出(P25)": pct <= 25.0,
|
||||
}
|
||||
)
|
||||
if not rows:
|
||||
return
|
||||
df = pd.DataFrame(rows)
|
||||
print(f"\n=== run {run_id} 卖出决策:去重前 vs 去重后 ===")
|
||||
print(df.pivot_table(
|
||||
index=["symbol", "signal_date"], columns="口径",
|
||||
values=["分位%", "触发卖出(P25)"], aggfunc="first",
|
||||
).to_string())
|
||||
before = df[df["口径"] == "去重前"].set_index(["symbol", "signal_date"])["触发卖出(P25)"]
|
||||
after = df[df["口径"] == "去重后"].set_index(["symbol", "signal_date"])["触发卖出(P25)"]
|
||||
flipped = before[before & ~after.reindex(before.index).fillna(False)]
|
||||
print(f"\n去重后**不再触发**卖出的成交:{len(flipped)} / {len(before)} 笔")
|
||||
for k in flipped.index:
|
||||
print(f" {k[0]} 信号日 {k[1]}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="股息率毛刺诊断")
|
||||
ap.add_argument("--symbol", help="股票代码,如 600690.SH")
|
||||
ap.add_argument("--start", help="起始日 YYYY-MM-DD")
|
||||
ap.add_argument("--end", help="结束日 YYYY-MM-DD")
|
||||
ap.add_argument("--run-id", help="反查某次回测的所有卖出成交")
|
||||
ap.add_argument("--duplicate-survey", action="store_true",
|
||||
help="全库统计:同一除权日重复记录的股票数")
|
||||
args = ap.parse_args()
|
||||
|
||||
db.load_dotenv_once()
|
||||
|
||||
if args.duplicate_survey:
|
||||
df = db.read_sql(
|
||||
"SELECT symbol, COUNT(*) AS rows_, COUNT(DISTINCT ex_date) AS ex_dates "
|
||||
"FROM hd_dividend WHERE cash_div_tax > 0 AND ex_date IS NOT NULL "
|
||||
" AND div_proc = '实施' GROUP BY symbol",
|
||||
{},
|
||||
)
|
||||
bad = df[df["rows_"] > df["ex_dates"]]
|
||||
print(f"有已实施现金分红的股票:{len(df)}")
|
||||
print(f"**存在同一除权日重复记录的股票:{len(bad)}**"
|
||||
f"(多出 {int((bad['rows_'] - bad['ex_dates']).sum())} 行被重复累加)")
|
||||
print(bad.sort_values("rows_", ascending=False).head(20).to_string(index=False))
|
||||
return 0
|
||||
|
||||
if args.run_id:
|
||||
diagnose_run(args.run_id)
|
||||
if args.symbol:
|
||||
start = date.fromisoformat(args.start) if args.start else date(2026, 1, 1)
|
||||
end = date.fromisoformat(args.end) if args.end else date(2026, 12, 31)
|
||||
diagnose_symbol(args.symbol, start, end)
|
||||
if not (args.run_id or args.symbol or args.duplicate_survey):
|
||||
ap.print_help()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user