feat: 股息率案例口径 + 策略库与图表统一 + 回测存档完整化
汇总三轮未提交的开发(每轮均在本机 MariaDB + 真实浏览器上验证):
1) 股息率案例(全市场股息率最高 n 只,默认 20,每 m 月择股)
- 新增日频估值表 daily_basic + 迁移;股息率因子(dv_ratio / dividend_yield / TTM)
- 名称历史表 stock_name_history:剔除 ST 按**择股日当时名称**判定,消除
「曾高股息后 ST」的股息陷阱(实测 3.70pp 偏差)
- 区间择股/调仓双周期(m 择股 / y 调仓)、指数成分与白名单、停牌近似剔除
- 复权因子口径核对(4,164,742 行、缺失 0.0%)、收盘价成交与涨跌停拦单
- 案例实测:2020-01-01~2026-09-04 总收益 +24.86%(年化 3.52%、回撤 -28.58%)
2) 策略库与前端统一
- strategy 表 + CRUD/PUT 原地更新 + `describe_strategy` 按 spec 真实推导
「一句话说明 + 计算公式 + 执行步骤 + 注意事项」(与引擎实执行规则同源)
- 任何出现股票代码处都成对显示名称且可点击进个股页
- 全站图表基座统一 TradingView Lightweight Charts(ECharts 依赖、
锁文件、组件与文档标注一并清除),买卖点标记只落在真实交易日上
3) 回测存档完整化(可往复查看)
- 同步端点(POST /api/backtests、/api/factor-tests)此前完全不落库 → 现在同样归档,
归档 id 经响应头 X-Experiment-Id 返回(不破坏 response_model)
- data_version 首次真实写入(数据快照指纹:最新交易日 + 各表规模)
- 个股收益曲线默认**全量保存**(此前硬截断 60 只);超出体积预算才裁剪,
并写 archive_meta(机器可读)+ unimplemented(人可读)如实标注
- 列表 kind/q 过滤 + X-Total-Count(此前 limit=50 静默截断)、DELETE 归档
- 只读归档页 /experiments/{id}(Server Component,SSR 直出**选股条件**与
**交易执行依据**);结果视图按 kind 分发(backtest/factor_test/selection),
非回测归档不套用回测口径
- 新增 CLI:prune_experiments(保留策略,默认 dry-run)、
restore_experiment_from_job(从 Job 副本按原 id 重建被删的历史归档,默认 dry-run)
门禁:pytest 388 passed、ruff All checks passed、tsc 0 错误、图表单测 7 passed、
next build 成功、契约脚本 verify_strategy_workspace 59/59(含按 kind 逐类验证归档页)。
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
"""实验归档保留策略 CLI(归档体积治理)。
|
||||
|
||||
用法(cd backend):
|
||||
# 只看会删什么(**默认就是 dry-run**,不写库)
|
||||
uv run python -m app.cli.prune_experiments --keep 30
|
||||
uv run python -m app.cli.prune_experiments --keep 20 --kind backtest
|
||||
uv run python -m app.cli.prune_experiments --older-than 180 # 180 天前的归档
|
||||
# 真正删除(必须显式加 --apply)
|
||||
uv run python -m app.cli.prune_experiments --keep 30 --apply
|
||||
|
||||
为什么默认 dry-run:归档是**研究结论的唯一凭据**(结果 + spec + 代码/数据版本),
|
||||
误删不可恢复。因此:
|
||||
- 默认只打印候选清单(含 id / 类型 / 时间 / 体积 / 摘要)与合计释放空间,不写库;
|
||||
- 只有显式 `--apply` 才执行删除,且删除前**再打印一次**清单;
|
||||
- 按 `created_at` **由新到旧保留**:先按过滤条件选出候选集合,再保留最新的 N 条,
|
||||
其余删除(`--keep` 与 `--older-than` 可同时给,取交集);
|
||||
- 只删 `experiment` 表(归档本身),**不动 `job_record`**(执行历史仍可追溯);
|
||||
但要注意:完整结果只存归档一份(`job.result_json` 对新记录为 NULL),
|
||||
因此**删除归档 = 该次回测结果不可再查看**。要留底先导出(归档页「导出完整 JSON」);
|
||||
- `--kind` 可只清理某一类(如只清 `factor_test`,保留回测结论)。
|
||||
|
||||
本模块是组装层(composition root):装配 Session 与 Repository,删除逻辑走仓储。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from app.infrastructure.persistence.sqlalchemy.repositories.jobs_impl import (
|
||||
SqlAlchemyExperimentRepository,
|
||||
)
|
||||
from app.infrastructure.persistence.sqlalchemy.session import SessionLocal
|
||||
|
||||
|
||||
def select_prune_candidates(
|
||||
rows: list,
|
||||
*,
|
||||
keep: int | None,
|
||||
older_than_days: int | None,
|
||||
now: datetime | None = None,
|
||||
) -> list:
|
||||
"""从归档列表里挑出**该删除**的那些(纯函数,便于单测)。
|
||||
|
||||
`rows`:任意具备 `id` / `created_at` 的归档摘要(不依赖 ORM 与数据库)。
|
||||
规则(两个条件同时给出时取交集):
|
||||
- `keep=N`:按 `created_at` 由新到旧排序后,**保留最新 N 条**,其余为候选;
|
||||
- `older_than_days=D`:创建时间早于 `now - D 天` 的才是候选;
|
||||
- `created_at` 为空的归档视为**最旧**(排序末位)——缺失时间时宁可被列为候选,
|
||||
也不静默把它当作「最新」而永久留在库里。
|
||||
"""
|
||||
ordered = sorted(rows, key=lambda e: (e.created_at or datetime.min), reverse=True)
|
||||
candidates = list(ordered)
|
||||
if older_than_days is not None:
|
||||
cutoff = (now or datetime.now()) - timedelta(days=older_than_days)
|
||||
candidates = [e for e in candidates if (e.created_at or datetime.min) < cutoff]
|
||||
if keep is not None:
|
||||
keep_ids = {e.id for e in ordered[: max(keep, 0)]}
|
||||
candidates = [e for e in candidates if e.id not in keep_ids]
|
||||
# 输出仍按新 → 旧,便于人核对
|
||||
candidates.sort(key=lambda e: (e.created_at or datetime.min), reverse=True)
|
||||
return candidates
|
||||
|
||||
|
||||
def _parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(
|
||||
prog="python -m app.cli.prune_experiments",
|
||||
description="实验归档保留策略(默认 dry-run,需 --apply 才真正删除)",
|
||||
)
|
||||
p.add_argument("--keep", type=int, default=None, help="按时间由新到旧保留的最新条数")
|
||||
p.add_argument("--kind", default=None, help="只处理某类归档(backtest/factor_test/selection)")
|
||||
p.add_argument(
|
||||
"--older-than",
|
||||
type=int,
|
||||
default=None,
|
||||
metavar="DAYS",
|
||||
help="只处理创建时间早于 N 天的归档",
|
||||
)
|
||||
p.add_argument("--apply", action="store_true", help="真正执行删除(否则仅预览)")
|
||||
args = p.parse_args(argv)
|
||||
if args.keep is None and args.older_than is None:
|
||||
p.error("至少给出 --keep 或 --older-than 之一,避免误删全部归档")
|
||||
if args.keep is not None and args.keep < 0:
|
||||
p.error("--keep 不能为负数")
|
||||
return args
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(sys.argv[1:] if argv is None else argv)
|
||||
with SessionLocal() as session:
|
||||
repo = SqlAlchemyExperimentRepository(session)
|
||||
# 列表查询刻意不取 result_json,体积用库侧算出的 result_bytes(单位:字符)
|
||||
rows = repo.list_filtered(kind=args.kind, limit=1_000_000, offset=0)
|
||||
candidates = select_prune_candidates(
|
||||
rows, keep=args.keep, older_than_days=args.older_than
|
||||
)
|
||||
|
||||
total_bytes = sum(e.result_bytes or 0 for e in candidates)
|
||||
print(f"归档总数 {len(rows)} 条,命中删除条件 {len(candidates)} 条,"
|
||||
f"预计释放约 {total_bytes / 1024 / 1024:.1f} MB")
|
||||
for e in candidates:
|
||||
created = e.created_at.strftime("%Y-%m-%d %H:%M") if e.created_at else "未知时间"
|
||||
print(
|
||||
f" - {e.id} {e.kind:<12} {created} "
|
||||
f"{((e.result_bytes or 0) / 1024):>8.0f} KB {(e.summary_text or '')[:60]}"
|
||||
)
|
||||
if not candidates:
|
||||
print("没有需要删除的归档。")
|
||||
return 0
|
||||
if not args.apply:
|
||||
print("\n[dry-run] 未删除任何记录;确认无误后加 --apply 执行。")
|
||||
return 0
|
||||
|
||||
deleted = 0
|
||||
for e in candidates:
|
||||
if repo.delete(e.id):
|
||||
deleted += 1
|
||||
session.commit()
|
||||
print(f"\n已删除 {deleted} 条归档(job_record 未改动)。")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,172 @@
|
||||
"""从 Job 结果副本重建 Experiment 归档(删除后的可选恢复路径)。
|
||||
|
||||
**为什么会有这个工具**:完整存档上线前,历史归档在 `job.result_json` 里另存了一份完整结果
|
||||
(双写遗留)。删除归档只删 `experiment` 行、不动 `job` 行,因此这些历史记录
|
||||
**可以从 Job 副本原样重建** —— 删除不等于数据永久消失。
|
||||
新归档(`job.result_json IS NULL`)没有副本,删除即不可恢复,本工具会明确拒绝而不是假装能救。
|
||||
|
||||
注意:本工具是**恢复手段**,不是"删除的撤销键"。删除本身是正常功能:
|
||||
已有历史归档可重建、新归档不可;是否恢复由人决定,工具默认 dry-run、不做任何自动动作。
|
||||
|
||||
用法(**默认 dry-run**,只打印将要写入的内容;`--apply` 才写库):
|
||||
|
||||
cd backend
|
||||
PYTHONPATH=. .venv/bin/python -m app.cli.restore_experiment_from_job --job-id JOB-XXXX
|
||||
PYTHONPATH=. .venv/bin/python -m app.cli.restore_experiment_from_job --job-id JOB-XXXX \
|
||||
--code-version 92627f5 --apply
|
||||
|
||||
诚实性要求(AGENT.md §7/§24):
|
||||
- **归档 id 用回原 id**(`job.experiment_id`),移动端/书签里的旧链接继续有效;
|
||||
- `spec_json` / `result_json` 逐字复制 Job 副本,不重新计算、不"顺手修正";
|
||||
- `summary_text` 由副本 JSON 反序列化后按归档同一函数重新生成(口径一致);
|
||||
- `code_version` **必须显式提供**,不做猜测:显式传 `--code-version ""` 表示"未知则留空";
|
||||
- `data_version` 留空 —— 历史归档当年没有数据指纹,补一个今天的指纹是伪造复现依据;
|
||||
- 重建前会检查该 job 的归档是否已存在,已存在则拒绝(除非 `--force`)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
from app.application.services.experiment_archive import _summary_text
|
||||
from app.domain.entities.research import (
|
||||
BacktestResult,
|
||||
ExperimentRecord,
|
||||
FactorTestReport,
|
||||
)
|
||||
from app.domain.entities.selection import SelectionResult
|
||||
from app.infrastructure.persistence.sqlalchemy.repositories.jobs_impl import (
|
||||
SqlAlchemyExperimentRepository,
|
||||
)
|
||||
from app.infrastructure.persistence.sqlalchemy.session import SessionLocal
|
||||
|
||||
|
||||
def _parse_result(kind: str, raw: str):
|
||||
"""按 kind 把 Job 里的结果 JSON 反序列化成领域对象(用于生成同一个摘要口径)。"""
|
||||
payload = json.loads(raw)
|
||||
if kind == "backtest":
|
||||
return BacktestResult.model_validate(payload)
|
||||
if kind == "factor_test":
|
||||
return FactorTestReport.model_validate(payload)
|
||||
if kind == "selection":
|
||||
return SelectionResult.model_validate(payload)
|
||||
return None
|
||||
|
||||
|
||||
def _load_job(session, job_id: str):
|
||||
"""按 ORM 读 Job(**不要用裸 SQL**)。
|
||||
|
||||
裸 `text()` 查询在 SQLite 下把 DateTime 列原样返回成字符串,写回 ORM 的 DateTime
|
||||
字段会抛 `TypeError: SQLite DateTime type only accepts Python datetime...`
|
||||
(MySQL+pymysql 返回 datetime 所以当时看不出问题)。ORM 读法跨方言类型一致,
|
||||
这个坑由 `tests/test_restore_experiment.py::test_restores_archive_from_job_copy` 钉住。
|
||||
"""
|
||||
from app.infrastructure.persistence.sqlalchemy.models.jobs import JobModel
|
||||
|
||||
return session.get(JobModel, job_id)
|
||||
|
||||
|
||||
def _experiment_exists(session, exp_id: str) -> bool:
|
||||
from app.infrastructure.persistence.sqlalchemy.models import ExperimentModel
|
||||
|
||||
return session.get(ExperimentModel, exp_id) is not None
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="从 Job 结果副本重建 Experiment 归档(删除后的可选恢复路径)")
|
||||
ap.add_argument("--job-id", required=True, help="来源 Job id(如 JOB-D3C120DC)")
|
||||
ap.add_argument(
|
||||
"--code-version",
|
||||
default=None,
|
||||
help="重建记录的 code_version(必填;传空串表示未知留空)。不猜版本。",
|
||||
)
|
||||
ap.add_argument("--force", action="store_true", help="归档已存在时也覆盖(默认拒绝)")
|
||||
ap.add_argument("--apply", action="store_true", help="真正写库(默认 dry-run)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if args.code_version is None:
|
||||
print(
|
||||
"❌ 必须显式给出 --code-version(不猜版本);确定未知请传 --code-version \"\"。",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
session = SessionLocal()
|
||||
try:
|
||||
row = _load_job(session, args.job_id)
|
||||
if row is None:
|
||||
print(f"❌ Job {args.job_id} 不存在", file=sys.stderr)
|
||||
return 1
|
||||
job_id = row.id
|
||||
kind = row.kind
|
||||
status = row.status
|
||||
spec_json = row.spec_json
|
||||
result_json = row.result_json
|
||||
exp_id = row.experiment_id
|
||||
finished_at = row.finished_at
|
||||
created_at = row.created_at
|
||||
if not exp_id:
|
||||
print(f"❌ Job {job_id} 没有 experiment_id,无法确定重建为哪个归档", file=sys.stderr)
|
||||
return 1
|
||||
if result_json is None:
|
||||
print(
|
||||
f"❌ Job {job_id} 的 result_json 为空(完整存档上线后结果只存归档一份),"
|
||||
"本工具无法重建 —— 这类归档删除后不可恢复。",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
exists = _experiment_exists(session, exp_id)
|
||||
result = _parse_result(kind, result_json)
|
||||
summary = _summary_text(kind, result) if result is not None else None
|
||||
restored_at = finished_at or created_at or datetime.now()
|
||||
|
||||
print(f"来源 Job : {job_id}(status={status},finished_at={finished_at})")
|
||||
print(f"归档 id : {exp_id}(已存在: {exists})")
|
||||
print(f"kind : {kind}")
|
||||
print(f"spec 长度 : {len(spec_json)} 字符")
|
||||
print(f"result 长度 : {len(result_json)} 字符")
|
||||
print(f"summary_text : {summary!r}")
|
||||
print(f"code_version : {args.code_version!r}")
|
||||
print("data_version : None(历史归档无指纹,不伪造)")
|
||||
print(f"created_at : {restored_at}")
|
||||
|
||||
if exists and not args.force:
|
||||
print("❌ 该归档已存在;如确要覆盖请加 --force", file=sys.stderr)
|
||||
return 1
|
||||
if not args.apply:
|
||||
print("\n(dry-run) 未写库。确认无误后加 --apply。")
|
||||
return 0
|
||||
|
||||
# 走仓储而不是直接操作 ORM 模型(AGENT §10:持久化只经 Repository)。
|
||||
# 这样 experiment 表 ↔ 领域实体的字段映射只有仓储一份:将来加列(尤其 NOT NULL)
|
||||
# 不会在这里静默漏写。覆盖语义由 `upsert`(merge)承担,归档 id 保持不变。
|
||||
record = ExperimentRecord(
|
||||
id=exp_id,
|
||||
kind=kind,
|
||||
spec_json=spec_json,
|
||||
result_json=result_json,
|
||||
summary_text=summary,
|
||||
code_version=args.code_version or None,
|
||||
data_version=None,
|
||||
job_id=job_id,
|
||||
created_at=restored_at,
|
||||
)
|
||||
repo = SqlAlchemyExperimentRepository(session)
|
||||
repo.upsert(record)
|
||||
session.commit()
|
||||
got = repo.get(exp_id)
|
||||
print(
|
||||
f"\n✅ 已重建:{got.id} / {got.kind} / 体积 {len(got.result_json)} 字符 / "
|
||||
f"code_version={got.code_version}"
|
||||
)
|
||||
return 0
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -16,8 +16,17 @@ import sys
|
||||
|
||||
|
||||
def _apply_memory_limit() -> None:
|
||||
"""设置 RLIMIT_AS(Linux)。达到上限时分配抛 MemoryError → Job 归档 failed,
|
||||
而不是让内核 OOM 杀掉整机(8G Pi5 上保护同机其它服务)。"""
|
||||
"""尝试施加 RLIMIT_AS:达到上限时分配抛 MemoryError → Job 归档 failed,
|
||||
而不是让内核 OOM 杀掉整机(8G Pi5 上保护同机其它服务)。
|
||||
|
||||
不按平台硬门禁:RLIMIT_AS 在 Linux / macOS 上都能设置,但**限额必须高于进程
|
||||
当前的虚拟地址空间基线**,否则内核返回 EINVAL(CPython 表现为
|
||||
`ValueError: current limit exceeds maximum limit`)。macOS 上 Python 进程的
|
||||
地址空间基线可达数十 GB(共享缓存 + malloc 区预留),故 `job.max_memory_gb: 6`
|
||||
在这类机器上会失败 —— 这是**配置阈值问题,不是平台不支持**,不应掩盖。
|
||||
|
||||
设置失败不阻断 Job,但如实打印原因与后果(AGENT.md §24「未实现项须如实标注」)。
|
||||
"""
|
||||
try:
|
||||
gb = int(os.environ.get("QLIB_JOB_MEM_LIMIT_GB") or "")
|
||||
except (TypeError, ValueError):
|
||||
@@ -27,7 +36,15 @@ def _apply_memory_limit() -> None:
|
||||
limit = gb * 1024**3
|
||||
import resource
|
||||
|
||||
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))
|
||||
try:
|
||||
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))
|
||||
except (ValueError, OSError) as exc:
|
||||
print(
|
||||
f"[run_job] 内存上限 {gb}GB 设置失败({type(exc).__name__}: {exc});"
|
||||
"本 Job 无内存隔离保护。若限额低于进程虚拟地址空间基线(macOS 常见),"
|
||||
"可调大 config.yaml job.max_memory_gb 后重试。",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
+192
-4
@@ -8,6 +8,7 @@
|
||||
uv run python -m app.cli.sync financial --all # 财务指标(增量)
|
||||
uv run python -m app.cli.sync financial --all --full # 财务指标(强制全量重拉)
|
||||
uv run python -m app.cli.sync verify --symbol 600519.SH # 新浪交叉验证
|
||||
uv run python -m app.cli.sync daily_basic --start 20200101 # 每日指标(股息率等)
|
||||
|
||||
增量与兜底:
|
||||
- daily --resume:从本地最新交易日续传(已有);Tushare 失败时走新浪校验兜底,
|
||||
@@ -31,8 +32,10 @@ from datetime import date, datetime, timedelta
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.application.services.data_sync import (
|
||||
DailyBasicSyncer,
|
||||
DailySymbolResult,
|
||||
FinancialSymbolResult,
|
||||
NameHistorySyncer,
|
||||
VerifiedDailySyncer,
|
||||
VerifiedFinancialSyncer,
|
||||
)
|
||||
@@ -47,7 +50,9 @@ from app.infrastructure.persistence.sqlalchemy.repositories.index_impl import (
|
||||
from app.infrastructure.persistence.sqlalchemy.repositories.market_impl import (
|
||||
SqlAlchemyAdjustFactorRepository,
|
||||
SqlAlchemyDailyBarRepository,
|
||||
SqlAlchemyDailyBasicRepository,
|
||||
SqlAlchemyFinancialRepository,
|
||||
SqlAlchemyStockNameHistoryRepository,
|
||||
SqlAlchemyStockRepository,
|
||||
SqlAlchemySyncLogRepository,
|
||||
SqlAlchemyTradingCalendarRepository,
|
||||
@@ -80,13 +85,44 @@ def _session_ctx():
|
||||
|
||||
|
||||
def cmd_basic(args) -> int:
|
||||
"""股票基础信息。--include-delisted 同时拉取已退市/暂停上市(幸存者偏差修正)。"""
|
||||
statuses = ["L", "P", "D"] if getattr(args, "include_delisted", False) else ["L"]
|
||||
with _session_ctx() as session:
|
||||
provider = _failover_provider(session)
|
||||
stocks = provider.get_stock_basic()
|
||||
repo = SqlAlchemyStockRepository(session)
|
||||
touched = repo.upsert_many(stocks)
|
||||
session.commit()
|
||||
print(f"[basic] 拉取 {len(stocks)} 只,落库 {touched} 条")
|
||||
total = 0
|
||||
failed: list[str] = []
|
||||
for st in statuses:
|
||||
try:
|
||||
stocks = provider.get_stock_basic(st)
|
||||
except DataSourceError as exc:
|
||||
# failover 会把备用源(新浪)的 NotSupported 包装成 DataSourceError,
|
||||
# 因此这里必须捕获 DataSourceError 而不是 DataSourceNotSupported,
|
||||
# 否则 --include-delisted 在主源抖动时会整体抛栈退出(L 已写、P/D 静默缺失)。
|
||||
print(f"[error] list_status={st} 拉取失败:{exc}", file=sys.stderr)
|
||||
failed.append(st)
|
||||
continue
|
||||
touched = repo.upsert_many(stocks)
|
||||
session.commit()
|
||||
total += touched
|
||||
n_delisted = sum(1 for s in stocks if s.delist_date is not None)
|
||||
print(
|
||||
f"[basic] list_status={st} 拉取 {len(stocks)} 只(含 delist_date {n_delisted} 只),"
|
||||
f"落库 {touched} 条"
|
||||
)
|
||||
if len(statuses) > 1:
|
||||
print(
|
||||
"[basic] 已退市股票已入库;其历史行情需另行同步(否则回测仍无法使用):\n"
|
||||
" 请执行 sync daily --symbols <退市代码逗号分隔> --start 20200101"
|
||||
)
|
||||
print(f"[basic] 合计落库 {total} 条")
|
||||
if failed:
|
||||
print(
|
||||
f"[error] 以下 list_status 拉取失败:{','.join(failed)};"
|
||||
"退市股缺失会让回测重新出现幸存者偏差,请重跑",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
@@ -104,6 +140,48 @@ def cmd_calendar(args) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_namechange(args) -> int:
|
||||
"""股票名称变更历史同步(时点 ST 判定依据;按自然年分片)。"""
|
||||
start = _parse_day(args.start) if args.start else date(1990, 1, 1)
|
||||
end = _parse_day(args.end) if args.end else date.today()
|
||||
if start > end:
|
||||
print("[error] --start 不能晚于 --end", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
with _session_ctx() as session:
|
||||
audit_repo = SqlAlchemySyncLogRepository(session)
|
||||
primary = TushareProvider(token=get_settings().tushare_token)
|
||||
repo = SqlAlchemyStockNameHistoryRepository(session)
|
||||
syncer = NameHistorySyncer(primary=primary, repo=repo, audit=audit_repo.add)
|
||||
|
||||
total_span = (end.year - start.year) + 1
|
||||
print(f"[namechange] {start}~{end} 共 {total_span} 个年度分片")
|
||||
ok = failed = written = fetched = 0
|
||||
t0 = time.time()
|
||||
|
||||
def _on(idx: int, total: int, chunk_start: date) -> None:
|
||||
_progress(idx, total, t0, f"{chunk_start.year} 年")
|
||||
|
||||
results = syncer.sync_range(start, end, on_progress=_on)
|
||||
for res in results:
|
||||
if res.status == "ok":
|
||||
ok += 1
|
||||
written += res.rows_written
|
||||
fetched += res.rows_fetched
|
||||
else:
|
||||
failed += 1
|
||||
for note in res.notes:
|
||||
print(f"\n[warn] {res.start}~{res.end}: {note}", file=sys.stderr)
|
||||
session.commit()
|
||||
lo, hi = repo.namechange_dates()
|
||||
print(
|
||||
f"\n[namechange] 分片成功 {ok} / 失败 {failed};"
|
||||
f"拉取 {fetched} 行、写入 {written} 行;"
|
||||
f"本地生效起点 {lo} ~ {hi}"
|
||||
)
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
def _symbols_of(args) -> list[str]:
|
||||
if getattr(args, "all", False):
|
||||
with _session_ctx() as session:
|
||||
@@ -382,11 +460,102 @@ def cmd_export(args) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
_DAILY_BASIC_DEFAULT_START = date(2020, 1, 1)
|
||||
|
||||
|
||||
def cmd_daily_basic(args) -> int:
|
||||
"""每日指标同步(按交易日整表;新浪不支持 → 失败如实记录,不留静默缺口)。"""
|
||||
start = _parse_day(args.start) if args.start else _DAILY_BASIC_DEFAULT_START
|
||||
end = _parse_day(args.end) if args.end else date.today()
|
||||
if start > end:
|
||||
print("[error] --start 不能晚于 --end", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
with _session_ctx() as session:
|
||||
audit_repo = SqlAlchemySyncLogRepository(session)
|
||||
primary = TushareProvider(token=get_settings().tushare_token)
|
||||
repo = SqlAlchemyDailyBasicRepository(session)
|
||||
syncer = DailyBasicSyncer(primary=primary, repo=repo, audit=audit_repo.add)
|
||||
|
||||
if args.full:
|
||||
cal = SqlAlchemyTradingCalendarRepository(session)
|
||||
days = [d.calendar_date for d in cal.list_range(start, end) if d.is_open]
|
||||
else:
|
||||
days = repo.missing_dates(start, end)
|
||||
total = len(days)
|
||||
if total == 0:
|
||||
print(f"[daily_basic] {start}~{end} 无待补交易日(本地已完整)")
|
||||
return 0
|
||||
print(f"[daily_basic] {start}~{end} 待同步 {total} 个交易日")
|
||||
|
||||
ok = failed = written = 0
|
||||
failures: list[date] = []
|
||||
t0 = time.time()
|
||||
for idx, day in enumerate(days, start=1):
|
||||
try:
|
||||
res = syncer.sync_day(day)
|
||||
except DataSourceError as exc:
|
||||
print(f"\n[error] 第 {idx}/{total} 日 {day} 权限/凭证故障,中止:{exc}", file=sys.stderr)
|
||||
session.commit()
|
||||
return 1
|
||||
if res.status == "ok":
|
||||
ok += 1
|
||||
written += res.rows_written
|
||||
else:
|
||||
failed += 1
|
||||
failures.append(day)
|
||||
_progress(idx, total, t0, f"{day} 行数={res.rows_fetched}")
|
||||
if idx % 20 == 0 or idx == total:
|
||||
session.commit() # 分批提交:中断时已完成的部分保持有效
|
||||
if args.sleep:
|
||||
time.sleep(args.sleep)
|
||||
session.commit()
|
||||
|
||||
span = time.time() - t0
|
||||
print(
|
||||
f"\n[daily_basic] 完成:成功 {ok} 日 / 失败 {failed} 日,累计写入 {written} 行,"
|
||||
f"耗时 {span:.1f}s"
|
||||
)
|
||||
if failures:
|
||||
print(
|
||||
"[daily_basic] 失败交易日(可重跑本命令补齐):"
|
||||
+ ", ".join(d.isoformat() for d in failures[:20])
|
||||
+ (" …" if len(failures) > 20 else ""),
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def _progress(idx: int, total: int, t0: float, extra: str = "") -> None:
|
||||
"""单行进度条(同步长任务可读性)。"""
|
||||
elapsed = time.time() - t0
|
||||
rate = idx / elapsed if elapsed > 0 else 0.0
|
||||
eta = (total - idx) / rate if rate > 0 else 0.0
|
||||
pct = idx / total * 100 if total else 100.0
|
||||
end = "\n" if idx >= total else "\r"
|
||||
print(
|
||||
f" 进度 {idx}/{total} ({pct:5.1f}%) 已用 {elapsed:6.1f}s 预计剩余 {eta:6.1f}s {extra}",
|
||||
end=end,
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(prog="app.cli.sync", description="Tushare 数据同步 CLI")
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_nc = sub.add_parser("namechange", help="同步股票名称变更历史(时点 ST 判定)")
|
||||
p_nc.add_argument("--start", help="起始日 YYYYMMDD(默认 19900101,覆盖全历史)")
|
||||
p_nc.add_argument("--end", help="结束日 YYYYMMDD(默认今天)")
|
||||
p_nc.set_defaults(func=cmd_namechange)
|
||||
|
||||
p_basic = sub.add_parser("basic", help="同步股票基础信息")
|
||||
p_basic.add_argument(
|
||||
"--include-delisted",
|
||||
action="store_true",
|
||||
help="同时拉取已退市(D)与暂停上市(P),填充 delist_date(幸存者偏差修正)",
|
||||
)
|
||||
p_basic.set_defaults(func=cmd_basic)
|
||||
|
||||
p_cal = sub.add_parser("calendar", help="同步交易日历")
|
||||
@@ -394,6 +563,25 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
p_cal.add_argument("--end", required=True, help="YYYYMMDD")
|
||||
p_cal.set_defaults(func=cmd_calendar)
|
||||
|
||||
p_db = sub.add_parser(
|
||||
"daily_basic",
|
||||
help="同步每日指标(估值/股息率/市值,按交易日整表;新浪不支持本接口)",
|
||||
)
|
||||
p_db.add_argument("--start", default="20200101", help="YYYYMMDD(默认 20200101)")
|
||||
p_db.add_argument("--end", default="", help="YYYYMMDD(默认今天)")
|
||||
p_db.add_argument(
|
||||
"--full",
|
||||
action="store_true",
|
||||
help="忽略本地已有日期,重拉区间内全部开市日(默认只补缺失日)",
|
||||
)
|
||||
p_db.add_argument(
|
||||
"--sleep",
|
||||
type=float,
|
||||
default=0,
|
||||
help="每个交易日请求间隔秒数(限速时加大,如 0.2 或 1)",
|
||||
)
|
||||
p_db.set_defaults(func=cmd_daily_basic)
|
||||
|
||||
p_daily = sub.add_parser("daily", help="同步日线与复权因子(Tushare 失败 → 新浪校验兜底补缺)")
|
||||
p_daily.add_argument("--symbols", default="", help="600519.SH,000001.SZ")
|
||||
p_daily.add_argument("--all", action="store_true", help="遍历 stock 表全部股票")
|
||||
|
||||
Reference in New Issue
Block a user