perf(backend): 内存优化三项——全市场研究不再占满 8G
1) 数据装配流式+列裁剪:Repository 新增 stream_range_many_columns(只 SELECT 所需列、SQL 侧转 REAL、yield_per 分批),引擎按 required_columns 取数 (LocalEngine 仅 close+因子字段),消除 ORM/Decimal 全量物化; 2) 研究 Job 独立子进程执行(job.mode=subprocess):python -m app.cli.run_job 在子进程内设 RLIMIT_AS 上限,OOM 归档 failed 而非拖垮 API worker; 子进程异常退出由父进程补记 failed;并发上限 2; 3) 服务启动清理:残留 queued/running Job 标记 failed(防永久 running)。 实测同款全市场回测:uvicorn worker RSS 稳定 ~220MB,任务峰值内存由 4.1GB+ 降至 ~470MB,24s 完成并归档(此前 43s 未完成即 OOM)。 新增/更新测试 96 passed,ruff 干净。
This commit is contained in:
@@ -39,10 +39,21 @@ class QlibEngine(QuantEngine):
|
||||
|
||||
name = "qlib"
|
||||
|
||||
# qlib 落盘需要 OHLCV(vwap/factor 由本地合成,不来自行情表)
|
||||
_DUMP_COLUMNS = {"open", "high", "low", "close", "volume", "amount"}
|
||||
|
||||
def __init__(self, qlib_dir: Path | None = None) -> None:
|
||||
# 默认落盘到 data/qlib(与 storage.qlib_dir 一致);可注入临时目录便于测试
|
||||
self.qlib_dir = qlib_dir or _default_qlib_dir()
|
||||
|
||||
def required_columns(self, spec: ResearchSpec) -> set[str]:
|
||||
# 回测需把全字段落盘成 qlib 数据集;因子测试走共享实现,只需 close+因子字段
|
||||
if spec.type == "backtest":
|
||||
return set(self._DUMP_COLUMNS)
|
||||
from app.quant.engine import factor_required_columns
|
||||
|
||||
return factor_required_columns(spec)
|
||||
|
||||
def run_factor_test(
|
||||
self, daily: pd.DataFrame, spec: ResearchSpec, horizon_days: int = 21
|
||||
) -> FactorTestReport:
|
||||
|
||||
Reference in New Issue
Block a user