perf(backend): 内存优化三项——全市场研究不再占满 8G
1) 数据装配流式+列裁剪:Repository 新增 stream_range_many_columns(只 SELECT 所需列、SQL 侧转 REAL、yield_per 分批),引擎按 required_columns 取数 (LocalEngine 仅 close+因子字段),消除 ORM/Decimal 全量物化; 2) 研究 Job 独立子进程执行(job.mode=subprocess):python -m app.cli.run_job 在子进程内设 RLIMIT_AS 上限,OOM 归档 failed 而非拖垮 API worker; 子进程异常退出由父进程补记 failed;并发上限 2; 3) 服务启动清理:残留 queued/running Job 标记 failed(防永久 running)。 实测同款全市场回测:uvicorn worker RSS 稳定 ~220MB,任务峰值内存由 4.1GB+ 降至 ~470MB,24s 完成并归档(此前 43s 未完成即 OOM)。 新增/更新测试 96 passed,ruff 干净。
This commit is contained in:
@@ -71,6 +71,19 @@ class TestBacktestMain:
|
||||
assert all(p.value <= 0 for p in res.drawdown)
|
||||
assert res.summary.max_drawdown_pct <= 0
|
||||
|
||||
def test_required_columns_follow_factor_dependencies(self) -> None:
|
||||
"""数据装配按引擎所需列裁剪:momentum 只取 close;量比因子需 volume。"""
|
||||
engine = LocalEngine()
|
||||
momentum = _spec() # momentum_20
|
||||
assert engine.required_columns(momentum) == {"close"}
|
||||
volume = _spec().model_copy(
|
||||
update={"factors": [FactorSpec(name="volume_ratio_5_60")]}
|
||||
)
|
||||
assert engine.required_columns(volume) == {"close", "volume"}
|
||||
unknown = _spec().model_copy(update={"factors": [FactorSpec(name="no_such")]})
|
||||
# 未知因子不参与列裁剪,交由执行期统一报错
|
||||
assert engine.required_columns(unknown) == {"close"}
|
||||
|
||||
|
||||
def _limit_up_scenario_daily() -> pd.DataFrame:
|
||||
"""Y 在 2024-07-01 相对前一交易日跳涨 10.5%(主板涨停不可追),且动量高于 X。"""
|
||||
|
||||
Reference in New Issue
Block a user