perf(backend): 内存优化三项——全市场研究不再占满 8G

1) 数据装配流式+列裁剪:Repository 新增 stream_range_many_columns(只 SELECT
   所需列、SQL 侧转 REAL、yield_per 分批),引擎按 required_columns 取数
   (LocalEngine 仅 close+因子字段),消除 ORM/Decimal 全量物化;
2) 研究 Job 独立子进程执行(job.mode=subprocess):python -m app.cli.run_job
   在子进程内设 RLIMIT_AS 上限,OOM 归档 failed 而非拖垮 API worker;
   子进程异常退出由父进程补记 failed;并发上限 2;
3) 服务启动清理:残留 queued/running Job 标记 failed(防永久 running)。

实测同款全市场回测:uvicorn worker RSS 稳定 ~220MB,任务峰值内存由 4.1GB+
降至 ~470MB,24s 完成并归档(此前 43s 未完成即 OOM)。
新增/更新测试 96 passed,ruff 干净。
This commit is contained in:
Simon
2026-09-06 22:12:44 +08:00
parent 02e42184be
commit 195f5d41f4
18 changed files with 593 additions and 31 deletions
+22
View File
@@ -10,6 +10,7 @@ from typing import Protocol
import pandas as pd
from app.domain.entities.research import BacktestResult, FactorTestReport, ResearchSpec
from app.quant.factors import FactorError, get_factor
from app.quant.local_engine import (
TopKBacktestRunner,
build_factor_panels,
@@ -17,12 +18,30 @@ from app.quant.local_engine import (
run_spec_factor_test,
)
# LocalEngine 路径恒需 close(TopK 收盘撮合 / 前瞻收益)
_CLOSE = {"close"}
def factor_required_columns(spec: ResearchSpec) -> set[str]:
"""spec 因子计算 + 回测撮合所需的行情数值列(含 close)。"""
needed = set(_CLOSE)
for fs in spec.factors:
try:
defn, _fn = get_factor(fs.name)
except FactorError:
continue # 未知因子由执行期统一报错
needed.update(defn.requires)
return needed
class QuantEngine(Protocol):
"""研究引擎端口:因子面板构建 / 因子测试 / 回测。"""
name: str
def required_columns(self, spec: ResearchSpec) -> set[str]:
"""执行该 spec 所需的行情数值列(数据装配按此裁剪,控制内存)。"""
def run_factor_test(
self, daily: pd.DataFrame, spec: ResearchSpec, horizon_days: int = 21
) -> FactorTestReport: ...
@@ -35,6 +54,9 @@ class LocalEngine:
name = "local"
def required_columns(self, spec: ResearchSpec) -> set[str]:
return factor_required_columns(spec)
def run_factor_test(
self, daily: pd.DataFrame, spec: ResearchSpec, horizon_days: int = 21
) -> FactorTestReport: