perf(backend): 内存优化三项——全市场研究不再占满 8G

1) 数据装配流式+列裁剪:Repository 新增 stream_range_many_columns(只 SELECT
   所需列、SQL 侧转 REAL、yield_per 分批),引擎按 required_columns 取数
   (LocalEngine 仅 close+因子字段),消除 ORM/Decimal 全量物化;
2) 研究 Job 独立子进程执行(job.mode=subprocess):python -m app.cli.run_job
   在子进程内设 RLIMIT_AS 上限,OOM 归档 failed 而非拖垮 API worker;
   子进程异常退出由父进程补记 failed;并发上限 2;
3) 服务启动清理:残留 queued/running Job 标记 failed(防永久 running)。

实测同款全市场回测:uvicorn worker RSS 稳定 ~220MB,任务峰值内存由 4.1GB+
降至 ~470MB,24s 完成并归档(此前 43s 未完成即 OOM)。
新增/更新测试 96 passed,ruff 干净。
This commit is contained in:
Simon
2026-09-06 22:12:44 +08:00
parent 02e42184be
commit 195f5d41f4
18 changed files with 593 additions and 31 deletions
+2 -3
View File
@@ -8,8 +8,7 @@ from datetime import datetime
from fastapi import APIRouter, BackgroundTasks, HTTPException
from app.api.deps import DbSession, ExperimentRepoDep, JobRepoDep
from app.api.jobs import _bg_factories
from app.application.services.job_executor import execute_job, new_id
from app.application.services.job_executor import new_id, run_job_background
from app.domain.entities.research import (
ExperimentRecord,
JobRecord,
@@ -77,5 +76,5 @@ def rerun_experiment(
)
job_repo.create(job)
session.commit()
background.add_task(execute_job, job.id, **_bg_factories())
background.add_task(run_job_background, job.id)
return {"job_id": job.id, "status": job.status, "origin_experiment": exp.id}
+6 -9
View File
@@ -1,8 +1,11 @@
"""异步研究 Job API(Phase 4):提交 / 查询 / SSE 进度。
POST /api/jobs 创建 Job(BackgroundTasks 本地执行),立即返回 job_id
POST /api/jobs 创建 Job(BackgroundTasks 后台执行),立即返回 job_id
GET /api/jobs/{id} 状态 + 结果(成功时内嵌 result)
GET /api/jobs/{id}/events SSE 进度(queued→running→success|failed)
执行模式见 config job.mode:subprocess 时研究任务在独立子进程跑(内存隔离),
API worker 不被重任务拖垮(内存优化专项)。
"""
from __future__ import annotations
@@ -15,7 +18,7 @@ from fastapi import APIRouter, BackgroundTasks, HTTPException
from fastapi.responses import StreamingResponse
from app.api.deps import DbSession, JobRepoDep
from app.application.services.job_executor import execute_job, new_id
from app.application.services.job_executor import new_id, run_job_background
from app.domain.entities.research import (
BacktestResult,
FactorTestReport,
@@ -31,12 +34,6 @@ from app.infrastructure.persistence.sqlalchemy.session import SessionLocal
router = APIRouter(prefix="/jobs", tags=["jobs"])
def _bg_factories():
from app.application.services.job_executor import default_factories
return default_factories()
def _decode_result(kind: str, result_json: str | None):
if result_json is None:
return None
@@ -68,7 +65,7 @@ def create_job(
)
job_repo.create(job)
session.commit()
background.add_task(execute_job, job.id, **_bg_factories())
background.add_task(run_job_background, job.id)
return {"job_id": job.id, "status": job.status}