docs: 文档重构 — 清理 AI agent 残留，整合 docs/ 目录结构
- 删除 11 个残留文件: continuation.md, init_plan.md, reasonix.toml, djapi/continuation.md, djapi/.serena/, djapi/.claude/, djapi/.mcp.json, .claude/skills/, docs/usage.html, docs/db_schema.md, docs/report_db_design.md - 7 个 CLAUDE-*.md 移入 docs/ 并重命名去 CLAUDE- 前缀 - 新增 4 个文档: architecture.md, development.md, api.md, deployment.md - 重写 usage.md, README.md - 修复所有过时引用和交叉链接
This commit is contained in:
Simon
+1
-1
@@ -13,7 +13,7 @@ MYSQL_PASSWORD=your-mysql-password
|
||||
MYSQL_DATABASE=myquant
|
||||
|
||||
# 日报结构化入库 (news_report / news_event) 只读查询
|
||||
# 与 report_db_design.md §7 一致；密码必填，缺失时接口直接报错
|
||||
# 与 docs/db_schema_v1.1.md 一致；密码必填，缺失时接口直接报错
|
||||
NEWS_DB_HOST=127.0.0.1
|
||||
NEWS_DB_PORT=3306
|
||||
NEWS_DB_USER=myquant
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"serena-djapi": {
|
||||
"command": "uv",
|
||||
"args": [
|
||||
"run",
|
||||
"--directory",
|
||||
"/Users/summer/Downloads/cc-cursor/mcp-servers/serena",
|
||||
"serena",
|
||||
"start-mcp-server",
|
||||
"--project",
|
||||
"/Users/summer/Downloads/cc-cursor/djapi"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,2 +0,0 @@
|
||||
/cache
|
||||
/project.local.yml
|
||||
@@ -1,23 +0,0 @@
|
||||
# Code Style & Conventions
|
||||
|
||||
## Python
|
||||
- Django app: all business logic in `api/stock/`, not in views
|
||||
- views.py is thin forwarding layer: extract params -> call function -> return Response
|
||||
- Double import pattern for standalone scripts: try relative import first, fall back to absolute
|
||||
- Use `viewFunc_tsCodeAndDate()` wrapper for ts_code + date_range endpoints
|
||||
- Use `viewFunc_singleParam()` wrapper for single-param endpoints
|
||||
- DRF `@api_view(['GET'])` + `@extend_schema` on all views
|
||||
- DRF `Response` (not `JsonResponse`) — no `safe=False` parameter
|
||||
- Configuration split: config.py (token) / strategy_config.py / scan_config.py
|
||||
|
||||
## Constraints
|
||||
- `api/video/` is protected — do NOT modify unless user explicitly asks
|
||||
- No python-dotenv dependency — use stdlib env loaders only
|
||||
- Backward compatibility: keep re-exports when splitting modules
|
||||
- Server `.env` file manages all secrets; uwsgi.ini only has DJANGO_SETTINGS_MODULE
|
||||
|
||||
## Secrets
|
||||
- All API keys/tokens/passwords via os.getenv()
|
||||
- Local: .env file (not committed)
|
||||
- Server: /home/simon/myquant/djapi/.env
|
||||
- Django loads via djapi/env_loader.py, video loads via api/video/env.py
|
||||
@@ -1,32 +0,0 @@
|
||||
# Project Overview
|
||||
|
||||
djapi is a Django 5.2 project providing financial data APIs for A-share stocks and CCTV news broadcast video processing.
|
||||
|
||||
## Tech Stack
|
||||
- Python 3.10, Django 5.2, uWSGI, nginx
|
||||
- Tushare (stock data), akshare (alternative stock data)
|
||||
- DRF + drf-spectacular (API documentation)
|
||||
- MySQL (business data), SQLite (Django admin only)
|
||||
- yt-dlp + ffmpeg + pydub (video/audio processing)
|
||||
- DashScope (ASR), DeepSeek API (AI text processing)
|
||||
|
||||
## Architecture
|
||||
- Single Django app: `api`
|
||||
- `api/stock/` — stock data module (Tushare/akshare -> pandas -> JsonResponse/DRF Response)
|
||||
- `api/video/` — independent video processing pipeline (download -> audio -> ASR -> AI split -> MySQL)
|
||||
- views.py is thin: extracts params, calls stock functions, returns Response
|
||||
|
||||
## Key Files
|
||||
- `api/views.py` — all ~15 API views, using @api_view + @extend_schema
|
||||
- `api/stock/stock_utils.py` — shared utilities: tscodeCheck, viewFunc_tsCodeAndDate, viewFunc_singleParam
|
||||
- `api/stock/config.py` — Tushare token + re-exports from strategy_config, scan_config
|
||||
- `api/serializers.py` — 13 DRF Serializer classes
|
||||
- `djapi/env_loader.py` — .env file loader (stdlib, no python-dotenv)
|
||||
- `api/video/env.py` — standalone .env loader for video module
|
||||
- `api/utils/mysql_handler.py` — shared MySQLDB class
|
||||
|
||||
## Deployment
|
||||
- Server: simon@doorcome.cn, path: /home/simon/myquant/djapi/
|
||||
- Virtual env: /opt/miniconda/envs/django/
|
||||
- uWSGI on port 5004, nginx reverse proxy
|
||||
- Domains: api.doorcome.cn, echart.doorcome.cn
|
||||
@@ -1,44 +0,0 @@
|
||||
# Suggested Commands
|
||||
|
||||
## Development
|
||||
```bash
|
||||
python manage.py runserver 0.0.0.0:8000 # dev server
|
||||
python manage.py check --deploy # check config
|
||||
python manage.py test api # run tests
|
||||
```
|
||||
|
||||
## uWSGI
|
||||
```bash
|
||||
uwsgi --ini uwsgi.ini # start
|
||||
uwsgi --reload uwsgi.pid # hot reload
|
||||
uwsgi --stop uwsgi.pid # stop
|
||||
# On server:
|
||||
/opt/miniconda/envs/django/bin/uwsgi --ini /home/simon/myquant/djapi/uwsgi.ini
|
||||
kill $(lsof -ti:5004) # force stop
|
||||
```
|
||||
|
||||
## Deploy
|
||||
```bash
|
||||
# Full sync (exclude production data)
|
||||
rsync -avz --delete \
|
||||
--exclude='.env' --exclude='db.sqlite3' \
|
||||
--exclude='*.log' --exclude='uwsgi.pid' \
|
||||
--exclude='__pycache__/' --exclude='*.pyc' \
|
||||
--exclude='xwlb_video/' --exclude='audio_processing/' \
|
||||
/Users/summer/Downloads/cc-cursor/djapi/ \
|
||||
simon@doorcome.cn:/home/simon/myquant/djapi/
|
||||
|
||||
# Single file sync MUST use full target path
|
||||
rsync -avz api/views.py simon@doorcome.cn:/home/simon/myquant/djapi/api/views.py
|
||||
```
|
||||
|
||||
## API Docs
|
||||
- /api/docs/ — Swagger UI
|
||||
- /api/redoc/ — ReDoc
|
||||
- /api/schema/ — OpenAPI JSON
|
||||
|
||||
## Video Processing
|
||||
```bash
|
||||
cd api/video
|
||||
python main.py
|
||||
```
|
||||
@@ -1,120 +0,0 @@
|
||||
# the name by which the project can be referenced within Serena
|
||||
project_name: "djapi"
|
||||
|
||||
|
||||
# list of languages for which language servers are started; choose from:
|
||||
# al ansible bash clojure cpp
|
||||
# cpp_ccls crystal csharp csharp_omnisharp dart
|
||||
# elixir elm erlang fortran fsharp
|
||||
# go groovy haskell haxe hlsl
|
||||
# java json julia kotlin lean4
|
||||
# lua luau markdown matlab msl
|
||||
# nix ocaml pascal perl php
|
||||
# php_phpactor powershell python python_jedi python_ty
|
||||
# r rego ruby ruby_solargraph rust
|
||||
# scala solidity swift systemverilog terraform
|
||||
# toml typescript typescript_vts vue yaml
|
||||
# zig
|
||||
# (This list may be outdated. For the current list, see values of Language enum here:
|
||||
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
|
||||
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# Note:
|
||||
# - For C, use cpp
|
||||
# - For JavaScript, use typescript
|
||||
# - For Free Pascal/Lazarus, use pascal
|
||||
# Special requirements:
|
||||
# Some languages require additional setup/installations.
|
||||
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
|
||||
# When using multiple languages, the first language server that supports a given file will be used for that file.
|
||||
# The first language is the default language and the respective language server will be used as a fallback.
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
languages:
|
||||
- typescript
|
||||
- python
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
encoding: "utf-8"
|
||||
|
||||
# line ending convention to use when writing source files.
|
||||
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
|
||||
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
|
||||
line_ending:
|
||||
|
||||
# The language backend to use for this project.
|
||||
# If not set, the global setting from serena_config.yml is used.
|
||||
# Valid values: LSP, JetBrains
|
||||
# Note: the backend is fixed at startup. If a project with a different backend
|
||||
# is activated post-init, an error will be returned.
|
||||
language_backend:
|
||||
|
||||
# whether to use project's .gitignore files to ignore files
|
||||
ignore_all_files_in_gitignore: true
|
||||
|
||||
# advanced configuration option allowing to configure language server-specific options.
|
||||
# Maps the language key to the options.
|
||||
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
|
||||
# No documentation on options means no options are available.
|
||||
ls_specific_settings: {}
|
||||
|
||||
# list of additional paths to ignore in this project.
|
||||
# Same syntax as gitignore, so you can use * and **.
|
||||
# Note: global ignored_paths from serena_config.yml are also applied additively.
|
||||
ignored_paths: []
|
||||
|
||||
# whether the project is in read-only mode
|
||||
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
|
||||
# Added on 2025-04-18
|
||||
read_only: false
|
||||
|
||||
# list of tool names to exclude.
|
||||
# This extends the existing exclusions (e.g. from the global configuration)
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
excluded_tools: []
|
||||
|
||||
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
|
||||
# This extends the existing inclusions (e.g. from the global configuration).
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
included_optional_tools: []
|
||||
|
||||
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
|
||||
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
fixed_tools: []
|
||||
|
||||
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
|
||||
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
|
||||
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
|
||||
# for this project.
|
||||
# This setting can, in turn, be overridden by CLI parameters (--mode).
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
default_modes:
|
||||
|
||||
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
added_modes:
|
||||
|
||||
# initial prompt for the project. It will always be given to the LLM upon activating the project
|
||||
# (contrary to the memories, which are loaded on demand).
|
||||
initial_prompt: ""
|
||||
|
||||
# time budget (seconds) per tool call for the retrieval of additional symbol information
|
||||
# such as docstrings or parameter information.
|
||||
# This overrides the corresponding setting in the global configuration; see the documentation there.
|
||||
# If null or missing, use the setting from the global configuration.
|
||||
symbol_info_budget:
|
||||
|
||||
# list of regex patterns which, when matched, mark a memory entry as read‑only.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
read_only_memory_patterns: []
|
||||
|
||||
# list of regex patterns for memories to completely ignore.
|
||||
# Matching memories will not appear in list_memories or activate_project output
|
||||
# and cannot be accessed via read_memory or write_memory.
|
||||
# To access ignored memory files, use the read_file tool on the raw file path.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
# Example: ["_archive/.*", "_episodes/.*"]
|
||||
ignored_memory_patterns: []
|
||||
+1
-1
@@ -99,7 +99,7 @@ python api/video/main.py
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `config.py` 中的 TS_TOKEN 和 `deepseek.py`/`ai.py`/`audioRead.py` 中的 API key、`mysqlHandle.py` 中的数据库密码均为硬编码 —— 生产环境应迁移到环境变量
|
||||
- 所有密钥已迁移到环境变量，通过 `.env` 统一管理
|
||||
- `api/stock/` 下的模块支持两种导入方式（相对导入和绝对导入），这是为了兼容「作为 Django app 被调用」和「直接命令行运行脚本」两种场景
|
||||
- `api/video/` 模块设计为独立命令行运行，不依赖 Django 框架
|
||||
- `db.sqlite3` 已提交到代码库，包含 Django admin 的用户数据
|
||||
|
||||
+1
-1
@@ -98,7 +98,7 @@ uwsgi --stop uwsgi.pid
|
||||
| `news/reports/` | report_type, start_date, end_date, id | 日报查询（默认最近 24h；传 id 返回单份详情含事件） |
|
||||
| `news/events/` | days, importance, report_type, section, limit | 重要事件聚合（跨日报，最近 N 天 importance≥阈值） |
|
||||
|
||||
日报查询接口详细说明见 [`docs/news_report_api.md`](../docs/news_report_api.md)（表结构见 `djapi/docs/db_schema.md`）。
|
||||
日报查询接口详细说明见 [`docs/news_report_api.md`](../docs/news_report_api.md)（表结构见 `../docs/db_schema_v1.1.md`）。
|
||||
|
||||
API 文档（Swagger）：`/api/docs/`
|
||||
OpenAPI Schema：`/api/schema/`
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
news_report / news_event 只读查询层（日报结构化入库，见 docs/db_schema.md）。
|
||||
news_report / news_event 只读查询层（日报结构化入库，见 docs/db_schema_v1.1.md）。
|
||||
|
||||
连接配置来自环境变量（与 docs/report_db_design.md §7 保持一致）：
|
||||
连接配置来自环境变量（与 docs/db_schema_v1.1.md 保持一致）：
|
||||
NEWS_DB_HOST / NEWS_DB_PORT / NEWS_DB_USER / NEWS_DB_PASSWORD / NEWS_DB_NAME
|
||||
NEWS_DB_PASSWORD 缺失时直接报错，禁止默认密码。
|
||||
|
||||
@@ -45,6 +45,18 @@ def _connect():
|
||||
return mysql.connector.connect(**load_db_config())
|
||||
|
||||
|
||||
def _parse_json(value):
|
||||
"""把 JSON 字符串列（如 sources）解析为 dict/list；已是对象则原样返回。"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return json.loads(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def _row_to_dict(row: dict) -> dict:
|
||||
"""序列化行：stats JSON 解析、日期/时间转 ISO 字符串。"""
|
||||
d = dict(row)
|
||||
@@ -87,11 +99,14 @@ def fetch_reports(
|
||||
report = _row_to_dict(row)
|
||||
cur.execute(
|
||||
"SELECT id, section, rank, importance, event_type, title, "
|
||||
"summary, sentiment, source, url "
|
||||
"summary, sentiment, source, sources, url "
|
||||
"FROM news_event WHERE report_id = %s ORDER BY section, rank",
|
||||
(report_id,),
|
||||
)
|
||||
report["events"] = [dict(r) for r in cur.fetchall()]
|
||||
report["events"] = [
|
||||
{**dict(r), "sources": _parse_json(r["sources"])}
|
||||
for r in cur.fetchall()
|
||||
]
|
||||
return report
|
||||
|
||||
where, params = [], []
|
||||
@@ -153,7 +168,7 @@ def fetch_important_events(
|
||||
sql = (
|
||||
"SELECT r.report_date, r.report_type, e.id, e.section, e.rank, "
|
||||
"e.importance, e.event_type, e.title, e.summary, e.sentiment, "
|
||||
"e.source, e.url "
|
||||
"e.source, e.sources, e.url "
|
||||
"FROM news_event e "
|
||||
"JOIN news_report r ON r.id = e.report_id "
|
||||
"WHERE " + " AND ".join(where)
|
||||
@@ -163,6 +178,10 @@ def fetch_important_events(
|
||||
)
|
||||
params.append(int(limit))
|
||||
cur.execute(sql, tuple(params))
|
||||
return [dict(r) for r in cur.fetchall()]
|
||||
rows = cur.fetchall()
|
||||
return [
|
||||
{**dict(r), "sources": _parse_json(r["sources"])}
|
||||
for r in rows
|
||||
]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
@@ -14,6 +14,7 @@ class EventSerializer(serializers.Serializer):
|
||||
summary = serializers.CharField(allow_null=True)
|
||||
sentiment = serializers.CharField(allow_null=True)
|
||||
source = serializers.CharField(allow_null=True)
|
||||
sources = serializers.JSONField(allow_null=True)
|
||||
url = serializers.CharField(allow_null=True)
|
||||
|
||||
|
||||
@@ -47,4 +48,5 @@ class ImportantEventSerializer(serializers.Serializer):
|
||||
summary = serializers.CharField(allow_null=True)
|
||||
sentiment = serializers.CharField(allow_null=True)
|
||||
source = serializers.CharField(allow_null=True)
|
||||
sources = serializers.JSONField(allow_null=True)
|
||||
url = serializers.CharField(allow_null=True)
|
||||
|
||||
@@ -122,6 +122,17 @@ class NewsEventsAPITest(TestCase):
|
||||
self.assertEqual(kwargs['report_type'], 'intl')
|
||||
self.assertEqual(kwargs['section'], 'intl')
|
||||
|
||||
@patch('api.report.query.fetch_important_events',
|
||||
return_value=[{'id': 1, 'title': 'x', 'source': 'yicai',
|
||||
'sources': ['yicai', 'stcn']}])
|
||||
def test_sources_field_passthrough(self, mock_fetch):
|
||||
"""事件聚合响应原样透传 sources（JSON 数组）"""
|
||||
resp = self.client.get(self.url)
|
||||
self.assertEqual(resp.status_code, 200)
|
||||
body = resp.json()
|
||||
self.assertEqual(body[0]['sources'], ['yicai', 'stcn'])
|
||||
self.assertEqual(body[0]['source'], 'yicai')
|
||||
|
||||
def test_invalid_days(self):
|
||||
resp = self.client.get(self.url, {'days': 'abc'})
|
||||
self.assertEqual(resp.status_code, 400)
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
# continuation.md
|
||||
|
||||
## 当前项目状态
|
||||
|
||||
djapi — Django 5.2 金融数据 API 项目，2026-06-17 已部署。
|
||||
|
||||
## Checkpoint 记录
|
||||
|
||||
| 日期 | 内容 |
|
||||
|------|------|
|
||||
| 2026-08-03 | 新增日报查询 API ×2（news/reports/ + news/events/），基于 news_report/news_event 表，已部署 doorcome ✅ |
|
||||
|
||||
服务器：`simon@doorcome.cn`，路径 `/home/simon/myquant/djapi/`，虚拟环境 `/opt/miniconda/envs/django/`。
|
||||
|
||||
## 已完成
|
||||
|
||||
### 1. 安全：密钥统一管理
|
||||
- 所有密钥 → 环境变量，`.env` 统一管理
|
||||
- `djapi/env_loader.py`（Django 端）+ `api/video/env.py`（video 端）双加载器
|
||||
- 共享 `MySQLDB` → `api/utils/mysql_handler.py`
|
||||
- `.env.example`、`.gitignore`
|
||||
|
||||
### 2. 代码质量
|
||||
- `api/views.py`：227 → ~130 行，消除重复
|
||||
- `api/stock/stock_utils.py`：`viewFunc_singleParam()` 包装器
|
||||
- `api/stock/config.py`：拆分为 config / strategy_config / scan_config
|
||||
|
||||
### 3. drf-spectacular 集成
|
||||
- 14 端点 `@api_view` + `@extend_schema`，8 tag 分组
|
||||
- 13 Serializer，Swagger `/api/docs/`
|
||||
|
||||
### 4. 股息率 API 优化
|
||||
- **删除** `api/stock/getDivData_AK.py`（akshare 版），`/api/getdivak/` 路由移除
|
||||
- **优化** `api/stock/getStockDiv2.py`：
|
||||
- TTM 计算：`calculate_ttm_div` 行级循环 O(n²) → `rolling('360D').sum()` O(n)
|
||||
- 删除向前填充逻辑（~30 行），避免与毛刺平滑冲突
|
||||
- **修复** `api/stock/smoothBrush.py`：if/elif 分支中 prev_valid/next_valid 赋值反了
|
||||
|
||||
### 5. video 模块重构与 Bug 修复
|
||||
- 新增 `api/video/env.py` — .env 加载
|
||||
- `newsRedo.py` 重写 — 三分支智能重处理
|
||||
- **P0 修复**：`getVideo5.py` 日期校验 bug（`start_date > start_date` → `start_date > end_date`）
|
||||
- **P1 清理**：删除 `ai.py`（两个函数均为死代码），清理 `newsProcess.py` 冗余 import
|
||||
- **P2 修复**：`audioRead.py` — `transcribe_audio` 异常时返回 `['', '']` 统一类型；`analyze_and_correct_text` 防御 None
|
||||
- **P3 修复**：`newsProcess.py` — `news_to_db()` JSON 解析自适应 dict/list（DeepSeek json_object 模式返回 dict 包装）
|
||||
|
||||
### 6. 文档与测试
|
||||
- `CLAUDE.md`、`README.md`、`continuation.md`
|
||||
- 18 个单元测试
|
||||
|
||||
## video 目录文件现状（10 个 .py）
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `env.py` | .env 加载 |
|
||||
| `getVideo5.py` | 主流程：抓取→下载→ASR→入库 |
|
||||
| `audioRead.py` | 音频转换、分割、ASR 识别、文本纠错 |
|
||||
| `deepseek.py` | DeepSeek API 封装（类 + `deepseek_text` 函数，支持 `response_format`） |
|
||||
| `newsProcess.py` | AI 新闻分割+标题提取，JSON 自适应解析 |
|
||||
| `newsRedo.py` | 手动重处理（三分支） |
|
||||
| `main.py` | 定时任务入口（当天） |
|
||||
| `main_videos.py` | 批量补缺（扫描缺失日期） |
|
||||
| `mysqlHandle.py` | MySQLDB 重新导出 |
|
||||
|
||||
## 所有 API 端点（16 个）
|
||||
|
||||
| 端点 | 数据源 | 说明 |
|
||||
|------|--------|------|
|
||||
| `stockbasic/` | Tushare | 日线行情 |
|
||||
| `stockinfo/` | Tushare | 个股基本信息 |
|
||||
| `industrys/` | Tushare | 行业股票列表 |
|
||||
| `stockparam/` | Tushare | 个股参数 |
|
||||
| `stockep/` | Tushare | TTM EPS |
|
||||
| `quarterlyEps/` | Tushare | 季度 EPS |
|
||||
| `indexByName/` | Tushare | 指数查询 |
|
||||
| `indexDatas/` | Tushare | 指数行情 |
|
||||
| `dailymargin/` | Tushare | 每日融资融券汇总 |
|
||||
| `stockmargin/` | Tushare | 个股融资融券 |
|
||||
| `finance/` | Tushare | 财务报表分析 |
|
||||
| `getdiv/` | Tushare | 股息率（TTM rolling + 毛刺平滑） |
|
||||
| `xwlbNews/` | MySQL | 新闻联播原始文本 |
|
||||
| `xwlbFine/` | MySQL | 新闻联播 AI 精编 |
|
||||
| `news/reports/` | MySQL (news_) | 日报查询：默认最近 24h；传 id 返回详情含事件 |
|
||||
| `news/events/` | MySQL (news_) | 重要事件聚合：最近 N 天 importance≥阈值 |
|
||||
|
||||
---
|
||||
|
||||
## 日报查询 API（2026-08-03 新增）
|
||||
|
||||
### 模块
|
||||
|
||||
- 新增 `api/report/` 包（独立于 stock）：`query.py`（连库+查询 SQL）/ `views.py`（2 视图）/ `serializers.py`（OpenAPI）/ `tests.py`（17 个单测，mock 查询层）
|
||||
- `api/urls.py` 注册 `news/reports/`、`news/events/`；`settings.py` SPECTACULAR TAGS 加「日报」
|
||||
- 数据库：doorcome 本机 MariaDB `myquant` 库 `news_report`（180 行）+ `news_event`（4372 条），与现有 `MYSQL_*` 同库同用户
|
||||
- 连接配置：服务器 `djapi/.env` 新增 `NEWS_DB_*`（复用 MYSQL_* 值，密码必填否则 500）
|
||||
- 文档：`docs/news_report_api.md`（使用手册，含线上地址/curl/真实样例）；README API 概览表已加两行
|
||||
|
||||
### 部署（2026-08-03 完成）
|
||||
|
||||
- rsync 增量同步（**未用文档中的 --delete**，见下）→ 重启 uWSGI → 冒烟通过（列表/详情/聚合/400/404）
|
||||
- 线上：`https://api.doorcome.cn/api/news/reports/`、`/api/news/events/`，Swagger `/api/docs/`「日报」tag
|
||||
|
||||
### 已知事项
|
||||
|
||||
1. **服务器顶层历史平铺文件未清理**：`/home/simon/myquant/djapi/` 顶层有 views.py/urls.py/smoothBrush.py/getStockDiv2.py/env_loader.py/akshare_data.py（历史 rsync 陷阱产物），`--delete` 会删除它们，但 `divSearch.py`（离线脚本）仍绝对导入顶层 getStockDiv2/smoothBrush → 本次增量同步保留；清理前需先修 divSearch.py 的导入
|
||||
2. **既有失败测试**：`api.tests.DateFormatCorrectionTest.test_empty_string`（date_format_correction('') 期望 None 实得 ''），与本次无关
|
||||
3. 冒烟曾发现 fetch_reports 列表 SQL 缺 `r.` 别名前缀（1052 ambiguous），已修复
|
||||
4. 本地验证需绕过 macOS TCC：`HOME=/tmp/djtest_home PYTHONPATH=/tmp/djtest_pkgs`（tushare 写 ~/tk.csv 被拦 + quant 环境缺 mysql-connector-python）
|
||||
|
||||
## 部署
|
||||
|
||||
```bash
|
||||
# 全量同步
|
||||
rsync -avz --delete \
|
||||
--exclude='.env' --exclude='db.sqlite3' \
|
||||
--exclude='*.log' --exclude='uwsgi.pid' \
|
||||
--exclude='__pycache__/' --exclude='*.pyc' \
|
||||
--exclude='xwlb_video/' --exclude='audio_processing/' \
|
||||
/Users/summer/Downloads/cc-cursor/djapi/ \
|
||||
simon@doorcome.cn:/home/simon/myquant/djapi/
|
||||
|
||||
# 单文件同步必须写完整路径
|
||||
# 正确：rsync api/views.py simon@...:/.../djapi/api/views.py
|
||||
|
||||
# 重启
|
||||
ssh simon@doorcome.cn "kill \$(lsof -ti:5004); sleep 2; /opt/miniconda/envs/django/bin/uwsgi --ini /home/simon/myquant/djapi/uwsgi.ini"
|
||||
```
|
||||
|
||||
## 关键设计决策
|
||||
|
||||
- video 模块保护、向后兼容优先、不使用 python-dotenv
|
||||
- rsync 陷阱：多文件源会展平路径
|
||||
- `.env` 双加载：Django 端 `djapi/env_loader.py` + video 端 `api/video/env.py`
|
||||
- 股息率 TTM 用 `rolling('360D').sum()` 向量化，不手动循环
|
||||
- DeepSeek json_object 模式返回 dict，newsProcess 自适应提取 list
|
||||
@@ -1,111 +0,0 @@
|
||||
# 日报结构化入库：数据库表结构与数据契约
|
||||
|
||||
> 版本：v1.0 | 2026-08-03
|
||||
> 用途：供 API / 前端对接读取日报数据。表位于 MySQL `myquant` 库，表前缀 `news_`。
|
||||
> 连接：`192.168.1.10:13306`（pi 上 autossh 隧道 → doorcome.cn:3306 MariaDB 10.11），用户 `myquant`（密码在服务器 `.env` 的 `NEWS_DB_PASSWORD`）。
|
||||
|
||||
---
|
||||
|
||||
## 1. 表结构
|
||||
|
||||
### 1.1 news_report（日报主表，一行 = 一份日报）
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| id | BIGINT UNSIGNED PK | 自增主键 |
|
||||
| report_date | DATE | 日报日期 |
|
||||
| report_type | VARCHAR(16) | `finance`=A 股日报 / `intl`=国际财经日报 |
|
||||
| file_name | VARCHAR(160) | 历史文件源文件名；**新生成日报为空字符串 `""`** |
|
||||
| generated_at | DATETIME | 生成时间 |
|
||||
| ai_summary | TEXT | AI 摘要全文（含换行，按条目分行） |
|
||||
| stats | JSON | 数据总览统计快照（见第 3 节），可为 NULL |
|
||||
| created_at | DATETIME | 入库时间 |
|
||||
|
||||
唯一键：`(report_date, report_type, file_name)` —— 历史同一天多次生成（intl 一日 3 次）保留多行；新生成日报 `file_name=''` 每天每类型仅一行，重复生成覆盖。
|
||||
|
||||
### 1.2 news_event（日报事件明细，一行 = 一条事件）
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| id | BIGINT UNSIGNED PK | 自增主键 |
|
||||
| report_id | BIGINT UNSIGNED | FK → news_report.id |
|
||||
| section | VARCHAR(16) | 板块：`xwlb`=新闻联播 / `news`=财经新闻 / `cninfo`=公告调研 / `intl`=国际重要事件 |
|
||||
| rank | INT | 板块内序号（1 起） |
|
||||
| importance | INT NULL | 重要度 1-5 |
|
||||
| event_type | VARCHAR(64) NULL | 事件类型（如 宏观经济/地缘政治/新闻联播/公告） |
|
||||
| title | VARCHAR(512) | 标题 |
|
||||
| summary | TEXT NULL | 摘要/正文 |
|
||||
| sentiment | VARCHAR(8) NULL | `positive` / `negative` / `neutral` |
|
||||
| source | VARCHAR(64) NULL | 来源（如 `cls`、`investinglive.com`） |
|
||||
| url | VARCHAR(512) NULL | 原文链接（新闻联播为空） |
|
||||
| created_at | DATETIME | 入库时间 |
|
||||
|
||||
索引：`idx_report_section (report_id, section)`。
|
||||
|
||||
---
|
||||
|
||||
## 2. 数据契约
|
||||
|
||||
- **幂等语义**：同一 `(report_date, report_type, file_name)` 重复写入会覆盖主表并全量替换事件（DELETE + INSERT），不会产生重复行。
|
||||
- **取最新**：同一天存在多份时（历史 intl 一日 3 次），前端按 `generated_at` 取最新；新日报 `file_name=''` 每天唯一。
|
||||
- **板块差异**：finance 日报含 `xwlb`+`news`+`cninfo` 三板块；intl 日报仅 `intl` 板块。前端按 `section` 过滤展示。
|
||||
- **历史覆盖范围**：2026-06-16 ~ 2026-08-03，共 177 行（finance 49 + intl 128；finance 少 1 因为两个目录存在同名文件被幂等合并）。事件总计 4222 条。
|
||||
|
||||
---
|
||||
|
||||
## 3. stats JSON 结构
|
||||
|
||||
`news_report.stats` 为数据总览快照，前端自行解析。finance 与 intl 的 key 集合不同：
|
||||
|
||||
| key | finance | intl | 内容 |
|
||||
| --- | --- | --- | --- |
|
||||
| `pipeline` | ✅ | ✅ | M1→M6 管道各环节数量：`{label: 数量}` |
|
||||
| `sources` | ✅ | — | 各新闻源文章数：`{源名: 数量}` |
|
||||
| `news` | ✅ | — | 新闻统计：`{total, hi_threshold, sentiments, importances, event_types}` |
|
||||
| `cninfo` | ✅ | — | 公告调研统计：`{total, hi_threshold, by_day, announcement, research, irm}` |
|
||||
| `xwlb` | ✅ | — | 联播统计：`{total, date}`（有数据时才有） |
|
||||
| `sentiment` | ✅ | ✅ | 情绪分布（历史文件为图例文本列表；新生成在 `news.sentiments`） |
|
||||
| `importance` | ✅ | ✅ | 重要度分布：`[{重要度, 数量}, ...]` |
|
||||
| `event_types` | ✅ | ✅ | 事件类型 TOP：`[{事件类型, 数量}, ...]` |
|
||||
| `source_dist` | — | ✅ | 文章来源分布：`[{来源, 文章数}, ...]` |
|
||||
|
||||
> 历史文件与新生成日报的 stats 结构存在差异（历史为 HTML 解析快照，新生成为结构化组装），前端建议按 key 防御性读取。
|
||||
|
||||
---
|
||||
|
||||
## 4. 常用查询示例（API 实现参考）
|
||||
|
||||
```sql
|
||||
-- 某类型日报列表（取每天最新一份）
|
||||
SELECT r.* FROM news_report r
|
||||
JOIN (
|
||||
SELECT report_date, report_type, MAX(generated_at) AS g
|
||||
FROM news_report GROUP BY report_date, report_type
|
||||
) t ON r.report_date = t.report_date AND r.report_type = t.report_type
|
||||
AND r.generated_at = t.g
|
||||
WHERE r.report_type = 'finance' AND r.report_date >= '2026-07-01'
|
||||
ORDER BY r.report_date DESC;
|
||||
|
||||
-- 某日报的全部事件（按板块）
|
||||
SELECT section, rank, importance, event_type, title, summary, sentiment, source, url
|
||||
FROM news_event WHERE report_id = ? ORDER BY section, rank;
|
||||
|
||||
-- 最近 N 天重要事件聚合（跨日报检索）
|
||||
SELECT e.* FROM news_event e
|
||||
JOIN news_report r ON r.id = e.report_id
|
||||
WHERE r.report_date >= DATE_SUB(CURDATE(), INTERVAL 7 DAY)
|
||||
AND e.importance >= 4
|
||||
ORDER BY e.importance DESC, r.report_date DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 相关命令（数据生产侧）
|
||||
|
||||
```bash
|
||||
uv run a-share report --date YYYYMMDD # 生成当日日报并入库（finance）
|
||||
uv run a-share report-import # 历史 HTML 全量解析入库（幂等）
|
||||
uv run a-share report-import --date YYYYMMDD --type intl
|
||||
```
|
||||
|
||||
代码：`report_db/`（连接/写入）、`report_import/`（历史解析/导入）、`scheduler/reporter.py`（日报生成）。
|
||||
@@ -1,330 +0,0 @@
|
||||
# Milestone 10 后端实现逻辑：日报结构化入库
|
||||
|
||||
> 版本：v0.1（设计稿） | 2026-07
|
||||
> 对应 project_plan.md「十八、Milestone 10」
|
||||
> **范围**：本项目侧"后端"= 数据生产层（日报内容生成 + 结构化写入 MySQL）。
|
||||
> 不包含 API 服务与前端页面（由用户另行实现），但表结构与数据契约以本文档为准，供 API/前端对接。
|
||||
|
||||
---
|
||||
|
||||
## 1. 定位
|
||||
|
||||
现有链路：`reporter.py` 收集数据 → `_render_html()` 渲染 HTML → scp 上传 doorcome。
|
||||
改造后：`reporter.py` 收集数据 → 组装结构化 `ReportData` → 写入 MySQL（`news_report` / `news_event`），不再产出 HTML。
|
||||
|
||||
另需：把 doorcome 上 178 份历史日报 HTML（`finance_news_daily_*` ×50、`intl_news_daily_*` ×128）解析成同一 `ReportData` 结构入库。
|
||||
|
||||
---
|
||||
|
||||
## 2. 数据流总览
|
||||
|
||||
```
|
||||
[历史 HTML ×178] [每日 pipeline]
|
||||
doorcome:/var/www/html/echart/research/ crawler→extractor→dedup→llm→embed→qdrant
|
||||
（一次性 scp 到 data/reports_history/） │
|
||||
│ ▼
|
||||
▼ reporter.generate_report()
|
||||
report_import/parser.py │
|
||||
│ （BeautifulSoup 解析） ▼
|
||||
▼ 组装 ReportData 组装 ReportData
|
||||
report_import/importer.py │
|
||||
│ （幂等 upsert） ▼
|
||||
▼ │
|
||||
┌────────────────────── MySQL (myquant 库) ──────────────────────┐
|
||||
│ news_report（主表） news_event（事件明细） │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
▲
|
||||
API / 前端（用户另行实现，只读）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据模型（Pydantic，`report_db/models.py`）
|
||||
|
||||
```python
|
||||
class EventRow(BaseModel):
|
||||
"""一条事件记录，对应 news_event 一行。"""
|
||||
section: str # xwlb | news | cninfo | intl
|
||||
rank: int # 板块内序号（从 1 开始）
|
||||
importance: int | None = None
|
||||
event_type: str | None = None
|
||||
title: str
|
||||
summary: str | None = None
|
||||
sentiment: str | None = None # positive | negative | neutral | ''
|
||||
source: str | None = None # 来源（如 cls / ForexLive）
|
||||
url: str | None = None
|
||||
|
||||
class ReportData(BaseModel):
|
||||
"""一份完整日报，对应 news_report 一行 + news_event 多行。"""
|
||||
report_date: date # 日报日期（YYYY-MM-DD）
|
||||
report_type: str # finance | intl
|
||||
file_name: str # 源文件名（新生成时可为 ""）
|
||||
generated_at: datetime # 生成时间
|
||||
ai_summary: str | None = None
|
||||
stats: dict[str, Any] = Field(default_factory=dict) # 数据总览统计快照 → JSON 列
|
||||
events: list[EventRow] = Field(default_factory=list)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 字段映射（核心契约）
|
||||
|
||||
### 4.1 事件 JSON（data/events/）→ news_event
|
||||
|
||||
现有事件文件结构与 news_event 字段对应关系（reporter 收集时直接转换）：
|
||||
|
||||
| news_event 字段 | 事件 JSON 来源 |
|
||||
| --- | --- |
|
||||
| section | 来源判定：`source_id=="cninfo"` → `cninfo`；`source_id=="xwlb"` → `xwlb`；否则 `news`；intl 解析固定 `intl` |
|
||||
| importance | `event.importance` |
|
||||
| event_type | `event.event_type` |
|
||||
| title | `title` |
|
||||
| summary | `event.summary` |
|
||||
| sentiment | `event.sentiment` |
|
||||
| source | `source_id` |
|
||||
| url | `url`（xwlb 为空） |
|
||||
|
||||
### 4.2 历史 HTML → ReportData
|
||||
|
||||
解析策略：**表头驱动列映射**。不同日报表格列集合不同：
|
||||
|
||||
| 板块 | 表格列（<th>） | section |
|
||||
| --- | --- | --- |
|
||||
| 新闻联播（finance） | `# / (空) / 标题 / 重要度 / 事件类型` | xwlb |
|
||||
| 重要事件：新闻（finance） | `# / (空) / 标题 / 源 / 重要度 / 事件类型 / 摘要` | news |
|
||||
| 重要事件：公告调研（finance） | 同上 | cninfo |
|
||||
| 重要事件（intl） | `# / (空) / 标题 / 重要度 / 事件类型 / 摘要` | intl |
|
||||
|
||||
要点：
|
||||
|
||||
- 以表头文本定位列索引（"标题""重要度""事件类型""摘要""源"），空 `<th>` 为情绪图标列（⚪/🔴/🟢 → neutral/negative/positive），**不要依赖列位置**。
|
||||
- 情绪图标仅存在于有图标列的表；intl 表情绪列存在，finance 表情绪列存在（空 th 首列后）。
|
||||
- intl 无"源"列时，尝试从标题尾部 `[来源]` 或摘要尾部提取，提取不到则 `source=None`。
|
||||
- 标题中的股票代码标注 `(600519, ...)` 与 ⭐（自选股标记）需剥除，只保留纯标题。
|
||||
- AI 摘要：取 `h2`（"一、AI 摘要"）之后紧随的 `.ai-summary` 区块纯文本（保留换行）。
|
||||
- 数据总览 → `stats` JSON：按 `h3` 标题映射 key（见 4.3），解析该 h3 后的首个 `<table>`，缺失的板块跳过、不报错。
|
||||
- 容错：任一板块解析失败 → 记 WARNING 日志，该板块置空，不影响整份入库；整份文件解析失败 → 抛 `ReportParseError`（由 importer 捕获计数）。
|
||||
|
||||
### 4.3 数据总览 → stats JSON
|
||||
|
||||
| h3 标题（含板块名） | stats key |
|
||||
| --- | --- |
|
||||
| M1→M6 管道 / 管道 | `pipeline`（保留原始行） |
|
||||
| 各源数据 | `sources` |
|
||||
| 情绪分布 | `sentiment` |
|
||||
| 重要度分布 | `importance` |
|
||||
| 事件类型（TOP 10 / 分布） | `event_types` |
|
||||
| 文章来源分布 | `source_dist` |
|
||||
|
||||
`stats` 存 MySQL `JSON` 列，前端自行解析展示。历史文件与未来新日报的 stats 结构可能不同（finance 与 intl 板块不同），一律按快照存储，不做跨版本规范化。
|
||||
|
||||
---
|
||||
|
||||
## 5. 模块设计
|
||||
|
||||
### 5.1 新包 `report_db/`（DB 层）
|
||||
|
||||
```
|
||||
report_db/
|
||||
├── __init__.py # 导出 connect / init_schema / save_report
|
||||
├── models.py # EventRow / ReportData（Pydantic）
|
||||
├── schema.py # DDL 常量（news_report / news_event，见 project_plan.md 十八）
|
||||
└── db.py # 连接、事务、写入
|
||||
```
|
||||
|
||||
`db.py` 关键函数：
|
||||
|
||||
```python
|
||||
def load_db_config() -> DbConfig:
|
||||
"""从环境变量读取 NEWS_DB_HOST/PORT/USER/PASSWORD/NAME。
|
||||
缺失 PASSWORD 时记 ERROR 并 raise，禁止默认密码。"""
|
||||
|
||||
def connect(cfg: DbConfig) -> Connection:
|
||||
"""pymysql.connect(autocommit=False, charset="utf8mb4", cursorclass=DictCursor)。
|
||||
失败时 logger.exception + raise。"""
|
||||
|
||||
def init_schema(conn: Connection) -> None:
|
||||
"""执行 schema.py 中的 CREATE TABLE IF NOT EXISTS ×2。"""
|
||||
|
||||
def save_report(conn: Connection, report: ReportData) -> int:
|
||||
"""事务内：
|
||||
1. INSERT INTO news_report (...) VALUES (...) 或按 (report_date, report_type, file_name)
|
||||
唯一键命中时 UPDATE（新生成日报重复执行 = 覆盖同 file_name/同日期，幂等）；
|
||||
2. 取 report_id，DELETE 旧事件后批量 INSERT news_event（保证整份覆盖一致）。
|
||||
返回 report_id。"""
|
||||
|
||||
def transaction(conn: Connection) -> contextmanager:
|
||||
"""提交/回滚上下文管理器。"""
|
||||
|
||||
def fetch_report(conn: Connection, report_id: int) -> dict | None:
|
||||
"""读侧辅助（联调/测试用），API 侧由用户自行实现。"""
|
||||
```
|
||||
|
||||
要点：
|
||||
|
||||
- 所有 SQL 为 MySQL/MariaDB 方言（`JSON` 列、`ENGINE=InnoDB`、`COMMENT`），**不依赖 ORM**。
|
||||
- 连接生命周期：每次 `save_report` 短连接（report 一天跑几次，量小，无需连接池；如未来加大再换）。
|
||||
- 字符集 utf8mb4，`SET NAMES utf8mb4` 由 pymysql charset 参数处理。
|
||||
|
||||
### 5.2 新包 `report_import/`（历史解析）
|
||||
|
||||
```
|
||||
report_import/
|
||||
├── __init__.py
|
||||
├── parser.py # parse_finance_report / parse_intl_report（BeautifulSoup）
|
||||
└── importer.py # import_history(dir, date=None, type=None) -> ImportStats
|
||||
```
|
||||
|
||||
`parser.py`：
|
||||
|
||||
```python
|
||||
class ReportParseError(Exception): ...
|
||||
|
||||
def parse_finance_report(html: str, file_name: str) -> ReportData: ...
|
||||
def parse_intl_report(html: str, file_name: str) -> ReportData: ...
|
||||
def parse_report(html: str, file_name: str) -> ReportData:
|
||||
"""按文件名前缀分流：finance_news_daily_* / intl_news_daily_*。"""
|
||||
```
|
||||
|
||||
- 依赖复用现有 `beautifulsoup4`（已在 pyproject 依赖），**不新增解析库**。
|
||||
- `report_date` 从文件名解析（`*_daily_{YYYYMMDD}_*.html`），不信任目录名。
|
||||
- `generated_at` 从文件名时间（`{HHMMSS}`）或 `<header>` 中"生成于"文本解析，解析不到用文件 mtime。
|
||||
|
||||
`importer.py`：
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ImportStats:
|
||||
scanned: int = 0 # 扫描到的日报文件数
|
||||
imported: int = 0 # 新入库
|
||||
skipped: int = 0 # 已存在（幂等跳过）
|
||||
failed: int = 0 # 解析失败
|
||||
errors: list[str] = field(default_factory=list)
|
||||
|
||||
def import_history(report_dir: Path, date: str | None = None,
|
||||
report_type: str | None = None) -> ImportStats:
|
||||
"""遍历 {report_dir}/{YYYYMMDD}/*_news_daily_*.html，
|
||||
过滤 date / type，逐个 parse → save_report。"""
|
||||
```
|
||||
|
||||
### 5.3 `scheduler/reporter.py` 改造（完全切换）
|
||||
|
||||
- 新增 `_build_report_data(news, cninfo, pipeline, ai_summary, day_str, xwlb) -> ReportData`：
|
||||
- 事件转换：`news["high"]` → `EventRow(section="news", ...)`；`cninfo["high"]` → `section="cninfo"`；`xwlb["items"]` → `section="xwlb"`；
|
||||
- `stats` 组装：`{"pipeline": pipeline, "sources": {...}, "sentiment": news["sentiments"], "importance": news["importances"], "event_types": news["event_types"], "cninfo": {...}}`；
|
||||
- 事件 `rank` 按板块内顺序编号。
|
||||
- `generate_report(day_str, *, upload=True)` 改为：收集（逻辑不变）→ `_build_report_data` → `connect()` + `save_report()`；删除 `_render_html`/`_upload` 调用。
|
||||
- `_render_*` 函数**保留但标记 deprecated**（注释说明"完全切换后不再调用"），不删除，保证最小改动、可回退。
|
||||
- 返回值由 `Path | None` 改为 `report_id: int | None`；`scheduler/pipeline.py` 中 report 步骤仅判断非 None（实施时核实该处调用，保持兼容）。
|
||||
- `stock_reporter.py` **不改动**（个股日报不在本期范围）。
|
||||
|
||||
### 5.4 `a_share_cli/main.py` 新增子命令
|
||||
|
||||
```
|
||||
uv run a-share report-import [--dir data/reports_history] [--date YYYYMMDD] [--type finance|intl]
|
||||
```
|
||||
|
||||
- 默认全量扫描 `REPORT_HISTORY_DIR`（.env 可配，默认 `data/reports_history/`）。
|
||||
- 输出 ImportStats 汇总（扫描/导入/跳过/失败）。
|
||||
|
||||
---
|
||||
|
||||
## 6. 关键流程
|
||||
|
||||
### 6.1 历史导入（一次执行，可重复）
|
||||
|
||||
```
|
||||
1. scp -r doorcome:/var/www/html/echart/research/2026* → data/reports_history/
|
||||
（一次手工操作，不进代码）
|
||||
2. uv run a-share report-import
|
||||
for each {date}/{file}:
|
||||
report_type = 文件名前缀（finance|intl）
|
||||
ReportData = parse_report(html, file_name)
|
||||
try: save_report(conn, ReportData) → imported += 1
|
||||
except DuplicateKey: skipped += 1 # 已导入过
|
||||
except ReportParseError as e: failed += 1; errors.append(str(e))
|
||||
3. 校验: SELECT report_type, COUNT(*) FROM news_report GROUP BY report_type
|
||||
期望 50 / 128
|
||||
```
|
||||
|
||||
### 6.2 每日日报生成（pipeline 07:00 步骤）
|
||||
|
||||
```
|
||||
generate_report(day_str):
|
||||
news = _collect_news_events(day_str) # 不变
|
||||
cninfo = _collect_cninfo_events(day_str) # 不变
|
||||
xwlb = _collect_xwlb(day_str) # 不变
|
||||
pipeline = _collect_pipeline_stats(day_str) # 不变
|
||||
ai_summary = _generate_ai_summary(...) # 不变
|
||||
report = _build_report_data(...) # 新增
|
||||
save_report(connect(), report) # 新增（替代渲染+上传）
|
||||
```
|
||||
|
||||
### 6.3 幂等策略
|
||||
|
||||
- 唯一键 `(report_date, report_type, file_name)`：
|
||||
- 历史导入：命中 → 跳过（或 `--force` 覆盖）；
|
||||
- 新日报：`file_name=""` 时唯一键退化为 `(report_date, report_type, "")`，同一天重复跑 → UPDATE 覆盖，事件表 DELETE+INSERT 全量替换，**不产生历史残留**。
|
||||
|
||||
---
|
||||
|
||||
## 7. 配置项（.env / .env.example）
|
||||
|
||||
```env
|
||||
# ---- 日报结构化入库 (M10) ----
|
||||
NEWS_DB_HOST=127.0.0.1 # 开发走 ssh 隧道: ssh -L 13306:127.0.0.1:13306 pi
|
||||
NEWS_DB_PORT=13306
|
||||
NEWS_DB_USER=myquant
|
||||
NEWS_DB_PASSWORD= # 填真实值，禁止写入源码/文档
|
||||
NEWS_DB_NAME=myquant
|
||||
REPORT_HISTORY_DIR=data/reports_history
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 错误处理
|
||||
|
||||
| 场景 | 行为 |
|
||||
| --- | --- |
|
||||
| DB 不可达/凭据错误 | `connect()` 抛异常 → `generate_report` 记 ERROR 并返回 None（pipeline 该步骤失败，其余步骤不受影响） |
|
||||
| 单份历史文件解析失败 | 记 WARNING，`failed += 1`，继续下一份；结束输出失败清单 |
|
||||
| 事件字段缺失（如无摘要列） | 对应字段留 None，不抛错 |
|
||||
| 全部失败 | `report-import` 返回非 0 退出码，便于排查 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 测试策略（tests/）
|
||||
|
||||
| 文件 | 内容 |
|
||||
| --- | --- |
|
||||
| `tests/test_report_parser.py` | 用 fixtures（从 178 份中拷贝 finance/intl 各 1 份真实样例到 `tests/fixtures/`）断言：板块数、事件行数、字段映射、标题净化、幂等文件日期解析 |
|
||||
| `tests/test_report_db.py` | 纯逻辑：`_build_report_data` 组装正确；SQL 层用 sqlite3 内存库建同构（简化 DDL）验证 upsert/覆盖语义 |
|
||||
| `tests/test_report_import.py` | 临时目录构造 2-3 份假 HTML → 全流程导入 → 断言 ImportStats 计数与幂等 |
|
||||
| 集成（`@pytest.mark.integration`，默认跳过） | 连真实 MySQL：init_schema + save_report + 查询回读 |
|
||||
|
||||
新增 pytest marker 说明：真实 DB 连接一律走 integration，**单元测试不得依赖生产库**。
|
||||
|
||||
---
|
||||
|
||||
## 10. 依赖变更
|
||||
|
||||
- `uv add pymysql`（纯 Python 驱动，唯一新增依赖）
|
||||
- 解析复用现有 `beautifulsoup4`，不新增
|
||||
|
||||
---
|
||||
|
||||
## 11. 开放问题（沿自 project_plan.md 十八，不阻塞开发）
|
||||
|
||||
1. 生产连接：pi5 无法直连 `192.168.1.10:13306`（隧道仅绑 loopback）——需决定改 pi 的 autossh 绑定 / pi5 自建隧道。
|
||||
2. intl 日报生成方不在本项目，未来 intl 新日报需按同一表结构写入（本项目仅负责解析历史 + finance 新日报）。
|
||||
3. 个股日报（research 根目录文件）本期不处理。
|
||||
|
||||
---
|
||||
|
||||
## 12. 实施顺序（供开发排期）
|
||||
|
||||
1. `report_db/`（models/schema/db）+ `.env` 配置 + 建表验证
|
||||
2. `report_import/parser.py` + fixtures + 单测
|
||||
3. `report_import/importer.py` + CLI `report-import` + 178 份全量导入验收
|
||||
4. `reporter.py` 改造（_build_report_data + save_report）+ pipeline 兼容性验证
|
||||
5. docs/db_schema.md 定稿（给 API/前端）、README / continuation.md 更新
|
||||
Reference in New Issue
Block a user